> > The current versions of mlogger SVN 4215 is flooding our message system with
> > stuff like
> > 
> > > Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> > > Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> > > Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events
> 
> Problem confirmed on the M11 DAQ system at TRIUMF. We definitely do nothing funny
> there, so what is going on? Will investigate.

The only place I see where this could happen is in mlogger.c, lines 3064ff:

   /* check if event size has changed */
   db_get_record_size(hDB, hKey, 0, &size);
   if (size != hist_log[i].buffer_size) {
      close_history();
      open_history();
      return;
   }

The record size corresponds to /Equipment/<name>/Variables. If this array changes in
size, it will trigger the re-definition of the history. So please have a look there
and check why the record size changes.

Some improvement of custom pages have been implemented. The idea behind is that
a custom page would contain a large background image containing indicators but
also controls. While indicators (values, bars) are already available, the field
of controls have been improved.

Edit boxes floating on top of a graphic
---------------------------------------

The first option has been there from the beginning, but was never documented. It
makes it possible to put an edit box right on top of a graphic by means of a CSS
style tag. The custom page code could look like this:

<div style="position:absolute; top:100px; left:50px;">
<odb src="/Runinfo/run number" edit=1>
</div>
<img src="cusgom.gif">

The "div" tag surrounding the "odb" tag places this directly on top of the
"custom.gif" image, where it can be clicked to be edited.

Password protection of an edit box
----------------------------------

Being able to control an experiment through a web interface of course rises the
question about safety. This is not so much about external access (for which we
have other protection schemes like host lists etc.) but it's about accidental
access by the normal shift crew. If a single click on a web page opens a
critical valve, this might be a problem. In order to restrict access to some
"experts", an additional password can be chosen for all or some controls on a
custom page. This is done by a new option in the "odb" tag and by adding a small
JavaScript function into the custom page:

<script type="text/javascript">
<!--
function promptpwd(path)
{
    pwd = prompt('Please enter password', '');
    document.cookie = "cpwd=" + pwd;
    location.href = path;
}
//-->
</script>

...

<odb src="/..." edit=1 pwd="CustomPwd">

...

If the "pwd" option is present in the "odb" tag, mhttpd establishes a call to
the promptpwd() function if one click on the value. The password is then asked
from the user and submitted as a cookie. mhttpd then check this password against
the ODB entry

/Custom/Pwd/CustomPwd

and shows an error if they don't match. By using an explicit name ("CustomPwd"
in the above example) one can use a single password for all controls on a page,
or one could use several passwords on the same page. Like a shift crew password
for the less severe controls (/Custom/Pwd/ShiftPwd), and an "expert" password
(/Custom/Pwd/ExpertPwd) for the critical things. This password is of course not
secure in the sense that it's placed in plain text into the ODB, it's more to
prevent accidental modifications of things.

Area map to toggle values
-------------------------

Sometimes it's desirable to toggle a value, like the state of a valve. This can
be done now with a new function like this:

<map name="Custom1">
<area shape="rect" coords="40,200,100,300" alt="Main Valve"
  href="Custom1?cmd=Toggle&odb=/Equipment/Environment/Variables/Output[2]">
</map>
<img src="cusgom.gif" usemap="#Custom1">

This defines a clickable map on top of the custom image. The area(s) should
match with some areas on the image like the box of a valve. By clicking on it,
the supplied path to the ODB is used (in this case
"/Equpiment/Environment/Variables[2]") and it's value is toggled (set to 0 if it
is 1, set to 0 if it is 1). If the valve value is then used in the image via a
"fill" statement to change the color of the valve, it can turn green or red
depending on it's state.

Are map with password check
---------------------------

The above area map can be combined with the password check. To do so, one needs:

<area shape="rect" coords="40,200,100,300" alt="Main Valve"
  href="#"
onClick="promptpwd('Custom1?cmd=toggle&pnam=CustomPwd?odb=/Equipment/Environment/Variables/Output[2]')">

in combination with the JavaScript from above.


An example of the are map technology is shown in the attachment. This page from the MEG experiment at PSI
shows a complex gas system. The valves are represented as green circles. If they are clicked, they close
and become red (after the user successfully supplied the correct password).

I implemented improved alarm conditions in the alarm system. Now one can write
conditions like

  /Equipment/HV/Variables/Input[*] < 100

or 

  /Equipment/HV/Variables/Input[2-3] < 100

to check all values from an array or a certain range. If one array element
fulfills the alarm condition, the alarm is trigger. In addition, bit-wise alarm
conditions are possible

  /Equipment/Environment/Variables/Input[0] & 8

is triggered if bit #2 is set in Input[0].

The changes are committed to SVN revision 4242.

The current implementation of ODB links has the problem that once a link is
created, it cannot be changed any more through odbedit. This is because each
"set" command works on the destination of the link instead of the link. The same
happens when one loads a *.odb file. To overcome this problem, two new functions
db_set_link_data() and db_set_link_data_index() have been implemented. They
resemble their counterparts db_set_data() and db_set_data_index(), but they can
be used to directly modify a link instead of the link target. I use these
functions now in odbedit and db_paste() so that the above described problems are
fixed now. I do not expect any side effect of this, but if people experience
problems with db_paste(), please let me know.

<script type="text/javascript" src="../mhttpd.js"></script>

Display of cursor location

<img id="refimg" src="...">

Retrieving ODB values

  ODBGet(<path>);

  ODBGet('/Equipment/Environment/Variables/Input[3]');

  ODBGet('/Equipment/Environment/Variables/Input[*]');

  var a = ODBGet('/Equipment/Environment/Variables/Input[*]');

  for (i=0 ; i<a.length ; i++)
    alert(a[i]);

  window.setInterval("Refresh()", 10000);

  function Refresh() {
    document.getElementById("run_number").innerHTML = ODBGet('/Runinfo/Run number');
  }

  <td id="run_number"></td>

  ODBGet('/Equipment/Environment/Variables/Input[3]&format=%1.2lf');

Retrieving System Messages

Setting ODB values

  ODBSet(<path>,<value>);

  ODBSet(<path>,<value>,<password_name>);

  <input type="checkbox" onClick="ODBSet('/Logger/Write data',this.checked?'1':'0')">

  <odb src="/Logger/Write data" type="checkbox" edit="2" onclick="ODBSet('/Logger/Write data',this.checked?'1':'0')">

A new ODB flag has been introduced. When the logger is configured for automatic 
stop and restart (/Logger/Auto restart = y), the restart delay was hard-wired 
to 20 sec., which might be too long or short for some experiments. Therefore a 
new parameter "/Logger/Auto restart delay" has been introduced which can be 
used to accommodate different delays. A non-zero delay is necessary for 
experiments where some lengthy activities occur during the stop of a run, like 
an analyzer writing many histograms to disk.

We had recently some problems at our experiment which I would like to share 
with the community. This affects however only experiments which have a slow 
control front-end in multi-threaded mode.

The problem is related with the fact that the midas API is not thread safe, so 
a device driver or bus driver from the slow control system may not call any ODB 
function. We found several drivers (mainly psi_separator.c, psi_beamline.c etc) 
which use inside read/write function the midas PAI function cm_msg() to report 
any error. While this is ok for the init section (which is executed in the main 
frontend thread) this is not ok for the read/write function inside the driver. 
If this is done anyhow, it can happen that the main thread locks the ODB (via 
db_lock_database()) and the thread interrupts that call and locks the ODB 
again. In rare cases this can cause a stale lock on the ODB. This blocks all 
other programs to access the ODB and the experiment will die loudly. It is hard 
to identify, since error messages cannot be produced any more, and remote 
programs (not affected by the lock) just show a rpc timeout.

I fixed all drivers now in our experiment which solved the problem for us, but 
I urge other people to double check their device drivers as well.

In case of problems, there is a thread ID check in 
db_lock_database()/db_unlock_database() which can be activated by supplying 

-DCHECK_THREAD_ID

in the compile command line. If then these functions are called from different 
threads, the program aborts with an assertion failure, which can then be 
debugged. 

There is also a stack history system implemented with new functions 
ss_stack_xxxx. Using this system, one can check which functions called 
db_lock_database() *before* an error occurs. Using this system, I identified 
the malicious drivers. Maybe this system can also be used in other error 
debugging scenarios.

I modified the logging behavior of lazylogger. Originally, it was writing 
messages (run copied, removed, ...) both into midas.log and 
lazy_log_update.log. Since we have many files, it kind of clutters up the 
logging files. I think it is a good idea to have a separate file (which I 
changed not to "lazy.log" instead of "lazy_log_update.log" which I guess was a 
bug), so I put the logging into the main file under a conditional compile:

#ifdef WRITE_MIDAS_LOG
   cm_msg(MINFO, "lazy_log_update", str);
#endif

so it can be turned on again by adding -DWRITE_MIDAS_LOG to the compile line. 
If other experiments have different needs, one could make the logging behavior 
controllable through the ODB. In that case, I would suggest a single parameter 
"Logging file" which can be either "midas.log" for the normal logging or 
"lazy.log" for logging into the extra file. I guess having the messages twice 
on the system is not needed by any experiment.

- Stefan

> mhttpd "messages" page stopped working after svn revision 4327 because of uninitialized variable 
> "filename2" in midas.c:cm_message_retrieve(). Attached patch fixes the problem for me.
> K.O.
> 
> 
> --- src/midas.c (revision 4342)
> +++ src/midas.c (working copy)
> @@ -978,6 +978,8 @@
>        size = sizeof(filename);
>        db_get_value(hDB, 0, "/Logger/Message file", filename, &size, TID_STRING, TRUE);
>  
> +      strlcpy(filename2, filename, sizeof(filename2));
> +
>        if (strchr(filename, '%')) {
>           /* replace strings such as midas_%y%m%d.mid with current date */
>           tzset();

Ups, was my fault, sorry. I committed your change.

Previously, one mhttpd server could sever several experiments at the same time. 
This caused however sometimes problems and was hard to maintain. Starting from 
SVN revision 4348, I removed the multi-experiment support, which I believe is 
now a much cleaner implementation. So if several experiments are defined on a 
computer, each one need a separate mhttpd process listening on a different 
port. The experiment name can now be supplied on the command line to mhttpd 
like for any other midas program. I have tested this so far at two experiments 
at PSI, but this does not cover all possibilities. What I did not try was 
experiments with web passwords and odb passwords. If there is any problem after 
upgrading to 4348, please report.

> I suspect mlogger uses ASYNC transactions exactly to avoid
> this type of deadlock (mlogger used ASYNC transactions since svn revision 2, the
> beginning of time).

That's exactly the case. If you would have asked me, I would have told you 
immediately, but it is also good that you re-confirmed the deadlock behavior with 
the SYNC flag. I didn't check this for the last ten years or so.

Making the buffers bigger is only a partial solution. Assume that the disk gets 
slow for some reason, then any buffer will fill up and you get the dead lock.

The only real solution is to put the logic into a separate thread. So the thread 
does all the RPC communication with the clients, while the main logger thread logs 
data as usual in parallel. The problem is that the RPC layer is not yet completely 
tested to be thread safe. I put some mutex and you correctly realized that these 
are system wide, but you want a local mutex just for the logger process. You need 
also some basic communication between the "run stop thread" and the "logger main 
thread". Maybe Pierre remembers that once there was the problem that the logger did 
not know when all events "came down the pipe" and could close the file. He added 
some delay which helped most of the time. But if we would have some communication 
from the "run stop thread" telling the main thread that all programs except the 
logger have stopped the run, then the logger only has to empty the local system 
buffer and knows 100% that everything is done.

In the MEG experiment we have the same problem. We need a certain sequence 
(basically because we have 9 front-ends and one event builder, which has to be 
called after the front-ends). We realized quickly that the logger cannot stop the 
run, so we wrote a little tool "RunSubmit", which is a run sequence with scripting 
facility. So you write a XML file, telling RunSubmit to start 10 runs, each with 
5000 events. RunSubmit now watches the run statistics and stops the run. Since it's 
outside the logger process, there is no dead lock. Unfortunately RunSubmit was 
written by one of our students and contains some MEG specific code. Otherwise it 
could be committed to the distribution.

So I feel that a separate thread for run stop (and maybe even start) would be a 
good thing, but I'm not sure when I will have time to address this issue.

- Stefan

> Dear MIDAS-addicts,
> 
> I would like to hear your opinion on this:
> We've until now used CAMAC with Hytec 1331 controllers. We're using Yale FADCs 
> whose readout takes ages in CAMAC (2048 samples take 2 milliseconds to be 
> read). We've got 20+ FADC channels (we usually read only 2-3)
> 
> Now we've had the brilliant idea to replace the Yale FADCs with some VME 
> digitizer and we now plan to buy a Struck SIS 1100/3100 PCI-VME controller,
> plus 4 pc. CAEN 1720 8ch 12bit, 250MHz WFD.
> 
> (1) Can anybody comment on this choice? Good experiences/problems?
> 
> We are still using the CAMAC stuff for all other modules (TDCs, ADCs, 
> scalers). So my plan is to have ONE frontend who reads both the CAMAC modules 
> and the VME modules.
> 
> (2) Is it possible to build and run a dual-controller frontend for both CAMAC 
> and VME? Does anybody have experience with that? Or is it a stupid idea?
> 
> I'd appreciate any hints.
> 
> [Edit: We're using Linux]
> 
> Thanks a lot,
> 
> Randolf

Dear Randolf,

I used some time ago several HYTEC 1331 controllers together with the Struck 
SIS3100. Since the HYTEC is IO-mapped and the SIS3100 is memory mapped, there was 
no problem in running them in parallel. Note however that there will soon be an 
improved version of the SIS3100 with improved speed, and also CAEN plans a WFD 
with 32 channels, 6 GSPS, 12 bit, using the DRS chip for the next year. I don't 
know if you need that, but just that you know.

Best regards, 
  Stefan

> In midas.c there are several places where client last activity time stamps are checked against the 
> watchdog timeout and the clients are declared dead if they fail to update their activity time stamps. 
> ODB time stamps and data buffer time stamps appear to be handled in a similar manner.
> 
> Most checks are done like this:
> 
> now = ss_millitime();
> if (client->watchdog > 0      <----- check that the watchdog is enabled
>     && now > client->last_activity    <---- check for crazy time stamps from the future
>     && now - client->last_activity > client->watchdog_timeout)   <--- normal timeout
>         remove_client(client);
> 
> But in a few places, the extra checks are missing:
> 
> now = ss_millitime();
> if (now - client->last_activity > client->watchdog_timeout)
>         remove_client(client);
> 
> Is this an oversight from when additional checks were added?
> Should I make all checks read like the first one?
> 
> K.O.

This is on purpose. Inside cm_watchdog(), the system check for client->watchdog > 0. If the watchdog 
timeout is zero, the client is not removed. This feature is used if you debug a program. If you come to a 
breakpoint and sit there for a while, you might be declared dead and the application is removed from the 
ODB, meaning that you cannot continue debugging (on the next ODB access the application asserts). This 
can be avoided by setting the watchdog to zero, which is implemented in most applications by supplying 
"-d" on the command line. Now assume you debug a program, so you set the watchdog timeout to zero, but in 
the debugging session you decide to quit. Since the watchdog timeout is zero, you will never be removed 
from the ODB. Therefore, the code inside cm_cleanup() doe NOT check client->watchdog > 0. Therefore, a 
"cleanup" inside odbedit will even remove clients having the timeout set to zero. 

Now there might be more clever ways to accomplish that, but that's how it is implemented right now.

> When I run odbedit remotely (odbedit -h ladd09), the "cleanup" command unexpectedly produces the 
> output of the "sor" command (sure enough, there is a call to db_get_open_records() there), but when I run 
> it locally, I do not get this output (but db_get_open_records() is still called). Strange. K.O.

The db_get_open_records() call was by mistake there, I removed it. What remains is that the notification 
message if a client is removed from the ODB goes through the system messages. When running locally, odbedit 
echoes it's own messages, but when running remotely, this is not the case. So the messages can be seen by 
everybody else (plus it ends up in the message file), but not by the remote odbedit where the cleanup is 
started. The quick fix for that is to say "old" in odbedit which shows the last few lines of the message 
file, so one can see any successful cleanup. 

> A new set of functions was added to mscb.h to adjust mscb timeouts and retries to better match specific 
> applications:
> 
> +   int EXPRT mscb_get_max_retry();
> +   int EXPRT mscb_set_max_retry(int max_retry);
> +   int EXPRT mscb_get_usb_timeout();
> +   int EXPRT mscb_set_usb_timeout(int timeout);
> +   int EXPRT mscb_get_eth_max_retry();
> +   int EXPRT mscb_set_eth_max_retry(int eth_max_retry);

In the spirit of this, a variable retry scheme has been implemented in the mscbdev.c device driver. At the 
MEG experiment, we have one mscb device which is pretty slow, while the others are fast. Therefore it is 
necessary to have a per-device max retry count which can be different for different submasters. I moved 
therefore the max_eth_retry variable into the mscb_fd structure and adjusted a few functions accordingly. I 
did not bother with the other timeouts and retries, since I don't need this for the moment, but it would be 
nice if they would be handled in the same way. Then I added code into mscbdev.c to read the retry variable 
form the ODB under /Equipment/<name>/Settings/Device/<Name>/Retries. The default is 10, but it can be 
changed and becomes valid after the program has been restarted. 

To fix this problem, do the following:

- Update to the current SVN revision 4368 of mhttpd.c
- Add following tag into your custom page:

  <input type=hidden name="redir" value="name">

  where "name" is the name of your custom page which follows the CS/ in the URL. Like 
if you have a custom page which you access through httpd://localhost/CS/junk then the 
tag would be 

  <input type=hidden name="redir" value="junk">

The "redir" parameter is now evaluated inside mhttpd and brings you back to the proper 
custom page. You can also define another custom page as the target, if that makes 
sense in your application.

Pierre: Would be nice to document this somewhere more officially.

> Not sure if this is a bug or a feature:
> 
> Writing a boolean label on an image seems to produce rather strange behaviour.
> 
> For example,
> odb>ls /Equipment/gas/settings/my_bool -lt
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> my_bool                         BOOL    1     4     14m  0   RWD  y
> 
> 
> odb>cd /custom/images/my_image.gif/labels
> odb>ls
> Src                             /Equipment/gas/settings/my_bool
> Format                          val: %d (bool)
> Font                            Medium
> X                               10
> Y                               10
> Align                           0
> FGColor                         FFFFFF
> BGColor                         FF8800
> 
> Instead of the expected string "val: y (bool)", only the value of the key
> appears, i.e. "y". 
> The behaviour is the same whether I use %d, %u, %s, %c etc as the format character. 

That has been fixed in rev. 4379

> When writing custom webpages, it would be nice to be able to write code such as
> 
> <odb src="/Equipment/TITAN_ACQ/ppg cycle/trans3/time offset (ms)" edit=1>
> 
> from Javascript, e.g.
> <script  type="text/javascript">
> if ( flag != 3)
>    document.write('<odb src="/Equipment/TITAN_ACQ/ppg cycle/trans3/time offset
> (ms)" edit=1>ms');
> else
>    document.write('<odb src="/Equipment/TITAN_ACQ/ppg cycle/trans4/time offset
> (ms)" edit=1>ms');
> </script>
> 
> This is not translated correctly by mhttpd; the final quote and bracket get
> stripped off, and it gives Javascript error
> 
>  Error: unterminated string literal
> Source File: http://titan04:8089/CS/ppg_cycle?cmd=Edit&index=11
> Line: 477, Column: 18
> Source Code:
>    document.write('<input type=text size=10 maxlength=80 name=value value="1">
> 
> I can get round this by using an input box and a combination of ODBGet and
> ODBSet, but it would be easier if the edit=1 form above worked correctly, or
> there was a command like ODBSet that would accept input from the user.
> 
> Thanks.
> 
>  would be nice is there was a command such as ODBGet or ODBSet that would work
> with javascript to 

Actually that won't work, even if I would fix it. The <odb> tag is evaluated on the
server side (mhttpd), where is gets replaced by the actual ODB value. But if you
use JavaScript to generate the <odb> tag dynamically, this only happens on the
client side, so the server has no chance to substitute them. So you have to go with
ODBGet's I'm afraid. Nevertheless, I changed the code such that any ODB tags inside
a JavaScript is not interpreted by mhttpd.

> Thanks Stefan. 
> Your fix works nicely with the start/stop buttons not returning to the same or to a
> different web page.
> 
> However, it does not seem to have fixed the problem with the Customscript button. It does
> not seem to pick up the redirect, nor do the Pause/Resume buttons (which are programmed to
> appear when the run starts).

That has been fixed in rev. 4377

> Dear All,
> 
> Is there any recommended platforms for running MIDAS? Have anyone encountered 
> problems when running MIDAS on Scientific Linux?
> 
> Thanks.
> 
> Jimmy

I run MIDAS on scientific Linux 5.1 without any problem.