mhttpd "messages" page stopped working after svn revision 4327 because of uninitialized variable 
"filename2" in midas.c:cm_message_retrieve(). Attached patch fixes the problem for me.
K.O.


--- src/midas.c (revision 4342)
+++ src/midas.c (working copy)
@@ -978,6 +978,8 @@
       size = sizeof(filename);
       db_get_value(hDB, 0, "/Logger/Message file", filename, &size, TID_STRING, TRUE);
 
+      strlcpy(filename2, filename, sizeof(filename2));
+
       if (strchr(filename, '%')) {
          /* replace strings such as midas_%y%m%d.mid with current date */
          tzset();

midas revision 4342 implements non-default tcp port numbers for the mserver.

To use, run "mserver -p 7070" and say "setenv MIDAS_SERVER_HOST
host.example.com:7070".

This is useful when multiple experiments share the same computer, but one does
not want to setup a global /etc/exptab (non-root users cannot change it) or one
does not want to run the mserver from xinetd (i.e. all experiments run different
versions of midas and cannot use the same common mserver executable).

Changed files:
src/mserver.c
src/midas.c
doxfiles/utilities.dox
doxfiles/appendixD.dox

Revision 4342.

K.O.

I modified the logging behavior of lazylogger. Originally, it was writing 
messages (run copied, removed, ...) both into midas.log and 
lazy_log_update.log. Since we have many files, it kind of clutters up the 
logging files. I think it is a good idea to have a separate file (which I 
changed not to "lazy.log" instead of "lazy_log_update.log" which I guess was a 
bug), so I put the logging into the main file under a conditional compile:

#ifdef WRITE_MIDAS_LOG
   cm_msg(MINFO, "lazy_log_update", str);
#endif

so it can be turned on again by adding -DWRITE_MIDAS_LOG to the compile line. 
If other experiments have different needs, one could make the logging behavior 
controllable through the ODB. In that case, I would suggest a single parameter 
"Logging file" which can be either "midas.log" for the normal logging or 
"lazy.log" for logging into the extra file. I guess having the messages twice 
on the system is not needed by any experiment.

- Stefan

We had recently some problems at our experiment which I would like to share 
with the community. This affects however only experiments which have a slow 
control front-end in multi-threaded mode.

The problem is related with the fact that the midas API is not thread safe, so 
a device driver or bus driver from the slow control system may not call any ODB 
function. We found several drivers (mainly psi_separator.c, psi_beamline.c etc) 
which use inside read/write function the midas PAI function cm_msg() to report 
any error. While this is ok for the init section (which is executed in the main 
frontend thread) this is not ok for the read/write function inside the driver. 
If this is done anyhow, it can happen that the main thread locks the ODB (via 
db_lock_database()) and the thread interrupts that call and locks the ODB 
again. In rare cases this can cause a stale lock on the ODB. This blocks all 
other programs to access the ODB and the experiment will die loudly. It is hard 
to identify, since error messages cannot be produced any more, and remote 
programs (not affected by the lock) just show a rpc timeout.

I fixed all drivers now in our experiment which solved the problem for us, but 
I urge other people to double check their device drivers as well.

In case of problems, there is a thread ID check in 
db_lock_database()/db_unlock_database() which can be activated by supplying 

-DCHECK_THREAD_ID

in the compile command line. If then these functions are called from different 
threads, the program aborts with an assertion failure, which can then be 
debugged. 

There is also a stack history system implemented with new functions 
ss_stack_xxxx. Using this system, one can check which functions called 
db_lock_database() *before* an error occurs. Using this system, I identified 
the malicious drivers. Maybe this system can also be used in other error 
debugging scenarios.

A new ODB flag has been introduced. When the logger is configured for automatic 
stop and restart (/Logger/Auto restart = y), the restart delay was hard-wired 
to 20 sec., which might be too long or short for some experiments. Therefore a 
new parameter "/Logger/Auto restart delay" has been introduced which can be 
used to accommodate different delays. A non-zero delay is necessary for 
experiments where some lengthy activities occur during the stop of a run, like 
an analyzer writing many histograms to disk.

The code for storing midas history in an odbc sql database has been committed.
Changes:
include/history_odbc.h, src/history_odbc.cxx --- implementation
src/mlogger.c --- call the history_odbc functions
utils/mh2sql.cxx --- import existing midas history files (*.hst) into an odbc
sql database.

This new code is enabled by the HAVE_ODBC gunk in the Makefile. If compilation
bombs, please let me know and as a work around, comment out all instances of
HAVE_ODBC from your Makefile.

Limitations:
- mhttpd support for reading history data from odbc sql database is missing
- many sql functions are implemented in a very minimalistic form (i.e. when
defining a history event, we blindly ask sql to create the tables, even if they
already exist - this works, but spams the midas log with sql errors).
- error handling is incomplete: after any sql error, the odbc connection is closed.
- only MySQL (and ascii output) are supported: we use mysql-specific data types
as they match midas types exactly. Code to support PgSQL is present and it used
to work, but is commented out. (At TRIUMF/T2K, we intend to use MySQL exclusively).
- ODBC ascii interface is used, instead of the potentially more efficient binary
interface.

To enable:
- create a MySQL database,
- create $HOME/.odbc.ini (see attached example)
- set ODB "/History/PerVariableHistory" to "1" - the new code is intended to be
used with per-variable history. Per-equipment (traditional) history would work,
but will result in suboptimal layout of SQL tables.
- set ODB "/Logger/ODBC_DSN" to the DSN defined in .odc.ini.
- set ODB "/Logger/ODBC_Debug" to non-zero to enable debugging output from the
new code.

To use the "ascii output" mode:
Included is code to write "ascii" sql output into a text file, instead of using
an actual SQL database. To enable it, set "ODBC_DSN" to
"/path/to/some/text/file" and all SQL output will be written to this file. No
actual SQL database required. This mode exists mostly for debugging the SQL syntax.

Despite limitations, the committed code is fully functional - we are presently
using it to record history data from slow controls of T2K detector tests
(voltages, currents, temperatures).

Comments and suggestions on naming and mapping from odb structures to SQL tables
is very much welcome.

K.O.

Following changes to midas produced from the TRIUMF T2K project have been
committed to svn:
1) cm_shutdown() will now SIGKILL clients that cannot be stopped via normal
means. Previously cm_shutdown() would print a message to the effect "please kill
this client yourself manually". The user action in this case (assuming they did
not issue cm_shutdown() by mistake) has been to find out the client pid using
"ps", kill -KILL it, then "odbedit clean". cm_shutdown() now performs all this
automatically.
2) rpc_send_event() did not correctly detect loss of connection to the remote
mserver (i.e. in case it was killed by cm_shutdown() above). Now, correct error
handling is in place and the remote frontend should gracefully shutdown if
mserver connection is lost. (However I observe that some of my remote frontends
fail to exit unless I do "exit(1);" from my frontend_exit() function.
3) mhttpd bug fixed: when editing odb entries, the "cancel" button did not work
correctly.
4) lazylogger "script" backup type is now fully tested and documented. Example
scripts for writing to dcache are available by request.
5) mlogger and mhttpd changes for writing history data to an sql database are
mostly completed and will be committed after some more debugging. (If you are
interested in details, please contact me directly).
6) (committed some time ago) Makefile changes for cross-compiling midas are now
in: "make linux32", "make linux64", "make crosscompile".
K.O.

<script type="text/javascript" src="../mhttpd.js"></script>

Display of cursor location

<img id="refimg" src="...">

Retrieving ODB values

  ODBGet(<path>);

  ODBGet('/Equipment/Environment/Variables/Input[3]');

  ODBGet('/Equipment/Environment/Variables/Input[*]');

  var a = ODBGet('/Equipment/Environment/Variables/Input[*]');

  for (i=0 ; i<a.length ; i++)
    alert(a[i]);

  window.setInterval("Refresh()", 10000);

  function Refresh() {
    document.getElementById("run_number").innerHTML = ODBGet('/Runinfo/Run number');
  }

  <td id="run_number"></td>

  ODBGet('/Equipment/Environment/Variables/Input[3]&format=%1.2lf');

Retrieving System Messages

Setting ODB values

  ODBSet(<path>,<value>);

  ODBSet(<path>,<value>,<password_name>);

  <input type="checkbox" onClick="ODBSet('/Logger/Write data',this.checked?'1':'0')">

  <odb src="/Logger/Write data" type="checkbox" edit="2" onclick="ODBSet('/Logger/Write data',this.checked?'1':'0')">

The current implementation of ODB links has the problem that once a link is
created, it cannot be changed any more through odbedit. This is because each
"set" command works on the destination of the link instead of the link. The same
happens when one loads a *.odb file. To overcome this problem, two new functions
db_set_link_data() and db_set_link_data_index() have been implemented. They
resemble their counterparts db_set_data() and db_set_data_index(), but they can
be used to directly modify a link instead of the link target. I use these
functions now in odbedit and db_paste() so that the above described problems are
fixed now. I do not expect any side effect of this, but if people experience
problems with db_paste(), please let me know.

I implemented improved alarm conditions in the alarm system. Now one can write
conditions like

  /Equipment/HV/Variables/Input[*] < 100

or 

  /Equipment/HV/Variables/Input[2-3] < 100

to check all values from an array or a certain range. If one array element
fulfills the alarm condition, the alarm is trigger. In addition, bit-wise alarm
conditions are possible

  /Equipment/Environment/Variables/Input[0] & 8

is triggered if bit #2 is set in Input[0].

The changes are committed to SVN revision 4242.

Dear All,

I have a problem when testing the V792N 16 CH QDC with the V1718 VME-USB 
Bridge on Scientific Linux 5.1 i386 (kernel 2.6.18-53.1.21.e15).

The problem is that the V792N does not response normally after a few minutes 
of continuous polling and readout of data. It seems like the V792N is hanged 
and a hardware reset of the VME system is required to bring it working again. 
If I do not poll for DREADY first and directly read the Output Buffer 
continuously, the system can work properly.

I have worked on this problem many days but I cannot find any clues to solve 
it. I have tried to use the CAENVMEDemo program (with some modifications) to 
do the same thing (polling and readout) and it works fine. CAEN technical 
support also doesn't know why the VME system is hanged. I think it might be a 
problem of MIDAS itself. I have tried with MIDAS revision 4132 and the trunk 
version, but the problem is still there. Is there any parameter in MIDAS 
(buffer size etc?) which may cause this problem? I have attached my frontend 
code and drivers for your reference.

Thank you for your kind attention.

Best Regards,
Jimmy

Some improvement of custom pages have been implemented. The idea behind is that
a custom page would contain a large background image containing indicators but
also controls. While indicators (values, bars) are already available, the field
of controls have been improved.

Edit boxes floating on top of a graphic
---------------------------------------

The first option has been there from the beginning, but was never documented. It
makes it possible to put an edit box right on top of a graphic by means of a CSS
style tag. The custom page code could look like this:

<div style="position:absolute; top:100px; left:50px;">
<odb src="/Runinfo/run number" edit=1>
</div>
<img src="cusgom.gif">

The "div" tag surrounding the "odb" tag places this directly on top of the
"custom.gif" image, where it can be clicked to be edited.

Password protection of an edit box
----------------------------------

Being able to control an experiment through a web interface of course rises the
question about safety. This is not so much about external access (for which we
have other protection schemes like host lists etc.) but it's about accidental
access by the normal shift crew. If a single click on a web page opens a
critical valve, this might be a problem. In order to restrict access to some
"experts", an additional password can be chosen for all or some controls on a
custom page. This is done by a new option in the "odb" tag and by adding a small
JavaScript function into the custom page:

<script type="text/javascript">
<!--
function promptpwd(path)
{
    pwd = prompt('Please enter password', '');
    document.cookie = "cpwd=" + pwd;
    location.href = path;
}
//-->
</script>

...

<odb src="/..." edit=1 pwd="CustomPwd">

...

If the "pwd" option is present in the "odb" tag, mhttpd establishes a call to
the promptpwd() function if one click on the value. The password is then asked
from the user and submitted as a cookie. mhttpd then check this password against
the ODB entry

/Custom/Pwd/CustomPwd

and shows an error if they don't match. By using an explicit name ("CustomPwd"
in the above example) one can use a single password for all controls on a page,
or one could use several passwords on the same page. Like a shift crew password
for the less severe controls (/Custom/Pwd/ShiftPwd), and an "expert" password
(/Custom/Pwd/ExpertPwd) for the critical things. This password is of course not
secure in the sense that it's placed in plain text into the ODB, it's more to
prevent accidental modifications of things.

Area map to toggle values
-------------------------

Sometimes it's desirable to toggle a value, like the state of a valve. This can
be done now with a new function like this:

<map name="Custom1">
<area shape="rect" coords="40,200,100,300" alt="Main Valve"
  href="Custom1?cmd=Toggle&odb=/Equipment/Environment/Variables/Output[2]">
</map>
<img src="cusgom.gif" usemap="#Custom1">

This defines a clickable map on top of the custom image. The area(s) should
match with some areas on the image like the box of a valve. By clicking on it,
the supplied path to the ODB is used (in this case
"/Equpiment/Environment/Variables[2]") and it's value is toggled (set to 0 if it
is 1, set to 0 if it is 1). If the valve value is then used in the image via a
"fill" statement to change the color of the valve, it can turn green or red
depending on it's state.

Are map with password check
---------------------------

The above area map can be combined with the password check. To do so, one needs:

<area shape="rect" coords="40,200,100,300" alt="Main Valve"
  href="#"
onClick="promptpwd('Custom1?cmd=toggle&pnam=CustomPwd?odb=/Equipment/Environment/Variables/Output[2]')">

in combination with the JavaScript from above.


An example of the are map technology is shown in the attachment. This page from the MEG experiment at PSI
shows a complex gas system. The valves are represented as green circles. If they are clicked, they close
and become red (after the user successfully supplied the correct password).

> > The current versions of mlogger SVN 4215 is flooding our message system with
> > stuff like
> > 
> > > Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> > > Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> > > Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events
> 
> Problem confirmed on the M11 DAQ system at TRIUMF. We definitely do nothing funny
> there, so what is going on? Will investigate.

The only place I see where this could happen is in mlogger.c, lines 3064ff:

   /* check if event size has changed */
   db_get_record_size(hDB, hKey, 0, &size);
   if (size != hist_log[i].buffer_size) {
      close_history();
      open_history();
      return;
   }

The record size corresponds to /Equipment/<name>/Variables. If this array changes in
size, it will trigger the re-definition of the history. So please have a look there
and check why the record size changes.

> The current versions of mlogger SVN 4215 is flooding our message system with
> stuff like
> 
> > Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> > Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> > Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events

Problem confirmed on the M11 DAQ system at TRIUMF. We definitely do nothing funny
there, so what is going on? Will investigate.

K.O.

Since the very beginning midas had the problem that the last event of a run was
sometimes missing in the data. While for most experiments this is not an issue,
it starts to hurt on experiments using event building (front-end 1 and front-end
2 in the example below). A missing event can screw up the event builder on the
next begin of run, where the "missing event" would show up as the first event of
the new run, triggering an event mismatch error in the event builder.

After some analysis, we identified the problem as follows. Assume FE1 controls
the trigger, while FE2 generates the second event fragment.

1) Stop is requested to FE1
2) tr_stop gets called on FE1
3) tr_stop calls end_of_run() use code
4) end_of_run() disables the trigger
5) FE1 finishes stop transition
6) Stop is requested to FE2
7) FE2 finishes stop transition

What can now happen is the following: An additional event occurs between 2) and
4). This event triggers ADCs and TDCs, and is then stored in the front-end
hardware. FE2 sees this event, since it has not yet done the stop transition,
and reads it out. FE1 is however already in the end_of_run() routine, and simply
disables the trigger, without reading this last event, and thus causing the
event mismatch at the beginning of the next run.

To fix the problem, the framework in mfe.c was changed:

1) Stop is requested to FE1
2) tr_stop gets called on FE1
3) tr_stop calls end_of_run() use code
4) end_of_run() disables the trigger
4b) tr_stop calls check_polled_events()
5) FE1 finishes stop transition
6) Stop is requested to FE2
7) FE2 finishes stop transition

The new routine check_polled_events checks if there is any more event in the
hardware by calling the user polling routine. If there is one more event, calls
the user readout routine and sends it to the back-end before concluding the run
transition. 

This modification solved our problem at the MEG experiment at PSI, but it might
be good that all experiments using event building update midas to revision 4225.

I do not expect any bad side effect, but one never knows. So if there are new
problems caused by this modification, please report.

- Stefan

> The current versions of mlogger SVN 4215 is flooding our message system with
> stuff like
> 
> > Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> > Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> > Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events
> 
> This is fatal to us and blowing up the midas.log like hell. I would prefer if
> one could flag these kind of messages (ODB /Logger/..), i.e. enable and disable
> it. At the moment I have to comment it out in the source code since we cannot
> work with it.

I just sent the attached message to Stefan - please read it.

Before we take any action, we need to understand why history is being
reconfigured every 10 seconds (according to your logfile snippet).

Are you starting a new run every 10 seconds?

If that is what you do and that is your intent, I guess it is atypical usage of
MIDAS and the message from the mlogger is offensive and should be removed/disabled.

If something else is going on, we need to understand it before we sweep trouble
under the carpet by disabling this message.

K.O.

Stefan - there is more bad news - the message is produced when the history
is being reconfigured. This only is supposed to happen when the mlogger
starts or at the begin of run.

So these messages are just a tip of an iceberg of some other trouble.

The logic of when history is reconfigured I did not change. So likely
the trouble existed before, but you did not know about it.

We can kill the message, but why is the history being reconfigured
at a rate that "floods the log file"? That cannot possibly be good.

K.O.

The current versions of mlogger SVN 4215 is flooding our message system with
stuff like

> Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events

This is fatal to us and blowing up the midas.log like hell. I would prefer if
one could flag these kind of messages (ODB /Logger/..), i.e. enable and disable
it. At the moment I have to comment it out in the source code since we cannot
work with it.

Cheers,
  Andreas 

Hi All,

I am testing the libraries provided by CAEN with the sample softwares in the 
bundle CD. The Windows sample program works fine, but I cannot get started with 
the Linux sample program. When I run CAENVMEDemo in Scientific Linux 5.1, it 
gives me a message "Error opening the device". I have followed the instructions 
in CAENVMElibReadme.txt: 
- compile and load the device driver v1718.ko
- install the library libCAENVME.so

Does anyone have any experience of using V1718 in Scientific Linux? Thanks.

Regards,
Jimmy


> Hi All,
> 
> Is there any example code for using MIDAS with the CAEN VME-USB Bridge V1718? 
> Thanks.
> 
> Regards,
> Jimmy

Hi All,

Is there any example code for using MIDAS with the CAEN VME-USB Bridge V1718? 
Thanks.

Regards,
Jimmy

> > Here is the list of known problems I am aware of and of fixes not yet committed to midas svn:
> > 1) added variable /equiment/foo/common/PerVariableHistory
> 
> corrected in svn revision 4203, read
> http://savannah.psi.ch/viewcvs/trunk/src/mlogger.c?root=midas&rev=4203&sortby=rev&view=log

Was still broken - all should work in revision 4207.

> > 2) writing compressed midas files (foo.mid.gz) crashes the mlogger when file
> > size reaches 2 GBytes. This problem could be new in SL5.1.

It turns out that on SL5 and SL5.1 (and others?) the 32-bit version of ZLIB opens the
compressed output file without the O_LARGEFILE flag, this limits the file size to 2 GB.

Fixed by opening the file ourselves, then attach compression stream using gzdopen().

Revision 4207. (Not tested on Windows - may be broken!)

> > 5) mhttpd history "export" button needs to be fixed (by request from ALPHA). At
> > present it either does not return all exiting data or crashes mhttpd. (no fix)
> 
> (no change)
> 
> > 6) mhttpd ODB editor in "set value" page, the "cancel" button is broken (needs
> > to be corrected for "relative URL").
> 
> Apply this patch to src/mhttpd.c
> 
> @@ -11156,10 +11190,7 @@
>           sprintf(str, "SC/%s/%s", eq_name, group);
>           redirect(str);
>        } else {
> -         strlcpy(str, path, sizeof(str));
> -         if (strrchr(str, '/'))
> -            strlcpy(str, strrchr(str, '/')+1, sizeof(str));
> -         redirect(str);
> +         redirect("./");
>        }
> 
> > 7) mhttpd needs AJAX-style methods for reading and writing ODB. (no fix)
> 
> (no change)
> 
> K.O.