> > Thu Dec 23 12:10:30 2010 [ODBEdit9,ERROR] [odb.c:3247:db_get_value,ERROR] "Name" is of type NULL, not STRING
> This is caused by a race condition between client removal in cm_delete_client_info() and cm_exist().
> ... this race condition seems to be benign.

Not so benign - after fixing cm_exist() to check the return value of db_get_value() and calling it without the "create" flag,
a crasher turned up inside db_find_key() called by db_get_value() with these stale hkeys. For invalid keys (not TID_KEY),
it would call db_get_path() and crash.

So after adding a check for valid key types, my test script runs much better - all the major weirdness is gone, I only see
rare messages from db_find_key(), db_get_key() and db_get_value() about invalid key and data types (after all,
I did not fix the underlying race condition).

The only remaining problem when running my script is some kind of deadlock between the ODB and SYSMSG semaphores...

K.O.

> Thu Dec 23 12:10:30 2010 [ODBEdit9,ERROR] [odb.c:3247:db_get_value,ERROR] "Name" is of type NULL, not STRING

This is caused by a race condition between client removal in cm_delete_client_info() and cm_exist().

The race condition in cm_exist() works like this:
- db_enum_key() returns the hkey (pointer to) the next /System/Clients/PID directory
- the client corresponding to PID is removed, our hkey now refers to a deleted entry
- db_get_value() tries to use the now stale hkey pointing to a deleted entry, complains about invalid key TID.

Because the offending db_get_value() is called with the "create if not found" argument set to TRUE, there is potential
for writing into ODB using a stale hkey, maybe leading to ODB corruption. Other than that, this race condition seems
to be benign.

cm_exist() is called from:
everybody->cm_yield()->al_check()->cm_exist()

Further analysis:
- cm_yield() calls al_check() every 10 sec, al_check() calls cm_exist() to check for "program is not running" alarms.
- in al_check() cm_exist() is called once for each entry in /Programs/xxx, even for programs with no alarms. (Maybe I should change this?)
- assuming 10 programs are running (10 clients), every 10 seconds, cm_exist() will be called 10 times and inside, will loop over 10 clients, exposing the enum-get race condition 10*10=100 times every 10 seconds. Usually, 
ODB /Programs/ has many more entries than there are active clients, further increasing the frequency of exposure of this race condition.

K.O.

The following script makes midas very unhappy and eventually causes odb corruption. I suspect the reason is some kind of race condition collision between client 
creation and destruction code and the watchdog activity (each client periodically runs cm_watchdog() to check if other clients are still alive, O(NxN) total complexity). 
Amongst messages appearing in midas.log:

Thu Dec 23 11:59:08 2010 [ODBEdit28,INFO] Client 'unknown' on buffer 'SYSMSG' removed by bm_open_buffer because client pid 20463 does not exist
Thu Dec 23 11:59:09 2010 [ODBEdit43,INFO] Client 'unknown' on buffer 'SYSMSG' removed by cm_watchdog because client pid 20465 does not exist
Thu Dec 23 12:11:21 2010 [ODBEdit,ERROR] [odb.c:1061:db_open_database,ERROR] Removing client 'ODBEdit11', pid 21536, index 27 because the pid no longer exists
Thu Dec 23 17:06:15 2010 [ODBEdit,ERROR] [odb.c:988:db_open_database,ERROR] maximum number of clients exceeded
Thu Dec 23 12:10:30 2010 [ODBEdit9,ERROR] [odb.c:3247:db_get_value,ERROR] "Name" is of type NULL, not STRING

The last message about <"Name" is of type NULL> appears during normal operation of the ND280 DAQ, leading me into these investigations.

Notes:
a) the script runs at most 50 copies of odbedit, never exceeding midas.h MAX_CLIENTS value 64, so one does not expect to see messages about "maximum number of 
clients exceeded"
b) the script runs 50 copies of odbedit in parallel, increasing the likelihood of whatever race condition is causing this. In the ND280 system, likelihood of failure is 
increased by the large number of running clients (10-20-30 clients), each client running periodic cm_watchdog, to collide with new client creation or destruction.
c) in other experiments, we do not see this (ok, we do have midas meltdowns once in a while) because (1) we tend to have fewer clients (reduced frequency of 
cm_watchdog), (2) we tend to not start and stop midas clients too often (reduced frequency of running client creation and destruction). (NB it seems like ND280 people 
tend to run many scripts containing odbedit commands, so they effectively start and stop midas clients more often than usual).


#!/usr/bin/perl -w
#$cmd = "odbedit -c \'scl -w\' &";
$cmd = "odbedit -c \'ls -l /system/clients\' &";
for (my $i=0; $i<50; $i++)
{
system $cmd;
}
#end

midas\mscb\embedded
  \include                <- place for framework include file mscbemb.h
  \lib                    <- precompiled TCP/IP library for SCS-260 submaster
  \src                    <- framework sources mscbmain.c and mscbutil.c
  \<project1>             <- separate folder for project1
      config.h            <- config file for project1
  \<project2>             <- separate folder for project2
      config.h            <- config file for project2
  ...
  \experiment
     \<experiment1>
     \<experiment2>

• The framework files mscbmain.c mscbutil.c and mscbemb.h have moved and therefore they have to be re-added to the projects in the Keil MicroVision Development Environment.
  
• The name of the device should not be defined under compiler settings (Project Options/C51/Preprocessor Symbols), but put directly into the config.h file associated with the project.
  
• The include paths in the compile have to be changed and point to \midas\mscb\embedded\include
  
• The file config.h has to be copied from a similar project and adjusted to fit the new project.
  

> very recent mhttpd mangles spaces in URL encoding-decoding and I cannot create or delete entries in for 
> example "/experiment/edit on start". For example attempt to delete "/experiment/Pedestals Run" 
> produces:
> <h1>Cannot find key Experiment/edit%20on%20start/Pedestals run</h1>
> (notice "%20" instead of spaces. further navigation sometimes replaces the "%" sign with "%25" making it 
> even more mangled)
> 
> this used to work. looks like a call to URL unmangling went missing somewhere.
> K.O.

Thanks for reporting. Fixed in SVN revision 4882. Actually I outcommented the fix some time ago and forgot to 
put it back. Now I hope that this does not blow anything else...

- Stefan

The general Midas documentation has been rejuvenated by Suzannah Daviel through 
a proof reading and with a collection of custom perl scripts to improve the 
Doxygen capabilities for the document itself. In particular, a contents list and 
alphabetical index of the documentation is generated automatically.

The new content is based on the previous version but with more cross-references, 
examples and descriptive images where necessary. Many of the previously 
undocumented features are now included.

The layout and organization is slightly different and requires getting used to, 
but hopefully will be an improvement.
Some of the changes are:
- The main topics are maintained, but we try to regroup all the aspects
  related to a particular topic in the same section.
- The Yellow icons provide navigation within the index section.
- The Blue icons provide navigation within the section content.

The full documentation is included under the midas/doc/src directory in the 
Midas distribution (SVN) and can be generated with the Doxygen tool.
The midasdoc-images.tar.gz from either https://midas.psi.ch/download/ or 
http://ladd00.triumf.ca/~daqweb/ftp/ needs to be extracted to the midas 
directory under doc/images for complete local web pages generation.

There are a few "ToDo" items which hopefully will be ironed out soon.
Feel free to contact us for pointing out omissions or improvements.

We hope this online documentation will serve as a better tool for your 
understanding of the Midas capabilities.

   A couple of experiments at triumf wanted certain important odb variables
displayed on their status page.  (There was already the possibility to show the
run comment)
   A new folder "/Experiment/Status Items" was created containing links to the
variables of interest, these items are show on the status page, under the run
comment (if any), in 3 columns.

the code from mhttpd.c:show_status_page()
between
   /*---- run comment ----*/
and
   /*---- Equipment list ----*/
is attached

Please note that mlogger does not work (crashes on run start) starting with svn
rev 4858, fixed in svn 4862. If you have to use this busted version of mlogger,
the crash is fixed by update of history_midas.c to svn rev 4862 or set ODB
/Logger/WriteFileHistory to 'n'. Sorry for the inconvenience. K.O.

very recent mhttpd mangles spaces in URL encoding-decoding and I cannot create or delete entries in for 
example "/experiment/edit on start". For example attempt to delete "/experiment/Pedestals Run" 
produces:
<h1>Cannot find key Experiment/edit%20on%20start/Pedestals run</h1>
(notice "%20" instead of spaces. further navigation sometimes replaces the "%" sign with "%25" making it 
even more mangled)

this used to work. looks like a call to URL unmangling went missing somewhere.
K.O.

> The attached errors all seem to be from cut-and-paste line breaks in the long "document.writeln()" statements. 
> When the page runs, there are no errors from Firefox and Safari.

Then it would be good if you re-submit the file as an attachment so that other people can use it.

> This example uses "document.writeln()" because the number of PostAmp devices displayed in the table is not 
> known in advance and is potentially read from ODB at page load time.

This was not a criticism but just to show that there are different ways of constructing such a page, depending on the 
needs. So people have the choice. Anyhow I think it's very good to have some working examples for people to start 
with.

> > We had javascript ODBGet() and ODBSet() functions for some time now, permitting 
> implementation of 
> > "page-reload-free" "self-updating" web pages. I finally got around to put all 
> the javascript bits together 
> > to actually implement such a page.
> 
> Unfortunately the page has tons of JavaScript errors, probably happened during 
> copy-and-paste to elog.

The attached errors all seem to be from cut-and-paste line breaks in the long "document.writeln()" statements. 
When the page runs, there are no errors from Firefox and Safari.

This example uses "document.writeln()" because the number of PostAmp devices displayed in the table is not 
known in advance and is potentially read from ODB at page load time.

K.O.

Please find attached another example of a JavaScript (JS) page using the 
ODBGet/Set functions. 

In contrast to the previous posting, the page is not constructed via the 
document.writeln() function, but written directly in HTML and modified through the 
"innerHTML = ..." functionality. 

It is a control page for our beamline, which gets updated in the background. In 
addition, the user can set the beamline to three predefined settings which are 
stored in an array at the top of the page. As an little extra there is a progress 
bar, which is updated locally via JS since changing the beamline takes a while. 
The progress bar is implemented as a table with variable width, and dynamically 
updated by the JS program. The second attachment is a screen dump from such a 
switching process. Since only values in the ODB are changed, you can try it 
yourself without actually modifying a PSI beam line ;-)

> We had javascript ODBGet() and ODBSet() functions for some time now, permitting 
implementation of 
> "page-reload-free" "self-updating" web pages. I finally got around to put all 
the javascript bits together 
> to actually implement such a page.

Unfortunately the page has tons of JavaScript errors, probably happened during 
copy-and-paste to elog. Note that 
such files are better places as attachment. I attached a screen dump from the JS 
debugger inside Chrome which I 
use normally to debug JS.

We had javascript ODBGet() and ODBSet() functions for some time now, permitting implementation of 
"page-reload-free" "self-updating" web pages. I finally got around to put all the javascript bits together 
to actually implement such a page. The main difference from a normal MIDAS "custom" page is the data 
update method - instead of fully reloading the page (via "<meta http-equiv="Refresh" content="60">" 
or javascript location.reload()) - I use ODBGet() to read new data from ODB and HTML DOM access to 
update it on the web page. Note that this is not quite AJAX because the load() function is synchronous 
and (i.e. on the MacOS 10.6 Safari web browser) completely freezes the web browser during data update 
(but no freeze on the Linux Firefox, go figure). An asynchronous ODBGet() should be easy to implement, 
but I can see how a fully asynchronous load() function would lose some of the simplicity of this 
example. (I hope elog does not mangle my example too much).
K.O.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <title>PostAmp control</title>
  </head>

  <body>
    <h1>PostAmp control</h1>

    <script src='mhttpd.js'></script>
    <script type="text/javascript">

var numcrates = 2;
var numpachan;
var reloadTimerId = 0;

function load()
{
  document.getElementById('LastUpdated').innerHTML = "Reloading..." + new Date;

  var crates_table = document.getElementById('crates');
  var slots_table = document.getElementById('cards');

  var Csn = ODBGet('/Equipment/PostAmp/Settings/PACtrlSerialNo[*]');
  for (var i = 0; i < numcrates; i++)
    crates_table.rows[1+i].cells[1].innerHTML = Csn[i];

  var MaxTemp = ODBGet('/Equipment/PostAmp/Variables/MaxTemp[*]');
  for (var i = 0; i < numcrates; i++)
    crates_table.rows[1+i].cells[2].innerHTML = MaxTemp[i];

  var D_TP = ODBGet('/Equipment/PostAmp/Variables/D_TP[*]');
  var M_TP = ODBGet('/Equipment/PostAmp/Variables/M_TP[*]');

  for (var i = 0; i < numcrates; i++)
    crates_table.rows[1+i].cells[3].innerHTML = D_TP[i] + " / " + M_TP[i];

  var sn = ODBGet('/Equipment/PostAmp/Settings/PASerialNo[*]');
  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[1].innerHTML = sn[i];

  var VoltageP  = ODBGet('/Equipment/PostAmp/Variables/VoltageP[*]');
  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[2].innerHTML = VoltageP[i];

  var VoltageM  = ODBGet('/Equipment/PostAmp/Variables/VoltageM[*]');
  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[3].innerHTML = VoltageM[i];

  var Temp      = ODBGet('/Equipment/PostAmp/Variables/Temp[*]');
  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[4].innerHTML = Temp[i];

  var D_VTp     = ODBGet('/Equipment/PostAmp/Variables/D_VTp[*]');
  var M_VTp     = ODBGet('/Equipment/PostAmp/Variables/M_VTp[*]');

  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[5].innerHTML = D_VTp[i] + " / " + M_VTp[i];

  var D_Thresh  = ODBGet('/Equipment/PostAmp/Variables/D_Thresh[*]');
  var M_ThreshA = ODBGet('/Equipment/PostAmp/Variables/M_ThreshA[*]');
  var M_ThreshB = ODBGet('/Equipment/PostAmp/Variables/M_ThreshB[*]');

  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[6].innerHTML = D_Thresh[i] + " / " + M_ThreshA[i] + " / " + M_ThreshB[i];

  document.getElementById('LastUpdated').innerHTML = "Last updated: " + new Date;
}

function reload()
{
  clearTimeout(reloadTimerId);
  load();
  reloadTimerId = setTimeout('reload()', 10000);
}

function main()
{
  clearTimeout(reloadTimerId);

  document.writeln("<p id=LastUpdated>Last updated: </p>");

  document.writeln("<input type=button value='Reload' onClick='reload();'></input>");
  document.writeln("<input type=button value='TP enable' onClick='clearTimeout(reloadTimerId); 
ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gtp 1\"); reload();'></input>");
  document.writeln("<input type=button value='TP disable' onClick='clearTimeout(reloadTimerId); 
ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gtp 0\"); reload();'></input>");
  //document.writeln("<input type=button value='Thresh 100' onClick='clearTimeout(reloadTimerId); 
ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gvth 100\"); reload();'></input>");
  //document.writeln("<input type=button value='Vtest 200' onClick='clearTimeout(reloadTimerId); 
ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gvtest 200\"); reload();'></input>");

  document.writeln("Set VTp: ");
  document.writeln("<input type=input size=5 value='200' onKeyPress='if (event.keyCode==13) { 
clearTimeout(reloadTimerId); ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gvtest \" + 
this.value); reload(); }'></input>");

  document.writeln("Set Thresh: ");
  document.writeln("<input type=input size=5 value='100' onKeyPress='if (event.keyCode==13) { 
clearTimeout(reloadTimerId); ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gvth \" + 
this.value); reload(); }'></input>");

  document.write("<table id=crates border=1>");
  
  document.writeln("<tr align=center>");
  document.writeln("<th>Crate");
  document.writeln("<th>SerialNo");
  document.writeln("<th>MaxTemp");
  document.writeln("<th>D_TP / M_TP");
  document.writeln("</tr>");
  

  for (c = 0; c < numcrates; c++) {
    document.writeln("<tr align=center>");
    document.writeln("<td>" + c);
    document.writeln("<td>sn");
    document.writeln("<td>maxtemp");
    document.writeln("<td>d_tp/m_tp");
    document.writeln("</tr>");
  }
  document.writeln("</table>");
  
  document.write("<table id=cards border=1>");
  
  document.writeln("<tr align=center>");
  document.writeln("<th>Crate/Slot");
  document.writeln("<th>SerialNo");
  document.writeln("<th>V+5");
  document.writeln("<th>V-5");
  document.writeln("<th>Temp");
  document.writeln("<th>VTp");
  document.writeln("<th>Thresh");
  document.writeln("</tr>");
  
  for (c = 0; c < numcrates; c++) {
    for (s = 1; s <= 24; s++) {
      xchan = (c*24) + (s-1);
      document.writeln("<tr align=center>");
      document.writeln("<td>" + c + "/" + s + "/" + xchan);
      document.writeln("<td>sn");
      document.writeln("<td>vp");
      document.writeln("<td>vm");
      document.writeln("<td>temp");
      document.writeln("<td>d_vtpm/m_vtp");
      document.writeln("<td>d_thresh/m_thresha/m_threshb");
      document.writeln("</tr>");
    }
  }
  document.writeln("</table>");

  numpachan = xchan+1;
}

main();
reload();

//ODBSet('/Equipment/FgdWiener01/Settings/outputSwitch[8]', value);

    </script>

    <hr>
    <address><a href="xxx@xxx">Expt S1249</a></address>
<!-- Created: Tue Sep 21 15:44:39 PDT 2010 -->
<!-- hhmts start -->
Last modified: Wed Sep 22 08:30:31 PDT 2010
<!-- hhmts end -->
  </body>
</html>

Prior to odb.c rev 4829 and mhttpd.c rev 4830 committed a few minutes ago, HTML javascript 
ODBGet("/non_existant_odb_entry") caused ODB corruption requiring ODB reload from backup file.

It turns out that ODBGet() tries to create ODB entries if they do not already exist, but because ODBGet() was 
called without the "type", "length", etc arguments, the mhttpd "jset" command was issued with "type" set to 
zero. This resulted in a db_create_key() call with "type" set to zero which created an invalid ODB entry. 
odb.c rev 4829 adds a check for "type<=0" (check for "type>=TID_LAST" was already there).

In addition, mhttpd.c rev 4830 adds a "jset" check for type==0.
K.O.

Just some idea:

The ultimate solution to that would be to do that completely JavaScript driven. You load ONCE the list of all 
variables into a local array, then sort this into your history panel LOCALLY. When I did the original mhttpd 
history config page, there was not much JavaScript around, but today this would be the ultimate option. It 
even supports drag-and-drop. So let's keep that in mind for the future.

- Stefan

In svn rev 4825, I added host based access control to mserver (the MIDAS RPC server). The implementation 
is a verbatim copy mhttpd host based access control list (-a command line switch).

Same as for mhttpd, "mserver -a hostname" enables access control and only permits access from listed 
host names (supply multiple -a switches for multiple hostnames).

This access control does not apply yet for the MIDAS RPC socket connections between MIDAS clients used 
to do RPC callbacks, i.e. to request run transitions. Each MIDAS program is listening for MIDAS RPC 
connections on a high TCP port and at present accepts connections from anybody. To implement access 
controls one could add "-a" switches to every midas application (lot of work) or fill the access control list 
automatically from ODB. mserver still has to use the "-a" command line switches because there is no ODB 
connection when it has to accept or reject remote sockets.

svn rev 4825
K.O.

> mhttpd.c svn rev 4823 implements a modified history planel editor. all previous functions should work 
> as before (minus new bugs). New experimental functions added:
> 
> a) there is a new column "Order" containing numbers 10, 20, 30, etc. ...

While this seems to work well enough, it might remain a function for "advanced users". For novice
users, a simpler gui, i.e. with "move up" and "move down" buttons, would have been "better", or
at least more familiar. (However I have double plus negative experience using nice
looking "move up and down buttons" to rearrange something I actually need to rearrange,
so I have no interest in implementing something I do not want to use. Think about moving
an item all the way from the bottom of a 10 item list to the very top. No do this not as a mental
exercise, but on a slow loading mhttpd web page running somewhere in Japan).

> b) there is a new button "List all variables" to list all existing variables

Some improvement here (mhttpd.c svn 4823): variables are organized by equipment and by history event
into an expandable list. (I already know that this list expansion does not play well with web page
scrolling, same problem exists in the ODB inline editor).

Again, midas users who have a small number of history events may find this new function
not so useful, but the old way was pretty much unusable for T2K/ND280.

Also, for users with a large number of history events, there 2 new ODB variables
/History/MaxDisplayEvents and /History/MaxDisplayTags which limit the maximum
number of events and tags listed in the old scrollable "option" selector history editor.
For the T2K/ND280 case, this reduces the size of the web page and reduces the page load
time quite substantially. (I picked default values of 20 events and 200 tags quite arbitrary,
perhaps the default should have been "no limit", but then nobody would benefit from this
possibility to substantially reduce web page load times - unless they read documentation (yea, right!)
that is not yet written).

K.O.

mhttpd.c svn rev 4823 implements a modified history planel editor. all previous functions should work 
as before (minus new bugs).

New experimental functions added:

a) there is a new column "Order" containing numbers 10, 20, 30, etc. If you change "30" to (say) "15" 
and press "refresh", the history variables will be reordered according to the new values. If you change 
(say) "10" to "" (empty) or "0" and press "refresh", this variable will be deleted. (But there is a UI wart - if 
you accidentally change the order value to something non-numeric (i.e. "aaa1" or " 1" (leading space) 
and press "enter", the variable will be immediately deleted from odb - "enter" works as "refresh + save" 
- should probably work as "refresh" requiring explicit press on the "save" button).

b) there is a new button "List all" to list all existing variables - next to each variable is a checkbox - 
select any checkboxes and press "add selected" to add selected variables to the history plot. You may 
find this function useful (or not), depending on how many variables you have in your history. For 
T2K/ND280 this is still not good enough (there are still too many variables) and I want to change this to 
a 3 level (equipment, history event, history tag) expandable/collapsable tree (or whatever is simplest to 
implement) - to permit the user to quickly zoom on the interesting variables.

I may still tweak with the UI of these new functions, but the basic functionality (reorder+delete and 
selection of multiple variables from a list) seems to be solid. Comments and suggestions on how to 
make it work the best for your experiment are very welcome.

K.O.

> It looks like some example drivers in .../drivers/class want to link against YBOS libraries. 
> This fails because ybos.o is missing from the MIDAS library.

I fixed the class drivers in meantime (SVN 4814).

There is however another problem: The lazylogger needs YBOS support compiled in if the FTP transfer mode is used. 
At PSI we are stuck at the moment to FTP, so we still need YBOS there (although none of the data is in YBOS format). 
Maybe there is a chance that this will be fixed some time and we can get rid of YBOS.