By request from Renee, I increased the MIDAS BANKLIST_MAX from 64 to 1024 and
after fixing a few buglets where YB_BANKLIST_MAX is used instead of (now bigger)
BANKLIST_MAX, I can do a full dump of ND280 FGD events (96 banks).

I also noticed that "mdump -b BANK" did not work, it turns out that it could not
handle 32bit-banks at all. This is now fixed, too.

svn rev 4624
K.O.

I notice that "mserver -s" (a non-default mode of operation) does not work right
- if I connect odbedit for the first time, all is okey, if I connect the second
time, mserver crashes - because after the first connection closed,
rpc_deregister_functions() was called, rpc_list is deleted and causes a crash
later on. Because everybody uses the default "mserver -m" mode, I am not sure
how important it is to fix this.
K.O.

mserver update svn rev 4625 fixes an anomaly in the MIDAS RPC network code where
in some network configurations MIDAS mserver connections work, but some RPC
transactions, such as starting and stopping runs, do not (use the wrong network
names or are routed over the wrong network).

The problem is a possible discrepancy between network addresses used to
establish the mserver connection and the value of "/System/Clients/xxx/Host"
which is ultimately set to the value of "hostname" of the remote client. This
ODB setting is then used to establish additional network connections, for
example to start or stop runs.

Use the client "hostname" setting works well for standard configurations, when
there is only one network interface in the machine, with only one IP address,
and with "hostname" set to the value that this IP address resolves to using DNS.

However, if there are private networks, multiple network interfaces, or multiple
network routes between machines, "/System/Clients/xxx/Host" may become set to an
undesirable value resulting in asymmetrical network routing or complete failure
to establish RPC connections.

Svn rev 4625 updates mserver.c to automatically set "/System/clients/xxx/Host"
to the same network name as was used to establish the original mserver connection.

As always with networking, any fix always breaks something somewhere for
somebody, in which case the old behavior can be restored by "setenv
MIDAS_MSERVER_DO_NOT_USE_CALLBACK_ADDR 1" before starting mserver.

The specific problem fixed by this change is when the MIDAS client and server
are on machines connected by 2 separate networks ("client.triumf.ca" and
"client.daq"; "server.triumf.ca" and "server.daq"). The ".triumf.ca" network
carries the normal SSH, NFS, etc traffic, and the ".daq" network carries MIDAS
data traffic.

The client would use the "server.daq" name to connect to the server and this
traffic would go over the data network (good).

However, previously, the client "/System/Clients/xxx/Host" would be set to
"client.triumf.ca" and any reverse connections (i.e. RPC to start/stop runs)
would go over the normal ".triumf.ca" network (bad).

With this modification, mserver will set "/System/Clients/xxx/Host" to
"client.daq" (the IP address of the interface on the ".daq" network) and all
reverse connections would also go over the ".daq" network (good).

P.S. This modification definitely works only for the default "mserver -m" mode,
but I do not think this is a problem as using "-s" and "-t" modes is not
recommended, and the "-s" mode is definitely broken (see my previous message).

svn rev 4625
K.O.

> 
> "mserver -s" is there for historical reasons and for debugging.
>

I confirm that my modification also works for "mserver -s". I also added an assert() to the
place in midas.c were it eventually crashes, to make it more obvious for the next guys.

K.O.

Notes on the implementation of the MIDAS alarm system.

Alarms are checked inside alarm.c::al_check(). This function is called by
cm_yield() every 10 seconds and by rpc_server_thread(), also every 10 seconds.

For remote midas clients, their al_check() issues an RPC_AL_CHECK RPC call into
the mserver, where rpc_server_dispatch() calls the local al_check().

As result, all alarm checks run inside a process directly attached to the local
midas shared memory (inside a local client or inside an mserver process for a
remote client).

Each and every midas client runs the alarm checks. To prevent race conditions
between different midas clients, access to al_check() is serialized using the
ALARM semaphore.

Inside al_check(), alarms are triggered using al_trigger_alarm(), which in turn
calls al_trigger_class(). Inside al_trigger_class(), the alarm is recorded into
an elog or into midas.log using cm_msg(MTALK).

Special note should be made of the ODB setting "/Alarm/Classes/xxx/System
message interval", which has a surprising effect - after an alarm is recorded
into system messages (using cm_msg(MTALK)), no record is made of any subsequent
alarms until the time interval set by this variable elapses. With default value
of 60 seconds, after one alarm, no more alarms are recorded for 60 seconds.
Also, because all the alarms are checked at the same time, only the first
triggered alarm will be recorded.

As of alarm.c rev 4683, "System message interval" set to 0 ensures that every
alarm is recorded into the midas log file. (In previous revisions, this setting
may still miss some alarms).

There are 3 types of alarms:

1) "program not running" alarms.

These alarms are enabled in ODB by setting "/Programs/ppp/Alarm class". Each
time al_check() runs, every program listed in "/Programs" is tested using
"cm_exist()" and if the program is not running, the time of first failure is
remembered in "/Programs/ppp/First failed".

If the program has not been running for longer than the time set in ODB
"/Programs/ppp/Check interval", an alarm is triggered (if enabled by
"/Programs/ppp/Alarm class" and the program is restarted (if enabled by
"/Programs/ppp/Auto restart").

The "not running" condition is tested every 10 seconds (each time al_check() is
called), but the frequency of "program not running" alarms can be reduced by
increasing the value of "/Alarms/Alarms/ppp/Check interval" (default value 60
seconds). This can be useful if "System message interval" is set to zero.

2) "evaluated" alarms
3) "periodic" alarms

There is nothing surprising in these alarms. Each alarm is checked with a time
period set by "/Alarm/xxx/Check interval". The value of an evaluated alarm is
computed using al_evaluate_condition().

K.O.

> This is my first time running an experiment on separate computers. I followed 
> the documentation (https://midas.psi.ch/htmldoc/quickstart.html) to setup the 
> files:
> /etc/services
> /etc/xinetd.d/midas


Hi, there. I have not recently run mserver through inetd, and we usually do not do
that at TRIUMF. We do this:

a) on the main computer: start mserver: "mserver -p 7070 -D" (note - use non-default
port - can use different ports for different experiments)
b) on remote computer: "odbedit -h main:7070" ("main" is the hostname of your main
computer). Use same "-h" switch for all other programs, including the frontends.

This works well when all computers are on the same network, but if you have some
midas clients running on private networks you may get into trouble when they try to
connect to each other and fail because network routing is funny.


K.O.

> > The way connections work under Midas is there is a callback scheme. The client starts 
> > mserver on the back-end, then the back-end connects back to the front-end on three 
> > different ports. These ports are assigned dynamically by the operating system and are 
> > typically in the range 40000-60000. So you also have to allow the reverse connection on 
> > your firewalls.
> 
> It works now after allowing ports 40000-60000 in the front-end computer. Thanks!


Yes, right. Midas networking does not like firewalls.

In the nutshell, TCP connections on all TCP ports have to be open between all computers
running MIDAS. I think in practice it is not a problem: you only ever have a finite (a small
integer) number of computers running MIDAS and you can be added them as exceptions to the
firewall rules. These exceptions should not create any security problem because you still have
the MIDAS computers firewalled from the outside world and one hopes that they will not be
attacking each other.

P.S. Permitting ports 40000-60000 is not good enough. TCP ports are allocated to TCP
connections semi-randomly from a 16-bit address space (0..65535) and your system will bomb
whenever port numbers like 39999 or 60001 get used.


K.O.

As of svn rev 4794, midas builds, runs and should be fully usable on MacOS 10.6.4. Previous revisions did 
not compile due to assorted Linuxisms and did not run because of a sizeof() problem in ss_gettid(). Also 
one of the system header files (mtio.h?) present in MacOS 10.5 vanished from 10.6.

Please continue reporting all problems with midas on macos to this forum.
K.O.

As of svn rev 4800, YBOS support was made optional, disabled by default. (But note that ybos.c is still used 
by mdump). See HAVE_YBOS in the Makefile.
K.O.

> As of svn rev 4800, YBOS support was made optional, disabled by default. (But note that ybos.c is still used 
> by mdump). See HAVE_YBOS in the Makefile.


It looks like some example drivers in .../drivers/class want to link against YBOS libraries. This fails because ybos.o is missing from the MIDAS library.

After discussions with SR and PAA, we think YBOS support can be removed or made optional, but there are too many of these drivers for me to fix 
them all right now in five minutes. Please accept my apology and use these workarounds:

If you get linker errors because of missing YBOS functions:

1) enable YBOS suport in the Makefile (uncomment HAVE_YBOS=1), or
2) "#ifdef HAVE_YBOS" all places that call YBOS functions

Solution (2) is preferable as it permits us to eventually remove YBOS completely. If you fix files from MIDAS svn, please do send me patches or diffs (or 
post them here).


K.O.

As of svn rev 4807, src/system.c has an experimental implementation of POSIX shared memory. It is 
similar to the already existing implementation of MMAP shared memory, but uses POSIX shm_open() 
instead of directly mmapping the .xxx.SHM file.

There are several benefits to using POSIX shared memory:
1) on MacOS, the (unchangable?) maximum SYSV shared memory is about 2 Mbytes, too small for most 
MIDAS experiments. POSIX shared memory does not seem to have such a limit;
2) on Linux, when using SYSV shared memory, the .xxx.SHM files are tied to the shared memory keys 
using ftok(). If the .xxx.SHM files are located on an NFS-mounted filesystem, ftok() has been observed 
to malfunction and return the same key for different shared memory buffers, causing mighty confusing 
behaviour of MIDAS. (while "man ftok" discusses a theoretical possibility for such collisions, I have 
observed ftok() collisions first hand on a running experiment and it cost us several hours to understand 
why all the events go into the wrong event buffers). The present POSIX shared memory implementation 
does not have such a problem.

This implementation has received limited testing on Linux and MacOS, and it is now the default shared 
memory implementation on MacOS. Linux continues to use SYSV shared memory (ipcs & co). Windows 
uses it's own implementation of shared memory (same as mmap, the best I can tell).

svn 4807
K.O.

> As of svn rev 4794, midas builds, runs and should be fully usable on MacOS 10.6.4. Previous revisions did 
> not compile due to assorted Linuxisms and did not run because of a sizeof() problem in ss_gettid(). Also 
> one of the system header files (mtio.h?) present in MacOS 10.5 vanished from 10.6.

It turns out that on MacOS 10.6 the default maximum SYSV shared memory size is about 2 Mbytes, too small even for the default MIDAS SYSTEM 
event buffer.

Svn revision 4807 implements POSIX shared memory, which does not seem to have such a small size limit and makes it the default on MacOS.

This update fixes the last issue that I am aware of for running MIDAS on MacOS.

svn rev 4807
K.O.

mhttpd.c svn rev 4823 implements a modified history planel editor. all previous functions should work 
as before (minus new bugs).

New experimental functions added:

a) there is a new column "Order" containing numbers 10, 20, 30, etc. If you change "30" to (say) "15" 
and press "refresh", the history variables will be reordered according to the new values. If you change 
(say) "10" to "" (empty) or "0" and press "refresh", this variable will be deleted. (But there is a UI wart - if 
you accidentally change the order value to something non-numeric (i.e. "aaa1" or " 1" (leading space) 
and press "enter", the variable will be immediately deleted from odb - "enter" works as "refresh + save" 
- should probably work as "refresh" requiring explicit press on the "save" button).

b) there is a new button "List all" to list all existing variables - next to each variable is a checkbox - 
select any checkboxes and press "add selected" to add selected variables to the history plot. You may 
find this function useful (or not), depending on how many variables you have in your history. For 
T2K/ND280 this is still not good enough (there are still too many variables) and I want to change this to 
a 3 level (equipment, history event, history tag) expandable/collapsable tree (or whatever is simplest to 
implement) - to permit the user to quickly zoom on the interesting variables.

I may still tweak with the UI of these new functions, but the basic functionality (reorder+delete and 
selection of multiple variables from a list) seems to be solid. Comments and suggestions on how to 
make it work the best for your experiment are very welcome.

K.O.

> mhttpd.c svn rev 4823 implements a modified history planel editor. all previous functions should work 
> as before (minus new bugs). New experimental functions added:
> 
> a) there is a new column "Order" containing numbers 10, 20, 30, etc. ...

While this seems to work well enough, it might remain a function for "advanced users". For novice
users, a simpler gui, i.e. with "move up" and "move down" buttons, would have been "better", or
at least more familiar. (However I have double plus negative experience using nice
looking "move up and down buttons" to rearrange something I actually need to rearrange,
so I have no interest in implementing something I do not want to use. Think about moving
an item all the way from the bottom of a 10 item list to the very top. No do this not as a mental
exercise, but on a slow loading mhttpd web page running somewhere in Japan).

> b) there is a new button "List all variables" to list all existing variables

Some improvement here (mhttpd.c svn 4823): variables are organized by equipment and by history event
into an expandable list. (I already know that this list expansion does not play well with web page
scrolling, same problem exists in the ODB inline editor).

Again, midas users who have a small number of history events may find this new function
not so useful, but the old way was pretty much unusable for T2K/ND280.

Also, for users with a large number of history events, there 2 new ODB variables
/History/MaxDisplayEvents and /History/MaxDisplayTags which limit the maximum
number of events and tags listed in the old scrollable "option" selector history editor.
For the T2K/ND280 case, this reduces the size of the web page and reduces the page load
time quite substantially. (I picked default values of 20 events and 200 tags quite arbitrary,
perhaps the default should have been "no limit", but then nobody would benefit from this
possibility to substantially reduce web page load times - unless they read documentation (yea, right!)
that is not yet written).

K.O.

In svn rev 4825, I added host based access control to mserver (the MIDAS RPC server). The implementation 
is a verbatim copy mhttpd host based access control list (-a command line switch).

Same as for mhttpd, "mserver -a hostname" enables access control and only permits access from listed 
host names (supply multiple -a switches for multiple hostnames).

This access control does not apply yet for the MIDAS RPC socket connections between MIDAS clients used 
to do RPC callbacks, i.e. to request run transitions. Each MIDAS program is listening for MIDAS RPC 
connections on a high TCP port and at present accepts connections from anybody. To implement access 
controls one could add "-a" switches to every midas application (lot of work) or fill the access control list 
automatically from ODB. mserver still has to use the "-a" command line switches because there is no ODB 
connection when it has to accept or reject remote sockets.

svn rev 4825
K.O.

Prior to odb.c rev 4829 and mhttpd.c rev 4830 committed a few minutes ago, HTML javascript 
ODBGet("/non_existant_odb_entry") caused ODB corruption requiring ODB reload from backup file.

It turns out that ODBGet() tries to create ODB entries if they do not already exist, but because ODBGet() was 
called without the "type", "length", etc arguments, the mhttpd "jset" command was issued with "type" set to 
zero. This resulted in a db_create_key() call with "type" set to zero which created an invalid ODB entry. 
odb.c rev 4829 adds a check for "type<=0" (check for "type>=TID_LAST" was already there).

In addition, mhttpd.c rev 4830 adds a "jset" check for type==0.
K.O.

We had javascript ODBGet() and ODBSet() functions for some time now, permitting implementation of 
"page-reload-free" "self-updating" web pages. I finally got around to put all the javascript bits together 
to actually implement such a page. The main difference from a normal MIDAS "custom" page is the data 
update method - instead of fully reloading the page (via "<meta http-equiv="Refresh" content="60">" 
or javascript location.reload()) - I use ODBGet() to read new data from ODB and HTML DOM access to 
update it on the web page. Note that this is not quite AJAX because the load() function is synchronous 
and (i.e. on the MacOS 10.6 Safari web browser) completely freezes the web browser during data update 
(but no freeze on the Linux Firefox, go figure). An asynchronous ODBGet() should be easy to implement, 
but I can see how a fully asynchronous load() function would lose some of the simplicity of this 
example. (I hope elog does not mangle my example too much).
K.O.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <title>PostAmp control</title>
  </head>

  <body>
    <h1>PostAmp control</h1>

    <script src='mhttpd.js'></script>
    <script type="text/javascript">

var numcrates = 2;
var numpachan;
var reloadTimerId = 0;

function load()
{
  document.getElementById('LastUpdated').innerHTML = "Reloading..." + new Date;

  var crates_table = document.getElementById('crates');
  var slots_table = document.getElementById('cards');

  var Csn = ODBGet('/Equipment/PostAmp/Settings/PACtrlSerialNo[*]');
  for (var i = 0; i < numcrates; i++)
    crates_table.rows[1+i].cells[1].innerHTML = Csn[i];

  var MaxTemp = ODBGet('/Equipment/PostAmp/Variables/MaxTemp[*]');
  for (var i = 0; i < numcrates; i++)
    crates_table.rows[1+i].cells[2].innerHTML = MaxTemp[i];

  var D_TP = ODBGet('/Equipment/PostAmp/Variables/D_TP[*]');
  var M_TP = ODBGet('/Equipment/PostAmp/Variables/M_TP[*]');

  for (var i = 0; i < numcrates; i++)
    crates_table.rows[1+i].cells[3].innerHTML = D_TP[i] + " / " + M_TP[i];

  var sn = ODBGet('/Equipment/PostAmp/Settings/PASerialNo[*]');
  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[1].innerHTML = sn[i];

  var VoltageP  = ODBGet('/Equipment/PostAmp/Variables/VoltageP[*]');
  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[2].innerHTML = VoltageP[i];

  var VoltageM  = ODBGet('/Equipment/PostAmp/Variables/VoltageM[*]');
  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[3].innerHTML = VoltageM[i];

  var Temp      = ODBGet('/Equipment/PostAmp/Variables/Temp[*]');
  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[4].innerHTML = Temp[i];

  var D_VTp     = ODBGet('/Equipment/PostAmp/Variables/D_VTp[*]');
  var M_VTp     = ODBGet('/Equipment/PostAmp/Variables/M_VTp[*]');

  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[5].innerHTML = D_VTp[i] + " / " + M_VTp[i];

  var D_Thresh  = ODBGet('/Equipment/PostAmp/Variables/D_Thresh[*]');
  var M_ThreshA = ODBGet('/Equipment/PostAmp/Variables/M_ThreshA[*]');
  var M_ThreshB = ODBGet('/Equipment/PostAmp/Variables/M_ThreshB[*]');

  for (var i = 0; i < numpachan; i++)
    slots_table.rows[1+i].cells[6].innerHTML = D_Thresh[i] + " / " + M_ThreshA[i] + " / " + M_ThreshB[i];

  document.getElementById('LastUpdated').innerHTML = "Last updated: " + new Date;
}

function reload()
{
  clearTimeout(reloadTimerId);
  load();
  reloadTimerId = setTimeout('reload()', 10000);
}

function main()
{
  clearTimeout(reloadTimerId);

  document.writeln("<p id=LastUpdated>Last updated: </p>");

  document.writeln("<input type=button value='Reload' onClick='reload();'></input>");
  document.writeln("<input type=button value='TP enable' onClick='clearTimeout(reloadTimerId); 
ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gtp 1\"); reload();'></input>");
  document.writeln("<input type=button value='TP disable' onClick='clearTimeout(reloadTimerId); 
ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gtp 0\"); reload();'></input>");
  //document.writeln("<input type=button value='Thresh 100' onClick='clearTimeout(reloadTimerId); 
ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gvth 100\"); reload();'></input>");
  //document.writeln("<input type=button value='Vtest 200' onClick='clearTimeout(reloadTimerId); 
ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gvtest 200\"); reload();'></input>");

  document.writeln("Set VTp: ");
  document.writeln("<input type=input size=5 value='200' onKeyPress='if (event.keyCode==13) { 
clearTimeout(reloadTimerId); ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gvtest \" + 
this.value); reload(); }'></input>");

  document.writeln("Set Thresh: ");
  document.writeln("<input type=input size=5 value='100' onKeyPress='if (event.keyCode==13) { 
clearTimeout(reloadTimerId); ODBSet(\"/Equipment/PostAmp/Settings/Command\", \"gvth \" + 
this.value); reload(); }'></input>");

  document.write("<table id=crates border=1>");
  
  document.writeln("<tr align=center>");
  document.writeln("<th>Crate");
  document.writeln("<th>SerialNo");
  document.writeln("<th>MaxTemp");
  document.writeln("<th>D_TP / M_TP");
  document.writeln("</tr>");
  

  for (c = 0; c < numcrates; c++) {
    document.writeln("<tr align=center>");
    document.writeln("<td>" + c);
    document.writeln("<td>sn");
    document.writeln("<td>maxtemp");
    document.writeln("<td>d_tp/m_tp");
    document.writeln("</tr>");
  }
  document.writeln("</table>");
  
  document.write("<table id=cards border=1>");
  
  document.writeln("<tr align=center>");
  document.writeln("<th>Crate/Slot");
  document.writeln("<th>SerialNo");
  document.writeln("<th>V+5");
  document.writeln("<th>V-5");
  document.writeln("<th>Temp");
  document.writeln("<th>VTp");
  document.writeln("<th>Thresh");
  document.writeln("</tr>");
  
  for (c = 0; c < numcrates; c++) {
    for (s = 1; s <= 24; s++) {
      xchan = (c*24) + (s-1);
      document.writeln("<tr align=center>");
      document.writeln("<td>" + c + "/" + s + "/" + xchan);
      document.writeln("<td>sn");
      document.writeln("<td>vp");
      document.writeln("<td>vm");
      document.writeln("<td>temp");
      document.writeln("<td>d_vtpm/m_vtp");
      document.writeln("<td>d_thresh/m_thresha/m_threshb");
      document.writeln("</tr>");
    }
  }
  document.writeln("</table>");

  numpachan = xchan+1;
}

main();
reload();

//ODBSet('/Equipment/FgdWiener01/Settings/outputSwitch[8]', value);

    </script>

    <hr>
    <address><a href="xxx@xxx">Expt S1249</a></address>
<!-- Created: Tue Sep 21 15:44:39 PDT 2010 -->
<!-- hhmts start -->
Last modified: Wed Sep 22 08:30:31 PDT 2010
<!-- hhmts end -->
  </body>
</html>

> > We had javascript ODBGet() and ODBSet() functions for some time now, permitting 
> implementation of 
> > "page-reload-free" "self-updating" web pages. I finally got around to put all 
> the javascript bits together 
> > to actually implement such a page.
> 
> Unfortunately the page has tons of JavaScript errors, probably happened during 
> copy-and-paste to elog.

The attached errors all seem to be from cut-and-paste line breaks in the long "document.writeln()" statements. 
When the page runs, there are no errors from Firefox and Safari.

This example uses "document.writeln()" because the number of PostAmp devices displayed in the table is not 
known in advance and is potentially read from ODB at page load time.

K.O.

very recent mhttpd mangles spaces in URL encoding-decoding and I cannot create or delete entries in for 
example "/experiment/edit on start". For example attempt to delete "/experiment/Pedestals Run" 
produces:
<h1>Cannot find key Experiment/edit%20on%20start/Pedestals run</h1>
(notice "%20" instead of spaces. further navigation sometimes replaces the "%" sign with "%25" making it 
even more mangled)

this used to work. looks like a call to URL unmangling went missing somewhere.
K.O.

Please note that mlogger does not work (crashes on run start) starting with svn
rev 4858, fixed in svn 4862. If you have to use this busted version of mlogger,
the crash is fixed by update of history_midas.c to svn rev 4862 or set ODB
/Logger/WriteFileHistory to 'n'. Sorry for the inconvenience. K.O.