Dear All,

Is there any recommended platforms for running MIDAS? Have anyone encountered 
problems when running MIDAS on Scientific Linux?

Thanks.

Jimmy

> Dear All,
> 
> Is there any recommended platforms for running MIDAS? Have anyone encountered 
> problems when running MIDAS on Scientific Linux?
> 
> Thanks.
> 
> Jimmy

I run MIDAS on scientific Linux 5.1 without any problem.

Dear All,

We have a temperature/humidity sensor in MIDAS now and will add a liquid level 
sensor to MIDAS soon. We want the operators to get alerted ASAP when the 
laboratory environment or the liquid level reached some critical levels. Can 
MIDAS send email alerts or SMS alerts to cell phones when the alarms are 
triggered? If yes, how can I config it?

Many thanks!

Best Regards,
Jimmy

> We have a temperature/humidity sensor in MIDAS now and will add a liquid level 
> sensor to MIDAS soon. We want the operators to get alerted ASAP when the 
> laboratory environment or the liquid level reached some critical levels. Can 
> MIDAS send email alerts or SMS alerts to cell phones when the alarms are 
> triggered? If yes, how can I config it?

Sure that's possible, that's why MIDAS contains an alarm system. To use it, define 
an ODB alarm on your liquid level, like

/Alarms/Alarms/Liquid Level
Active	                 y
Triggered	         0 (0x0)
Type	                 3 (0x3)
Check interval	        60 (0x3C)
Checked last	1227690148 (0x492D10A4)
Time triggered first	(empty)
Time triggered last	(empty)
Condition	        /Equipment/Environment/Variables/Input[0] < 10
Alarm Class	        Level Alarm
Alarm Message	        Liquid Level is only %s

The Condition if course might be different in your case, just select the correct 
variable from your equipment. In this case, the alarm triggers an alarm of class 
"Level Alarm". Now you define this alarm class:

/Alarms/Classes/Level Alarm
Write system message	y
Write Elog message	n
System message interval	600 (0x258)
System message last	0 (0x0)
Execute command	        /home/midas/level_alarm '%s'
Execute interval	1800 (0x708)
Execute last	        0 (0x0)
Stop run	        n
Display BGColor	        red
Display FGColor	        black

The key here is to call a script "level_alarm", which can send emails. Use 
something like:

#/bin/csh
echo $1 | mail -s \"Level Alarm\" your.name@domain.edu
odbedit -c 'msg 2 level_alarm \"Alarm was sent to your.name@domain.edu\"'

The second command just generates a midas system message for confirmation. Most 
cell phones (depends on the provider) have an email address. If you send an email 
there, it gets translated into a SMS message.

The script file above can of course be more complicated. We use a perl script 
which parses an address list, so everyone can register by adding his/her email 
address to that list. The script collects also some other slow control variables 
(like pressure, temperature) and combines this into the SMS message.

For very sensitive systems, having an alarm via SMS is not everything, since the 
alarm system could be down (computer crash or whatever). In this case we use 
'negative alarms' or however you might call it. The system sends every 30 minutes 
an SMS with the current levels etc. If the SMS is missing for some time, it might 
be an indication that something in the midas system is wrong and one can go there 
and investigate.

lazylogger was updated to improve handling of the list of runs still on disk
(odb /Lazy/xxx/List).

Previously, each and every run was listed in the List arrays. With modern
Terabyte-sized data disks, many many days worth of runs tend to remain on disk
and these List arrays were getting too big, inflating the size of ODB dumps
written by mlogger into the output data file and slowing down starting and
stopping of runs considerably.

Now, the runs are listed as ranges of "first run" - "last run", (see example below).

This significantly reduces the size of the "List" arrays and makes lazylogger
usable for the ALPHA experiment at CERN and for T2K/ND280 prototype DAQ at
TRIUMF (writing to Castor and Dcache respectively, using the newly added
"Script" method).

The new List format is fully compatible with the old format and you can update
and run the new lazylogger without changing anything in ODB. New runs will be
added to the List arrays in the new format and data in the old format will
eventually go away as old runs are removed from disk.

svn revision 4394.
K.O.

Example: this reads like this:
range from 7100 to 7154
range from 7157 to 7161 (7155-7156 are missing)
range from 7163 to 7168 (7162 is missing)
runs 7170, 7173, 7176
range from 7179 to 7182
and so forth.

ODB /Lazy/Dcache/List
007100
[0] 7100 (0x1BBC)
[1] -7154 (0xFFFFE40E)
[2] 7157 (0x1BF5)
[3] -7161 (0xFFFFE407)
[4] 7163 (0x1BFB)
[5] -7168 (0xFFFFE400)
[6] 7170 (0x1C02)
[7] 7173 (0x1C05)
[8] 7176 (0x1C08)
[9] 7179 (0x1C0B)
[10] -7182 (0xFFFFE3F2)
[11] 7184 (0x1C10)
[12] 7188 (0x1C14)
[13] -7199 (0xFFFFE3E1)
007200
[0] 7200 (0x1C20)
[1] -7225 (0xFFFFE3C7)

> revision 4142+4143 are minor fixes, refactoring (switch the code to use helper
> functions) and implementation of history for structured banks

The implementation of "history for structured banks" had a bug - tags inside
structured banks were counted incorrectly, leading to memory overwrites and mlogger
crash in open_history().

This is problem is now fixed (plus added assert() checks to crash-out if overwrite of
tags[] array is detected).

svn revision 4398.
K.O.

Dear All,

How can I add a button at the top of the "Status" webpage which will show a 
page similar to the "CNAF" one after I click on it? and how can I make a 
custom page similar to "CNAF" which allow me to call some custom funtions? I 
want to make a page which is particularly for doing calibration.

Thank you for your attention!

Best Regards,
Jimmy Ngai

> How can I add a button at the top of the "Status" webpage which will show a 
> page similar to the "CNAF" one after I click on it? and how can I make a 
> custom page similar to "CNAF" which allow me to call some custom funtions? I 
> want to make a page which is particularly for doing calibration.

The CNAF page calls directly functions through the RPC layer of midas, which is 
not possible from custom pages. All you can do is to execute a scrip on the 
server side, which then causes some action. For details please consult the 
documentation.

Here are my notes on the MIDAS ODB "hot link" function. Perhaps others can find them useful.

Using db_open_record(key,function), the user can tell MIDAS to call the specified user function when 
the specified ODB key is modified by any other MIDAS program. This function works both locally 
(shared memory odb access) and remotely (odb access through mserver tcp rpc). For example, the 
MIDAS "history" mechanism is implemented in the mlogger by "hot-linking" ODB 
"/equipment/xxx/Variables".

First, the relevant data structures defined in midas.h and msystem.h (ODB database headers, etc)

(in midas.h)
#define NAME_LENGTH            32            /**< length of names, mult.of 8! */
#define MAX_CLIENTS            64            /**< client processes per buf/db */
#define MAX_OPEN_RECORDS       256           /**< number of open DB records   */

(in msystem.h)
DATABASE buf <--- local, private to each client)
  DATABASE_HEADER* database_header <--- odb in shared memory
    char name[NAME_LENGTH]
    DATABASE_CLIENT client[MAX_CLIENTS]
      char name[NAME_LENGTH]
      OPEN_RECORD open_record[MAX_OPEN_RECORDS]
        handle
        access_mode
        flags

(the above means that each midas client has access to the list of all open records through
buf->database_header.client[i].open_record[j])

Second, the data path through db_set_data & co: (other odb "write" functions work the same way)

db_set_data(key)
  lock db
  update odb <--- memcpy(), really
  db_notify_clients(key)
  unlock db
  return

db_notify_clients(key)
  loop: <--- data for this key changed and so data for all keys containing it
             also changed, and we need to notify anybody who has an open record
             on the parents of this key. need to loop over parents of this key (follow "..")
  if (key->notify_count)
    foreach client
      foreach open_record
        if (open_record.handle == key)
          ss_resume(client->port, "O hDB hKey")
  key = key.parent
  goto loop;

ss_resume(port, message)
  idx = ss_suspend_get_index()  <--- magic here
  send udp message ("O hDB hKey") to localhost:port <-- notifications sent only to local host!

note 1: I do not completely understand the ss_suspend_xxx() stuff. The best I can tell
is it creates a number of udp sockets bound to the local host and at least one udp rpc
receive socket ultimately connected to the cm_dispatch_rpc() function.

note 2: More magic here: database_header->client[i].port appears to be the udp rpc server
port of the mserver, while ODB /Clients/xxx/Port is the tcp rpc server port
of the client itself, on the remote host

note 3: the following is for remote odb clients connected through the mserver. For local
clients, cm_dispatch_rpc() calls the local db_update_record() as shown at the very end.

note 4: this uses udp rpc. If the udp datagram is lost inside the os kernel (it looks like these udp/rpc 
datagrams never go out to the network), "hot-link" silently fails: code below is not executed. Some 
OSes (namely, Linux) are known to lose udp datagrams with high probability under certain
not very well understood conditions.

local mserver receives the udp datagram
  ...
  cm_dispatch_ipc()
    if (message=="O hDB hKey")
      decode message (hDB, hKey)
      db_update_record(hDB, hKey)
        send tcp rpc with args(MSG_ODB, hDB, hKey)

(note- unlike udp rpc, tcp rpc are never "lost")

remote client receives tcp rpc:
rpc_client_dispatch()
  recv_tcp(net_buffer)
  if (net_buffer.routine_id == MSG_ODB)
    db_update_record(hDB, hKey)

db_update_record(hDB, hKey)
  if remote delivery, see cm_dispatch_ipc() above
  <--- local delivery
  foreach (_recordlist)
    if (recordlist.handle == hKey)
      if (!recordlist.access_mode&MODE_WRITE)
        db_get_record(hDB,hKey,recordlist.data,recordlist.size)
        recordlist.dispatcher(hDB,hKey,recordlist.info); <-- user-supplied handler

Note: the dispatcher() above is the function supplied by the user in db_open_record().

K.O.

> How can I add a button at the top of the "Status" webpage which will show a 
> page similar to the "CNAF" one after I click on it? and how can I make a 
> custom page similar to "CNAF" which allow me to call some custom funtions? I 
> want to make a page which is particularly for doing calibration.


I was going to say that you can do this by using the MIDAS "hot-link" function.

In your equipment program, you create a string /eq/xxx/Settings/Command, and hot-link
it to the function you want to be called. (See midas function db_open_record() for details
and examples). (To test it, you put a call to printf("Hello world!\n") into your handler function,
then change the value of "command" using odbedit or the mhttpd odb editor
and observe that your function gets called and that it receives the correct value of "command").

Then on your custom web page you create 2 buttons "aaa" and "bbb" attached to javascript
ODBset("/eq/xxx/Settings/Command","aaa") and "bbb" respectively. When you push the button,
the specified string is written into ODB, and your hot-link handler function is called with the contents
of "command", which you can then look at to find out which web button was pushed.


But after looking at the hot-link data paths (see https://ladd00.triumf.ca/elog/Midas/546), I see 2 
problems that make the above scheme unreliable and maybe unusable in some applications:

1) the data path contains one UDP communication and it is well known that UDP datagrams can be (and 
are) lost with low or high probability, depending on not-well-understood external factors.

The effect is that the hot-link fails to "fire": odb contents is changed but your function is not called.

2) there is a timing problem with multiple odb writes: the odb lock is dropped before the "hot-link" gets 
to see the new contents of odb: db_data_set()->lock odb->change data->send notification->unlock 
odb->xxx->notification received by client->read the data->call user function. If something else is 
written into odb during "xxx" above, the client may never see the data written by the first odb write. For 
local clients, the delay between "send notification" and "notification is received by client" is not bounded in 
time (can be arbitrary long, depending on the system load, etc). For remote clients, there is an additional 
delay as the udp datagram is received by the local mserver and is forwarded to the remote client through 
a tcp rpc connection (another source of unbounded delay).

The effect is that if buttons "aaa" and "bbb" are pushed quickly one right after the other, while your 
function will be called 2 times (if neither udp packet is dropped), you may never see the value of "aaa"
as is it will be overwritten by "bbb" by the time you receive the first notification.

Probability of malfunction increases with code written like this: { ODBset("command", "open door"); 
ODBset("command", "walk through doorway"); }. You may see the "open door" command sometimes 
mysteriously disappear...

The net effect is that sometimes you will push the button but nothing will happen. This may be okey,
depending on your application and depending on how often it happens in practice on your specific system 
If you are lucky, you may never see either of the 2 problems listed above ad hot-links will work for you 
perfectly. At TRIUMF, in the past, we have seen hot-links misbehave in the TWIST experiment, and now I 
think I understand why (because of the 2 problems described above).

K.O.

The UDP connection you mention is only used locally for inter-process communication. When I implemented that, I 
made extensive tests and found that there is never a packet being dropped. This happens for UDP only if the packet 
goes over a physical network. Maybe this is different in modern Linux versions, so one should double check this 
again.

For remote hot-link notification, the notification is sent over the TCP link, so it should not be lost either. But 
your second point is correct. The hot-link mechanism was developed to change parameters in front-end programs for 
example. So by design it is guaranteed that if you change a value in the ODB, any client hot-linked to that will 
see the change (sooner or later). If there are many changes in short intervals (or the callback function on the 
remote client takes long time), only the last change is guaranteed to arrive. Therefore, as you correctly state, 
the hot-link mechanism is not a save replacement for the RPC layer (That's why the RPC layer is there after all).

mhttpd and mlogger have been updated with potentially troublesome changes.
Before using these latest versions, please make a backup of your ODB. This is
svn revisions 4434 (mhttpd.c) and 4435 (mlogger.c).

These new features are now available:
- a "feature complete" implementation of "history in an SQL database". We use
this new code to write history data from the T2K test setup in the TRIUMF M11
beam line to a MySQL database (mlogger) and to make history plots directly from
this database (mhttpd). We still write normal midas history files and we have a
utility to import midas .hst files into an SQL database (utils/mh2sql). The code
is functional, but incomplete. For best SQL database data layout, you should
enable the "per variable history" (but backup your ODB before you do this!). All
are welcome to try it, kick the tires, report any problems. Documentation TBW.
- experimental implementation of "ODBRpc" added to the midas javascript library
(ODBSet, ODBGet & co). This permits buttons on midas "custom" web pages to
invoke RPC calls directly into user frontend programs, for example to turn
things on or off. Documentation TBW.
- the mlogger/mhttpd implementation of /History/Tags has proved troublesome and
we are moving away from it. The SQL database history implementation already does
not use it. During the present transition period:
- mlogger and mhttpd will now work without /History/Tags. This implementation
reads history tags directly from the history files themselves. Two downsides to
this: it is slower and tags become non-persistent: if some frontends have not
been running for a while, their variables may vanish from the history panel
editor. To run in this mode, set "/History/DisableTags" to "y". Existing
/History/Tags will be automatically deleted.
- for the above 2 reasons, I still recommend using /History/Tags, but the format
of the tags is now changed to simplify management and reduce odb size. mlogger
will automatically convert the tags to this new format (this is why you should
make a backup of your ODB).
- using old mlogger with new mhttpd is okey: new mhttpd understands both formats
of /History/Tags.
- using old mhttpd with new mlogger is okey: please set ODB
"/History/CreateOldTags" to "y" (type TID_BOOL/"boolean") before starting mlogger.

K.O.

• Set /Channels/x/Settings/Subrun Byte limit to the number of bytes for a subrun
  
• Set /Channels/x/Settings/Filename to something like run%05d_%02d.mid. The first %05d gets replaced by the run number, while the second one gets replaced by the subrun number. This will result in files such as

  run00001_00.mid    run #1
  run00001_01.mid      "
  run00001_02.mid      "
  run00001_03.mid      "
  run00002_00.mid    run #2
  run00002_01.mid      "
  run00002_02.mid      "
  run00002_03.mid      "

> mhttpd and mlogger have been updated with potentially troublesome changes.
> These new features are now available:
> - a "feature complete" implementation of "history in an SQL database".

The mlogger SQL history driver has been updated with improvements that make this new system usable in 
production environment: the silly "create all tables on startup, every time, even if they already exist" is fixed,
mlogger survives restarts of mysqld and checks that existing sql columns have data types compatible with the 
data we are trying to write.

There are still a few trouble spots remaining. For example, in mapping midas names into sql names (sql names 
have more restrictions on permitted characters) and in reverse mapping of sql data types to midas data types. 
To properly solve this, I may have to save the midas names and data types into an additional index table.

Included is the mh2sql utility for importing existing history files into an SQL database (in the same way as if 
they were written into the database by mlogger).

The mhttpd side of this system still needs polishing, but should be already fully functional.

A preliminary version of documentation for this new SQL history system is here. After additional review and 
editing it will be committed to the midas midox documentation. Included are full instructions on enabling 
writing of midas history into a MySQL database.
http://ladd00.triumf.ca/~olchansk/midas/Internal.html#History_sql_internal

svn revision 4452
K.O.

(catching up on recent changes from t2k and pienu)

Various timeout problems fixed:
- cm_transition() timeouts now settable from ODB (/experiment/transition timeout, transition connect 
timeout). Rev 4479
- rpc_client_call() timeout did not work because of bad select() and alarm() interaction. Rev 4479
- implement rpc connect timeout (was hardwired 10 sec) via rpc_{set,get}_option(-2, RPC_OTIMEOUT). Rev 
4478
- ss_mutex_wait_for() timeout only worked if 1Hz alarm() interrupts are present. Now I use semtimedop() 
and timeout should always work. Rev 4472

K.O.

When using remote midas clients with mserver, you may have noticed the zero-size .RPC.SHM files 
these clients create in the directory where you run them. These files are associated with the semaphore 
created by the midas rpc layer (rpc_call) to synchronize rpc calls between multiple threads. This 
semaphore is always created, even for single-threaded midas applications. Also normally midas 
semaphore files are created in the midas experiment directory specified in exptab (same place as 
.ODB.SHM), but for remote clients, we do not know that location until we start making rpc calls, so the 
semaphore file is created in the current directory (and it is on a remote machine anyway, so this 
location may not be visible locally).

There are 2 problems with these semaphores:
1) in multiple experiments, we have observed the RPC.SHM semaphore stuck in a locked state, 
requiring manual cleanup (ipcrm -s xxx). So far, I have failed to duplicate this lockup using test 
programs and test experiments. The code appears to be coded correctly to automatically unlock the 
semaphore when the program exits or is killed.
2) RPC.SHM is created as a global shared semaphore so it synchronizes rpc calls not just for all threads 
inside one application, but across all threads in all applications (excessive locking - separate 
applications are connected to separate mservers and do not need this locking); but only for applications 
that run from the same current directory - RPC.SHM files in different directories are "connected" to 
different semaphores.

To try to fix this, I implemented "private semaphores" in system.c and made rpc_call() use them.

This introduced a major bug - a semaphore leak - quickly using up all sysv semaphores (see sysctl 
kernel.sem).

The code was now reverted back to using RPC.SHM as described above.

The "bad" svn revisions start with rev 4472, the problem is fixed in rev 4480.

If you use remote midas clients and have one of these bad revisions, either update midas.c to rev 4480 
or apply this patch to midas.c::rpc_call():
ss_mutex_create("", &_mutex_rpc);
should read
ss_mutex_create("RPC", &_mutex_rpc);

Apologies for any inconvenience caused by this problem
K.O.

mhttpd function for starting and stopping runs now uses cm_transition(DETACH) which spawns an 
external helper program called mtransition to handle the transition sequencing. This helps with the old 
problem of looking at a blank screen for a long time if some frontends take a long time to process run 
transitions. Now mhttpd returns right back and shows start "starting run", "stopping run", etc as 
appropriate.

svn rev 4484 (some bits of this feature are present in rev 4473 and later).
K.O.

P.S. In one of our experiments, I sometimes see mhttpd getting "stuck" when starting or stopping a run 
using this feature. strace shows it is stuck in repeated calls to wait(), but I am unable to reproduce this 
problem in a test system and it happens only sometimes in the experiment. When it does, mhttpd has to 
be restarted. Replacing system("mtransition ...") with ss_sysem("mtransition ...") seems to fix this problem, 
but there are downsides to this (mtransition debug output vanishes) so I am not committing this yet.
K.O.