Notes on the implementation of the MIDAS alarm system.

Alarms are checked inside alarm.c::al_check(). This function is called by
cm_yield() every 10 seconds and by rpc_server_thread(), also every 10 seconds.

For remote midas clients, their al_check() issues an RPC_AL_CHECK RPC call into
the mserver, where rpc_server_dispatch() calls the local al_check().

As result, all alarm checks run inside a process directly attached to the local
midas shared memory (inside a local client or inside an mserver process for a
remote client).

Each and every midas client runs the alarm checks. To prevent race conditions
between different midas clients, access to al_check() is serialized using the
ALARM semaphore.

Inside al_check(), alarms are triggered using al_trigger_alarm(), which in turn
calls al_trigger_class(). Inside al_trigger_class(), the alarm is recorded into
an elog or into midas.log using cm_msg(MTALK).

Special note should be made of the ODB setting "/Alarm/Classes/xxx/System
message interval", which has a surprising effect - after an alarm is recorded
into system messages (using cm_msg(MTALK)), no record is made of any subsequent
alarms until the time interval set by this variable elapses. With default value
of 60 seconds, after one alarm, no more alarms are recorded for 60 seconds.
Also, because all the alarms are checked at the same time, only the first
triggered alarm will be recorded.

As of alarm.c rev 4683, "System message interval" set to 0 ensures that every
alarm is recorded into the midas log file. (In previous revisions, this setting
may still miss some alarms).

There are 3 types of alarms:

1) "program not running" alarms.

These alarms are enabled in ODB by setting "/Programs/ppp/Alarm class". Each
time al_check() runs, every program listed in "/Programs" is tested using
"cm_exist()" and if the program is not running, the time of first failure is
remembered in "/Programs/ppp/First failed".

If the program has not been running for longer than the time set in ODB
"/Programs/ppp/Check interval", an alarm is triggered (if enabled by
"/Programs/ppp/Alarm class" and the program is restarted (if enabled by
"/Programs/ppp/Auto restart").

The "not running" condition is tested every 10 seconds (each time al_check() is
called), but the frequency of "program not running" alarms can be reduced by
increasing the value of "/Alarms/Alarms/ppp/Check interval" (default value 60
seconds). This can be useful if "System message interval" is set to zero.

2) "evaluated" alarms
3) "periodic" alarms

There is nothing surprising in these alarms. Each alarm is checked with a time
period set by "/Alarm/xxx/Check interval". The value of an evaluated alarm is
computed using al_evaluate_condition().

K.O.

Hi,

On a custom web page, can a "filled" area be made to flash (i.e. cycle between 
two colours)? This area would have to update faster than the whole page update.
 
I have a custom page representing a gas system, and the users
want the heaters to flash when they are on, as is done in their EPICS page.

Thanks,
Suzannah 

> > The custom and alias links in the standard midas status page were shown as HTML 
> > links so far. If there are many links with names having spaces in their names, 
> > it's a bit hard to distinguish between them. Therefore, they are packed now into 
> > individual buttons (see attachment) starting from SVN revision 4633 on. This makes 
> > also the look more homogeneous. If there is any problem with that, please report.
> 
> Would you consider using a different colour for the alias buttons (or background
> colour)? At present it's hard to know whether a button is an alias link, a custom page
> link or a user-button especially if you are not familiar with the button layout. 

Ok, I changed the background colors for the button rows. There are now four different 
colors: Main menu buttons, Scripts, Manually triggered events, Alias & Custom pages. Hope 
this is ok. Of course one could have each button in a different color, but then it gets 
complicated... In that case I would recommend to make a dedicated custom page with all these 
buttons, which you can then tailor exactly to your needs.

> The custom and alias links in the standard midas status page were shown as HTML 
> links so far. If there are many links with names having spaces in their names, 
> it's a bit hard to distinguish between them. Therefore, they are packed now into 
> individual buttons (see attachment) starting from SVN revision 4633 on. This makes 
> also the look more homogeneous. If there is any problem with that, please report.

Would you consider using a different colour for the alias buttons (or background
colour)? At present it's hard to know whether a button is an alias link, a custom page
link or a user-button especially if you are not familiar with the button layout. 

/Experiment/Menu Buttons = Start, ODB, Messages, ELog, Alarms, Programs, History, MSCB, Config, Help

[local:MEG:R]/MSCB>ls -r
MSCB
    Submaster
        mscb004
            Pwd                 xxxxx
            Comment             BTS
            Address             1
        mscb034
            Pwd                 xxxxx
            Comment             XEC HV & LED
            Address
                                0
                                1
                                2

/Equipment/<name>/Settings/Devices/Input/Device

/Equipment/<name>/Settings/Devices/<name>/MSCB Device

Dear Stefan,

I followed your suggestion to try the sample front-ends from the distribution and 
they work fine. They also work fine with any one of my front-ends. Only my two 
front-ends cannot run concurrently in the same directory. I later found that the 
problem is in the CAEN HV wrapper library. The problem arises when the front-ends 
are both linked to that library and it is solved now.

Thanks & Best Regards,
Jimmy


> Hi Jimmy,
> 
> ok, now I understand. Well, I don't see your problem. I just tried with the 
> current SVN 
> version to start
> 
> midas/examples/experiment/frontend
> midas/examples/slowcont/scfe
> 
> in the same directory (without "-h localhost") and it works just fine (see 
> attachemnt). I even started them from the same directory. Yes there are *.SHM 
> files and they correspond to shared memory, but both front-ends use this shared 
> memory together (that's why it's called 'shared').
> 
> Your error message 'Semaphore already present' is strange. The string is not 
> contained in any midas program, so it must come from somewhere else. Do you 
> maybe try to access the same hardware with the two front-end programs?
> 
> I would propose you do the following: Use the two front-ends from the 
> distribution (see above). They do not access any hardware. See if you can run 
> them with the current SVN version of midas. If not, report back to me.
> 
> Best regards,
> 
>   Stefan
> 
> 
> > Dear Stefan,
> > 
> > Thanks for the reply. I have tried your patch and it didn't solve my problem. 
> Maybe I 
> > have not written my question clearly. The two frontends could run on the same 
> computer 
> > if I use the remote method, i.e. by setting up the mserver and connect to the 
> > experiment by specifying "-h localhost", also the frontend programs need to be 
> put in 
> > different directory. What I want to know is whether I can simply start 
> multiple 
> > frontends in the same directory without setting up the mserver etc. I noticed 
> that 
> > there are several *.SHM files, I'm not familiar with semaphore, but I guess 
> they are 
> > the key to the problem. Please correct me if I misunderstood something.
> > 
> > Best Regards,
> > Jimmy
> > 
> > 
> > > > Dear All,
> > > > 
> > > > I want to run two frontend programs (one for trigger and one for slow 
> control)
> > > > concurrently on the same computer, but I failed. The second frontend said: 
> > > > 
> > > > Semaphore already present
> > > >  There is another process using the semaphore.
> > > >  Or a process using the semaphore exited abnormally.
> > > >  In That case try to manually release the semaphore with:
> > > >    ipcrm sem XXX.
> > > > 
> > > > The two frontends are connected to the same experiment. Is there any way I 
> can
> > > > overcome this problem?
> > > 
> > > That might be related to the RPC mutex, which gets created system wide now. 
> I 
> > > modified this in midas.c rev. 4628, so there will be one mutex per process. 
> Can you 
> > > try that temporary patch and tell me if it works for you?

Hi Jimmy,

ok, now I understand. Well, I don't see your problem. I just tried with the 
current SVN 
version to start

midas/examples/experiment/frontend
midas/examples/slowcont/scfe

in the same directory (without "-h localhost") and it works just fine (see 
attachemnt). I even started them from the same directory. Yes there are *.SHM 
files and they correspond to shared memory, but both front-ends use this shared 
memory together (that's why it's called 'shared').

Your error message 'Semaphore already present' is strange. The string is not 
contained in any midas program, so it must come from somewhere else. Do you 
maybe try to access the same hardware with the two front-end programs?

I would propose you do the following: Use the two front-ends from the 
distribution (see above). They do not access any hardware. See if you can run 
them with the current SVN version of midas. If not, report back to me.

Best regards,

  Stefan


> Dear Stefan,
> 
> Thanks for the reply. I have tried your patch and it didn't solve my problem. 
Maybe I 
> have not written my question clearly. The two frontends could run on the same 
computer 
> if I use the remote method, i.e. by setting up the mserver and connect to the 
> experiment by specifying "-h localhost", also the frontend programs need to be 
put in 
> different directory. What I want to know is whether I can simply start 
multiple 
> frontends in the same directory without setting up the mserver etc. I noticed 
that 
> there are several *.SHM files, I'm not familiar with semaphore, but I guess 
they are 
> the key to the problem. Please correct me if I misunderstood something.
> 
> Best Regards,
> Jimmy
> 
> 
> > > Dear All,
> > > 
> > > I want to run two frontend programs (one for trigger and one for slow 
control)
> > > concurrently on the same computer, but I failed. The second frontend said: 
> > > 
> > > Semaphore already present
> > >  There is another process using the semaphore.
> > >  Or a process using the semaphore exited abnormally.
> > >  In That case try to manually release the semaphore with:
> > >    ipcrm sem XXX.
> > > 
> > > The two frontends are connected to the same experiment. Is there any way I 
can
> > > overcome this problem?
> > 
> > That might be related to the RPC mutex, which gets created system wide now. 
I 
> > modified this in midas.c rev. 4628, so there will be one mutex per process. 
Can you 
> > try that temporary patch and tell me if it works for you?

Dear Stefan,

Thanks for the reply. I have tried your patch and it didn't solve my problem. Maybe I 
have not written my question clearly. The two frontends could run on the same computer 
if I use the remote method, i.e. by setting up the mserver and connect to the 
experiment by specifying "-h localhost", also the frontend programs need to be put in 
different directory. What I want to know is whether I can simply start multiple 
frontends in the same directory without setting up the mserver etc. I noticed that 
there are several *.SHM files, I'm not familiar with semaphore, but I guess they are 
the key to the problem. Please correct me if I misunderstood something.

Best Regards,
Jimmy


> > Dear All,
> > 
> > I want to run two frontend programs (one for trigger and one for slow control)
> > concurrently on the same computer, but I failed. The second frontend said: 
> > 
> > Semaphore already present
> >  There is another process using the semaphore.
> >  Or a process using the semaphore exited abnormally.
> >  In That case try to manually release the semaphore with:
> >    ipcrm sem XXX.
> > 
> > The two frontends are connected to the same experiment. Is there any way I can
> > overcome this problem?
> 
> That might be related to the RPC mutex, which gets created system wide now. I 
> modified this in midas.c rev. 4628, so there will be one mutex per process. Can you 
> try that temporary patch and tell me if it works for you?

The mhttpd program shows some standard buttons in the top row for 
starting/stopping runs, accessing the ODB, Alarms, etc. Since not all experiments 
make use of all buttons, they have been customized. By default mhttpd creates 
following entry in the ODB:

/Experiment/Menu Buttons = Start, ODB, Messages, ELog, Alarms, Programs, History, 
Config, Help

Which is the standard set (except the old CNAF). People can customize this now by 
removing unnecessary buttons or by changing their order. The "Start" entry above 
actually causes the whole set of Start/Stop/Pause/Resume buttons to appear, 
depending on the current run state. 

Since the column on the main midas status page with fraction of analyzed events is 
barely used, I decided to drop it. Anyhow it does not make sense for all slow 
control events. If this feature is required in some experiment, I propose to move it 
into a custom page and calculate this ratio in JavaScript, where one has much more 
flexibility. 

This modification frees up more space on the status page for the "Status" column, where 
front-end programs can report errors etc.

The custom and alias links in the standard midas status page were shown as HTML 
links so far. If there are many links with names having spaces in their names, 
it's a bit hard to distinguish between them. Therefore, they are packed now into 
individual buttons (see attachment) starting from SVN revision 4633 on. This makes 
also the look more homogeneous. If there is any problem with that, please report.

> Dear All,
> 
> I want to run two frontend programs (one for trigger and one for slow control)
> concurrently on the same computer, but I failed. The second frontend said: 
> 
> Semaphore already present
>  There is another process using the semaphore.
>  Or a process using the semaphore exited abnormally.
>  In That case try to manually release the semaphore with:
>    ipcrm sem XXX.
> 
> The two frontends are connected to the same experiment. Is there any way I can
> overcome this problem?

That might be related to the RPC mutex, which gets created system wide now. I 
modified this in midas.c rev. 4628, so there will be one mutex per process. Can you 
try that temporary patch and tell me if it works for you?

> 
> "mserver -s" is there for historical reasons and for debugging.
>

I confirm that my modification also works for "mserver -s". I also added an assert() to the
place in midas.c were it eventually crashes, to make it more obvious for the next guys.

K.O.

> I notice that "mserver -s" (a non-default mode of operation) does not work right
> - if I connect odbedit for the first time, all is okey, if I connect the second
> time, mserver crashes - because after the first connection closed,
> rpc_deregister_functions() was called, rpc_list is deleted and causes a crash
> later on. Because everybody uses the default "mserver -m" mode, I am not sure
> how important it is to fix this.
> K.O.

"mserver -s" is there for historical reasons and for debugging. I started originally 
with a single process server back in the 90's, and only afterwards developed the multi 
process scheme. The single process server now only works for one connection and then 
crashes, as you described. But it can be used for debugging any server connection, 
since you don't have to follow the creation of a subprocess with your debugger, and 
therefore it's much easier. But after the first connection has been closed, you have 
to restart that single server process. Maybe one could add some warning about that, or 
even fix it, but it's nowhere used in production mode.

mserver update svn rev 4625 fixes an anomaly in the MIDAS RPC network code where
in some network configurations MIDAS mserver connections work, but some RPC
transactions, such as starting and stopping runs, do not (use the wrong network
names or are routed over the wrong network).

The problem is a possible discrepancy between network addresses used to
establish the mserver connection and the value of "/System/Clients/xxx/Host"
which is ultimately set to the value of "hostname" of the remote client. This
ODB setting is then used to establish additional network connections, for
example to start or stop runs.

Use the client "hostname" setting works well for standard configurations, when
there is only one network interface in the machine, with only one IP address,
and with "hostname" set to the value that this IP address resolves to using DNS.

However, if there are private networks, multiple network interfaces, or multiple
network routes between machines, "/System/Clients/xxx/Host" may become set to an
undesirable value resulting in asymmetrical network routing or complete failure
to establish RPC connections.

Svn rev 4625 updates mserver.c to automatically set "/System/clients/xxx/Host"
to the same network name as was used to establish the original mserver connection.

As always with networking, any fix always breaks something somewhere for
somebody, in which case the old behavior can be restored by "setenv
MIDAS_MSERVER_DO_NOT_USE_CALLBACK_ADDR 1" before starting mserver.

The specific problem fixed by this change is when the MIDAS client and server
are on machines connected by 2 separate networks ("client.triumf.ca" and
"client.daq"; "server.triumf.ca" and "server.daq"). The ".triumf.ca" network
carries the normal SSH, NFS, etc traffic, and the ".daq" network carries MIDAS
data traffic.

The client would use the "server.daq" name to connect to the server and this
traffic would go over the data network (good).

However, previously, the client "/System/Clients/xxx/Host" would be set to
"client.triumf.ca" and any reverse connections (i.e. RPC to start/stop runs)
would go over the normal ".triumf.ca" network (bad).

With this modification, mserver will set "/System/Clients/xxx/Host" to
"client.daq" (the IP address of the interface on the ".daq" network) and all
reverse connections would also go over the ".daq" network (good).

P.S. This modification definitely works only for the default "mserver -m" mode,
but I do not think this is a problem as using "-s" and "-t" modes is not
recommended, and the "-s" mode is definitely broken (see my previous message).

svn rev 4625
K.O.

I notice that "mserver -s" (a non-default mode of operation) does not work right
- if I connect odbedit for the first time, all is okey, if I connect the second
time, mserver crashes - because after the first connection closed,
rpc_deregister_functions() was called, rpc_list is deleted and causes a crash
later on. Because everybody uses the default "mserver -m" mode, I am not sure
how important it is to fix this.
K.O.

By request from Renee, I increased the MIDAS BANKLIST_MAX from 64 to 1024 and
after fixing a few buglets where YB_BANKLIST_MAX is used instead of (now bigger)
BANKLIST_MAX, I can do a full dump of ND280 FGD events (96 banks).

I also noticed that "mdump -b BANK" did not work, it turns out that it could not
handle 32bit-banks at all. This is now fixed, too.

svn rev 4624
K.O.

We were debugging a strange problem in the event builder, where out of 14
fragments, two fragments were always getting serial number mismatches and the
serial numbers were not sequentially increasing (the other 12 fragments were
just fine).

Then we noticed in the event builder debug output that these 2 fragments were
getting assigned the same buffer handle number, despite having different names -
BUF09 and BUFTPC.

Then we looked at "ipcs", counted the buffers, and there are only 13 buffers for
14 frontends.

Aha, we went, maybe we have unlucky buffer names, renamed BUFTPC to BUFAAA and
everything started to work just fine.

It turns out that the MIDAS ss_shm_open() function uses "ftok" to convert buffer
names to SystemV shared memory keys. This "ftok" function promises to create
unique keys, but I guess, just not today.

Using a short test program, I confirmed that indeed we have unlucky buffer names
and ftok() returns duplicate keys, see below.

Apparently ftok() uses the low 16 bits of the file inode number, but in our
case, the files are NFS mounted and inode numbers are faked inside NFS. When I
run my test program on a different computer, I get non-duplicate keys. So I
guess we are double unlucky.

Test program:

#include <stdio.h>
#include <sys/types.h>
#include <sys/ipc.h>

int main(int argc, char* argv[])
{
  //key_t ftok(const char *pathname, int proj_id);
  
  int k1 = ftok("/home/t2kdaq/midas/nd280/backend/.BUF09.SHM", 'M');
  int k2 = ftok("/home/t2kdaq/midas/nd280/backend/.BUFTPC.SHM", 'M');
  int k3 = ftok("/home/t2kdaq/midas/nd280/backend/.BUFFGD.SHM", 'M');

  printf("key1: 0x%08x, key2: 0x%08x, key3: 0x%08x\n", k1, k2, k3);
  return 0;
}

[t2kfgd@t2knd280logger ~/xxx]$ g++ -o ftok -Wall ftok.cxx
[t2kfgd@t2knd280logger ~/xxx]$ ./ftok
key1: 0x4d138154, key2: 0x4d138154, key3: 0x4d138152

Also:

[t2kfgd@t2knd280logger ~/xxx]$ ls -li ...
14385492 -rw-r--r-- 1 t2kdaq t2kdaq  8405052 Nov 24 17:42
/home/t2kdaq/midas/nd280/backend/.BUF09.SHM
36077906 -rw-r--r-- 1 t2kdaq t2kdaq 67125308 Nov 26 10:19
/home/t2kdaq/midas/nd280/backend/.BUFFGD.SHM
36077908 -rw-r--r-- 1 t2kdaq t2kdaq  8405052 Nov 25 15:53
/home/t2kdaq/midas/nd280/backend/.BUFTPC.SHM

(hint: print the inode numbers in hex and compare to shm keys printed by the
program)

K.O.