> Midas has a nice alarm system. I am wondering whether it is easily possible to
> get the Alarm/Warning banner also on top of custom pages?!

K.O. made nice JavaScript routines to access the alarm status. The new alarm page is completely 
made dynamically from JavaScript code (mhttpd does not supply any HTML code any more, only 
functions to obtain ODB values etc). Part of this new dynamic page must be some code to display 
the alarm status. You just need to copy this to your custom page. K.O. can tell you details.

Stefan

> > Midas has a nice alarm system. I am wondering whether it is easily possible to
> > get the Alarm/Warning banner also on top of custom pages?!
> 
> K.O. made nice JavaScript routines to access the alarm status. The new alarm page is completely 
> made dynamically from JavaScript code (mhttpd does not supply any HTML code any more, only 
> functions to obtain ODB values etc). Part of this new dynamic page must be some code to display 
> the alarm status. You just need to copy this to your custom page. K.O. can tell you details.
> 

Yes, please look at resources/alarm.html and the "get_alarms" JSON-RPC method. The "get_alarms" example in 
resources/example.html probably already does exactly what you need. Also note the presence of "al_reset_alarm" and 
"al_trigger_alarm" JSON_RPC methods.

K.O.

I want midas alarms to go off when I cannot ping arbitrary remote hosts. Is
there is easy/preferred way to do this? K.O.

> I want midas alarms to go off when I cannot ping arbitrary remote hosts. Is
> there is easy/preferred way to do this? K.O.

There are "internal alarms" with type AT_EVALUATED. Just find a program 
where you can put some code which gets periodically executed (like the idle 
loop in the frontend), and so something like:

DWORD last = 0;

  if (ss_time() > last+60)
    {
    last = ss_time();

    /* do a ping via socket(), bind() and connect() */
    ...

    if (status != CM_SUCCESS)
      al_trigger_alarm("XYZ Ping", str, "Warning", 
                       "Host is dead", AT_INTERNAL);
    }

Pierre does the same thing in lazylogger.c, just have a look. I don't know 
how to do a ping correctly in C, I guess you have to send an UDP packet 
somewhere, but I never did it. If you find it out, please post it.


 

Dear MIDAS users,

I want to add several .root-files produced by the MIDAS analyzer, in a fast 
and convenient way. ROOT's hadd fails because it does not know how to treat 
TFolders. I guess this problem is not unique to me, so I hope that somebody of 
you might already have found a solution.

Why don't I just run "analyzer -r 1 10000"?
We have taken lots of runs under (rapidly) varying conditions, so it would be 
lots of "-r". And the analysis is quite involved, so rerunning all data takes 
about one hour on a fast PC making this quite painful.
Therefore, I would like to rerun all data only once, and then add the result 
files depending on different criteria.

Of course, I tried to write a script that does the adding. But somehow it is 
incredibly slow. And I am not the Master Of C++, too.

Is there any deeper reason for MIDAS using TFolders, not TDirectorys? ROOT's 
hadd can treat TDirectory. Can I simply patch "my" MIDAS? Is there general 
interest in a change like this? (Does anyone have experience with the speed of 
hadd?)

Looking forward to comments from the Forum.

Cheers,

Randolf

The following program handles regular directories in a file, or folders (ugh).
Most histograms are added bin by bin.

For scaler events it is convenient to see the counts as a function of time (ala
sclaer history plots in mhttpd).  If the histogram looks like a scaler plot versus
time, then new bins are added on to the end (or into the middle!) of the histogram.

All different versions of cuts are kept.

TTrees are not explicitly supported, so probably don't do the right thing...

John.

> Dear MIDAS users,
> 
> I want to add several .root-files produced by the MIDAS analyzer, in a fast 
> and convenient way. ROOT's hadd fails because it does not know how to treat 
> TFolders. I guess this problem is not unique to me, so I hope that somebody of 
> you might already have found a solution.
> 
> Why don't I just run "analyzer -r 1 10000"?
> We have taken lots of runs under (rapidly) varying conditions, so it would be 
> lots of "-r". And the analysis is quite involved, so rerunning all data takes 
> about one hour on a fast PC making this quite painful.
> Therefore, I would like to rerun all data only once, and then add the result 
> files depending on different criteria.
> 
> Of course, I tried to write a script that does the adding. But somehow it is 
> incredibly slow. And I am not the Master Of C++, too.
> 
> Is there any deeper reason for MIDAS using TFolders, not TDirectorys? ROOT's 
> hadd can treat TDirectory. Can I simply patch "my" MIDAS? Is there general 
> interest in a change like this? (Does anyone have experience with the speed of 
> hadd?)
> 
> Looking forward to comments from the Forum.
> 
> Cheers,
> 
> Randolf

Hello,

we are using the Struck SIS 3100 VME interface for our experiment, but the midas
driver doesn't have interrupt control integrated. Previously we were happy with
just periodic readout, but our requirements have changed so I thought I could
just implement this as there is a demo program provided by Struck on how to use
their driver with interrupts.

Could you recommend an existing midas driver that has a good implementation of
the midas interrupt functions (mvme_interrupt_*) just for me too use as a guideline?

Best regards,
Clemens Sauerzopf

> Hello,
> 
> we are using the Struck SIS 3100 VME interface for our experiment, but the midas
> driver doesn't have interrupt control integrated. Previously we were happy with
> just periodic readout, but our requirements have changed so I thought I could
> just implement this as there is a demo program provided by Struck on how to use
> their driver with interrupts.
> 
> Could you recommend an existing midas driver that has a good implementation of
> the midas interrupt functions (mvme_interrupt_*) just for me too use as a guideline?
> 
> Best regards,
> Clemens Sauerzopf

Hi Clemens,

We did have interrupt handling at some point under VxWorks and later with Linux, but it 
has always been a challenge.
As you may have found, the current frontend (mfe.c) still has some code to that purpose. 
But I wouldn't guarantee that recent development related to multi-threading didn't 
affect the expected interrupt operation (not been tested).

Now-a-days, I would suggest that you encapsulate your interrupt handling function based 
on the provided software into a independent thread started by a standard midas frontend. 
While the main frontend task could operate a periodic equipment as you've done so far, a 
polling equipment would poll on the data availability from the ring buffer. The readout 
function would compose the appropriate data bank.

This method has the advantage to decouple all the interrupt timing/restriction related 
issues from midas and run a conventional frontend. The ring buffer functions are part of 
midas (rb_...()).
Example for multi-threading can be found in examples/mtfe which include the use of the 
ring buffer as well.

Cheers, PAA

Hi Pierre-Andre,

thanks for your comments. If I understand you correctly you are advising to separate the
triggering based on the interrupt signal and the actual data readout. In principal wouldn't
it be also possible to facilitate the multi-threading equipment type to poll the trigger
signal? Then veto new triggers and start the readout of the different detector modules by a
"manual trigger" ?

I'll check the example you've recommended to compare the different solutions.

By the way I've written a driver for the CAEN V1742 VME module, it's working but the code is
currently not in a "nice" state. but if you are interested I could provide the driver code.

Cheers,
Clemens

> > Hello,
> > 
> > we are using the Struck SIS 3100 VME interface for our experiment, but the midas
> > driver doesn't have interrupt control integrated. Previously we were happy with
> > just periodic readout, but our requirements have changed so I thought I could
> > just implement this as there is a demo program provided by Struck on how to use
> > their driver with interrupts.
> > 
> > Could you recommend an existing midas driver that has a good implementation of
> > the midas interrupt functions (mvme_interrupt_*) just for me too use as a guideline?
> > 
> > Best regards,
> > Clemens Sauerzopf
> 
> Hi Clemens,
> 
> We did have interrupt handling at some point under VxWorks and later with Linux, but it 
> has always been a challenge.
> As you may have found, the current frontend (mfe.c) still has some code to that purpose. 
> But I wouldn't guarantee that recent development related to multi-threading didn't 
> affect the expected interrupt operation (not been tested).
> 
> Now-a-days, I would suggest that you encapsulate your interrupt handling function based 
> on the provided software into a independent thread started by a standard midas frontend. 
> While the main frontend task could operate a periodic equipment as you've done so far, a 
> polling equipment would poll on the data availability from the ring buffer. The readout 
> function would compose the appropriate data bank.
> 
> This method has the advantage to decouple all the interrupt timing/restriction related 
> issues from midas and run a conventional frontend. The ring buffer functions are part of 
> midas (rb_...()).
> Example for multi-threading can be found in examples/mtfe which include the use of the 
> ring buffer as well.
> 
> Cheers, PAA

Hello Clemens,

The hardware readout is triggered by the interrupt within this thread. The main thread poll on  
the data availability (from the rb) to filter/compose the frontend event.
In a similar multi-threaded implementation presently used in a dark matter experiment we start 
as many thread as necessary to constantly poll on the hardware for "data fragment" collection.
The event composition is done in the main thread through polling on the RBs.

Depending on the trigger rate and readout time, we can afford to analyze the data fragment at 
the thread level and add computed/summary information to the ring buffer on a event-by-event 
basis. This facilitate the overall event filtering happening later on in our event builder. 

"polling the trigger signal?", I don't understand. You can poll on the trigger condition but 
then you don't need interrupt.

The original Midas interrupt implementation was to let the interrupt function set a acknowledge 
flag which is picked up by the standard midas polling function (user code) for triggering the 
readout. This method ensure a minimal time spent in the IRQ and works fine for a single thread.

In regards of the CAEN V1742, we do have a VME driver for it, but it hasn't been added to the 
Midas yet (quite recent), but please don't hesitate to send us a copy.

Cheers, PAA

 

> Hi Pierre-Andre,
> 
> thanks for your comments. If I understand you correctly you are advising to separate the
> triggering based on the interrupt signal and the actual data readout. In principal wouldn't
> it be also possible to facilitate the multi-threading equipment type to poll the trigger
> signal? Then veto new triggers and start the readout of the different detector modules by a
> "manual trigger" ?
> 
> I'll check the example you've recommended to compare the different solutions.
> 
> By the way I've written a driver for the CAEN V1742 VME module, it's working but the code is
> currently not in a "nice" state. but if you are interested I could provide the driver code.
> 
> Cheers,
> Clemens
> 
> > > Hello,
> > > 
> > > we are using the Struck SIS 3100 VME interface for our experiment, but the midas
> > > driver doesn't have interrupt control integrated. Previously we were happy with
> > > just periodic readout, but our requirements have changed so I thought I could
> > > just implement this as there is a demo program provided by Struck on how to use
> > > their driver with interrupts.
> > > 
> > > Could you recommend an existing midas driver that has a good implementation of
> > > the midas interrupt functions (mvme_interrupt_*) just for me too use as a guideline?
> > > 
> > > Best regards,
> > > Clemens Sauerzopf
> > 
> > Hi Clemens,
> > 
> > We did have interrupt handling at some point under VxWorks and later with Linux, but it 
> > has always been a challenge.
> > As you may have found, the current frontend (mfe.c) still has some code to that purpose. 
> > But I wouldn't guarantee that recent development related to multi-threading didn't 
> > affect the expected interrupt operation (not been tested).
> > 
> > Now-a-days, I would suggest that you encapsulate your interrupt handling function based 
> > on the provided software into a independent thread started by a standard midas frontend. 
> > While the main frontend task could operate a periodic equipment as you've done so far, a 
> > polling equipment would poll on the data availability from the ring buffer. The readout 
> > function would compose the appropriate data bank.
> > 
> > This method has the advantage to decouple all the interrupt timing/restriction related 
> > issues from midas and run a conventional frontend. The ring buffer functions are part of 
> > midas (rb_...()).
> > Example for multi-threading can be found in examples/mtfe which include the use of the 
> > ring buffer as well.
> > 
> > Cheers, PAA

Hello Pierre-Andre,

thank you for your help with the interrupt handling. To close this case I'll
attach my interrupt
handling code for the SIS 3100 to this post as a reference. Maybe someone wants
to do something
similar in the future. 

I've decide to go for a C++ frontend therefore it is a class that handles
everything. The user only
has to provide a function pointer to the constructor that handles the interrupt
bitmask. The
interrupt handling is done with a timedwait within a separate thread. 

Cheers,
Clemens

> Hello Clemens,
> 
> The hardware readout is triggered by the interrupt within this thread. The
main thread poll on  
> the data availability (from the rb) to filter/compose the frontend event.
> In a similar multi-threaded implementation presently used in a dark matter
experiment we start 
> as many thread as necessary to constantly poll on the hardware for "data
fragment" collection.
> The event composition is done in the main thread through polling on the RBs.
> 
> Depending on the trigger rate and readout time, we can afford to analyze the
data fragment at 
> the thread level and add computed/summary information to the ring buffer on a
event-by-event 
> basis. This facilitate the overall event filtering happening later on in our
event builder. 
> 
> "polling the trigger signal?", I don't understand. You can poll on the trigger
condition but 
> then you don't need interrupt.
> 
> The original Midas interrupt implementation was to let the interrupt function
set a acknowledge 
> flag which is picked up by the standard midas polling function (user code) for
triggering the 
> readout. This method ensure a minimal time spent in the IRQ and works fine for
a single thread.
> 
> In regards of the CAEN V1742, we do have a VME driver for it, but it hasn't
been added to the 
> Midas yet (quite recent), but please don't hesitate to send us a copy.
> 
> Cheers, PAA
> 

Creating an ODB key requires users to know the Type ID that are defined in 
https://bitbucket.org/tmidas/midas/src/develop/include/midas.h starting at line 320.

I can't find any information on the Midas Wiki about these values or how to find 
them.

Am I missing something obvious?  Is there a way to improve how to find these values?  
Or is this not the best way to interact with the ODB?

> Creating an ODB key requires users to know the Type ID that are defined in 
> https://bitbucket.org/tmidas/midas/src/develop/include/midas.h starting at line 320.
> 
> I can't find any information on the Midas Wiki about these values or how to find 
> them.
> 
> Am I missing something obvious?  Is there a way to improve how to find these values?  
> Or is this not the best way to interact with the ODB?

Well, you found them in midas.h, so where is the problem?

If you want a more detailed description, just look in the midas documentation (RTFM):

https://midas.triumf.ca/MidasWiki/index.php/Midas_Data_Types

If you want a more modern interface to the ODB without these data types, look here:

https://midas.triumf.ca/MidasWiki/index.php/Odbxx

Best regards,
Stefan

In svn rev 4825, I added host based access control to mserver (the MIDAS RPC server). The implementation 
is a verbatim copy mhttpd host based access control list (-a command line switch).

Same as for mhttpd, "mserver -a hostname" enables access control and only permits access from listed 
host names (supply multiple -a switches for multiple hostnames).

This access control does not apply yet for the MIDAS RPC socket connections between MIDAS clients used 
to do RPC callbacks, i.e. to request run transitions. Each MIDAS program is listening for MIDAS RPC 
connections on a high TCP port and at present accepts connections from anybody. To implement access 
controls one could add "-a" switches to every midas application (lot of work) or fill the access control list 
automatically from ODB. mserver still has to use the "-a" command line switches because there is no ODB 
connection when it has to accept or reject remote sockets.

svn rev 4825
K.O.

Commited to CVS is the preliminary driver for the Wiener CC-USB CAMAC interface.
The driver implements all the mcstd.h camac access functions, except for those
not supported by hardware (8-bit operations, interrupts) and a few esoteric
functions not implemented in any other camac driver. The driver uses the
musbstd.h library to access USB, also commited in preliminary form.

Affected files:
midas/Makefile (added musbstd.c to libmidas.{a,so})
include/musbstd.h, src/musbstd.c (preliminary USB access library)
drivers/bus/ccusb.{c,h}

Most of the CAMAC access functions have been tested (see comments in ccusb.c).
If you find errors and problems, please email me (olchansk@triumf.ca) or write
an elog reply to this elog message.

Missing is the documentation and finalization of USB access library.
Missing is conformity to some MIDAS coding conventions.

Enjoy,
K.O.

I now commited the changes to mlogger (mlogger.c, msystem.h) implementing data
compression using zlib (svn revision 3845)

To enable compression, observe that mlogger is compiled with -DHAVE_ZLIB (see
the Makefile), in "/Logger/Channels/NNN/Settings", set "compression" to "1" and
the filename to "run%05d.mid.gz" (note the suffix ".gz").

In the Makefile, I only enabled HAVE_ZLIB for Linux, as that is the only
platform I tested. If somebody can test compression on Windows, please do and
let us know.

My ROOT analyzer (rootana) package can read compressed MIDAS files directly and
if one wants to add this capability to other MIDAS-related packages, one is
welcome to use my TMidasFile.cxx as an example
(http://ladd00.triumf.ca/viewcvs/rootana/trunk/TMidasFile.cxx?view=markup).

K.O.

mlogger (history_image) now supports rtsp cameras, in ALPHA we have 
acquisitioned several new network connected cameras. Unfortunately they dont 
have a way of just capturing a single frame using libcurl


========================================
Motivation to link to OpenCV libraries
========================================

After looking at the ffmpeg libraries, it seemed non trivial to use them to 
listen to a rtsp stream and write a series of jpgs.

OpenCV became an obvious choice (it is itself linked to ffmpeg and 
gstreamer), its a popular, multiplatform, open source library that's easy to 
use. It is available in the default package managers in centos 7 and ubuntu 
(an is installed by default on lxplus).

========================================
How it works:
========================================

The framework laid out in history_image.cxx is great. A separate thread is 
dedicated for each camera. This is continued with the rtsp support, using 
the same periodicity:

if (ss_time() >= o["Last fetch"] + o["Period"]) {
An rtsp camera is detected by its URL, if the URL starts with ‘rtsp://’ its 
obvious its using the rtsp protocol and the cv::VideoCapture object is 
created (line 147).

If the connection fails, it will continue to retry, but only send an error 
message on the first 10 attempts (line 150). This counter is reset on 
successful connection
If MIDAS has been built without OpenCV, mlogger will send an error message 
that OpenCV is required if a rtsp URL is given (line 166)
The VideoCapture ‘stays live' and will grab frames from the camera based on 
the sleep, saving to file based on the Period set in the ODB.

If the VideoCapture object is unable to grab a frame, it will release() the 
camera, send an error message to MIDAS, then destroy itself, and create a 
new version (this destroy and create fully resets the connection to a 
camera, required if its on flaky wifi)
If the VideoCapture gets an empty frame, it also follows the same reset 
steps.
If the VideoCaption fills a cv::Frame object successfully, the image is 
saved to disk in the same way as the curl tools.

========================================
Concerns for the future:
========================================

VideoCapture is decoding the video stream in the background, allowing us to 
grab frames at will. This is nice as we can be pretty agnostic to the video 
format in the stream (I tested with h264 from a TP-LINK TAPO C100, but the 
CPU usage is not negligible.

I noticed that this used ~2% of the CPU time on an intel i7-4770 CPU, given 
enough cameras this is considerable. In ALPHA, I have been testing with 10 
cameras:
elog:2220/1 

My suggestion / request would be to move the camera management out of 
mlogger and into a new program (mcamera?), so that users can choose to off 
load the CPU load to another system (I understand the OpenCV will use GPU 
decoders if available also, which can also lighten the CPU load).

> mlogger (history_image) now supports rtsp cameras

my goodness, we will drive the video surveillance industry out of business.

> My suggestion / request would be to move the camera management out of 
> mlogger and into a new program (mcamera?), so that users can choose to off 
> load the CPU load to another system (I understand the OpenCV will use GPU 
> decoders if available also, which can also lighten the CPU load).

every 2 years I itch to separate mlogger into two parts - data logger
and history logger.

but then I remember that the "I" in MIDAS stands for "integrated",
and "M" stands for "maximum" and I say, "nah..."

(I guess we are not maximum integrated enough to have mhttpd, mserver
and mlogger to be one monolithic executable).

There is also a line of thinking that mlogger should remain single-threaded
for maximum reliability and ease of debugging. So if we keep adding multithreaded
stuff to it, perhaps it should be split-apart after all. (anything that makes
the size of mlogger.cxx smaller is a good thing, imo).

K.O.