Dear experts,
              is there any way to launch an executable script on the host computer from the MIDAS 
sequencer? If not, is there any interest to develop such a feature?

Thank you,
         Francesco

> > Asume you have a variable foo and a link bar -> foo. When you go to the ODB in
> > mhttpd, click "Delete" and select bar, it actually deletes foo. bar stays,
> > stating "<cannot resolve link>". Trying the same in odbedit with rm gives the
> > expected result (bar is gone, foo is still there).
> > 
> > I'm on the develop branch.
> 
> I think I can confirm this. Created a bug report on bitbucket:
> 
> https://bitbucket.org/tmidas/midas/issues/148/mhttpd-odb-editor-deletes-wrong-symlink
> 
> K.O.

I fixed this and committed the change. Took me a while since it was in KO's code.

Stefan

Per changes at TRIUMF, the MIDAS forum mail relay was changed from trmail.triumf.ca to 
smtp.triumf.ca. K.O.

> Hi. We've just updated our midas installation to the newest version, and we now see repeated errors from the 
> mserver in messages. Mostly we see
> 
> 11:17:02.994 2018/08/21 [ODBEdit,TALK] Program mserver restarted
> 
> which happens 2-3 times per minute. 
> 
> We have also been seeing occasional dropped rpc connections to our frontends, which could be related. 
> 
> The version we were running with previously was ~1 year old, and we have just updated to the newest version 
> on bitbucket. 

Hmm... usually mserver will not restart automatically, maybe you have set it to autorestart on ODB (/programs/mserver/auto_restart 
set to "y").

It would be unusual for the main mserver program to crash, to debug it, you will need to run it in a terminal
and see if there is any error messages. Even better to run it in a terminal inside "gdb" and capture the stack trace
when it crashes.

Anyhow, crash of main mserver will not cause "dropped rpc connections" to clients - this would require for their
individual mserver subprocesses to crash. Such crashes would be highly unusual and are harder to debug.

Perhaps for the crashes you see there is some error messages in midas.log?

K.O.

> I would like to store the address of 1-Wire temperature sensors in a device
> driver. However, the supportet data types (as definded around
> include/midas.h:311) do not foresee a type large enough.
>

Hmm... you do not say what sensor you use and how many bits you actually need.

For up to 32 bits you can use TID_DWORD (uint32_t) (obviously)

For up to 48 bits (or so), you can use TID_DOUBLE (double) (wierd, but IEEE754 double precision variables would work as 48-bit (or so) integers).

For more, I would use arrays of TID_DWORD (64 bits, store low 32 bits into a[0], high bits into a[1]).

> 
> Is there a good reason against this?
> 

We had requests for implementing uint64_t 64-bit data types in MIDAS before. There are two problems:

a) in the MIDAS data banks, there is a problem with the bank header definition which only has 3 DWORDSs so causes
each alternating data bank to be 64-bit misaligned. And misaligned 64-bit data is very bad.

b) in ODB, 64-bit data support will need to be added from scratch and again it is not clear without doing it
if there will be any alignement problems. If one were to implement ODB from scratch, one would have everything
aligned to 64-bits or maybe even 128-bits, with uint64_t fully supported.

It is unlikely this kind of work will ever be done on ODB, but who knows.

> I know that other experiments use this kind of sensor, how do you store the
> addresses? I've noticed that most of the address is just zeroes, but I wouldn't
> like to store just half the address, assuming that half the address is always
> zeroes.

Cannot answer without knowing what sensor you use, but certainly you can use an array of bytes
or an array of integers to store arbitrarily long addresses. You can also use a TID_STRING
and store the address as a text string "0xabcdabcdabcdabcd" of arbitrary length.

K.O.

Hi, what you try should have worked. Perhaps your symlink is wrong and should say "/External/..." (with a leading slash). The "links period" would have
worked same as equipment/common/history period - as a rate limiter.

Anyhow, I suggest another way to do the same - create a fake equipment - the logger does
not care if the equipment is real or not and if you write into /eq/fake/variables from a proper frontend
or from a script. To hide the fake equipment from the status page, set /eq/fake/common/hidden to "true".

This will work for sure.

K.O.




> Hi,
> I am trying to set up virtual history events following
> https://midas.triumf.ca/MidasWiki/index.php/History_System#Virtual_History_Event 
> 
> Trying it the first way, using the following setup:
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> Links                           DIR
>     dirlink -> External/dir     KEY     1     12    >99d 0   RWD  <subdirectory>
> 
> 
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> External                        DIR
>     dir                         DIR
>         foo                     FLOAT   1     4     16s  0   RWD  12.5
> 
> 
> Then I get the following error message:
> ==================== History link "dirlink", ID 28150  =======================
> [Logger,ERROR] [mlogger.cxx:4942:open_history,ERROR] History event dirlink has
> no variables in ODB
> 
> 
> Trying the second way, I set up the following:
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> Links                           DIR
>     dir                         DIR
>         testlink -> External/foo
>                                 FLOAT   1     4     8m   0   RWD  5.2
> 
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> External                        DIR
>     foo                         FLOAT   1     4     6m   0   RWD  5.2
> 
> 
> Starting mlogger in verbose mode yields the following error:
> ==================== History link "dir", ID 28150  =======================
> [Logger,ERROR] [mlogger.cxx:4935:open_history,ERROR] History link
> /History/Links/dir/testlink is invalid
> Error in history system, aborting startup.
> 
> I'm not sure if this is a bug or just a case of PEBCAK.
> 
> Finally, to set the update period, do I need entries in /history/links periods
> with the tag name? Is there a way to only write them in the history file when
> they change? I want to use the virtual history events for measurements I get
> from external scripts, some periodic, some manual.
> 
> Thanks

> Asume you have a variable foo and a link bar -> foo. When you go to the ODB in
> mhttpd, click "Delete" and select bar, it actually deletes foo. bar stays,
> stating "<cannot resolve link>". Trying the same in odbedit with rm gives the
> expected result (bar is gone, foo is still there).
> 
> I'm on the develop branch.

I think I can confirm this. Created a bug report on bitbucket:

https://bitbucket.org/tmidas/midas/issues/148/mhttpd-odb-editor-deletes-wrong-symlink

K.O.

Hi,
I am trying to set up virtual history events following
https://midas.triumf.ca/MidasWiki/index.php/History_System#Virtual_History_Event 

Trying it the first way, using the following setup:
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Links                           DIR
    dirlink -> External/dir     KEY     1     12    >99d 0   RWD  <subdirectory>


Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
External                        DIR
    dir                         DIR
        foo                     FLOAT   1     4     16s  0   RWD  12.5


Then I get the following error message:
==================== History link "dirlink", ID 28150  =======================
[Logger,ERROR] [mlogger.cxx:4942:open_history,ERROR] History event dirlink has
no variables in ODB


Trying the second way, I set up the following:
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Links                           DIR
    dir                         DIR
        testlink -> External/foo
                                FLOAT   1     4     8m   0   RWD  5.2

Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
External                        DIR
    foo                         FLOAT   1     4     6m   0   RWD  5.2


Starting mlogger in verbose mode yields the following error:
==================== History link "dir", ID 28150  =======================
[Logger,ERROR] [mlogger.cxx:4935:open_history,ERROR] History link
/History/Links/dir/testlink is invalid
Error in history system, aborting startup.

I'm not sure if this is a bug or just a case of PEBCAK.

Finally, to set the update period, do I need entries in /history/links periods
with the tag name? Is there a way to only write them in the history file when
they change? I want to use the virtual history events for measurements I get
from external scripts, some periodic, some manual.

Thanks

Asume you have a variable foo and a link bar -> foo. When you go to the ODB in
mhttpd, click "Delete" and select bar, it actually deletes foo. bar stays,
stating "<cannot resolve link>". Trying the same in odbedit with rm gives the
expected result (bar is gone, foo is still there).

I'm on the develop branch.

> I would like to store the address of 1-Wire temperature sensors in a device
> driver. However, the supportet data types (as definded around
> include/midas.h:311) do not foresee a type large enough.
> 
> Is there a good reason against this?
> 
> I know that other experiments use this kind of sensor, how do you store the
> addresses? I've noticed that most of the address is just zeroes, but I wouldn't
> like to store just half the address, assuming that half the address is always
> zeroes.

Well, when this code was written, computers had 640kB  and operating systems had 16 bit. What 
you  can do for your 1-wire sensor is to store the address in two values, one 32-bit LSB and one 
32-bit MSB. Or store it in a string with hex representation.

Stefan 

I would like to store the address of 1-Wire temperature sensors in a device
driver. However, the supportet data types (as definded around
include/midas.h:311) do not foresee a type large enough.

Is there a good reason against this?

I know that other experiments use this kind of sensor, how do you store the
addresses? I've noticed that most of the address is just zeroes, but I wouldn't
like to store just half the address, assuming that half the address is always
zeroes.

Hi. We've just updated our midas installation to the newest version, and we now see repeated errors from the 
mserver in messages. Mostly we see

11:17:02.994 2018/08/21 [ODBEdit,TALK] Program mserver restarted

which happens 2-3 times per minute. 

We have also been seeing occasional dropped rpc connections to our frontends, which could be related. 

The version we were running with previously was ~1 year old, and we have just updated to the newest version 
on bitbucket. 

Thanks,
Wes

Dear Mr. Olchanski

Thank you for your comment.
We exect the number of readout channels is ~1000, boards ~100 and the frontend pc <10.	
We expect that trigger rate is a few kHz.

Writing monolithic c++ code may need complete understanding on midas,
and I will consider more about writing from scratch or modifying midas code.

Best regards
Hiroaki Natori

> > I'm going to develop MIDAS DAQ for COMET experiment.
> > I'm thinking to distribute the load of event building to different PCs.
> > I attach a schematics of one of the examples of the design.
> 
> Your schematic is reminiscent of the T2K/ND280 structure where the MIDAS DAQ
> was split into several separate MIDAS instances (separate "experiments": the FGD, the TPC, 
> the slow controls, etc).
> 
> They were joined together by the "cascade" equipment which provided a path
> for the data events to flow from subsidiary midas instances to the main system (the one
> with the final mlogger). It also provided a reverse path for run control, where starting
> a run in the main experiment also started the run in all the subsidiary experiments.
> 
> This cascade frontend was never included in the midas distribution (an oversight),
> but I still have the code for it somewhere.
> 
> How many "frontend PC" components do you envision? (10, 100, 1000?).
> 
> In T2K/ND280, each subsidiary experiment had it's own ODB which made sense
> because e.g. the FGD and the TPC were quite different and were managed by different
> groups.
> 
> But for you it probably makes sense to have one common ODB. This means a MIDAS
> structure where ODB is located on the main computer ("event builder PC"),
> all others connect to it via the mserver and midas rpc.
> 
> But you will need to have the MIDAS shared event buffers on each "frontend PC" to be local,
> which means the bm_xxx() functions have run locally instead of throuhg the mserver rpc.
> This is not how midas works right now, but it could be modified to do this.
> 
> On the other hand, you do not have to use midas to write the "frontend pc" code. Today's
> C++ provides enough features - threads, locks, mutexes, shared memories, event queues,
> etc so you can write the whole sub-event builder as one monolithic c++ program
> and use midas only to send the data to the main event builder. (plus midas rpc to handle
> run control). In this scheme, technically, this "frontend pc" program would
> be a multithreaded midas frontend.
> 
> K.O.

> I'm going to develop MIDAS DAQ for COMET experiment.
> I'm thinking to distribute the load of event building to different PCs.
> I attach a schematics of one of the examples of the design.

Your schematic is reminiscent of the T2K/ND280 structure where the MIDAS DAQ
was split into several separate MIDAS instances (separate "experiments": the FGD, the TPC, 
the slow controls, etc).

They were joined together by the "cascade" equipment which provided a path
for the data events to flow from subsidiary midas instances to the main system (the one
with the final mlogger). It also provided a reverse path for run control, where starting
a run in the main experiment also started the run in all the subsidiary experiments.

This cascade frontend was never included in the midas distribution (an oversight),
but I still have the code for it somewhere.

How many "frontend PC" components do you envision? (10, 100, 1000?).

In T2K/ND280, each subsidiary experiment had it's own ODB which made sense
because e.g. the FGD and the TPC were quite different and were managed by different
groups.

But for you it probably makes sense to have one common ODB. This means a MIDAS
structure where ODB is located on the main computer ("event builder PC"),
all others connect to it via the mserver and midas rpc.

But you will need to have the MIDAS shared event buffers on each "frontend PC" to be local,
which means the bm_xxx() functions have run locally instead of throuhg the mserver rpc.
This is not how midas works right now, but it could be modified to do this.

On the other hand, you do not have to use midas to write the "frontend pc" code. Today's
C++ provides enough features - threads, locks, mutexes, shared memories, event queues,
etc so you can write the whole sub-event builder as one monolithic c++ program
and use midas only to send the data to the main event builder. (plus midas rpc to handle
run control). In this scheme, technically, this "frontend pc" program would
be a multithreaded midas frontend.

K.O.

Dear expert,

I'm going to develop MIDAS DAQ for COMET experiment.
I'm thinking to distribute the load of event building to different PCs.
I attach a schematics of one of the examples of the design.
Please tell me how can I accomplish a kind of "sub-EventBuilder".
I'm reading the midas code to understand the scheme of MIDAS but
it is a lot and I want to know which one to focus on.
Can I do it writing user function based on either "mfe.c" or "mevb.c"?
Frontend program with multithread equipment is the one to do?
Or should I modify the original midas files?


Best regards,
Hiroaki Natori

Concurrence.

Normally, MIDAS data events are saved to ODB (via RO_ODB into /eq/xxx/variables) to make them go into the midas history (/eq/xxx/common/history > 0).

If you do not want events to go into the history, but still want them saved to ODB, it should work (as long as ODB itself
is big enough), but you may run into other problems, specifically ODB free space fragmentation, when no matter how big ODB is, there is never
enough continuous free space for saving a large event. If it happens you will also see random "odb full" errors.

K.O.



> Two options:
> 
> 1) Do NOT send your events into the ODB. This is controlled via the flag RO_ODB in your frontend setting. For simple experiments with small events, it might make sense to copy each 
> event into the ODB for debugging, but if you have large events, this does not make sense. Use the "mdump" utility to check your events instead.
> 
> 2) Increase the size of the ODB. See the first FAQ here: https://midas.triumf.ca/MidasWiki/index.php/FAQ
> 
> Stefan
> 
> 
> > Dear expert,
> >       I'm developing a frontend and I'm getting this kind of error at each event:
> > 
> > 10:14:56.564 2018/05/04 [Sample Frontend,ERROR] [odb.c:5911:db_set_data1,ERROR]
> > online database full
> > 
> > If I run the mem command in odbedit I get the result at the end of this post.
> > 
> > Notice that I need to use an event size which is significantly larger than the
> > default one. I don't know if it is relevant for this error. I have in the ODB:
> > 
> >   /Experiment/MAX_EVENT_SIZE = 900000000
> > 
> > and in the frontend code:
> > 
> >   /* maximum event size produced by this frontend */
> >   INT max_event_size = 300000000;
> > 
> >   /* maximum event size for fragmented events (EQ_FRAGMENTED) */
> >   INT max_event_size_frag = 5 * 1024 * 1024;
> > 
> >   /* buffer size to hold events */
> >   INT event_buffer_size = 600000000;
> > 
> > Events seem to be properly stored in the output files, but I'm afraid I could
> > get some other problem.
> > 
> > Thank you for your help,
> >          Francesco
> > 
> > -------------------------------------------------------------------------
> > 
> > Database header size is 0x21040, all following values are offset by this!
> > Key area  0x00000000 - 0x0007FFFF, size 524288 bytes
> > Data area 0x00080000 - 0x00100000, size 524288 bytes
> > 
> > Keylist:
> > --------
> > Free block at 0x00000B58, size 0x00000008, next 0x000053E0
> > Free block at 0x000053E0, size 0x00000008, next 0x00006560
> > Free block at 0x00006560, size 0x00079AA0, next 0x00000000
> > 
> > Free Key area: 498352 bytes out of 524288 bytes
> > 
> > Data:
> > -----
> > Free block at 0x000847F0, size 0x0007B810, next 0x00000000
> > 
> > Free Data area: 505872 bytes out of 524288 bytes
> > 
> > Free: 498352 (95.1%) keylist, 505872 (96.5%) data

> Invalid tag 0 'Comment' in event 21 'Run Parameters': cannot do history for 
> TID_STRING data, sorry!

The original MIDAS history API does not have provisions for storing TID_STRING data,
it is a very unfortunate limitation that has been with us for a very long time.

If I ever get around to rewrite the MIDAS history API, I will definitely add support for TID_STRING data.

But not today.

K.O.

P.S. Support for arbitrary binary blobs is also possible, but this will make the midas history
a kind of "a daq inside the daq" thing, probably we do not want to go this direction.

K.O.

The ROOT I/O workshop was held on June 20th at CERN. A few things of interest in MIDAS land:

- LZ4 is now used as default compression (replacing gzip-1)
- JSON class streamer is finally implemented (XML streamer updated/reworked)
- recursive read-write lock class implemented
- do not see any special mention of Javascript I/O or jsroot, but jsroot git repo seems to be quite active

Of these the recursive read-write lock is most interesting - using something similar would improve ODB performance
and presumably fix the existing lock fairness problems.

https://root.cern.ch/doc/master/TReentrantRWLock_8hxx_source.html
https://indico.cern.ch/event/715802/contributions/2942560/attachments/1670191/2680682/ROOT_IO_June_Workshop_v2.pdf
https://github.com/root-project/jsroot

K.O.

We run as follows:

* sis3316 digitizers in a vme crate
* 1-2 midas events /s
* data rate at 20 MB/s

At a rate of 30 MB/s the daq crashed because the I think the mlogger can`t follow:

  * it runs at 100% cpu
  * memory usage of mlogger process goes from 2% to 15%
  * All other processes < 50 % cpu and < 20% RAM

Both the vme frontend and the mlogger crash about 2.5 minutes into a run. Both
the logger and vme fe spit out:
bm_validate_client_pointers: Assertion `pclient->read_pointer >= 0 &&
pclient->read_pointer <= pheader->size' failed.
Aborted

I first thought that writing-to-disk could be a bottle neck. But when I write to
an SSD, same thing.

Is there another bottleneck which keeps the mlogger busy?

> > Uncaught ReferenceError: XMLHttpRequestGeneric is not defined
> >     at cs_button (Trend:165)
> >     at HTMLInputElement.onclick (Trend:90)  
> 
> That code was not written by me, so I'm must guessing here.
> 
> Probably the XMLHttpRequestGeneric() is some function hiding browser specialities to create 
> AJAX requests. These days most browser understand the standard request
> 
> XMLHttpRequest()
> 
> so why don't you try to just remove the "Generic"
> 
> Stefan

That removes the error, but script doesnt get called. It goes to the javascript function and 
callback, but nothing happens. 

When I change type=button to type=submit , the script gets called again, but with page refresh.