Hello Everybody,

I am a member of a satellite team with a scientific payload and I am considering
coordinating the payload using MIDAS. This looks to be challenging since MIDAS
would be implemented on an Xilinx Spartan 6 FPGA with minimal hardware
resources. The idea would be to install a soft processor on the Spartan 6 and
run MIDAS through UCLinux either on the FPGA or boot it from SPI Flash. Does
anybody have any comments on how feasible this would be or perhaps have
experience implementing a similar system?

-Devin

> Hello Everybody,
> 
> I am a member of a satellite team with a scientific payload and I am considering
> coordinating the payload using MIDAS. This looks to be challenging since MIDAS
> would be implemented on an Xilinx Spartan 6 FPGA with minimal hardware
> resources. The idea would be to install a soft processor on the Spartan 6 and
> run MIDAS through UCLinux either on the FPGA or boot it from SPI Flash. Does
> anybody have any comments on how feasible this would be or perhaps have
> experience implementing a similar system?
> 
> -Devin

While some people successfully implemented a midas *client* in an FPGA softcore, the full midas 
backend would probably not fit into a Spartan 6. Having done some FPGA programming and 
working on satellites, I doubt that midas would be well suited for such an environment. It's 
probably some kind of overkill. The complete GUI is likely useless since you want to minimize your 
communication load on the satellite link.

Stefan

> > Hello Everybody,
> > 
> > I am a member of a satellite team with a scientific payload and I am considering
> > coordinating the payload using MIDAS. This looks to be challenging since MIDAS
> > would be implemented on an Xilinx Spartan 6 FPGA with minimal hardware
> > resources. The idea would be to install a soft processor on the Spartan 6 and
> > run MIDAS through UCLinux either on the FPGA or boot it from SPI Flash. Does
> > anybody have any comments on how feasible this would be or perhaps have
> > experience implementing a similar system?
> > 
> > -Devin
> 
> While some people successfully implemented a midas *client* in an FPGA softcore, the full midas 
> backend would probably not fit into a Spartan 6. Having done some FPGA programming and 
> working on satellites, I doubt that midas would be well suited for such an environment. It's 
> probably some kind of overkill. The complete GUI is likely useless since you want to minimize your 
> communication load on the satellite link.
> 
> Stefan

Thank you for your comment Stefan. We do have some hardware resources on the board such as RAM, ROM and
Flash storage so we wouldn't necessarily have to virtualize everything. Ideally we would like a
completed and compressed file to be produced on board and regularly sent back to ground without
requiring remote access. MIDAS is appealing to us because its easily automated but we wouldn't
necessarily need functions like a GUI or web interface. Part of the discussion now is whether or not a
microblaze processor would be sufficient or if we need a dedicted ARM processor.

Devin 

> 
> Thank you for your comment Stefan. We do have some hardware resources on the board such as RAM, ROM and
> Flash storage so we wouldn't necessarily have to virtualize everything. Ideally we would like a
> completed and compressed file to be produced on board and regularly sent back to ground without
> requiring remote access. MIDAS is appealing to us because its easily automated but we wouldn't
> necessarily need functions like a GUI or web interface. Part of the discussion now is whether or not a
> microblaze processor would be sufficient or if we need a dedicted ARM processor.
> 

Hi, just recently I got a midas frontend to build and run on uclinux on a microblaze arm CPU (GRIFFIN CDM VME board).

It worked, but uncovered many problems inside midas - uclinux has no mmu, no multithreading, no recursive mutexes, no 
some of the other stuff assumed always available.

The worst problem I ran into was with uclinux giving us a very small stack so code like "int main() { char buf[10*1024]; }
crashes right away and there is a lot of code like this in midas.

My feeling about the xilinx soft-core CPU, if you can run uclinux, you can also run a midas frontend. We do not require 
memory beyond that needed to store one or two of your data events.

By design, the midas library can be built in a "minimal" configuration that only supports a frontend connected
to the mserver (no local ODB, no local event buffers, no local mhttpd/mlogger, etc).

As you have seen in the Makefile, there are provisions for cross-compilation and I cross-compile midas things quite often.

On the other side, if you have xilinx FPGA with build-in PowerPC CPU, most definitely you can run full linux
and you can run full midas on it, we have done this for the T2K/ND280 experiment in Japan.

K.O.

Hi all

There is a bug report in the ROME repository which says bm_receive_event timeouts.

https://bitbucket.org/muegamma/rome3/issues/8/rome-with-midas-produces-timeout-after

Does anybody have any ideas what could causing the problem ?

Ryu

> There is a bug report in the ROME repository which says bm_receive_event timeouts.
> https://bitbucket.org/muegamma/rome3/issues/8/rome-with-midas-produces-timeout-after
> Does anybody have any ideas what could causing the problem ?

There could be a problem with causing bm_receive_event() to wait for an event for a time longer than 
the rpc timeout. This rings a very small bell for me. But I do not remember the details.

As I now go through the midas event buffer code, I will check that bm_receive_event() connected 
through the mserver has correctly working timeouts.

Thank you for reminding me about this difficulty.

K.O.

the mxml library was updated to make it thread-safe.
https://bitbucket.org/tmidas/mxml/src/master/

I also take an opportunity to remind all to update your copy to the latest version
as I just stumbled on old bug that I fixed 1 year ago (crash of mlogger)
but forgot to update all and every of my copies of mxml.

I also looked at the xml encoder and I see that it has several places where it may
truncate the data, but none of these places can cause truncation of ODB data
because the fixed-size internal buffers are big enough to hold the longest
values sent by the odb xml encoder.

K.O.

One can collapse the side panel when looking at history pages with the button in
the top left, great! We want to see many pages so screen real estate is important

The issue we face is that when the page refreshes, the side panel expands. Can
we make the panel state more 'sticky'?

Many thanks
Joseph (ALPHA)

Version: 	2.1
Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
on branch feature/midas-2017-10

> 
> 
> One can collapse the side panel when looking at history pages with the button in
> the top left, great! We want to see many pages so screen real estate is important
> 
> The issue we face is that when the page refreshes, the side panel expands. Can
> we make the panel state more 'sticky'?
> 
> Many thanks
> Joseph (ALPHA)
> 
> Version: 	2.1
> Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
> on branch feature/midas-2017-10

Hi Joseph,

In principle a page refresh should now not be necessary, since pages should reload automatically 
the contents which changes. If a custom page needs a reload, it is not well designed. If necessary, I 
can explain the details. 

Anyhow I implemented your "stickyness" of the side panel in the last commit to the develop branch.

Best regards,
Stefan

> > 
> > 
> > One can collapse the side panel when looking at history pages with the button in
> > the top left, great! We want to see many pages so screen real estate is important
> > 
> > The issue we face is that when the page refreshes, the side panel expands. Can
> > we make the panel state more 'sticky'?
> > 
> > Many thanks
> > Joseph (ALPHA)
> > 
> > Version: 	2.1
> > Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
> > on branch feature/midas-2017-10
> 
> Hi Joseph,
> 
> In principle a page refresh should now not be necessary, since pages should reload automatically 
> the contents which changes. If a custom page needs a reload, it is not well designed. If necessary, I 
> can explain the details. 
> 
> Anyhow I implemented your "stickyness" of the side panel in the last commit to the develop branch.
> 
> Best regards,
> Stefan

Hi Stefan,

I apologise for miss using the word refresh. The re-appearing sidebar was also seen with the automatic
reload, I have implemented your fix here and it now works great!

Thank you very much!
Joseph

> I apologise for miss using the word refresh. The re-appearing sidebar was also seen with the automatic
> reload, I have implemented your fix here and it now works great!

Still did not get your point. Why is there "automatic reload"? The status page should not "completely reload" any more. 
Instead, all data is fetched in the background using AJAX calls, and only the data on the page is updated once per second. If 
there is a "complete reload", something is wrong.

Stefan

> > I apologise for miss using the word refresh. The re-appearing sidebar was also seen with the automatic
> > reload, I have implemented your fix here and it now works great!
> 
> Still did not get your point. Why is there "automatic reload"? The status page should not "completely reload" any more. 
> Instead, all data is fetched in the background using AJAX calls, and only the data on the page is updated once per second. 
If 
> there is a "complete reload", something is wrong.

Joseph's original message says that the problem is with the standard MIDAS history page, which currently use a complete reload 
when refreshing.  Of course we are planning to update this history pages to only grab what it needs (as well as changing the 
plotting to use newer HTML plotting). But until that upgrade happens your fix is helpful for the history page.

> Joseph's original message says that the problem is with the standard MIDAS history page, which currently use a complete reload 
> when refreshing.  Of course we are planning to update this history pages to only grab what it needs (as well as changing the 
> plotting to use newer HTML plotting). But until that upgrade happens your fix is helpful for the history page.

Ok, now I understand, and of course I agree with you.

Stefan

Dear experts,
              is there any way to launch an executable script on the host computer from the MIDAS 
sequencer? If not, is there any interest to develop such a feature?

Thank you,
         Francesco

> Dear experts,
>               is there any way to launch an executable script on the host computer from the MIDAS 
> sequencer? If not, is there any interest to develop such a feature?
> 
> Thank you,
>          Francesco

The SCRIPT command will do that (on the machine running MIDAS). I know it works with either python or
bash scripts. I tried without success to pass the parameters and I went around by setting ODB entries
prior to running the script and then access to them within the script.

> > Dear experts,
> >               is there any way to launch an executable script on the host computer from the MIDAS 
> > sequencer? If not, is there any interest to develop such a feature?
> > 
> > Thank you,
> >          Francesco
> 
> The SCRIPT command will do that (on the machine running MIDAS). I know it works with either python or
> bash scripts. I tried without success to pass the parameters and I went around by setting ODB entries
> prior to running the script and then access to them within the script.

Passing parameters should work. If it's confirmed to be broken, I'm willing to fix it.

Stefan

Asume you have a variable foo and a link bar -> foo. When you go to the ODB in
mhttpd, click "Delete" and select bar, it actually deletes foo. bar stays,
stating "<cannot resolve link>". Trying the same in odbedit with rm gives the
expected result (bar is gone, foo is still there).

I'm on the develop branch.

> Asume you have a variable foo and a link bar -> foo. When you go to the ODB in
> mhttpd, click "Delete" and select bar, it actually deletes foo. bar stays,
> stating "<cannot resolve link>". Trying the same in odbedit with rm gives the
> expected result (bar is gone, foo is still there).
> 
> I'm on the develop branch.

I think I can confirm this. Created a bug report on bitbucket:

https://bitbucket.org/tmidas/midas/issues/148/mhttpd-odb-editor-deletes-wrong-symlink

K.O.

> > Asume you have a variable foo and a link bar -> foo. When you go to the ODB in
> > mhttpd, click "Delete" and select bar, it actually deletes foo. bar stays,
> > stating "<cannot resolve link>". Trying the same in odbedit with rm gives the
> > expected result (bar is gone, foo is still there).
> > 
> > I'm on the develop branch.
> 
> I think I can confirm this. Created a bug report on bitbucket:
> 
> https://bitbucket.org/tmidas/midas/issues/148/mhttpd-odb-editor-deletes-wrong-symlink
> 
> K.O.

I fixed this and committed the change. Took me a while since it was in KO's code.

Stefan

Per changes at TRIUMF, the MIDAS forum mail relay was changed from trmail.triumf.ca to 
smtp.triumf.ca. K.O.

Hi. We've just updated our midas installation to the newest version, and we now see repeated errors from the 
mserver in messages. Mostly we see

11:17:02.994 2018/08/21 [ODBEdit,TALK] Program mserver restarted

which happens 2-3 times per minute. 

We have also been seeing occasional dropped rpc connections to our frontends, which could be related. 

The version we were running with previously was ~1 year old, and we have just updated to the newest version 
on bitbucket. 

Thanks,
Wes

> Hi. We've just updated our midas installation to the newest version, and we now see repeated errors from the 
> mserver in messages. Mostly we see
> 
> 11:17:02.994 2018/08/21 [ODBEdit,TALK] Program mserver restarted
> 
> which happens 2-3 times per minute. 
> 
> We have also been seeing occasional dropped rpc connections to our frontends, which could be related. 
> 
> The version we were running with previously was ~1 year old, and we have just updated to the newest version 
> on bitbucket. 

Hmm... usually mserver will not restart automatically, maybe you have set it to autorestart on ODB (/programs/mserver/auto_restart 
set to "y").

It would be unusual for the main mserver program to crash, to debug it, you will need to run it in a terminal
and see if there is any error messages. Even better to run it in a terminal inside "gdb" and capture the stack trace
when it crashes.

Anyhow, crash of main mserver will not cause "dropped rpc connections" to clients - this would require for their
individual mserver subprocesses to crash. Such crashes would be highly unusual and are harder to debug.

Perhaps for the crashes you see there is some error messages in midas.log?

K.O.

I would like to store the address of 1-Wire temperature sensors in a device
driver. However, the supportet data types (as definded around
include/midas.h:311) do not foresee a type large enough.

Is there a good reason against this?

I know that other experiments use this kind of sensor, how do you store the
addresses? I've noticed that most of the address is just zeroes, but I wouldn't
like to store just half the address, assuming that half the address is always
zeroes.

> I would like to store the address of 1-Wire temperature sensors in a device
> driver. However, the supportet data types (as definded around
> include/midas.h:311) do not foresee a type large enough.
> 
> Is there a good reason against this?
> 
> I know that other experiments use this kind of sensor, how do you store the
> addresses? I've noticed that most of the address is just zeroes, but I wouldn't
> like to store just half the address, assuming that half the address is always
> zeroes.

Well, when this code was written, computers had 640kB  and operating systems had 16 bit. What 
you  can do for your 1-wire sensor is to store the address in two values, one 32-bit LSB and one 
32-bit MSB. Or store it in a string with hex representation.

Stefan 

> I would like to store the address of 1-Wire temperature sensors in a device
> driver. However, the supportet data types (as definded around
> include/midas.h:311) do not foresee a type large enough.
>

Hmm... you do not say what sensor you use and how many bits you actually need.

For up to 32 bits you can use TID_DWORD (uint32_t) (obviously)

For up to 48 bits (or so), you can use TID_DOUBLE (double) (wierd, but IEEE754 double precision variables would work as 48-bit (or so) integers).

For more, I would use arrays of TID_DWORD (64 bits, store low 32 bits into a[0], high bits into a[1]).

> 
> Is there a good reason against this?
> 

We had requests for implementing uint64_t 64-bit data types in MIDAS before. There are two problems:

a) in the MIDAS data banks, there is a problem with the bank header definition which only has 3 DWORDSs so causes
each alternating data bank to be 64-bit misaligned. And misaligned 64-bit data is very bad.

b) in ODB, 64-bit data support will need to be added from scratch and again it is not clear without doing it
if there will be any alignement problems. If one were to implement ODB from scratch, one would have everything
aligned to 64-bits or maybe even 128-bits, with uint64_t fully supported.

It is unlikely this kind of work will ever be done on ODB, but who knows.

> I know that other experiments use this kind of sensor, how do you store the
> addresses? I've noticed that most of the address is just zeroes, but I wouldn't
> like to store just half the address, assuming that half the address is always
> zeroes.

Cannot answer without knowing what sensor you use, but certainly you can use an array of bytes
or an array of integers to store arbitrarily long addresses. You can also use a TID_STRING
and store the address as a text string "0xabcdabcdabcdabcd" of arbitrary length.

K.O.

Hi,
I am trying to set up virtual history events following
https://midas.triumf.ca/MidasWiki/index.php/History_System#Virtual_History_Event 

Trying it the first way, using the following setup:
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Links                           DIR
    dirlink -> External/dir     KEY     1     12    >99d 0   RWD  <subdirectory>


Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
External                        DIR
    dir                         DIR
        foo                     FLOAT   1     4     16s  0   RWD  12.5


Then I get the following error message:
==================== History link "dirlink", ID 28150  =======================
[Logger,ERROR] [mlogger.cxx:4942:open_history,ERROR] History event dirlink has
no variables in ODB


Trying the second way, I set up the following:
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Links                           DIR
    dir                         DIR
        testlink -> External/foo
                                FLOAT   1     4     8m   0   RWD  5.2

Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
External                        DIR
    foo                         FLOAT   1     4     6m   0   RWD  5.2


Starting mlogger in verbose mode yields the following error:
==================== History link "dir", ID 28150  =======================
[Logger,ERROR] [mlogger.cxx:4935:open_history,ERROR] History link
/History/Links/dir/testlink is invalid
Error in history system, aborting startup.

I'm not sure if this is a bug or just a case of PEBCAK.

Finally, to set the update period, do I need entries in /history/links periods
with the tag name? Is there a way to only write them in the history file when
they change? I want to use the virtual history events for measurements I get
from external scripts, some periodic, some manual.

Thanks

Hi, what you try should have worked. Perhaps your symlink is wrong and should say "/External/..." (with a leading slash). The "links period" would have
worked same as equipment/common/history period - as a rate limiter.

Anyhow, I suggest another way to do the same - create a fake equipment - the logger does
not care if the equipment is real or not and if you write into /eq/fake/variables from a proper frontend
or from a script. To hide the fake equipment from the status page, set /eq/fake/common/hidden to "true".

This will work for sure.

K.O.




> Hi,
> I am trying to set up virtual history events following
> https://midas.triumf.ca/MidasWiki/index.php/History_System#Virtual_History_Event 
> 
> Trying it the first way, using the following setup:
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> Links                           DIR
>     dirlink -> External/dir     KEY     1     12    >99d 0   RWD  <subdirectory>
> 
> 
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> External                        DIR
>     dir                         DIR
>         foo                     FLOAT   1     4     16s  0   RWD  12.5
> 
> 
> Then I get the following error message:
> ==================== History link "dirlink", ID 28150  =======================
> [Logger,ERROR] [mlogger.cxx:4942:open_history,ERROR] History event dirlink has
> no variables in ODB
> 
> 
> Trying the second way, I set up the following:
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> Links                           DIR
>     dir                         DIR
>         testlink -> External/foo
>                                 FLOAT   1     4     8m   0   RWD  5.2
> 
> Key name                        Type    #Val  Size  Last Opn Mode Value
> ---------------------------------------------------------------------------
> External                        DIR
>     foo                         FLOAT   1     4     6m   0   RWD  5.2
> 
> 
> Starting mlogger in verbose mode yields the following error:
> ==================== History link "dir", ID 28150  =======================
> [Logger,ERROR] [mlogger.cxx:4935:open_history,ERROR] History link
> /History/Links/dir/testlink is invalid
> Error in history system, aborting startup.
> 
> I'm not sure if this is a bug or just a case of PEBCAK.
> 
> Finally, to set the update period, do I need entries in /history/links periods
> with the tag name? Is there a way to only write them in the history file when
> they change? I want to use the virtual history events for measurements I get
> from external scripts, some periodic, some manual.
> 
> Thanks

Dear expert,

I'm going to develop MIDAS DAQ for COMET experiment.
I'm thinking to distribute the load of event building to different PCs.
I attach a schematics of one of the examples of the design.
Please tell me how can I accomplish a kind of "sub-EventBuilder".
I'm reading the midas code to understand the scheme of MIDAS but
it is a lot and I want to know which one to focus on.
Can I do it writing user function based on either "mfe.c" or "mevb.c"?
Frontend program with multithread equipment is the one to do?
Or should I modify the original midas files?


Best regards,
Hiroaki Natori

> I'm going to develop MIDAS DAQ for COMET experiment.
> I'm thinking to distribute the load of event building to different PCs.
> I attach a schematics of one of the examples of the design.

Your schematic is reminiscent of the T2K/ND280 structure where the MIDAS DAQ
was split into several separate MIDAS instances (separate "experiments": the FGD, the TPC, 
the slow controls, etc).

They were joined together by the "cascade" equipment which provided a path
for the data events to flow from subsidiary midas instances to the main system (the one
with the final mlogger). It also provided a reverse path for run control, where starting
a run in the main experiment also started the run in all the subsidiary experiments.

This cascade frontend was never included in the midas distribution (an oversight),
but I still have the code for it somewhere.

How many "frontend PC" components do you envision? (10, 100, 1000?).

In T2K/ND280, each subsidiary experiment had it's own ODB which made sense
because e.g. the FGD and the TPC were quite different and were managed by different
groups.

But for you it probably makes sense to have one common ODB. This means a MIDAS
structure where ODB is located on the main computer ("event builder PC"),
all others connect to it via the mserver and midas rpc.

But you will need to have the MIDAS shared event buffers on each "frontend PC" to be local,
which means the bm_xxx() functions have run locally instead of throuhg the mserver rpc.
This is not how midas works right now, but it could be modified to do this.

On the other hand, you do not have to use midas to write the "frontend pc" code. Today's
C++ provides enough features - threads, locks, mutexes, shared memories, event queues,
etc so you can write the whole sub-event builder as one monolithic c++ program
and use midas only to send the data to the main event builder. (plus midas rpc to handle
run control). In this scheme, technically, this "frontend pc" program would
be a multithreaded midas frontend.

K.O.

Dear Mr. Olchanski

Thank you for your comment.
We exect the number of readout channels is ~1000, boards ~100 and the frontend pc <10.	
We expect that trigger rate is a few kHz.

Writing monolithic c++ code may need complete understanding on midas,
and I will consider more about writing from scratch or modifying midas code.

Best regards
Hiroaki Natori

> > I'm going to develop MIDAS DAQ for COMET experiment.
> > I'm thinking to distribute the load of event building to different PCs.
> > I attach a schematics of one of the examples of the design.
> 
> Your schematic is reminiscent of the T2K/ND280 structure where the MIDAS DAQ
> was split into several separate MIDAS instances (separate "experiments": the FGD, the TPC, 
> the slow controls, etc).
> 
> They were joined together by the "cascade" equipment which provided a path
> for the data events to flow from subsidiary midas instances to the main system (the one
> with the final mlogger). It also provided a reverse path for run control, where starting
> a run in the main experiment also started the run in all the subsidiary experiments.
> 
> This cascade frontend was never included in the midas distribution (an oversight),
> but I still have the code for it somewhere.
> 
> How many "frontend PC" components do you envision? (10, 100, 1000?).
> 
> In T2K/ND280, each subsidiary experiment had it's own ODB which made sense
> because e.g. the FGD and the TPC were quite different and were managed by different
> groups.
> 
> But for you it probably makes sense to have one common ODB. This means a MIDAS
> structure where ODB is located on the main computer ("event builder PC"),
> all others connect to it via the mserver and midas rpc.
> 
> But you will need to have the MIDAS shared event buffers on each "frontend PC" to be local,
> which means the bm_xxx() functions have run locally instead of throuhg the mserver rpc.
> This is not how midas works right now, but it could be modified to do this.
> 
> On the other hand, you do not have to use midas to write the "frontend pc" code. Today's
> C++ provides enough features - threads, locks, mutexes, shared memories, event queues,
> etc so you can write the whole sub-event builder as one monolithic c++ program
> and use midas only to send the data to the main event builder. (plus midas rpc to handle
> run control). In this scheme, technically, this "frontend pc" program would
> be a multithreaded midas frontend.
> 
> K.O.

Dear expert,
      I'm developing a frontend and I'm getting this kind of error at each event:

10:14:56.564 2018/05/04 [Sample Frontend,ERROR] [odb.c:5911:db_set_data1,ERROR]
online database full

If I run the mem command in odbedit I get the result at the end of this post.

Notice that I need to use an event size which is significantly larger than the
default one. I don't know if it is relevant for this error. I have in the ODB:

  /Experiment/MAX_EVENT_SIZE = 900000000

and in the frontend code:

  /* maximum event size produced by this frontend */
  INT max_event_size = 300000000;

  /* maximum event size for fragmented events (EQ_FRAGMENTED) */
  INT max_event_size_frag = 5 * 1024 * 1024;

  /* buffer size to hold events */
  INT event_buffer_size = 600000000;

Events seem to be properly stored in the output files, but I'm afraid I could
get some other problem.

Thank you for your help,
         Francesco

-------------------------------------------------------------------------

Database header size is 0x21040, all following values are offset by this!
Key area  0x00000000 - 0x0007FFFF, size 524288 bytes
Data area 0x00080000 - 0x00100000, size 524288 bytes

Keylist:
--------
Free block at 0x00000B58, size 0x00000008, next 0x000053E0
Free block at 0x000053E0, size 0x00000008, next 0x00006560
Free block at 0x00006560, size 0x00079AA0, next 0x00000000

Free Key area: 498352 bytes out of 524288 bytes

Data:
-----
Free block at 0x000847F0, size 0x0007B810, next 0x00000000

Free Data area: 505872 bytes out of 524288 bytes

Free: 498352 (95.1%) keylist, 505872 (96.5%) data

Two options:

1) Do NOT send your events into the ODB. This is controlled via the flag RO_ODB in your frontend setting. For simple experiments with small events, it might make sense to copy each 
event into the ODB for debugging, but if you have large events, this does not make sense. Use the "mdump" utility to check your events instead.

2) Increase the size of the ODB. See the first FAQ here: https://midas.triumf.ca/MidasWiki/index.php/FAQ

Stefan


> Dear expert,
>       I'm developing a frontend and I'm getting this kind of error at each event:
> 
> 10:14:56.564 2018/05/04 [Sample Frontend,ERROR] [odb.c:5911:db_set_data1,ERROR]
> online database full
> 
> If I run the mem command in odbedit I get the result at the end of this post.
> 
> Notice that I need to use an event size which is significantly larger than the
> default one. I don't know if it is relevant for this error. I have in the ODB:
> 
>   /Experiment/MAX_EVENT_SIZE = 900000000
> 
> and in the frontend code:
> 
>   /* maximum event size produced by this frontend */
>   INT max_event_size = 300000000;
> 
>   /* maximum event size for fragmented events (EQ_FRAGMENTED) */
>   INT max_event_size_frag = 5 * 1024 * 1024;
> 
>   /* buffer size to hold events */
>   INT event_buffer_size = 600000000;
> 
> Events seem to be properly stored in the output files, but I'm afraid I could
> get some other problem.
> 
> Thank you for your help,
>          Francesco
> 
> -------------------------------------------------------------------------
> 
> Database header size is 0x21040, all following values are offset by this!
> Key area  0x00000000 - 0x0007FFFF, size 524288 bytes
> Data area 0x00080000 - 0x00100000, size 524288 bytes
> 
> Keylist:
> --------
> Free block at 0x00000B58, size 0x00000008, next 0x000053E0
> Free block at 0x000053E0, size 0x00000008, next 0x00006560
> Free block at 0x00006560, size 0x00079AA0, next 0x00000000
> 
> Free Key area: 498352 bytes out of 524288 bytes
> 
> Data:
> -----
> Free block at 0x000847F0, size 0x0007B810, next 0x00000000
> 
> Free Data area: 505872 bytes out of 524288 bytes
> 
> Free: 498352 (95.1%) keylist, 505872 (96.5%) data

Concurrence.

Normally, MIDAS data events are saved to ODB (via RO_ODB into /eq/xxx/variables) to make them go into the midas history (/eq/xxx/common/history > 0).

If you do not want events to go into the history, but still want them saved to ODB, it should work (as long as ODB itself
is big enough), but you may run into other problems, specifically ODB free space fragmentation, when no matter how big ODB is, there is never
enough continuous free space for saving a large event. If it happens you will also see random "odb full" errors.

K.O.



> Two options:
> 
> 1) Do NOT send your events into the ODB. This is controlled via the flag RO_ODB in your frontend setting. For simple experiments with small events, it might make sense to copy each 
> event into the ODB for debugging, but if you have large events, this does not make sense. Use the "mdump" utility to check your events instead.
> 
> 2) Increase the size of the ODB. See the first FAQ here: https://midas.triumf.ca/MidasWiki/index.php/FAQ
> 
> Stefan
> 
> 
> > Dear expert,
> >       I'm developing a frontend and I'm getting this kind of error at each event:
> > 
> > 10:14:56.564 2018/05/04 [Sample Frontend,ERROR] [odb.c:5911:db_set_data1,ERROR]
> > online database full
> > 
> > If I run the mem command in odbedit I get the result at the end of this post.
> > 
> > Notice that I need to use an event size which is significantly larger than the
> > default one. I don't know if it is relevant for this error. I have in the ODB:
> > 
> >   /Experiment/MAX_EVENT_SIZE = 900000000
> > 
> > and in the frontend code:
> > 
> >   /* maximum event size produced by this frontend */
> >   INT max_event_size = 300000000;
> > 
> >   /* maximum event size for fragmented events (EQ_FRAGMENTED) */
> >   INT max_event_size_frag = 5 * 1024 * 1024;
> > 
> >   /* buffer size to hold events */
> >   INT event_buffer_size = 600000000;
> > 
> > Events seem to be properly stored in the output files, but I'm afraid I could
> > get some other problem.
> > 
> > Thank you for your help,
> >          Francesco
> > 
> > -------------------------------------------------------------------------
> > 
> > Database header size is 0x21040, all following values are offset by this!
> > Key area  0x00000000 - 0x0007FFFF, size 524288 bytes
> > Data area 0x00080000 - 0x00100000, size 524288 bytes
> > 
> > Keylist:
> > --------
> > Free block at 0x00000B58, size 0x00000008, next 0x000053E0
> > Free block at 0x000053E0, size 0x00000008, next 0x00006560
> > Free block at 0x00006560, size 0x00079AA0, next 0x00000000
> > 
> > Free Key area: 498352 bytes out of 524288 bytes
> > 
> > Data:
> > -----
> > Free block at 0x000847F0, size 0x0007B810, next 0x00000000
> > 
> > Free Data area: 505872 bytes out of 524288 bytes
> > 
> > Free: 498352 (95.1%) keylist, 505872 (96.5%) data

I am setting up a sqlite db to serve as a run database.

The easiest option is to use the history sqlite feature, and add run information 
as virtual history events

however:

Invalid tag 0 'Comment' in event 21 'Run Parameters': cannot do history for 
TID_STRING data, sorry!

I'd like to save e.g. the edit on start information , with shift crew checks. 
Would it be easy to allow for text, or is this inherent to the history system 
handling binary data?

> Invalid tag 0 'Comment' in event 21 'Run Parameters': cannot do history for 
> TID_STRING data, sorry!

The original MIDAS history API does not have provisions for storing TID_STRING data,
it is a very unfortunate limitation that has been with us for a very long time.

If I ever get around to rewrite the MIDAS history API, I will definitely add support for TID_STRING data.

But not today.

K.O.

P.S. Support for arbitrary binary blobs is also possible, but this will make the midas history
a kind of "a daq inside the daq" thing, probably we do not want to go this direction.

K.O.

The ROOT I/O workshop was held on June 20th at CERN. A few things of interest in MIDAS land:

- LZ4 is now used as default compression (replacing gzip-1)
- JSON class streamer is finally implemented (XML streamer updated/reworked)
- recursive read-write lock class implemented
- do not see any special mention of Javascript I/O or jsroot, but jsroot git repo seems to be quite active

Of these the recursive read-write lock is most interesting - using something similar would improve ODB performance
and presumably fix the existing lock fairness problems.

https://root.cern.ch/doc/master/TReentrantRWLock_8hxx_source.html
https://indico.cern.ch/event/715802/contributions/2942560/attachments/1670191/2680682/ROOT_IO_June_Workshop_v2.pdf
https://github.com/root-project/jsroot

K.O.

We run as follows:

* sis3316 digitizers in a vme crate
* 1-2 midas events /s
* data rate at 20 MB/s

At a rate of 30 MB/s the daq crashed because the I think the mlogger can`t follow:

  * it runs at 100% cpu
  * memory usage of mlogger process goes from 2% to 15%
  * All other processes < 50 % cpu and < 20% RAM

Both the vme frontend and the mlogger crash about 2.5 minutes into a run. Both
the logger and vme fe spit out:
bm_validate_client_pointers: Assertion `pclient->read_pointer >= 0 &&
pclient->read_pointer <= pheader->size' failed.
Aborted

I first thought that writing-to-disk could be a bottle neck. But when I write to
an SSD, same thing.

Is there another bottleneck which keeps the mlogger busy?

I am implementing buttons to launch scripts from a custom page. 

The simple way works, i.e.

  <input type=submit name=customscript value="run_script">

But I want to stay on the page. Copying "Customscript button without a page 
reload" from https://midas.triumf.ca/MidasWiki/index.php/Custom_Page_Features 
yields the following error:

Uncaught ReferenceError: XMLHttpRequestGeneric is not defined
    at cs_button (Trend:165)
    at HTMLInputElement.onclick (Trend:90)  

I included  <script src="mhttpd.js"></script> and call mhttpd_init on page load.

So why can`t it run this ajax request?


Or is there a better way to launch a script without messing up the page

> Uncaught ReferenceError: XMLHttpRequestGeneric is not defined
>     at cs_button (Trend:165)
>     at HTMLInputElement.onclick (Trend:90)  

That code was not written by me, so I'm must guessing here.

Probably the XMLHttpRequestGeneric() is some function hiding browser specialities to create 
AJAX requests. These days most browser understand the standard request

XMLHttpRequest()

so why don't you try to just remove the "Generic"

Stefan

> > Uncaught ReferenceError: XMLHttpRequestGeneric is not defined
> >     at cs_button (Trend:165)
> >     at HTMLInputElement.onclick (Trend:90)  
> 
> That code was not written by me, so I'm must guessing here.
> 
> Probably the XMLHttpRequestGeneric() is some function hiding browser specialities to create 
> AJAX requests. These days most browser understand the standard request
> 
> XMLHttpRequest()
> 
> so why don't you try to just remove the "Generic"
> 
> Stefan

That removes the error, but script doesnt get called. It goes to the javascript function and 
callback, but nothing happens. 

When I change type=button to type=submit , the script gets called again, but with page refresh. 

Hi,

So I finally got around to "publish" work I did in 2009/2010 with RTEMS.

The work was mainly between myself and Till Straumann (SLAC), and Dr. Joel 
Sherill, to get VME support for vme universe/vme tsi148 ( basic support ), into 
the i386 bsp.

https://bitbucket.org/lcpool2/midas-k600/src/develop/ ( our rtems port ).


What this did was to allow us to run our various VME single board controllers, 
with a single frontend application.

It is still classified testing but its been very successful, so
far, and I hope to use it in the next experiment, if possible.

The midas port, contains a makefile, and some changes to the 
midas.c/system.c/mfe.c files. I've not tested  the full functionality
as I'm super time limited.

Hope this is help full to others...

I am embedding histories into a custom page within an SVG,

<image x="21000" y="1000" width="6000" height="6000"
href="../HS/SampleCryo/SampleTemp.gif?width=230&scale=0.5h"/>

this works fine. However, I would like to update this regularly without
refreshing the full page via

<meta http-equiv="Refresh" content="60">

is there a good way to do that? By the way, the "Periodic update of parts of a
custom page" from the documentation does not seem to work here.

I started to use the new midas custom page features from 
https://midas.triumf.ca/MidasWiki/index.php/Custom_Page . I'd like to setup the 
editable odb values (.e.g <div name="modbvalue" data-odb-path="/Runinfo/Run 
number" data-odb-editable="1"                    
style="position:absolute;top:157px;left:288px;"></div>) from within javascript, 
which doesn`t seem to work. 

 Both 

document.getElementById("test").innerHTML = '<div data-odb-editable="1" data-
odb-path="/Runinfo/Run number" name="modbvalue" 
style="display:inline;position:absolute;top:157px;left:288px;"></div>'

or 

var elem = document.createElement("div");
	 //var id = "manifold0";
	    elem.setAttribute("name","modbvalue");
	 elem.setAttribute("data-odb-
path","/Equipment/Autofill/Variables/Output[6]");
	 elem.setAttribute("data-odb-editable","1");
	 elem.style="position:absolute;top:157px;left:288px;";
         document.getElementById("test").appendChild(elem);

fail on name="modbvalue" with the error 

mjsonrpc_error_alert: TypeError: Cannot set property 'innerHTML' of undefined


How should one do this? I don`t want hard code everything in the htlm body, as 
I have look up odb key indexes in the javascript code.

The function mhttpd_init() scans the custom page and installs handlers etc. for each modbxxx 
element. If you create an modbvalue dynamically in your code, this is probably done after you 
called mhttpd_init(), so the function has no chance to modify the dynamically created element.

To fix that, I separated mhttpd_init() into the old init function which installs the header and sidebar, 
and a function mhttpd_scan() which scans the custom page and processes all modbxxx elements. 
Next, I tapped the error you reported, and added an automatic call to mhttp_scan() in case that 
happens. I tried it on a test page and it worked for me. Please give it a try (commit 090394e8).

Stefan

> The function mhttpd_init() scans the custom page and installs handlers etc. for each modbxxx 
> element. If you create an modbvalue dynamically in your code, this is probably done after you 
> called mhttpd_init(), so the function has no chance to modify the dynamically created element.
> 
> To fix that, I separated mhttpd_init() into the old init function which installs the header and sidebar, 
> and a function mhttpd_scan() which scans the custom page and processes all modbxxx elements. 
> Next, I tapped the error you reported, and added an automatic call to mhttp_scan() in case that 
> happens. I tried it on a test page and it worked for me. Please give it a try (commit 090394e8).
> 
> Stefan

Also works for me

 thanks

> > The function mhttpd_init() scans the custom page and installs handlers etc. for each modbxxx 
> > element. If you create an modbvalue dynamically in your code, this is probably done after you 
> > called mhttpd_init(), so the function has no chance to modify the dynamically created element.
> > 
> > To fix that, I separated mhttpd_init() into the old init function which installs the header and sidebar, 
> > and a function mhttpd_scan() which scans the custom page and processes all modbxxx elements. 
> > Next, I tapped the error you reported, and added an automatic call to mhttp_scan() in case that 
> > happens. I tried it on a test page and it worked for me. Please give it a try (commit 090394e8).
> > 
> > Stefan
> 
> Also works for me
> 
>  thanks


This is more about aesthetic, but when I use the modbvalue div, it first only shows the odb value. However, 
after editing, the odb index gets added to the field, which is kinda ugly -> see attachments

There is a bug in the string handling when changing ODB string entries via the
mhttpd (git sha 07dfb83). It truncates the string length in the ODB.

For instance I create a string with length 32 and set it with odbedit to 'a'.
Then the string length stays 32, as expected. If the same is done through the
web-interface, the string length will be truncated to 2.

This can lead to problems if some frontend has a hotlink to a structure
containing this string since it will complain about structure size mismatch. 

> There is a bug in the string handling when changing ODB string entries via the
> mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
> 
> For instance I create a string with length 32 and set it with odbedit to 'a'.
> Then the string length stays 32, as expected. If the same is done through the
> web-interface, the string length will be truncated to 2.
> 
> This can lead to problems if some frontend has a hotlink to a structure
> containing this string since it will complain about structure size mismatch. 

I know about this problem since last summer. I mentioned it to KO, since it's deep down in his 
JSONRPC code. We had a long discussion, where he kind of insisted that this is not a bug but a 
feature. The ODB should store strings with variable lengths, and thus adapt it according to the 
current string length. This makes some sense, since in the future we plan to put C++ string 
support for the ODB, where strings have dynamically varying lengths. But this will take a while, so 
I asked KO to change the truncation of the strings though the web interface, because this breaks 
many experiments. He did not react so far. Several people complained. Maybe your request will 
help now.

Stefan

> > There is a bug in the string handling when changing ODB string entries via the
> > mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
> > 
> > For instance I create a string with length 32 and set it with odbedit to 'a'.
> > Then the string length stays 32, as expected. If the same is done through the
> > web-interface, the string length will be truncated to 2.
> > 
> > This can lead to problems if some frontend has a hotlink to a structure
> > containing this string since it will complain about structure size mismatch. 
> 
> I know about this problem since last summer. I mentioned it to KO, since it's deep down in his 
> JSONRPC code. We had a long discussion, where he kind of insisted that this is not a bug but a 
> feature. The ODB should store strings with variable lengths, and thus adapt it according to the 
> current string length. This makes some sense, since in the future we plan to put C++ string 
> support for the ODB, where strings have dynamically varying lengths. But this will take a while, so 
> I asked KO to change the truncation of the strings though the web interface, because this breaks 
> many experiments. He did not react so far. Several people complained. Maybe your request will 
> help now.
> 
> Stefan

Well I appreciate the direction towards more C++ string handling, yet it must not break the hotlink
functionality which is very important at many places.

I wanted to try to summarize the current situation with regards to the handling of strings through the MIDAS web interface.

During the last year, we started the switch-over to using the new mjson-rpc functions in the MIDAS webpages.  As part of this work, changes were made that allowed for 
resizing strings through the MIDAS web interface (specifically through the MJSON-RPC calls). This reflected desires from some users that strings could be allowed to 
grow larger than the default 256 size that is usually used for MIDAS strings (for instance, see [1]).  In this first set of changes the ODB strings would always be resized to 
the exact size of the string that it was set to.

The problem with this change was that it breaks the behaviour of db_get_record(), which typically expects strings to be a fixed length of 32 or 256 (for instance, see 
[2,3]).  Konstantin notes that we can work around this problem by using db_get_record1() function, which automatically resizes strings to expected values; this method 
has already been used in the MIDAS core code.  But the problem would still remain for some user code that uses db_get_record.

As a short-term compromise solution, Stefan implemented a change where the MJSON-RPC string setting will now expand the size of strings, but not truncate them; ie a 
string that is size 256 will stay 256 bytes, unless you set it to something larger.  So this should fix most cases of user code that call db_get_record() and hence expects 
fixed string lengths. Users who call that db_get_record with strings that exceed the expected length will still have problems; but now you will at least get an explicit 
MIDAS error message, rather than just silent string truncation (as was the case before).

In the longer run we still want to develop a new set of ODB methods that more naturally support C++-style variable-length strings. But that's a discussion for the longer 
term.

[1] https://midas.triumf.ca/elog/Midas/1150
[2] https://bitbucket.org/tmidas/midas/issues/121/odb-inline-editor-truncates-stings
[3] https://midas.triumf.ca/elog/Midas/1343