> I was wondering if one can use multiple pysequencers at the same time and if not
> if this feature is planned in the future? Our experiment (mu3e) has one frontend
> per detector and each sequencer would only access a subset of ODB entries.

I believe sequencer design supports this, same as mlogger supports multiple output 
files and mhttpd supports multiple listener ports.

But how to do this with pysequencer, I am not sure. Simplest is to wait a few more 
weeks for Ben to return from vacation.

K.O.

> Deferred transition is registered by the FPN00

FPN00, yes, this rings a bell.

> (main clock) process to make sure the logger has finished logging all events before continuing to stop readout
> of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
> stop without getting stuck waiting for the logger.

yes, that's right, if trigger control and run control are in the same frontend, you run into this trouble:

user pushes run stop button
you get the run stop callback
you stop the trigger
but have to wait for all the data to flush down the pipes all the way to mlogger
but cannot wait, must return from the callback otherwise run transition is stuck

if trigger control and run control are in different frontends, things are simpler:

user pushes run stop botton
trigger control frontend disables the trigger, returns without waiting for anything
data frontend (i.e. FGD, TPC), flush all the hardware FIFOs, etc to MIDAS SYSTEM buffer (trigger is already 
disabled, there is no new data)
mlogger flushes SYSTEM buffer to disk
run control frontend reports successful run stop to the run database (and whatever else).

> Interestingly, I just checked and this feature is
> disabled in the FGD ODB, but it is enabled in the TPC ODB.

I do not have access to the current code, but I can check what I have,
and I am pretty sure it did not have any deferred transitions. Maybe
it was added after my time.

K.O.

Bjarne Stroustrup gave a nice talk at CERN, good enough to bring Rene Brun out of retirement to ask a few 
questions at the 1:16:50 mark.

https://indico.cern.ch/event/1696048/
https://videos.cern.ch/record/3026440

Good listen, but closer to home:

Stroustrup slides have a wee reference to a CppCon-2024 video "C++ exceptions for smaller firmware", I gave 
it a listen. The guy talks fast and waves hands vigorously, but his material is solid.

In the first 30 minutes he busts the myth that embedded programming can not/should not use exceptions (with 
numbers and example code).

In the second 30 minutes he demonstrates the insane code complexity required to implement exceptions, and 
that also makes per-call error checking very cheap (compares all three "return bool", "return 
std::expected", "throw exception", with disassembly of example code)

The the third 30 minutes he talks about project level implications of using exceptions vs other methods of 
error handling (one example is audacity).

If you did not want to use exceptions before watching this talk, you will after!

https://www.youtube.com/watch?v=bY2FlayomlE

K.O.

> If you did not want to use exceptions before watching this talk, you will after!

Not me. One issue, exceptions are unpredictable, for example, does this code always print "hello, world"?

#include <stdio.h>
#include "foo.h"
int main(...)
{
  foo();
  printf("hello, world\n");
  return 0;
}

I can wrap foo() (and each and every function call) in try/catch.

I can check if today's version of foo.h declares foo() as "nothrow" (but what if tomorrow's foo() start 
throwing?)

Unpredictable code can be dangerous:

void boil_kettle()
{
   i2c_set_bit(10, 1); // bit 10 is stove heater control, turn it on
   foo(); // wait for kettle to start boiling
   i2c_set_bit(10, 0); // turn heater off
}

If foo() starts throwing exceptions, I will likely have a fire in my kitchen!

K.O.

I finished debugging the code I wrote to check and repair the ODB key area and data area free lists, this 
should fix the famous "ODB is full" error.

I have one computer that runs MIDAS that seems to have a hardware problem and crashes about every 1-2 
weeks. After each crash MIDAS ODB is always corrupted and I started writing the (long desired) free list 
checker as a way to try to understand the nature of the corruption. (more on that at the end)

Once I had a reliable checker and I remembered how the ODB free lists work,
writing the repair code was easy.

I tested it on my collection of corrupted .ODB.SHM files, and in each case,
after repair, I can start odbedit, look at ODB data and save it to JSON file.

Of course every corruption loses some data, for full recovery, ODB must be reloaded from a backup. But at 
least one does not have to kill ODB and recreate it from scratch anymore (in theory).

One type of corruption I cannot fix: if two ODB entries seem to overlap, in the key area (I cannot repair), 
or in the data area (I delete one of the overlapping entries).

Because repair of the ODB free list can be dangerous, I disabled it by default.

To run odbedit in "repair" mode, do this (we could also make it an odbedit option)

MIDAS_ODB_REPAIR_FREE_LIST=1 ./bin/odbedit

While looking at corrupted ODB files, I fixed two more buglets:

1) if ODB entry has num_values set to 0 (unusual, but not an error), save odb.json produces invalid JSON
2) if /Experiment/Buffer sizes/SYSMSG has a bogus value (I tend to see "0"), odbedit would not start and I 
cannot edit ODB to fix the problem.

I have this code on a private branch and I will merge and push it in the next few days.

P.S.

Cause of corruption is still not clear, best guess is that there is a hardware memory failure, but this 
explanation requires a difficult sequence: memory goes bad, ODB is corrupted, linux and ZFS still running, 
MIDAS writes corrupted ODB to file .ODB.SHM, linux and ZFS still running, more memory corruption, linux 
dies, reboots/resets.

Alternatively, memory corruption affects the ZFS cache, ODB in RAM is okey, but .ODB.SHM in ZFS cache is 
corrupted, but still gets written to disk, then more memory corruption kills linux. The ZFS volume itself 
does not show any corruption, "zfs scrub" comes back clean.

Garden variety "buggy midas frontend corrupted ODB" is not good enough to explain computer reset/reboot.

K.O.

> I finished debugging the code I wrote to check and repair the ODB key area and data area free lists, this 
> should fix the famous "ODB is full" error.

commit 67c2160891c2688a956189752e00ae1acb7b531c

develop branch before this merge tagged midas-2026-07-a

K.O.

> I updated the midas forum elog to the latest version from git: 083448f7
> 
> Also investigated elogd failure to start on reboot,
> it turned out to be a crasher bug, see
> https://elog.psi.ch/elogs/Forum/69919
>

Also investigating elogd crash last Friday, looks like an elog bug, see

https://elog.psi.ch/elogs/Forum/69933

K.O.

> Looks like the code formatting got messed up by the elog... here it is in the attachment instead.

unfortunately, your code does nothing because you do not call "i2c_set_bit(10, 1);".

if you add it in the destructor:

~i2c_temp_bit() {
std::cout << "Bit " << bit << " set to " << oldval << "\n";
i2c_set_bit(10, 0);
}

you will burn the kitchen down if "std::cout" and "<<" throw an exception (as we know they do). (I 
always use printf(), instead of c++ exceptions, it can throw the SIGPIPE signal, so main() must 
always have signal(SIGPIPE, SIG_IGN);).

also if usleep() throws an exception, the water does not get boiled to 100 degC, important for food 
safety.

also there is a logic error, if you are at a high enough elevation (Mount Everest), water starts 
boiling well below 100 degC, so the loop never ends, probably until all water is gone and the empty 
kettle is heated to 100 degC, at this point, both kettle and heater are probably damaged. the 
infinite loop should have some kind of safety limit.

K.O.

Also a crash soon after startup:

https://elog.psi.ch/elogs/Forum/69934

K.O.

crashed again, restarted. gcc address sanitizer disables core dumps by default, to enable, do this:
export ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1
K.O.

> crashed again, restarted. gcc address sanitizer disables core dumps by default, to enable, do this:
> export ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1

midas forum elogd would not stay up, bot traffic crashes it within hours.

as temporary mitigation:
- elogd is now running from an autorestart script (15 second delay between restart attempts)
- address sanitizer and core dumps enabled to collect and fix crashes as they are identified (I fixed 4-5 crashers, so far)

K.O.

this is a draft message, it will be updated with additional information. K.O.

Back in 2007, Stefan implemented the very useful feature, ODB links to array elements,
https://daq00.triumf.ca/elog-midas/Midas/418

This is handy for "Edit On Start", for history links, for feepics and other places.

If you have an array in ODB: 

ia5                             INT32   5     4     12m  0  RWD   
                                        [0]             10
                                        [1]             11
                                        [2]             12
                                        [3]             13
                                        [4]             14

a normal ODB link works as a UNIX filesystem symlink, whereas an ODB link that includes an array index refers to just one element of the link target array:

symlink_to_ia5_2 -> /test_odb/ia5[2]
                                INT32   1     4     12m  0  RWD   12

Because there was some confusion over how it works and how it is actually implements,
some ODB functions do not implement this feature consistently.

Here, I explain how it actually works:

0) expected syntax is: "link name" -> "/link/target/array[12345]":

- "/link/target/array" is the absolute ODB path to the link target (must start with "/", relative links are not permitted)
- array index "12345" should be an integer (converted using atoi())
- there should be nothing after final "]"
- there should be nothing between the "[" and the array index decimal numeric value

1) db_get_data():

- before calling db_get_data() we must call db_find_key()
- db_find_key() will resolve ODB links and return the final destination (or an error if dangling link or circular link or too many nested links)
- db_get_data() will return the data from this ODB key. this is the path for normal symlinks.

- if db_find_key() encounters an ODB link to an array element (target path contains "["), it returns this key (of type TID_LINK).
- db_get_data() checks the key type (normally it would be TID_INT, TID_STRING, etc). if it is TID_LINK, it means we have a link to an array index (as identified by db_find_key())
- in this case, ODB link is resolved using db_find_key(), array index is extracted from the link
- and db_get_data() returns data for the corresponding array element

If db_get_data() is called using an hKey returned by db_find_link() (instead of db_find_key())
and it happens to be an ODB link (TID_LINK), we have an inconsistent result:

- if it's a normal link, db_get_data() will return a type mismatch error
- db_get_data(TID_LINK) will return the link target string (NUL-terminated)
- if it's a link to an array element, db_get_data() will resolve it and return data from the link target array element.
- db_get_data(TID_LINK) will return a type mismatch error because it will resolve the link, then fail the type check as link target cannot be TID_LINK

2) db_get_value(), db_get_data_index()

- normal links work, resolved by db_find_key()
- links to array elements not implemented, will return a type mismatch error (i.e. TID_LINK vs user requested TID_STRING, TID_INT, etc)

3) mhttpd obsolete "jcopy"

- broken, returns malformed json (old code), returns the complete array, not just the linked array element (new code)

4) odbedit save odb.json

- broken, output file odb.json is empty (new code)

K.O.

> > I finished debugging the code I wrote to check and repair the ODB key area and data area free lists, this 
> > should fix the famous "ODB is full" error.
> 
> commit 67c2160891c2688a956189752e00ae1acb7b531c
> 
> develop branch before this merge tagged midas-2026-07-a
> 
> K.O.

first bug is in, incorrect calculation of data free space of empty ODB, reported by "make test".

commit bae880f1b793496a6f341054a8618dbdb9cc3b2c

K.O.

> > > I was wondering if one can use multiple pysequencers at the same time and if not
> > > if this feature is planned in the future? Our experiment (mu3e) has one frontend
> > > per detector and each sequencer would only access a subset of ODB entries.
> > 
> > I believe sequencer design supports this, same as mlogger supports multiple output 
> > files and mhttpd supports multiple listener ports.
> > 
> > But how to do this with pysequencer, I am not sure. Simplest is to wait a few more 
> > weeks for Ben to return from vacation.
> > 
> > K.O.
> 
> I looked into this, and it has been possible for at least a year. Just start the sequencer with `-c MyUniqueName` and 
> then all of the state etc. can be controlled from the ODB location `/PySequencerMyUniqueName/`.
> And obviously use different names for each instance.

that's what I remember, too. but was not sure. K.O.

There was a bit of confusion on the MIDAS Programs page with programs that have 
similar names. This confusion relates to the use of bUnique in cm_shutdown() and 
cm_exist().

bUnique is now removed from cm_shutdown(), midas client name passed to this call 
must match the client name exactly (as reported i.e. by odbedit "scl"). this is 
same as calling cm_shutdown() with bUnique set to TRUE.

bUnique remains in cm_exist() for now. everybody calls it with bUnique set to 
TRUE, except for the "program not running" alarm and for the "start program" 
code on the MIDAS Programs page. After they are updated, bUnique will be removed 
from cm_exist() as well. it will require the exact client name, same as calling 
it with bUnique set to TRUE.

the related confusion on the MIDAS Programs page was caused by the code for 
matching client entries in /System/Clients to program pages in /Programs. This 
is needed to report Alarm conditions, "start" and "stop" buttons, etc.

the MIDAS Programs page is constructed from ODB /Programs, each program entry 
gets one line on the web page. (non-running, non-required programs are omitted).

normally, there is only 1 copy of each MIDAS program running (i.e. mhttpd, 
mlogger, mserver, vme frontend, etc) and the client name is the same as the 
program name in /Programs, matching them is easy and they are always shown on 
the correct line on the web page.

some programs can be started with many copies, i.e. odbedit, mdump, etc. the 
client names will be "program name" plus a number, i.e. odbedit, odbedit1, 
odbedit2, etc. these programs are grouped together on one line of the web page. 
grouping is done by matching client names against entries in ODB /Programs and 
this test sometimes misfires.

for example if there is an entry for "/Programs/odb", odbedit, odbedit1 & co, 
will show up on two lines of the web page, the normal "odbedit" line and the 
unexpected "odb" line. this is because name matching used truncation instead of 
a check for "program name plus a number".

similar malfunction happens if experiment has 2 frontends named "frontend" and 
"frontend_for_vme_readout". there will be 2 lines on the web page, one for each 
frontend, and "frontend_for_vme_readout" will show up on both lines (name check 
is done by truncating the name to "frontend".

this is now fixed by using the correct check for client name: "program name plus 
a number".

similar thing happens for indexed frontends ("frontend -i 1"), depending on how 
frontend equipments are setup, the client name will be "program name plus 
frontend index". the MIDAS Programs page may or may not match all such indexed 
frontends as one group and provide a common "stop" button for all of them. I am 
not sure if this accidental feature survives the current update.

the last malfunction happens with the Programs page "stop" button, it called 
cm_shutdown() with bUnique set to FALSE, and in the above example, shut down 
both programs, "frontend" and "frontend_for_vme_readout". Unexpected and 
undesired.

removal of bUnique from cm_shutdown() fixed this. to shut down multiple clients, 
the Programs page now makes a separate call for each one of them.

K.O.

Hello Stefan,
is there a difference for the later data processing (after writing the ring buffer blocks)
if we write single events or multiple in one rb_get_wp - memcopy - rb_increment_wp cycle?
Both Marius and me have seen some inconsistencies in the number of events produced that is reported in the status page when writing multiple events in one go,
so I was wondering if this is due to us treating the buffer badly or the way midas handles the events after that.

Given that we produce the full event in our (FPGA) domain, an option would be to always copy one event from the dma to the midas-system buffer in a loop.
The question is if there is a difference (for midas) between
[pseudo code, much simplified]

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
    dma_read_index+=event_size;
    continue;   
  }
  copy_n(dma_buffer, pdata, event_size);
  rb_increment_wp(event_size);
  dma_read_index+=event_size;
} 

and

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
     ...
  };
  total_size=max_n_events_that_fit_in_rb_block();
  copy_n(dma_buffer, pdata, total_size);
  rb_increment_wp(total_size);
  dma_read_index+=total_size;
}

Cheers,
Konrad

> The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:
> 
> 1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. It's your responsibility to not write more bytes into pdata then 
> what you specified as max event size in rb_create()
> 
> 2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
> function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I don't see this behaviour in your code. You would need something 
> like
> 
> do {
>    status = rb_get_wp(rbh, (void **)&pdata, 10);
>    if (status == DB_TIMEOUT)
>       ss_sleep(10);
>    } while (status == DB_TIMEOUT);
> 
> Best,
> Stefan
> 
> 
> > Dear all,
> > 
> > we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:
> > 
> > // event 1
> > dma_buf[0] = 0x00000001; // Trigger and Event ID
> > dma_buf[1] = 0x00000001; // Serial number
> > dma_buf[2] = TIME; // time
> > dma_buf[3] = 18*4-4*4; // event size
> > dma_buf[4] = 18*4-6*4; // all bank size
> > dma_buf[5] = 0x11; // flags
> > // bank 0
> > dma_buf[6] = 0x46454230; // bank name
> > dma_buf[7] = 0x6; // bank type TID_DWORD
> > dma_buf[8] = 0x3*4; // data size
> > dma_buf[9] = 0xAFFEAFFE; // data
> > dma_buf[10] = 0xAFFEAFFE; // data
> > dma_buf[11] = 0xAFFEAFFE; // data
> > // bank 1
> > dma_buf[12] = 0x1; // bank name
> > dma_buf[12] = 0x46454231; // bank name
> > dma_buf[13] = 0x6; // bank type TID_DWORD
> > dma_buf[14] = 0x3*4; // data size
> > dma_buf[15] = 0xAFFEAFFE; // data
> > dma_buf[16] = 0xAFFEAFFE; // data
> > dma_buf[17] = 0xAFFEAFFE; // data
> > 
> > // event 2
> > .....
> > 
> > dma_buf[fpga_pointer] = 0xXXXXXXXX;
> > 
> > 
> > And we do something like:
> > 
> > while{true}
> >    // obtain buffer space
> >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> >    fpga_pointer = fpga.read_last_data_add();
> > 
> >    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
> >    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
> >    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> > 
> >    last_fpga_pointer = fpga_pointer;
> > 
> > Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> > 
> > The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> > 
> > Remark: dma_buf is volatile so memcpy is not possible here.
> > 
> > Cheers,
> > Marius

> Hi there:
> I have a question if there's anybody out there running MIDAS with event builder
> that assembles events from more that just a few front ends (say on the order of
> 0x10 or more)?
> Any experiences with scalability?
> 
> Cheers
>  Piotr

Mulan (which you hopefully remember with great fondness :-) is currently running
around ten frontends, six of which produce data at any rate.  If I'm remembering
correctly, the event builder handles about 30-40MB/s.  You could probably ping Tim
Gorringe or his current postdoc Volodya Tishenko (tishenko@pa.uky.edu) if you want
more details.  Volodya solved a significant number of throughput related
bottlenecks in the year leading up to our 2006 run.

creating a frontend on MAC Sierra OSX 10
include the midas.h file and when compiling with XCode I get an error based on
this entry in the midas.h include

#if !defined(OS_IRIX) && !defined(OS_VMS) && !defined(OS_MSDOS) &&
!defined(OS_UNIX) && !defined(OS_VXWORKS) && !defined(OS_WINNT)
#error MIDAS cannot be used on this operating system
#endif


Perhaps I should not use Xcode?
Perhaps I won't need Midas.h?

The MIDAS system is running on my MAC but I need to add a very simple front end
for testing and I encounted this error.

Dear all,

please do not be surprised if your 2D plots start looking different: The default
colour scheme changed to the (now kind of standard one) 'viridis' from
matplotlib to have perceptually uniform sequential colour scheme. So if you
print your plots in black and white or have problems colour vision deficiency
the plots should not have artificial artifacts anymore.


Cheers,
Julian

One can collapse the side panel when looking at history pages with the button in
the top left, great! We want to see many pages so screen real estate is important

The issue we face is that when the page refreshes, the side panel expands. Can
we make the panel state more 'sticky'?

Many thanks
Joseph (ALPHA)

Version: 	2.1
Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
on branch feature/midas-2017-10