Bjarne Stroustrup gave a nice talk at CERN, good enough to bring Rene Brun out of retirement to ask a few 
questions at the 1:16:50 mark.

https://indico.cern.ch/event/1696048/
https://videos.cern.ch/record/3026440

Good listen, but closer to home:

Stroustrup slides have a wee reference to a CppCon-2024 video "C++ exceptions for smaller firmware", I gave 
it a listen. The guy talks fast and waves hands vigorously, but his material is solid.

In the first 30 minutes he busts the myth that embedded programming can not/should not use exceptions (with 
numbers and example code).

In the second 30 minutes he demonstrates the insane code complexity required to implement exceptions, and 
that also makes per-call error checking very cheap (compares all three "return bool", "return 
std::expected", "throw exception", with disassembly of example code)

The the third 30 minutes he talks about project level implications of using exceptions vs other methods of 
error handling (one example is audacity).

If you did not want to use exceptions before watching this talk, you will after!

https://www.youtube.com/watch?v=bY2FlayomlE

K.O.

> If you did not want to use exceptions before watching this talk, you will after!

Not me. One issue, exceptions are unpredictable, for example, does this code always print "hello, world"?

#include <stdio.h>
#include "foo.h"
int main(...)
{
  foo();
  printf("hello, world\n");
  return 0;
}

I can wrap foo() (and each and every function call) in try/catch.

I can check if today's version of foo.h declares foo() as "nothrow" (but what if tomorrow's foo() start 
throwing?)

Unpredictable code can be dangerous:

void boil_kettle()
{
   i2c_set_bit(10, 1); // bit 10 is stove heater control, turn it on
   foo(); // wait for kettle to start boiling
   i2c_set_bit(10, 0); // turn heater off
}

If foo() starts throwing exceptions, I will likely have a fire in my kitchen!

K.O.

I finished debugging the code I wrote to check and repair the ODB key area and data area free lists, this 
should fix the famous "ODB is full" error.

I have one computer that runs MIDAS that seems to have a hardware problem and crashes about every 1-2 
weeks. After each crash MIDAS ODB is always corrupted and I started writing the (long desired) free list 
checker as a way to try to understand the nature of the corruption. (more on that at the end)

Once I had a reliable checker and I remembered how the ODB free lists work,
writing the repair code was easy.

I tested it on my collection of corrupted .ODB.SHM files, and in each case,
after repair, I can start odbedit, look at ODB data and save it to JSON file.

Of course every corruption loses some data, for full recovery, ODB must be reloaded from a backup. But at 
least one does not have to kill ODB and recreate it from scratch anymore (in theory).

One type of corruption I cannot fix: if two ODB entries seem to overlap, in the key area (I cannot repair), 
or in the data area (I delete one of the overlapping entries).

Because repair of the ODB free list can be dangerous, I disabled it by default.

To run odbedit in "repair" mode, do this (we could also make it an odbedit option)

MIDAS_ODB_REPAIR_FREE_LIST=1 ./bin/odbedit

While looking at corrupted ODB files, I fixed two more buglets:

1) if ODB entry has num_values set to 0 (unusual, but not an error), save odb.json produces invalid JSON
2) if /Experiment/Buffer sizes/SYSMSG has a bogus value (I tend to see "0"), odbedit would not start and I 
cannot edit ODB to fix the problem.

I have this code on a private branch and I will merge and push it in the next few days.

P.S.

Cause of corruption is still not clear, best guess is that there is a hardware memory failure, but this 
explanation requires a difficult sequence: memory goes bad, ODB is corrupted, linux and ZFS still running, 
MIDAS writes corrupted ODB to file .ODB.SHM, linux and ZFS still running, more memory corruption, linux 
dies, reboots/resets.

Alternatively, memory corruption affects the ZFS cache, ODB in RAM is okey, but .ODB.SHM in ZFS cache is 
corrupted, but still gets written to disk, then more memory corruption kills linux. The ZFS volume itself 
does not show any corruption, "zfs scrub" comes back clean.

Garden variety "buggy midas frontend corrupted ODB" is not good enough to explain computer reset/reboot.

K.O.

> I finished debugging the code I wrote to check and repair the ODB key area and data area free lists, this 
> should fix the famous "ODB is full" error.

commit 67c2160891c2688a956189752e00ae1acb7b531c

develop branch before this merge tagged midas-2026-07-a

K.O.

> I updated the midas forum elog to the latest version from git: 083448f7
> 
> Also investigated elogd failure to start on reboot,
> it turned out to be a crasher bug, see
> https://elog.psi.ch/elogs/Forum/69919
>

Also investigating elogd crash last Friday, looks like an elog bug, see

https://elog.psi.ch/elogs/Forum/69933

K.O.

> Looks like the code formatting got messed up by the elog... here it is in the attachment instead.

unfortunately, your code does nothing because you do not call "i2c_set_bit(10, 1);".

if you add it in the destructor:

~i2c_temp_bit() {
std::cout << "Bit " << bit << " set to " << oldval << "\n";
i2c_set_bit(10, 0);
}

you will burn the kitchen down if "std::cout" and "<<" throw an exception (as we know they do). (I 
always use printf(), instead of c++ exceptions, it can throw the SIGPIPE signal, so main() must 
always have signal(SIGPIPE, SIG_IGN);).

also if usleep() throws an exception, the water does not get boiled to 100 degC, important for food 
safety.

also there is a logic error, if you are at a high enough elevation (Mount Everest), water starts 
boiling well below 100 degC, so the loop never ends, probably until all water is gone and the empty 
kettle is heated to 100 degC, at this point, both kettle and heater are probably damaged. the 
infinite loop should have some kind of safety limit.

K.O.

Also a crash soon after startup:

https://elog.psi.ch/elogs/Forum/69934

K.O.

crashed again, restarted. gcc address sanitizer disables core dumps by default, to enable, do this:
export ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1
K.O.

> crashed again, restarted. gcc address sanitizer disables core dumps by default, to enable, do this:
> export ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1

midas forum elogd would not stay up, bot traffic crashes it within hours.

as temporary mitigation:
- elogd is now running from an autorestart script (15 second delay between restart attempts)
- address sanitizer and core dumps enabled to collect and fix crashes as they are identified (I fixed 4-5 crashers, so far)

K.O.

this is a draft message, it will be updated with additional information. K.O.

Back in 2007, Stefan implemented the very useful feature, ODB links to array elements,
https://daq00.triumf.ca/elog-midas/Midas/418

This is handy for "Edit On Start", for history links, for feepics and other places.

If you have an array in ODB: 

ia5                             INT32   5     4     12m  0  RWD   
                                        [0]             10
                                        [1]             11
                                        [2]             12
                                        [3]             13
                                        [4]             14

a normal ODB link works as a UNIX filesystem symlink, whereas an ODB link that includes an array index refers to just one element of the link target array:

symlink_to_ia5_2 -> /test_odb/ia5[2]
                                INT32   1     4     12m  0  RWD   12

Because there was some confusion over how it works and how it is actually implements,
some ODB functions do not implement this feature consistently.

Here, I explain how it actually works:

0) expected syntax is: "link name" -> "/link/target/array[12345]":

- "/link/target/array" is the absolute ODB path to the link target (must start with "/", relative links are not permitted)
- it is resolved using db_find_key() and can include additional ODB links
- array index "12345" should be an integer (converted using atoi())
- there should be nothing after final "]"
- there should be nothing between the "[" and the array index decimal numeric value

1) db_get_data():

- before calling db_get_data() we must call db_find_key()
- db_find_key() will resolve ODB links and return the final destination (or an error if dangling link or circular link or too many nested links)
- db_get_data() will return the data from this ODB key. this is the path for normal symlinks.

- if db_find_key() encounters an ODB link to an array element (target path contains "["), it returns this key (of type TID_LINK).
- db_get_data() checks the key type (normally it would be TID_INT, TID_STRING, etc). if it is TID_LINK, it means we have a link to an array index (as identified by db_find_key())
- in this case, ODB link is resolved using db_find_key(), array index is extracted from the link
- and db_get_data() returns data for the corresponding array element

If db_get_data() is called using an hKey returned by db_find_link() (instead of db_find_key())
and it happens to an ODB link (TID_LINK), we have an inconsistent result:

- if it's a normal link, db_get_data() will return a type mismatch error
- db_get_data(TID_LINK) will return the link target string (NUL-terminated)
- if it's a link to an array element, db_get_data() will resolve it and return data from the link target array element.
- db_get_data(TID_LINK) will return a type mismatch error because it will resolve the link, then fail the type check as link target cannot be TID_LINK

2) db_get_value():

- not implemented, best I can tell, will return a type mismatch error (i.e. TID_LINK vs user requested TID_STRING, TID_INT, etc)

K.O.

> > I finished debugging the code I wrote to check and repair the ODB key area and data area free lists, this 
> > should fix the famous "ODB is full" error.
> 
> commit 67c2160891c2688a956189752e00ae1acb7b531c
> 
> develop branch before this merge tagged midas-2026-07-a
> 
> K.O.

first bug is in, incorrect calculation of data free space of empty ODB, reported by "make test".

commit bae880f1b793496a6f341054a8618dbdb9cc3b2c

K.O.

Hello Stefan,
is there a difference for the later data processing (after writing the ring buffer blocks)
if we write single events or multiple in one rb_get_wp - memcopy - rb_increment_wp cycle?
Both Marius and me have seen some inconsistencies in the number of events produced that is reported in the status page when writing multiple events in one go,
so I was wondering if this is due to us treating the buffer badly or the way midas handles the events after that.

Given that we produce the full event in our (FPGA) domain, an option would be to always copy one event from the dma to the midas-system buffer in a loop.
The question is if there is a difference (for midas) between
[pseudo code, much simplified]

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
    dma_read_index+=event_size;
    continue;   
  }
  copy_n(dma_buffer, pdata, event_size);
  rb_increment_wp(event_size);
  dma_read_index+=event_size;
} 

and

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
     ...
  };
  total_size=max_n_events_that_fit_in_rb_block();
  copy_n(dma_buffer, pdata, total_size);
  rb_increment_wp(total_size);
  dma_read_index+=total_size;
}

Cheers,
Konrad

> The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:
> 
> 1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. It's your responsibility to not write more bytes into pdata then 
> what you specified as max event size in rb_create()
> 
> 2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
> function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I don't see this behaviour in your code. You would need something 
> like
> 
> do {
>    status = rb_get_wp(rbh, (void **)&pdata, 10);
>    if (status == DB_TIMEOUT)
>       ss_sleep(10);
>    } while (status == DB_TIMEOUT);
> 
> Best,
> Stefan
> 
> 
> > Dear all,
> > 
> > we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:
> > 
> > // event 1
> > dma_buf[0] = 0x00000001; // Trigger and Event ID
> > dma_buf[1] = 0x00000001; // Serial number
> > dma_buf[2] = TIME; // time
> > dma_buf[3] = 18*4-4*4; // event size
> > dma_buf[4] = 18*4-6*4; // all bank size
> > dma_buf[5] = 0x11; // flags
> > // bank 0
> > dma_buf[6] = 0x46454230; // bank name
> > dma_buf[7] = 0x6; // bank type TID_DWORD
> > dma_buf[8] = 0x3*4; // data size
> > dma_buf[9] = 0xAFFEAFFE; // data
> > dma_buf[10] = 0xAFFEAFFE; // data
> > dma_buf[11] = 0xAFFEAFFE; // data
> > // bank 1
> > dma_buf[12] = 0x1; // bank name
> > dma_buf[12] = 0x46454231; // bank name
> > dma_buf[13] = 0x6; // bank type TID_DWORD
> > dma_buf[14] = 0x3*4; // data size
> > dma_buf[15] = 0xAFFEAFFE; // data
> > dma_buf[16] = 0xAFFEAFFE; // data
> > dma_buf[17] = 0xAFFEAFFE; // data
> > 
> > // event 2
> > .....
> > 
> > dma_buf[fpga_pointer] = 0xXXXXXXXX;
> > 
> > 
> > And we do something like:
> > 
> > while{true}
> >    // obtain buffer space
> >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> >    fpga_pointer = fpga.read_last_data_add();
> > 
> >    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
> >    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
> >    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> > 
> >    last_fpga_pointer = fpga_pointer;
> > 
> > Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> > 
> > The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> > 
> > Remark: dma_buf is volatile so memcpy is not possible here.
> > 
> > Cheers,
> > Marius

> Hi there:
> I have a question if there's anybody out there running MIDAS with event builder
> that assembles events from more that just a few front ends (say on the order of
> 0x10 or more)?
> Any experiences with scalability?
> 
> Cheers
>  Piotr

Mulan (which you hopefully remember with great fondness :-) is currently running
around ten frontends, six of which produce data at any rate.  If I'm remembering
correctly, the event builder handles about 30-40MB/s.  You could probably ping Tim
Gorringe or his current postdoc Volodya Tishenko (tishenko@pa.uky.edu) if you want
more details.  Volodya solved a significant number of throughput related
bottlenecks in the year leading up to our 2006 run.

creating a frontend on MAC Sierra OSX 10
include the midas.h file and when compiling with XCode I get an error based on
this entry in the midas.h include

#if !defined(OS_IRIX) && !defined(OS_VMS) && !defined(OS_MSDOS) &&
!defined(OS_UNIX) && !defined(OS_VXWORKS) && !defined(OS_WINNT)
#error MIDAS cannot be used on this operating system
#endif


Perhaps I should not use Xcode?
Perhaps I won't need Midas.h?

The MIDAS system is running on my MAC but I need to add a very simple front end
for testing and I encounted this error.

One can collapse the side panel when looking at history pages with the button in
the top left, great! We want to see many pages so screen real estate is important

The issue we face is that when the page refreshes, the side panel expands. Can
we make the panel state more 'sticky'?

Many thanks
Joseph (ALPHA)

Version: 	2.1
Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
on branch feature/midas-2017-10

> > 
> > 
> > One can collapse the side panel when looking at history pages with the button in
> > the top left, great! We want to see many pages so screen real estate is important
> > 
> > The issue we face is that when the page refreshes, the side panel expands. Can
> > we make the panel state more 'sticky'?
> > 
> > Many thanks
> > Joseph (ALPHA)
> > 
> > Version: 	2.1
> > Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
> > on branch feature/midas-2017-10
> 
> Hi Joseph,
> 
> In principle a page refresh should now not be necessary, since pages should reload automatically 
> the contents which changes. If a custom page needs a reload, it is not well designed. If necessary, I 
> can explain the details. 
> 
> Anyhow I implemented your "stickyness" of the side panel in the last commit to the develop branch.
> 
> Best regards,
> Stefan

Hi Stefan,

I apologise for miss using the word refresh. The re-appearing sidebar was also seen with the automatic
reload, I have implemented your fix here and it now works great!

Thank you very much!
Joseph

Hi,

I have been looking at the 2019 workshop slides, I am interested in the C++ future of MIDAS. 

I am quite interested in using the object oriented 


ALPHA will start data taking in 2021

I have written a system monitor tool for MIDAS, that has been merged in the develop branch today: sysmon

https://bitbucket.org/tmidas/midas/pull-requests/8/system-monitoring-a-new-frontend-to-log/diff

To use it, simply run the new program
sysmon
on any host that you want to monitor, no configuring required.




The program is a frontend for MIDAS, there is no need for configuration, as upon initialisation it builds a history display for you. Simply run one instance per machine you want to monitor. By default, it only logs once per 10 seconds.

The equipment name is derived from the hostname, so multiple instances can be run across multiple machines without conflict. A new history display will be created for each host.

sysmon uses the /proc pseudo-filesystem, so unfortunately only linux is supported. It does however work with multiple architectures, so x86 and ARM processors are supported.

If the build machine has NVIDIA drivers installed, there is an additional version of sysmon that gets built: sysmon-nvidia. This will log the GPU temperature and usage, as well as CPU, memory and swap. A host should only run either sysmon or sysmon-nvidia

elog:1727/1 shows the History Display generated by sysmon-nvidia. sysmon would only generate the first two displays (sysmon/localhost and sysmon/localhost-CPU)

A little advertised feature of the modifications needed support the msysmon program is 
that MIDAS equipment names can support the injecting of the hostname of the system 
running the frontend at runtime (register_equipment(void)).

https://midas.triumf.ca/MidasWiki/index.php/Equipment_List_Parameters#Equipment_Name

A special string ${HOSTNAME} can be put in any position in the equipment name. It will 
be replaced with the hostname of the computer running the frontend at run-time. Note, 
the frontend_name string will be trimmed down to 32 characters.
Example usage: msysmon


EQUIPMENT equipment[] = {

  { "${HOSTNAME}_msysmon",   /* equipment name */    {
      EVID_MONITOR, 0,      /* event ID, trigger mask */
      "SYSTEM",             /* event buffer */
      EQ_PERIODIC,          /* equipment type */
      0,                    /* event source */
      "MIDAS",              /* format */
      TRUE,                 /* enabled */
      RO_ALWAYS,            /* Read when running */
      10000,                /* poll every so milliseconds */
      0,                    /* stop run after this event limit */
      0,                    /* number of sub events */
      1,                    /* history period */
      "", "", ""
    },
    read_system_load,/* readout routine */
  },
  { "" }
};

Hi all,

I have been experimenting with a frontend solution for my experiment 
(ALPHA). The intention to replace how we log data from PCs running LabVIEW. 

I am at the proof of concept stage. So far I have some promising 
performance, able to handle 10-100x more data in my test setup (current 
limitations now are just network bandwith, MIDAS is impressively efficient). 

==========================================================================

Our experiment has many PCs using LabVIEW which all log to MIDAS, the 
experiment has grown such that we need some sort of load balancing in our 
frontend.

The concept was to have a 'supervisor frontend' and an array of 'worker 
frontend' processes. 
-A LabVIEW client would connect to the supervisor, then be referred to a 
worker frontend for data logging. 
-The supervisor could start a 'worker frontend' process as the demand 
required.

To increase accountability within the experiment, I intend to have a 'worker 
frontend' per PC connecting. Then any rouge behavior would be clear from the 
MIDAS frontpage.

Presently there around 20-30 of these LabVIEW PCs, but given how the group 
is growing, I want to be sure that my data logging solution will be viable 
for the next 5-10 years. With the increased use of single board computers, I 
chose the target of benchmarking upto 1000 worker frontends... but I quickly 
hit the '64 MAX CLIENTS' and '64 RPC CONNECTION' limit. Ok...

branching and updating these limits:
https://bitbucket.org/tmidas/midas/branch/experimental-beyond_64_clients

I have two commits. 
1. update the memory layout assertions and use MAX_CLIENTS as a variable
https://bitbucket.org/tmidas/midas/commits/302ce33c77860825730ce48849cb810cf
366df96?at=experimental-beyond_64_clients
2. Change the MAX_CLIENTS and MAX_RPC_CONNECTION
https://bitbucket.org/tmidas/midas/commits/f15642eea16102636b4a15c8411330969
6ce3df1?at=experimental-beyond_64_clients

Unintended side effects:
I break compatibility of existing ODB files... the database layout has 
changed and I read my old ODB as corrupt. In my test setup I can start from 
scratch but this would be horrible for any existing experiment.

Edit: I noticed 'make testdiff' pipeline is failing... also fails locally... 
investigating

Early performance results:
In early tests, ~700 PCs logging 10 unique arrays of 10 doubles into 
Equipment variables in the ODB seems to perform well... All transactions 
from client PCs are finished within a couple of ms or less

==========================================================================

Questions:

Does the community here have strong opinions about increasing the 
MAX_CLIENTS and MAX_RPC_CONNECTION limits? 
Am I looking at this problem in a naive way?


Potential solutions other than increasing the MAX_CLIENTS limit:
-Make worker threads inside the supervisor (not a separate process), I am 
using TMFE, so I can dynamically create equipment. I have not yet taken a 
deep dive into how any multithreading is implemented
-One could have a round robin system to load balance between a limited pool 
of 'worker frontend' proccesses. I don't like this solution as I want to 
able to clearly see which client PCs have been setup to log too much data

==========================================================================