> > I will work today on the odbxx API to make sure there are no memory leaks when you switch form one file to another. I talked to KO so he agreed that yo then commit your proposed change of manalyzer 
> 
> That, and add a "clear()" method that resets odbxx state to "empty". I will call odbxx.clear() everywhere where I call "delete fOdb;" (TARunInfo::dtor and other places).

No need for clear(), since no memory gets allocated by midas::odd::set_odb_source(). All it does is to remember the file name. When you instantiate a midas::odd object, the file gets loaded, and the midas::odd object gets initialized from the file contents. Then the buffer 
gets deleted (actually it's a simple local variable). Of course this causes some overhead (each midas::odd() constructor reads the whole file), but since the OS will cache the file, it's probably not so bad.

Stefan

As part of discussions with Stefan during the MIDAS workshop,
an update to supported versions of c++ on different platforms.

- el6/SL6 is gone, gcc 4.4.7, no c++11
- el7/CentOS-7 is out the door, gcc 4.8.5, full c++11, no c++14, no c++17
- el8 was a still born child of RedHat
- el9  - gcc 11.5 with 12, 13, and 14 available, incomplete c++26, c++23, experimental c++20, default c++17
- el10 - gcc 14.2, incomplete c++26, c++23, experimental c++20, default c++17

- U-18 - gcc  7.5, experimental, incomplete c++17
- U-20 - gcc  9.4 default, 10.5 available, experimental, incomplete c++17
- U-22 - gcc 11.4 default, 12.3 available, default c++17
- U-24 - gcc 13.3 default, 14.2 available, default c++17

- D-11 - gcc 10.2, experimental c++17
- D-12 - gcc 12.2, default c++17

- MacOS 15.2 - llvm/clang 16, default c++17
- MacOS 15.7 - llvm/clang 17, default c++17

Taken from C++ level support:

GCC C++ support: https://gcc.gnu.org/projects/cxx-status.html
LLVM clang c++ support: https://clang.llvm.org/cxx_status.html
GLIBC c++ support: https://gcc.gnu.org/onlinedocs/libstdc++/manual/status.html

This suggests that c++17 is available on all current platforms.

K.O.

P.S. ROOT moved to C++17 as of release 6.30 on November 6, 2023.

> > ....Before commit of this patch, can you confirm the RunInfo destructor
> > deletes this ODB stuff from odbxx? manalyzer takes object life times very seriously.
> 
> The call stores the ODB string in static members of the midas::odb class. So these will have a lifetime of the process or until they're replaced by another 
> call. When a midas::odb is instantiated it reads from these static members and then that data has the lifetime of that instance.

this is the behavious we need to modify.

> > Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.
> Yes, I hadn't realised that was an option.

It is an option I would like to keep open. Not too many use cases, but imagine a "split brain" experiment
that has two MIDAS instances record data into two separate midas files. (if LIGO were to use MIDAS,
consider LIGO Hanford and LIGO Livingston).

Assuming data in these two data sets have common precision timestamps,
our task is to assemble data from two input files into single physics events. The analyzer will need
to read two input files, each file with it's run number, it's own ODB dump, etc, process the midas
events (unpack, calibrate, filter, etc), look at the timestamps, assemble the data into physics events.

This trivially generalizes into reading 2, 3, or more input files.

> For that to work I guess the aforementioned static members could be made thread local storage, and 
> processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
> class member or something.

manalyzer is already multithreaded, if you will need to keep track of which thread should see which odbxx global object,
seems like abuse of the thread-local storage idea and intent.

> Note that I missed doing the same for the end of run event, which should probably also be added.

Ideally, the memory sanitizer will flag this for us, complain about anything that odbxx.clear() failes to free.

K.O.

> I will work today on the odbxx API to make sure there are no memory leaks when you switch form one file to another. I talked to KO so he agreed that yo then commit your proposed change of manalyzer 

That, and add a "clear()" method that resets odbxx state to "empty". I will call odbxx.clear() everywhere where I call "delete fOdb;" (TARunInfo::dtor and other places).

K.O.

I will work today on the odbxx API to make sure there are no memory leaks when you switch form one file to another. I talked to KO so he agreed that yo then commit your proposed change of manalyzer 

Best,
Stefan

Dear all,

The original zoom link on the indico page was not correct.  We have created a new zoom link and updated the 
Indico overview page.  Please refresh the Indico overview page to get the correct link.

Thomas

> Dear all,
> 
> A final reminder that the MIDAS workshop will be occurring next Monday and Tuesday (at least in Europe 
> and the Americas).  See the full schedule is posted here
> 
> https://indico.psi.ch/event/17580/timetable/#20250922.detailed
> 
> The zoom link is on the indico overview page.
> 
> Note that for those wanting to attend the workshop in person we will be meeting in ISAC2 room 223 at 
> TRIUMF.
> 
> See you there.
> Thomas
> 
> > Dear all,
> > 
> > A reminder we will have our MIDAS workshop starting two weeks from today (Sept 22-23).  The 
> > meeting will be in the morning in Vancouver, evening in Europe.  A detailed schedule is available 
> > here
> > 
> > https://indico.psi.ch/event/17580/timetable/#20250922.detailed
> > 
> > The zoom link is available from the indico overview page. The schedule will allow for a fair bit 
> > of discussion time, so it is unlikely that talks will start exactly on time.
> > 
> > Looking forward to seeing people there.
> > 
> > Thomas
> > 
> > > Dear all,
> > > 
> > > We have setup an indico page for the MIDAS workshop on Sept 22-23.  The page is here
> > > 
> > > https://indico.psi.ch/event/17580/overview
> > > 
> > > As I mentioned, we are keen to hear reports from any users or developers; we want to hear  
> > > how MIDAS is working for you and what improvements you would like to see.  If you or your 
> > > experiment would like to give a talk about your MIDAS experiences then please submit an 
> > > abstract through the indico page.  
> > > 
> > > Also, feel free to also register for the workshop (no fees).  Registration is not 
> > > mandatory, but it would be useful for us to have an idea how many people will connect.
> > > 
> > > Thanks,
> > > Thomas
> > > 
> > > 
> > > > Dear MIDAS enthusiasts,
> > > > 
> > > > We are planning a fifth MIDAS workshop, following on from previous successful 
> > > > workshops in 2015, 2017, 2019 and 2023.  The goals of the workshop include:
> > > > 
> > > > - Getting updates from MIDAS developers on new features, bug fixes and planned 
> > > > changes.
> > > > - Getting reports from MIDAS users on how they are using MIDAS and what problems 
> > > > they are facing.
> > > > - Making plans for future MIDAS changes and improvements
> > > > 
> > > > We are planning to have an online workshop on Sept 22-23, 2025 (it will coincide 
> > > > with a visit of Stefan to TRIUMF).  We are tentatively planning to have a four 
> > > > hour session on each day, with the sessions timed for morning in Vancouver and 
> > > > afternoon/evening in Europe.  Sorry, the sessions are likely to again not be well 
> > > > timed for our colleagues in Asia.  
> > > > 
> > > > We will provide exact times and more details closer to the date.  But I hope 
> > > > people can mark the dates in their calendars; we are keen to hear from as much of 
> > > > the MIDAS community as possible.  
> > > > 
> > > > Best Regards,
> > > > Thomas Lindner

Dear all,

A final reminder that the MIDAS workshop will be occurring next Monday and Tuesday (at least in Europe 
and the Americas).  See the full schedule is posted here

https://indico.psi.ch/event/17580/timetable/#20250922.detailed

The zoom link is on the indico overview page.

Note that for those wanting to attend the workshop in person we will be meeting in ISAC2 room 223 at 
TRIUMF.

See you there.
Thomas

> Dear all,
> 
> A reminder we will have our MIDAS workshop starting two weeks from today (Sept 22-23).  The 
> meeting will be in the morning in Vancouver, evening in Europe.  A detailed schedule is available 
> here
> 
> https://indico.psi.ch/event/17580/timetable/#20250922.detailed
> 
> The zoom link is available from the indico overview page. The schedule will allow for a fair bit 
> of discussion time, so it is unlikely that talks will start exactly on time.
> 
> Looking forward to seeing people there.
> 
> Thomas
> 
> > Dear all,
> > 
> > We have setup an indico page for the MIDAS workshop on Sept 22-23.  The page is here
> > 
> > https://indico.psi.ch/event/17580/overview
> > 
> > As I mentioned, we are keen to hear reports from any users or developers; we want to hear  
> > how MIDAS is working for you and what improvements you would like to see.  If you or your 
> > experiment would like to give a talk about your MIDAS experiences then please submit an 
> > abstract through the indico page.  
> > 
> > Also, feel free to also register for the workshop (no fees).  Registration is not 
> > mandatory, but it would be useful for us to have an idea how many people will connect.
> > 
> > Thanks,
> > Thomas
> > 
> > 
> > > Dear MIDAS enthusiasts,
> > > 
> > > We are planning a fifth MIDAS workshop, following on from previous successful 
> > > workshops in 2015, 2017, 2019 and 2023.  The goals of the workshop include:
> > > 
> > > - Getting updates from MIDAS developers on new features, bug fixes and planned 
> > > changes.
> > > - Getting reports from MIDAS users on how they are using MIDAS and what problems 
> > > they are facing.
> > > - Making plans for future MIDAS changes and improvements
> > > 
> > > We are planning to have an online workshop on Sept 22-23, 2025 (it will coincide 
> > > with a visit of Stefan to TRIUMF).  We are tentatively planning to have a four 
> > > hour session on each day, with the sessions timed for morning in Vancouver and 
> > > afternoon/evening in Europe.  Sorry, the sessions are likely to again not be well 
> > > timed for our colleagues in Asia.  
> > > 
> > > We will provide exact times and more details closer to the date.  But I hope 
> > > people can mark the dates in their calendars; we are keen to hear from as much of 
> > > the MIDAS community as possible.  
> > > 
> > > Best Regards,
> > > Thomas Lindner

> > Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.
> 
> Yes, I hadn't realised that was an option. For that to work I guess the aforementioned static members could be made thread local storage, and 
> processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
> class member or something.

If we want to analyze several runs, I can easily add code to make this possible. In a new call to set_odb_source(), the previously allocated memory in that function can be freed. We can aldo make the memory handling 
thread-specific, allowing several thread to analyze different runs at the same time. But I will only invest work there once it's really needed by someone.

Stefan

> ....Before commit of this patch, can you confirm the RunInfo destructor
> deletes this ODB stuff from odbxx? manalyzer takes object life times very seriously.

The call stores the ODB string in static members of the midas::odb class. So these will have a lifetime of the process or until they're replaced by another 
call. When a midas::odb is instantiated it reads from these static members and then that data has the lifetime of that instance.

> Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.

Yes, I hadn't realised that was an option. For that to work I guess the aforementioned static members could be made thread local storage, and 
processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
class member or something.

Note that I missed doing the same for the end of run event, which should probably also be added.

Thanks,

Mark.

> The current develop branch no longer compiles on macOS. I get lots of errors of the form
> /Users/me/midas/src/history_schema.cxx:740:4: error: unknown type name 'off64_t' ...

Confirmed. No idea why off64_t is missing on MacOS. I will try to fix it next week.

K.O.

> Lots of users like the midas::odb interface for reading from the ODB in manalyzers.
> +#include "odbxx.h"

This is a useful improvement. Before commit of this patch, can you confirm the RunInfo destructor
deletes this ODB stuff from odbxx? manalyzer takes object life times very seriously.

There is also the issue that two different RunInfo objects would load two different ODB dumps
into odbxx. (inability to access more than 1 ODB dump is a design feature of odbxx).

This is not an actual problem in manalyzer because it only processes one run at a time
and only 1 or 0 RunInfo objects exists at any given time.

Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.

K.O.

Hi,
Lots of users like the midas::odb interface for reading from the ODB in manalyzers. It currently doesn't 
work offline however without a few manual lines to tell midas::odb to read from the ODB copy in the run 
header. The code also gets a bit messy to work out the current filename and get midas::odb to reopen the 
file currently being processed. This would be much cleaner if manalyzer set this up automatically, and then 
user code could be written that is completely ignorant of whether it is running online or offline.

The change I suggest is in the `set_offline_odb` branch, commit 4ffbda6, which is simply:

diff --git a/manalyzer.cxx b/manalyzer.cxx
index 371f135..725e1d2 100644
--- a/manalyzer.cxx
+++ b/manalyzer.cxx
@@ -15,6 +15,7 @@
 
 #include "manalyzer.h"
 #include "midasio.h"
+#include "odbxx.h"
 
 //////////////////////////////////////////////////////////
 
@@ -2075,6 +2076,8 @@ static int ProcessMidasFiles(const std::vector<std::string>& files, const std::v
                if (!run.fRunInfo) {
                   run.CreateRun(runno, filename.c_str());
                   run.fRunInfo->fOdb = MakeFileDumpOdb(event->GetEventData(), event->data_size);
+                  // Also set the source for midas::odb in case people prefer that interface
+                  midas::odb::set_odb_source(midas::odb::STRING, std::string(event->GetEventData(), event-
>data_size));
                   run.BeginRun();
                }


It happens at the point where the ODB record is already available and requires no effort from the user to 
be able to read the ODB offline.

Thanks,

Mark.

Hi,
The current develop branch no longer compiles on macOS. I get lots of errors of the form

/Users/me/midas/src/history_schema.cxx:740:4: error: unknown type name 'off64_t'; did you mean 'off_t'?
  740 |    off64_t fDataOffset = 0;
      |    ^~~~~~~
      |    off_t
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.5.sd
k/usr/include/sys/_types/_off_t.h:31:33: note: 'off_t' declared here
   31 | typedef __darwin_off_t          off_t;
      |                                 ^

There are also similar errors about lseek64. This appears to have come in with commit 9a6ad2e dated 
23rd July, but I think it was merged into develop with commit 2beeca0 on 3rd of September.

Googling around it seems that off64_t is a GNU extension. I don't know of a cross platform solution but I'm 
happy to test if someone has a suggestion.

Thanks,

Mark.

Dear all,

A reminder we will have our MIDAS workshop starting two weeks from today (Sept 22-23).  The 
meeting will be in the morning in Vancouver, evening in Europe.  A detailed schedule is available 
here

https://indico.psi.ch/event/17580/timetable/#20250922.detailed

The zoom link is available from the indico overview page. The schedule will allow for a fair bit 
of discussion time, so it is unlikely that talks will start exactly on time.

Looking forward to seeing people there.

Thomas

> Dear all,
> 
> We have setup an indico page for the MIDAS workshop on Sept 22-23.  The page is here
> 
> https://indico.psi.ch/event/17580/overview
> 
> As I mentioned, we are keen to hear reports from any users or developers; we want to hear  
> how MIDAS is working for you and what improvements you would like to see.  If you or your 
> experiment would like to give a talk about your MIDAS experiences then please submit an 
> abstract through the indico page.  
> 
> Also, feel free to also register for the workshop (no fees).  Registration is not 
> mandatory, but it would be useful for us to have an idea how many people will connect.
> 
> Thanks,
> Thomas
> 
> 
> > Dear MIDAS enthusiasts,
> > 
> > We are planning a fifth MIDAS workshop, following on from previous successful 
> > workshops in 2015, 2017, 2019 and 2023.  The goals of the workshop include:
> > 
> > - Getting updates from MIDAS developers on new features, bug fixes and planned 
> > changes.
> > - Getting reports from MIDAS users on how they are using MIDAS and what problems 
> > they are facing.
> > - Making plans for future MIDAS changes and improvements
> > 
> > We are planning to have an online workshop on Sept 22-23, 2025 (it will coincide 
> > with a visit of Stefan to TRIUMF).  We are tentatively planning to have a four 
> > hour session on each day, with the sessions timed for morning in Vancouver and 
> > afternoon/evening in Europe.  Sorry, the sessions are likely to again not be well 
> > timed for our colleagues in Asia.  
> > 
> > We will provide exact times and more details closer to the date.  But I hope 
> > people can mark the dates in their calendars; we are keen to hear from as much of 
> > the MIDAS community as possible.  
> > 
> > Best Regards,
> > Thomas Lindner

> > I would like to know how to retrieve the online data during the experiment so 
> > that I can create my own custom plot. I execute my own frontend.exe to start the 
> > experiment. I can get a midas file after the experiment, but I am not sure about 
> > how to retrieve the online data. I know that rootana can help us to get the 
> > online plots, but the instructions in rootana is not clear. Can anyone give me 
> > some suggestion?  Thank you.
> 
> The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
> but it can also be used stand-alone without MIDAS.
> 
> There is several examples:
> 
> manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
> manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
> manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
> manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer
> 
> Documentation is in README.md
> 
> Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
> write a one-size-fits-all recipe.
> 
> Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
> explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
> manalyzer module.
> 
> K.O.

Dear K.O.
I would like to create some online plots (also reading the mid) when the frontend code is running. I may try to break the task into small steps first.
In the frontend.cxx, I have a bank called "TDC0" which includes an array of integer. The code is as follow:

bk_create(pevent, "TDC0", TID_INT, (void**) &pdata);
for(int i =0; i<ch; ++i)
*pdata++ = data[i];
bk_close(pevent,pdata);

In the meantime, I want to run manalyzer to get some plots.

> > I would like to know how to retrieve the online data during the experiment so 
> > that I can create my own custom plot. I execute my own frontend.exe to start the 
> > experiment. I can get a midas file after the experiment, but I am not sure about 
> > how to retrieve the online data. I know that rootana can help us to get the 
> > online plots, but the instructions in rootana is not clear. Can anyone give me 
> > some suggestion?  Thank you.
> 
> The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
> but it can also be used stand-alone without MIDAS.
> 
> There is several examples:
> 
> manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
> manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
> manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
> manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer
> 
> Documentation is in README.md
> 
> Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
> write a one-size-fits-all recipe.
> 
> Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
> explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
> manalyzer module.
> 
> K.O.

> I would like to know how to retrieve the online data during the experiment so 
> that I can create my own custom plot. I execute my own frontend.exe to start the 
> experiment. I can get a midas file after the experiment, but I am not sure about 
> how to retrieve the online data. I know that rootana can help us to get the 
> online plots, but the instructions in rootana is not clear. Can anyone give me 
> some suggestion?  Thank you.

The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
but it can also be used stand-alone without MIDAS.

There is several examples:

manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer

Documentation is in README.md

Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
write a one-size-fits-all recipe.

Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
manalyzer module.

K.O.

Dear experts,
I would like to know how to retrieve the online data during the experiment so 
that I can create my own custom plot. I execute my own frontend.exe to start the 
experiment. I can get a midas file after the experiment, but I am not sure about 
how to retrieve the online data. I know that rootana can help us to get the 
online plots, but the instructions in rootana is not clear. Can anyone give me 
some suggestion?  Thank you.

Best,
Terry

FILE history code (mhf_*.dat files) did not support reading history files bigger than about 2GB, this is now 
fixed on branch "feature/history_off64_t" (in final testing, to be merged ASAP).

History files were never meant to get bigger than about 100 MBytes, but it turns out large files can still 
happen:

1) files are rotated only when history is closed and reopened
2) we removed history close and open on run start
3) so files are rotated only when mlogger is restarted

In the old code, large files would still happen if some equipment writes a lot of data (I have a file from 
Stefan with history record size about 64 kbytes, written at 1/second, MIDAS handles this just fine) or if 
there is no runs started and stopped for a long time.

There are reasons for keeping file size smaller:

a) I would like to use mmap() to read history files, and mmap() of a 100 Gbyte file on a 64 Gbyte RAM 
machine would not work very well.
b) I would like to implement compressed history files and decompression of a 100 Gbyte file takes much 
longer than decompression of a 100 Mbyte file. it is better if data is in smaller chunks.

(it is easy to write a utility program to break-up large history files into smaller chunks).

Why use mmap()? I note that the current code does 1 read() syscall per history record (it is much better to 
read data in bigger chunks) and does multiple seek()/read() syscalls to find the right place in the history 
file (plays silly buggers with the OS read-ahead and data caching). mmap() eliminates all syscalls and has 
the potential to speed things up quite a bit.

K.O.

> for (int i=0; i<array_of_10_elements.size(); i++)

becomes

for (size_t i=0; i<array.size(); i++)

but for a reverse loop, replacing "int" with "size_t" becomes a bug:

for (size_t i=array.size()-1; i>=0; i--)

explodes, last iteration should be with i set to 0, then i--
wraps it around a very big positive value and loop end condition
is still true (i>=0), the loop never ends. (why is there no GCC warning
that with "size_t i", "i>=0" is always true?

a kludge solution is:

for (size_t i=array.size()-1; ; i--) {
do_stuff(i, array[i]);
if (i==0) break;
}

if you do not need the index variable, you can use a reverse iterator (which is missing from a few 
container classes).

K.O.