> > Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.
> 
> Yes, I hadn't realised that was an option. For that to work I guess the aforementioned static members could be made thread local storage, and 
> processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
> class member or something.

If we want to analyze several runs, I can easily add code to make this possible. In a new call to set_odb_source(), the previously allocated memory in that function can be freed. We can aldo make the memory handling 
thread-specific, allowing several thread to analyze different runs at the same time. But I will only invest work there once it's really needed by someone.

Stefan

> ....Before commit of this patch, can you confirm the RunInfo destructor
> deletes this ODB stuff from odbxx? manalyzer takes object life times very seriously.

The call stores the ODB string in static members of the midas::odb class. So these will have a lifetime of the process or until they're replaced by another 
call. When a midas::odb is instantiated it reads from these static members and then that data has the lifetime of that instance.

> Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.

Yes, I hadn't realised that was an option. For that to work I guess the aforementioned static members could be made thread local storage, and 
processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
class member or something.

Note that I missed doing the same for the end of run event, which should probably also be added.

Thanks,

Mark.

> The current develop branch no longer compiles on macOS. I get lots of errors of the form
> /Users/me/midas/src/history_schema.cxx:740:4: error: unknown type name 'off64_t' ...

Confirmed. No idea why off64_t is missing on MacOS. I will try to fix it next week.

K.O.

> Lots of users like the midas::odb interface for reading from the ODB in manalyzers.
> +#include "odbxx.h"

This is a useful improvement. Before commit of this patch, can you confirm the RunInfo destructor
deletes this ODB stuff from odbxx? manalyzer takes object life times very seriously.

There is also the issue that two different RunInfo objects would load two different ODB dumps
into odbxx. (inability to access more than 1 ODB dump is a design feature of odbxx).

This is not an actual problem in manalyzer because it only processes one run at a time
and only 1 or 0 RunInfo objects exists at any given time.

Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.

K.O.

Hi,
Lots of users like the midas::odb interface for reading from the ODB in manalyzers. It currently doesn't 
work offline however without a few manual lines to tell midas::odb to read from the ODB copy in the run 
header. The code also gets a bit messy to work out the current filename and get midas::odb to reopen the 
file currently being processed. This would be much cleaner if manalyzer set this up automatically, and then 
user code could be written that is completely ignorant of whether it is running online or offline.

The change I suggest is in the `set_offline_odb` branch, commit 4ffbda6, which is simply:

diff --git a/manalyzer.cxx b/manalyzer.cxx
index 371f135..725e1d2 100644
--- a/manalyzer.cxx
+++ b/manalyzer.cxx
@@ -15,6 +15,7 @@
 
 #include "manalyzer.h"
 #include "midasio.h"
+#include "odbxx.h"
 
 //////////////////////////////////////////////////////////
 
@@ -2075,6 +2076,8 @@ static int ProcessMidasFiles(const std::vector<std::string>& files, const std::v
                if (!run.fRunInfo) {
                   run.CreateRun(runno, filename.c_str());
                   run.fRunInfo->fOdb = MakeFileDumpOdb(event->GetEventData(), event->data_size);
+                  // Also set the source for midas::odb in case people prefer that interface
+                  midas::odb::set_odb_source(midas::odb::STRING, std::string(event->GetEventData(), event-
>data_size));
                   run.BeginRun();
                }


It happens at the point where the ODB record is already available and requires no effort from the user to 
be able to read the ODB offline.

Thanks,

Mark.

Hi,
The current develop branch no longer compiles on macOS. I get lots of errors of the form

/Users/me/midas/src/history_schema.cxx:740:4: error: unknown type name 'off64_t'; did you mean 'off_t'?
  740 |    off64_t fDataOffset = 0;
      |    ^~~~~~~
      |    off_t
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.5.sd
k/usr/include/sys/_types/_off_t.h:31:33: note: 'off_t' declared here
   31 | typedef __darwin_off_t          off_t;
      |                                 ^

There are also similar errors about lseek64. This appears to have come in with commit 9a6ad2e dated 
23rd July, but I think it was merged into develop with commit 2beeca0 on 3rd of September.

Googling around it seems that off64_t is a GNU extension. I don't know of a cross platform solution but I'm 
happy to test if someone has a suggestion.

Thanks,

Mark.

Dear all,

A reminder we will have our MIDAS workshop starting two weeks from today (Sept 22-23).  The 
meeting will be in the morning in Vancouver, evening in Europe.  A detailed schedule is available 
here

https://indico.psi.ch/event/17580/timetable/#20250922.detailed

The zoom link is available from the indico overview page. The schedule will allow for a fair bit 
of discussion time, so it is unlikely that talks will start exactly on time.

Looking forward to seeing people there.

Thomas

> Dear all,
> 
> We have setup an indico page for the MIDAS workshop on Sept 22-23.  The page is here
> 
> https://indico.psi.ch/event/17580/overview
> 
> As I mentioned, we are keen to hear reports from any users or developers; we want to hear  
> how MIDAS is working for you and what improvements you would like to see.  If you or your 
> experiment would like to give a talk about your MIDAS experiences then please submit an 
> abstract through the indico page.  
> 
> Also, feel free to also register for the workshop (no fees).  Registration is not 
> mandatory, but it would be useful for us to have an idea how many people will connect.
> 
> Thanks,
> Thomas
> 
> 
> > Dear MIDAS enthusiasts,
> > 
> > We are planning a fifth MIDAS workshop, following on from previous successful 
> > workshops in 2015, 2017, 2019 and 2023.  The goals of the workshop include:
> > 
> > - Getting updates from MIDAS developers on new features, bug fixes and planned 
> > changes.
> > - Getting reports from MIDAS users on how they are using MIDAS and what problems 
> > they are facing.
> > - Making plans for future MIDAS changes and improvements
> > 
> > We are planning to have an online workshop on Sept 22-23, 2025 (it will coincide 
> > with a visit of Stefan to TRIUMF).  We are tentatively planning to have a four 
> > hour session on each day, with the sessions timed for morning in Vancouver and 
> > afternoon/evening in Europe.  Sorry, the sessions are likely to again not be well 
> > timed for our colleagues in Asia.  
> > 
> > We will provide exact times and more details closer to the date.  But I hope 
> > people can mark the dates in their calendars; we are keen to hear from as much of 
> > the MIDAS community as possible.  
> > 
> > Best Regards,
> > Thomas Lindner

> > I would like to know how to retrieve the online data during the experiment so 
> > that I can create my own custom plot. I execute my own frontend.exe to start the 
> > experiment. I can get a midas file after the experiment, but I am not sure about 
> > how to retrieve the online data. I know that rootana can help us to get the 
> > online plots, but the instructions in rootana is not clear. Can anyone give me 
> > some suggestion?  Thank you.
> 
> The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
> but it can also be used stand-alone without MIDAS.
> 
> There is several examples:
> 
> manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
> manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
> manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
> manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer
> 
> Documentation is in README.md
> 
> Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
> write a one-size-fits-all recipe.
> 
> Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
> explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
> manalyzer module.
> 
> K.O.

Dear K.O.
I would like to create some online plots (also reading the mid) when the frontend code is running. I may try to break the task into small steps first.
In the frontend.cxx, I have a bank called "TDC0" which includes an array of integer. The code is as follow:

bk_create(pevent, "TDC0", TID_INT, (void**) &pdata);
for(int i =0; i<ch; ++i)
*pdata++ = data[i];
bk_close(pevent,pdata);

In the meantime, I want to run manalyzer to get some plots.

> > I would like to know how to retrieve the online data during the experiment so 
> > that I can create my own custom plot. I execute my own frontend.exe to start the 
> > experiment. I can get a midas file after the experiment, but I am not sure about 
> > how to retrieve the online data. I know that rootana can help us to get the 
> > online plots, but the instructions in rootana is not clear. Can anyone give me 
> > some suggestion?  Thank you.
> 
> The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
> but it can also be used stand-alone without MIDAS.
> 
> There is several examples:
> 
> manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
> manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
> manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
> manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer
> 
> Documentation is in README.md
> 
> Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
> write a one-size-fits-all recipe.
> 
> Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
> explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
> manalyzer module.
> 
> K.O.

> I would like to know how to retrieve the online data during the experiment so 
> that I can create my own custom plot. I execute my own frontend.exe to start the 
> experiment. I can get a midas file after the experiment, but I am not sure about 
> how to retrieve the online data. I know that rootana can help us to get the 
> online plots, but the instructions in rootana is not clear. Can anyone give me 
> some suggestion?  Thank you.

The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
but it can also be used stand-alone without MIDAS.

There is several examples:

manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer

Documentation is in README.md

Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
write a one-size-fits-all recipe.

Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
manalyzer module.

K.O.

Dear experts,
I would like to know how to retrieve the online data during the experiment so 
that I can create my own custom plot. I execute my own frontend.exe to start the 
experiment. I can get a midas file after the experiment, but I am not sure about 
how to retrieve the online data. I know that rootana can help us to get the 
online plots, but the instructions in rootana is not clear. Can anyone give me 
some suggestion?  Thank you.

Best,
Terry

FILE history code (mhf_*.dat files) did not support reading history files bigger than about 2GB, this is now 
fixed on branch "feature/history_off64_t" (in final testing, to be merged ASAP).

History files were never meant to get bigger than about 100 MBytes, but it turns out large files can still 
happen:

1) files are rotated only when history is closed and reopened
2) we removed history close and open on run start
3) so files are rotated only when mlogger is restarted

In the old code, large files would still happen if some equipment writes a lot of data (I have a file from 
Stefan with history record size about 64 kbytes, written at 1/second, MIDAS handles this just fine) or if 
there is no runs started and stopped for a long time.

There are reasons for keeping file size smaller:

a) I would like to use mmap() to read history files, and mmap() of a 100 Gbyte file on a 64 Gbyte RAM 
machine would not work very well.
b) I would like to implement compressed history files and decompression of a 100 Gbyte file takes much 
longer than decompression of a 100 Mbyte file. it is better if data is in smaller chunks.

(it is easy to write a utility program to break-up large history files into smaller chunks).

Why use mmap()? I note that the current code does 1 read() syscall per history record (it is much better to 
read data in bigger chunks) and does multiple seek()/read() syscalls to find the right place in the history 
file (plays silly buggers with the OS read-ahead and data caching). mmap() eliminates all syscalls and has 
the potential to speed things up quite a bit.

K.O.

> for (int i=0; i<array_of_10_elements.size(); i++)

becomes

for (size_t i=0; i<array.size(); i++)

but for a reverse loop, replacing "int" with "size_t" becomes a bug:

for (size_t i=array.size()-1; i>=0; i--)

explodes, last iteration should be with i set to 0, then i--
wraps it around a very big positive value and loop end condition
is still true (i>=0), the loop never ends. (why is there no GCC warning
that with "size_t i", "i>=0" is always true?

a kludge solution is:

for (size_t i=array.size()-1; ; i--) {
do_stuff(i, array[i]);
if (i==0) break;
}

if you do not need the index variable, you can use a reverse iterator (which is missing from a few 
container classes).

K.O.

Over the last 10 years, the traditional C/C++ data types have been
displaced by a hodgepodge of new data types that promise portability
and generate useful (and not so useful) warnings, for example:

for (int i=0; i<array_of_10_elements.size(); i++)

is now a warning with a promise of crash (in theory, even if "int" is 64 bit).

"int" and "long" are dead, welcome "size_t", "off64_t" & co.

What to do, what to do? This is what I figured out:

1) for data returned from hardware: use uint16_t, uint32_t, uint64_t, uint128_t (u16, u32, u64 in 
the Linux kernel), they have well defined width to match hardware (FPGA, AXI, VME, etc) data 
widths.

2) for variables used with strlen(), array.size(), etc: use size_t, a data type wide enough to 
store the biggest data size possible on this hardware (32-bit on 32-bit machines, 64-bit on 64-bit 
machines). use with printf("%zu").

3) for return values of read() and write() syscalls: use ssize_t and observe an inconsistency, 
read() and write() syscalls take size_t (32/64 bits), return ssize_t (31/63 bits) and the error 
check code cannot be written without having to defeat the C/C++ type system (a cast to size_t):

size_t s = 100;
void* ptr = malloc(s);
ssize_t rd = read(fd, ptr, s);
if (rd < 0) { syscall error }
else if ((size_t)rd != s) { short read, important for TCP sockets }
else { good read }

use ssize_t with printf("%zd")

4) file access uses off64_t with lseek64() and ftruncate64(), this is a signed type (to avoid the 
cast in the error handling code) with max file size 2^63 (at $/GB, storage for a file of max size 
costs $$$$$, you cannot have enough money to afford one). use with printf("%jd", (intmax_t)v). 
intmax_t by definition is big enough for all off64_t values, "%jd" is the corresponding printf() 
format.

5) there is no inconsistency between 32-bit size_t and 64-bit off64_t, on 32-bit systems you can 
only read files in small chunks, but you can lseek64() to any place in the file.

BTW, 64-bit time_t has arrived with Ubuntu LTS 24.04, I will write about this some other time.

> Perhaps have:
> 
> set_write_cache_size("SYSTEM", 0);
> set_write_cache_size("BUF1", bigsize);
> 
> with an internal std::map<std::string,size_t>; for write cache size for each named buffer

Ok, this is implemented now in mfed.cxx and called from examples/experiment/frontend.cxx

Stefan

Something changed in our system and we started seeing memory leaks in mhttpd again.  I guess someone 
updated some front end or custom page code that interacted with mhttpd differently.
I found a few memory leaks in some (presumably) rarely seen corner cases and we now see steady 
memory usage.  The branch is fix/memory_leaks 
(https://bitbucket.org/tmidas/midas/branch/fix/memory_leaks) and I opened pull request #55 
(https://bitbucket.org/tmidas/midas/pull-requests/55).  I couldn't find a BitBucket account for you 
Konstantin to add as a reviewer, so it currently has none.

Thanks,

Mark.

Since this memory leak is quite obvious, I pushed the fix to develop.

Stefan

I added now code to the logger so it properly complains if there would be a leading space in a variable name.

Stefan

> By accident we had history variables with leading spaces. The history schema check then decides that this is a new variable (the leading space is not read from the history file) and starts a new file. We found this because the run start became slow due to the many, many history files created. It would be nice to just get an error if one has a malformed variable name like this.
> 
> How to reproduce: Try to put a variable with a leading space in the name into the history, repeatedly start runs.
> Sugested fix: Produce an error if a history variable has a leading space.

I have encounter this a few times
* Make a new history panel
* Use the web GUI to add history variables
* When I am at the "add history variables" panel, there is not scroll option. So 
depending on the size and zoom of my screen, some variables further down the list 
can not be selected

tried Chrome and Firefox