Following discussions at the MIDAS workshop, we propose to move MIDAS from c++11 to c++17. There is 
many new features and we want to start using some of them.

Per my previous message https://daq00.triumf.ca/elog-midas/Midas/3084,
c++17 is available on current MacOS, U-22 and newer, el9 and newer, D-12 and newer.

(ROOT moved to C++17 as of release 6.30 on November 6, 2023)

As I reported earlier, MIDAS already builds with c++23 on U-24, and this move does not require any 
actual code changes other than a bump of c++ version in CMakeLists.txt and Makefile.

Please let us know if this change will cause problems or if you think that we should move to an older 
c++ (c++14) or newer c++ (c++20 or c++23 or c++26).

If we do not hear anything, we will implement this change in about 2-3 weeks.

K.O.

Historically, building MIDAS with ROOT caused us many problems - build failures because of c++ version 
mismatch, CFLAGS mismatch; run-time failures because of ROOT library mismatches; etc, etc.

Following discussions at the MIDAS Workshop, we think we should finally bite the bullet and remove ROOT 
support from MIDAS:

- remove support for writing data in ROOT TTree format in mlogger (rmlogger)
- remove support for ROOT in mana.c based analyzer (which itself we propose to remove)
- remove ROOT helper functions in rmidas.h
- remove ROOT support in cmake
- remove rmlogger and rmana

This change will not affect the rootana and manalyzer packages, they will continue to be built with 
ROOT support if ROOT is available.

Right now, we cannot remember any experiment that uses the ROOT TTree output function in mlogger. 

If you use this feature, we very much would like to hear from you. Please contact Stefan or myself or 
reply to this message.

As replacement for rmlogger, we could implement identical or similar functionality using an manalyzer-
based custom logger, but we need at least one user who can test it for us and confirm that it works 
correctly.

K.O.

Following discussions at the MIDAS workshop and the proposed removal of support for ROOT, the very obsolete mana.c 
analyzer framework has reached the end of the line.

Right now we cannot remember any experiment that uses a mana.c based analyzer. Most experiments use analyzers based 
on the rootana package (developed for ALPHA-1 at CERN) and on the manalyzer package (developed for ALPHA-2 and ALPHA-
g at CERN, with multithreading support contributed by Joseph McKenna).

If you know of any experiment that uses a mana.c based analyzer, please let us know. We can help with building it 
using an outside-of-midas local copy of mana.c or help with migration to a newer framework (or migration to a 
framework-free standalone analyzer).

If we do not hear from anybody, we will remove mana.c (and rmana) at the same time as we remove ROOT support from 
mlogger (rmlogger).

K.O.

Hi,
at the LEM Experiment at PSI, we still use mana.c and would like to keep it until end of 2026, where we will enter a long shutdown.
There we will switch to the manalyzer. Before that, there is simply no time for the change over. One thing I already noticed is the "lack" of documentation, since 
for a lot of items I found simply "TBW". I know that writing documentation is boring and hard, but I hope at the time we will switch there is a more complete 
documentation available.

Thanks a lot for the ongoing development and support

Andreas

> Following discussions at the MIDAS workshop and the proposed removal of support for ROOT, the very obsolete mana.c 
> analyzer framework has reached the end of the line.
> 
> Right now we cannot remember any experiment that uses a mana.c based analyzer. Most experiments use analyzers based 
> on the rootana package (developed for ALPHA-1 at CERN) and on the manalyzer package (developed for ALPHA-2 and ALPHA-
> g at CERN, with multithreading support contributed by Joseph McKenna).
> 
> If you know of any experiment that uses a mana.c based analyzer, please let us know. We can help with building it 
> using an outside-of-midas local copy of mana.c or help with migration to a newer framework (or migration to a 
> framework-free standalone analyzer).
> 
> If we do not hear from anybody, we will remove mana.c (and rmana) at the same time as we remove ROOT support from 
> mlogger (rmlogger).
> 
> K.O.

> Hi, at the LEM Experiment at PSI, we still use mana.c and would like to keep it until end of 2026, where we will enter a long shutdown.

Excellent, good to hear from you! Once we remove ROOT support rmana.o will be gone, only mana.o (no ROOT) will remain. Will this break your builds?

One solution could be to copy mana.c from MIDAS into your source tree and compile/link it from there (not from MIDAS).

Perhaps the way to proceed is create a test branch with ROOT and mana.c removed, you can try it,
report success/fail and we go from there.

We should schedule this work for when both of us have a block of free time to work on it.

K.O.

To record discussion with Stefan regarding 64-bit time_t

To remember:

signed   32-bit time_t will overflow in 2038 (soon enough)
unsigned 32-bit time_t will overflow in 2106 ("not my problem")

https://en.wikipedia.org/wiki/Year_2038_problem
https://wiki.debian.org/ReleaseGoals/64bit-time

64-bit Linux uses 64-bit time_t since as far back as el6.

MIDAS uses unsigned 32-bit (DWORD) time-in-seconds in many places (ODB, event 
headers, event buffers, etc)

MIDAS also uses unsigned 32-bit (DWORD) time-in-milliseconds in many places.

All time arithmetic is done using unsigned 32-bit math, these time calculations
are good until year 2106, at which time they will wrap around
(but still work correctly).

So we do not need to do anything, but...

To reduce confusion between the different time types, we will probably
introduce a 32-bit-time-in-seconds data type (i.e. time32_t alias for uint32_t),
and a 32-bit-time-in-milliseconds data type (i.e. millitime32_t alias for 
uint32_t). Also rename ss_time() to ss_time32() and ss_millitime() to 
ss_millitime32().

This should help avoiding accidental mixing of MIDAS 32-bit time, system 64-bit 
time and MIDAS 32-bit-time-in-milliseconds.

(confusion between time-in-seconds and time-in-milliseconds happened several 
times in MIDAS code).

There will be additional discussion and announcements if we go ahead with these 
changes.

K.O.

perhaps c++20? - std::format is an immediately useful feature. --regards, Pasha

> Following discussions at the MIDAS workshop, we propose to move MIDAS from c++11 to c++17. There is 
> many new features and we want to start using some of them.
> 
> Per my previous message https://daq00.triumf.ca/elog-midas/Midas/3084,
> c++17 is available on current MacOS, U-22 and newer, el9 and newer, D-12 and newer.
> 
> (ROOT moved to C++17 as of release 6.30 on November 6, 2023)
> 
> As I reported earlier, MIDAS already builds with c++23 on U-24, and this move does not require any 
> actual code changes other than a bump of c++ version in CMakeLists.txt and Makefile.
> 
> Please let us know if this change will cause problems or if you think that we should move to an older 
> c++ (c++14) or newer c++ (c++20 or c++23 or c++26).
> 
> If we do not hear anything, we will implement this change in about 2-3 weeks.
> 
> K.O.

To record discussion with Stefan about long history variable names.

We have several requests to remove the 32-byte limit on history variable names.

Presently, history variable names are formed from two 32-byte strings: history event name and 
history tag name:

* history event name is usually same as the equipment name (also a 32-byte string)
* history tag name is composed from /eq/xxx/variables/yyy name (also a 32-byte string) or from 
names in /eq/xxx/variables/names and "names zzz", which can have arbitrary length (and tag name 
would have to be truncated).

This worked well for "per-equipment" history, history events corresponded to equipment/variables 
and all data from equipment/variables were written together in one go. (this very inefficient if 
values of only one variable is updated).

Then at some point we implemented "per-variable" history:

* history event name is a equipment name (32-byte string) plus /eq/xxx/variables/vvv variable name 
(also 32-byte string). (obviously truncation is quite possible)
* history tag name is unchanged (also can be truncated)

With "per-variable" history, history events correspond to individual variables (ODB entries) in 
/eq/xxx/variables. If value of one variable is updated, only that variable is written to ODB. This 
is much more efficient. (If variable is an array, the whole array is written, is variable is a 
subdirectory, the whole subdirectory is written).

We considered even finer granularity, writing to history file only the one value that changed, but 
decided against slicing the data too fine. (for arrays, MIDAS frontends usually update all values 
of an array, as in "array of 10 temperatures" or "array of 4000 high voltages").

Many years later, we have the SQL history and the FILE history which do not have the 32-byte limit 
on history event names and history tag names (no limit in the MIDAS C++ code. MySQL and PgSQL have 
limits on table name and column name lengths, 64 bytes and 31 bytes respectively, best I can 
tell).

But the API still uses the MIDAS "struct TAG" with the 32-byte tag name length limit.

It is pretty easy to change the API to use a new "struct TAG_CXX" with std::string unlimited-
length tag names, but the old MIDAS history will be unable to deal with long names. Hence
the discussion about removing the old MIDAS history and keeping only the FILE and SQL history 
(plus the mhdump and mh2sql tools to convert old history files to the new formats).

(some code in mhttpd may need to be corrected for long history names. javascript code should be 
okey, history plot code may need adjustment to display pathologically long names. use small font, 
truncate, etc).

K.O.

> perhaps c++20? - std::format is an immediately useful feature. --regards, Pasha

confirmed. std::format is an improvement over K&R C printf().

but seems unavailable on U-20 and older, requires --std=c++20 on U-24 and MacOS.

but also available as a standalone library: https://github.com/fmtlib/fmt

myself, I use printf() and msprintf(), I think std::format is in the "too little, too late to save C++" 
department.

K.O.

> > perhaps c++20? - std::format is an immediately useful feature. --regards, Pasha
> 
> confirmed. std::format is an improvement over K&R C printf().
> 
> but seems unavailable on U-20 and older, requires --std=c++20 on U-24 and MacOS.

agreed! - availability is significantly more important. -- regards, Pasha

Sorry, 

I have had now the time to dig deeper in our code and realized that we actually use rmana, i.e. WITH ROOT. If there is an easy way to incorporate the necessary parts temporarily to our side, we will do it. Without ROOT this would have been quite easy, with ROOT, I am not that sure. Anyhow, as said the timeline for this is only until end of 2026.

Andreas

> > Hi, at the LEM Experiment at PSI, we still use mana.c and would like to keep it until end of 2026, where we will enter a long shutdown.
> 
> Excellent, good to hear from you! Once we remove ROOT support rmana.o will be gone, only mana.o (no ROOT) will remain. Will this break your builds?
> 
> One solution could be to copy mana.c from MIDAS into your source tree and compile/link it from there (not from MIDAS).
> 
> Perhaps the way to proceed is create a test branch with ROOT and mana.c removed, you can try it,
> report success/fail and we go from there.
> 
> We should schedule this work for when both of us have a block of free time to work on it.
> 
> K.O.

Hi,
 has anyone tried runnning midas frontend on a Linux running 
on a Motorola MVME167 motorola embedded CPU?
I have seen people running Linux on a MV167 
(http://www.sleepie.demon.co.uk/linuxvme/)
so in principle this can be done.

The reason I am asking is that we have a lot of them in house 
and we would like to avoid paying for VxWorks
(I have succesfully run Midas on a mvme167/VxWorks node)

Or maybe one has come up with a much better solution 
[short of dumping mv167 into a sewer :)]

Piotr

I would like to know whether all parameters used by the frontend code have to be in the "Experiment/
Run Parameters" section.  This section can become big and difficult to maintain, because it is one single 
big section of experim.h (EXP_PARAM_DEFINED).  I have parameters the various frontends read at the 
beginning of each run, which set the hardware settings of various devices.  I would like to place these in 
a section all their own, organized by device.  Is this doable? 

Hi Jan,

I usually keep under /Experiment/Run Parameters only those settings which are kind of "global" and thus of
interest to frontend *and* analyzer, like a run mode (data/calibration/cosmic/...). Settings more specific to a
frontend I keep under /Equipment/<name>/Settings where <name> is the equipment name the specific frontend
produces. In your case each frontend will then get its own tree (related to each fragment). Please note that
both discussed trees can contain a whole tree with subdirectories, which lets you organize your data better.

Best regards, Stefan.

   We've had mhttpd aborting regularly since upgrading from midas-1.9.3.  This
happens during elog queries, and is due to an elog file that was incorrectly
modified by hand.  The modification to the file occurred 6 months ago.
   el_retrieve(midas.c:15683) now has several assert statements, one of which
aborts the program on reading the bad entry.

   Why is assert used, instead of an error return from the function (if
necessary), and maybe an error message in the log file?  Assert statements are
often removed, using NDEBUG, for normal use.

Chris

   The problem elog entry had one character removed, so end-of-file came before
the end of the message.  This could probably occur without the file being
altered, if the disk containing the elog fills.

>    We've had mhttpd aborting regularly since upgrading from midas-1.9.3.  This
> happens during elog queries, and is due to an elog file that was incorrectly
> modified by hand.

(sorry for delayed reply, for reasons unknown, I did not get an email notice when this was posted)

Yes, I agree, error handling in midas elog code is insufficient (note missing error checks for
read() and lseek() system calls). Anything but "perfect" elog files would cause funny errors and
malfunctions.

>  The modification to the file occurred 6 months ago.
>    el_retrieve(midas.c:15683) now has several assert statements, one of which
> aborts the program on reading the bad entry.

I added those to fix problems with "broken last NN days" and with infinite looping in the elog code
that we observed in TWIST.

You are welcome to replace the assert() statements with proper error handling. I used to have some code
that could report the filename of the bad elog file. Can we also report the exact file location for broken
files.

Please send me the diff, I will commit it to midas cvs.

>    Why is assert used, instead of an error return from the function (if
> necessary), and maybe an error message in the log file?  Assert statements are
> often removed, using NDEBUG, for normal use.

I use assert() in several ways:

0) I want a core dump each time X happens. (This is the only reasonable action when facing memory/stack
corruption. The problems in the elog code were stack corruption).
1) "I am too lazy to write proper error handling code" so I just crash and burn. This includes the
case where "proper error handling" would be "too invasive".
2) the error is too bad (or too deep) and there is no reasonable way to recover. Print an error message
and dump core (for later analysis). I sometimes use "cm_msg(); abort()". (assert is "printf("error"); abort()")

Please refer to literature for philosophic discussions on uses of assert() (Argh! Stefan will have my
head again!), but I will mention that "abort() early, abort() often" I find very effective. BTW, this technique
is heavily used in the Linux kernel (oops(), bug(), panic()) with some good effect, too.

>    The problem elog entry had one character removed, so end-of-file came before
> the end of the message.  This could probably occur without the file being
> altered, if the disk containing the elog fills.

Yes, I think you are right. In TWIST, we have seen disk-full conditions break both elog and history.

K.O.

What is the api call to determine the index of the frontend when specifying the
-i parameter during execution of the frontend? 

> What is the api call to determine the index of the frontend when specifying the
> -i parameter during execution of the frontend? 

INT get_frontend_index();

- Stefan

When i compile the experiment example of 1.9.5 the problem happened:

adccalib.c: In function `INT adc_calib_init()':
adccalib.c:114: `H1_BOOK' undeclared (first use this function)
adccalib.c:114: (Each undeclared identifier is reported only once for each
   function it appears in.)
make: *** [adccalib.o] Error 1

my ROOT is 4.01 and Zlib is 1.2.2

> When i compile the experiment example of 1.9.5 the problem happened:
> 
> adccalib.c: In function `INT adc_calib_init()':
> adccalib.c:114: `H1_BOOK' undeclared (first use this function)
> adccalib.c:114: (Each undeclared identifier is reported only once for each
>    function it appears in.)
> make: *** [adccalib.o] Error 1
> 
> my ROOT is 4.01 and Zlib is 1.2.2

We're in the process of fixing in the proper manner this problem, in the mean time
please add to the analyzer makefile the definition: -DUSE_ROOT at the line:
...
ROOTCFLAGS += -DHAVE_ROOT -DUSE_ROOT