> I deployed several Debian Linux boxes as the DAQ systems in our lab. But I
feel it's boring to build and install midas and its related softwares (such as
root) on each box.


Our solution at TRIUMF is to install such packages on a shared NFS filesystem
visible to all client computers. This works well for ROOT and but MIDAS we found
it nearly impossible to keep MIDAS versions in sync between different projects
and expiments, so each experiment uses it's own copy of MIDAS, usually located
in the experiment home directory ($HOME/packages/midas). Because we often need
to make local modifications to MIDAS sources (Makefile, etc), we do not
"install" MIDAS into non-user-writable /usr/local & etc.


> I use autoconf/automake


The promise (premise) of autoconf/automake is to "hide" system dependencies. The
scripts are supposed to automatically probe the build environment and construct
an appropriate Makefile.

In practice, the autotool scripts always have bugs and incorrect assumptions
about the build environment and only work well for a few standardized systems
(RHEL and Debian derivatives) where the differences are so trivial that
autotools is an overkill and a normal Makefile is adequate for the job.

In my experience, as soon as I try to build an autotool-ized package on anything
that does not look like RHEL or Debian, autotool scripts explode and have to be
debugged and kludged by hand. Anybody who has ever done that would agree with me
that one would rather hack the ugliest Makefile than any of the  autotool
generated gibberish.

And of course autotools have never handled cross-compilation in any reasonable
way. Since we do cross-compile MIDAS (for VxWorks and embedded Linux, see "make
crosscompile") a Makefile is required and it so happens that the same Makefile
also works for normal Linux and MacOS, thank you very much.



> Here are the installation:
> [*] executalbes -- /usr/lib/daq-midas/bin
> [*] library and objs -- /usr/lib/daq-midas/lib


Is this in violation of the LSB (or LFS)? I though they mandate that files
controlled by package manager should be /usr/bin/odbedit, /usr/lib64/libmidas.a,
etc (/usr/bin/midas/odbedit no permitted).


> gcc `mdaq-config --cflags` -c -o myfe.o myfe.c


Please check if your config scripts correctly handle the "-m32" and "-m64" flags
- we frequently cross-compile 32-bit MIDAS executables on 64-bit machines.


K.O.

Hi,

I am running a simple FE executable that is supposed to define a PRAW DWORD bank.
The issue is that, right after the start of the run, the logger crashes without messages.
Then the FE reports this error, which is rather confusing.
```
12:59:29.140 2021/05/24 [feTestDatastruct,ERROR] [odb.cxx:6986:db_set_data1,ERROR] "/Equipment/Trigger/Variables/PRAW" is of type UINT32, not UINT32
```

> Hi,
> 
> I am running a simple FE executable that is supposed to define a PRAW DWORD bank.
> The issue is that, right after the start of the run, the logger crashes without messages.
> Then the FE reports this error, which is rather confusing.
> ```
> 12:59:29.140 2021/05/24 [feTestDatastruct,ERROR] [odb.cxx:6986:db_set_data1,ERROR] "/Equipment/Trigger/Variables/PRAW" is of type UINT32, not UINT32
> ```

I think this is fixed in latest midas. There was a typo in this message, the same tid was printed twice,
with result you report "mismatch UINT32 and UINT32", instead of "mismatch of UINT32 vs what is actually there".

This fixes the message, after that you have to manually fix the mismatch in the data type in ODB (delete old one, I guess).

K.O.

Hello!
My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
University (Japan). I also happen to be a T2K collaborator.
I am currently developing the DAQ software for a T2K near detector called WAGASCI 
(different from ND280) and we recently decided to adopt MIDAS as a user 
interface.
Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
the remote repository every now and then with my local copy and that is fine ... 
but on the 25th of April our experiment is officially starting and I was 
wondering which version of MIDAS should I use for "production".
So my question is:
For a running experiment that needs software stability what branch of the MIDAS 
repository is better suited? The master branch or the develop branch? Moreover, 
what point in time do you think is more stable?
Best regards
Giorgio

Hi, Giorgio - you are asking excellent questions. I will try to answer them, but as ever, there are no 
easy answers.

In general, the top of the midas "develop" branch is "the best midas there is".

So for a new experiment, it is a reasonable place to start. Of course you can see
that there is quite a bit of commit activity going on, however, most substantial
changes are done on separate branches, where we try the new code, debug
it and test it. Only when the new code is "ready", we commit it to the "develop"
branch. Then, most often, we find some more last minute post-merge bugs,
and fix them right then and there on the head of the "develop" branch. Eventually
the dust settles and we have stable code that stays stable for a long time.

For example, right now we are waiting for the dust to settle on the change
of the MIDAS URL scheme, which was necessitated by needs of several experiments
that have more-complex-then-usual https proxy configurations. Unusual today,
but more common as we move forward, I think.

So if the head of the "develop" branch does not work for you, we encourage
that you file a bug report (here on this forum or on the bitbucket issue tracker).

While we try to sort out your problem, you can fall back to a previous version
of midas:

a) go back to one of the older "midas-YYYY-MM-X" tags
b) go back to one of the release candidate branches "feature/midas-YYYY-MM".

But if you have an already running system and you already have a working
instance of MIDAS, you do not have to update it to the latest version
unless you need some newest feature or you suffer from a bug
that has been fixed in a newer version.

In general, I find that it is fairly safe to update a working instance of MIDAS to
the latest code. But do keep your old working copy, if there is trouble, you can
always go back to something that works.

Now to your questions:

> For a running experiment that needs software stability what branch of the MIDAS 
> repository is better suited?

easy to answer. the most stable is the version you are using right now (doh!). each time
you upgrade, there is a risk that something will go wrong.

if you start from scratch, use the head of the "develop" branch (the latest and greatest),
if you run into trouble, report the trouble and update to a newer version with your
trouble fixed, or go back in time to previous tags and release candidate branches (as I described 
above).

> The master branch or the develop branch?

the "develop" branch.

> Moreover, what point in time do you think is more stable?

I find that it is impossible to have a stable-stable-stable version of midas
because the rest of the world flows forward in time. Old versions of midas
stop compiling because of OS and compiler changes; they stop running
because of OS or web browser or hardware changes. Then somebody
always asks for new features and new or old bugs surface constantly.

But we try to mark some "good" spots using git tags and release branches,
however the more far you go back in time, the more likely the code will
not even compile anymore.


K.O.



> Hello!
> My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
> University (Japan). I also happen to be a T2K collaborator.
> I am currently developing the DAQ software for a T2K near detector called WAGASCI 
> (different from ND280) and we recently decided to adopt MIDAS as a user 
> interface.
> Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
> the remote repository every now and then with my local copy and that is fine ... 
> but on the 25th of April our experiment is officially starting and I was 
> wondering which version of MIDAS should I use for "production".
> So my question is:
> For a running experiment that needs software stability what branch of the MIDAS 
> repository is better suited? The master branch or the develop branch? Moreover, 
> what point in time do you think is more stable?
> Best regards
> Giorgio

Hi Konstantin,
thank you for the very in-depth answer. Right now I am using the latest version of MIDAS (the head of the 
develop branch) and I have not noticed any bugs or problems in the MIDAS code, except the ones that I 
myself put in the part of the code that I am developing of course.
So I will do as you suggest and continue to use the head of the develop branch.
Thank you again for taking the time to answer all my questions!

PS other than the code peculiar to our experiment, I have made a little modification to the MIDAS install 
Makefile (I noticed that there is an "install" target but not an "uninstall" target so I wrote it).
If you are interested, I could make a merge request on BitBucket, just let me know.

Bests
Giorgio

> Hi, Giorgio - you are asking excellent questions. I will try to answer them, but as ever, there are no 
> easy answers.
> 
> In general, the top of the midas "develop" branch is "the best midas there is".
> 
> So for a new experiment, it is a reasonable place to start. Of course you can see
> that there is quite a bit of commit activity going on, however, most substantial
> changes are done on separate branches, where we try the new code, debug
> it and test it. Only when the new code is "ready", we commit it to the "develop"
> branch. Then, most often, we find some more last minute post-merge bugs,
> and fix them right then and there on the head of the "develop" branch. Eventually
> the dust settles and we have stable code that stays stable for a long time.
> 
> For example, right now we are waiting for the dust to settle on the change
> of the MIDAS URL scheme, which was necessitated by needs of several experiments
> that have more-complex-then-usual https proxy configurations. Unusual today,
> but more common as we move forward, I think.
> 
> So if the head of the "develop" branch does not work for you, we encourage
> that you file a bug report (here on this forum or on the bitbucket issue tracker).
> 
> While we try to sort out your problem, you can fall back to a previous version
> of midas:
> 
> a) go back to one of the older "midas-YYYY-MM-X" tags
> b) go back to one of the release candidate branches "feature/midas-YYYY-MM".
> 
> But if you have an already running system and you already have a working
> instance of MIDAS, you do not have to update it to the latest version
> unless you need some newest feature or you suffer from a bug
> that has been fixed in a newer version.
> 
> In general, I find that it is fairly safe to update a working instance of MIDAS to
> the latest code. But do keep your old working copy, if there is trouble, you can
> always go back to something that works.
> 
> Now to your questions:
> 
> > For a running experiment that needs software stability what branch of the MIDAS 
> > repository is better suited?
> 
> easy to answer. the most stable is the version you are using right now (doh!). each time
> you upgrade, there is a risk that something will go wrong.
> 
> if you start from scratch, use the head of the "develop" branch (the latest and greatest),
> if you run into trouble, report the trouble and update to a newer version with your
> trouble fixed, or go back in time to previous tags and release candidate branches (as I described 
> above).
> 
> > The master branch or the develop branch?
> 
> the "develop" branch.
> 
> > Moreover, what point in time do you think is more stable?
> 
> I find that it is impossible to have a stable-stable-stable version of midas
> because the rest of the world flows forward in time. Old versions of midas
> stop compiling because of OS and compiler changes; they stop running
> because of OS or web browser or hardware changes. Then somebody
> always asks for new features and new or old bugs surface constantly.
> 
> But we try to mark some "good" spots using git tags and release branches,
> however the more far you go back in time, the more likely the code will
> not even compile anymore.
> 
> 
> K.O.
> 
> 
> 
> > Hello!
> > My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
> > University (Japan). I also happen to be a T2K collaborator.
> > I am currently developing the DAQ software for a T2K near detector called WAGASCI 
> > (different from ND280) and we recently decided to adopt MIDAS as a user 
> > interface.
> > Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
> > the remote repository every now and then with my local copy and that is fine ... 
> > but on the 25th of April our experiment is officially starting and I was 
> > wondering which version of MIDAS should I use for "production".
> > So my question is:
> > For a running experiment that needs software stability what branch of the MIDAS 
> > repository is better suited? The master branch or the develop branch? Moreover, 
> > what point in time do you think is more stable?
> > Best regards
> > Giorgio

> 
> PS other than the code peculiar to our experiment, I have made a little modification to the MIDAS install 
> Makefile (I noticed that there is an "install" target but not an "uninstall" target so I wrote it).
> If you are interested, I could make a merge request on BitBucket, just let me know.
> 

Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.

K.O.



> Bests
> Giorgio
> 
> > Hi, Giorgio - you are asking excellent questions. I will try to answer them, but as ever, there are no 
> > easy answers.
> > 
> > In general, the top of the midas "develop" branch is "the best midas there is".
> > 
> > So for a new experiment, it is a reasonable place to start. Of course you can see
> > that there is quite a bit of commit activity going on, however, most substantial
> > changes are done on separate branches, where we try the new code, debug
> > it and test it. Only when the new code is "ready", we commit it to the "develop"
> > branch. Then, most often, we find some more last minute post-merge bugs,
> > and fix them right then and there on the head of the "develop" branch. Eventually
> > the dust settles and we have stable code that stays stable for a long time.
> > 
> > For example, right now we are waiting for the dust to settle on the change
> > of the MIDAS URL scheme, which was necessitated by needs of several experiments
> > that have more-complex-then-usual https proxy configurations. Unusual today,
> > but more common as we move forward, I think.
> > 
> > So if the head of the "develop" branch does not work for you, we encourage
> > that you file a bug report (here on this forum or on the bitbucket issue tracker).
> > 
> > While we try to sort out your problem, you can fall back to a previous version
> > of midas:
> > 
> > a) go back to one of the older "midas-YYYY-MM-X" tags
> > b) go back to one of the release candidate branches "feature/midas-YYYY-MM".
> > 
> > But if you have an already running system and you already have a working
> > instance of MIDAS, you do not have to update it to the latest version
> > unless you need some newest feature or you suffer from a bug
> > that has been fixed in a newer version.
> > 
> > In general, I find that it is fairly safe to update a working instance of MIDAS to
> > the latest code. But do keep your old working copy, if there is trouble, you can
> > always go back to something that works.
> > 
> > Now to your questions:
> > 
> > > For a running experiment that needs software stability what branch of the MIDAS 
> > > repository is better suited?
> > 
> > easy to answer. the most stable is the version you are using right now (doh!). each time
> > you upgrade, there is a risk that something will go wrong.
> > 
> > if you start from scratch, use the head of the "develop" branch (the latest and greatest),
> > if you run into trouble, report the trouble and update to a newer version with your
> > trouble fixed, or go back in time to previous tags and release candidate branches (as I described 
> > above).
> > 
> > > The master branch or the develop branch?
> > 
> > the "develop" branch.
> > 
> > > Moreover, what point in time do you think is more stable?
> > 
> > I find that it is impossible to have a stable-stable-stable version of midas
> > because the rest of the world flows forward in time. Old versions of midas
> > stop compiling because of OS and compiler changes; they stop running
> > because of OS or web browser or hardware changes. Then somebody
> > always asks for new features and new or old bugs surface constantly.
> > 
> > But we try to mark some "good" spots using git tags and release branches,
> > however the more far you go back in time, the more likely the code will
> > not even compile anymore.
> > 
> > 
> > K.O.
> > 
> > 
> > 
> > > Hello!
> > > My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
> > > University (Japan). I also happen to be a T2K collaborator.
> > > I am currently developing the DAQ software for a T2K near detector called WAGASCI 
> > > (different from ND280) and we recently decided to adopt MIDAS as a user 
> > > interface.
> > > Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
> > > the remote repository every now and then with my local copy and that is fine ... 
> > > but on the 25th of April our experiment is officially starting and I was 
> > > wondering which version of MIDAS should I use for "production".
> > > So my question is:
> > > For a running experiment that needs software stability what branch of the MIDAS 
> > > repository is better suited? The master branch or the develop branch? Moreover, 
> > > what point in time do you think is more stable?
> > > Best regards
> > > Giorgio

> Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.


... and change the documentation accordingly (Suzannah!?). Installing midas these days does not really make sense, since normally only one 
users uses it on a given machine.

Stefan

> > Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.
> 
> 
> ... and change the documentation accordingly (Suzannah!?). Installing midas these days does not really make sense, since normally only one 
> users uses it on a given machine.
> 
> Stefan

I understand. Anyway, I preferred to install MIDAS in the /opt/midas folder to be consistent with the other programs (Pyrame and Calicoes from LLR) that we are using for our experiment (they are all installed in the /opt folder). Moreover, I am using Linux systemd to enable mhttpd on startup (and other handy features like auto-restart after a crash). And quite unfortunately CentOS doesn't support to start systemd units as a non-root user.
So in my particular case, perhaps it made some sense to install MIDAS in a folder other than the source code folder.

Giorgio

> > Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.
> 
> 
> ... and change the documentation accordingly (Suzannah!?). Installing midas these days does not really make sense, since normally only one 
> users uses it on a given machine.
> 
> Stefan

I understand. Anyway, I preferred to install MIDAS in the /opt/midas folder to remain consistent with the other programs that we are using for 
our experiment (Pyrame and Calicoes from LLR). I am also using Linux systemd to enable mhttpd on startup (and other handy features like auto-
restart after a crash) and unfortunally CentOS doesn't support to enable systemd units as a non-root user.
So in my particular case, perhaps it made some sense to install MIDAS in a folder other than the source code folder
Giorgio

> > > Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.
>
> install MIDAS in the /opt/midas folder to remain consistent with the other programs that we are using for 
> our experiment (Pyrame and Calicoes from LLR).

I see. Would this work as well? Instead of "make install" do this:
su - root
cd /opt
git pull midas
cd midas
make
add /opt/midas/linux/bin to your PATH. (is it time to get rid of the "linux" part from the default build path?!?)

> I am also using Linux systemd to enable mhttpd on startup

We use cron @reboot to start mhttpd.

But. CentOS7 systemd cron unit file has a bug and they run @reboot cron jobs before NIS and autofs is ready, see
http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_crontab_.40reboot_for_MIDAS_.28CentOS7.29

Can you post your systemd unit file to this elog, others may find it useful.

K.O.

> I see. Would this work as well? Instead of "make install" do this:
> su - root
> cd /opt
> git pull midas
> cd midas
> make
> add /opt/midas/linux/bin to your PATH. (is it time to get rid of the "linux" part from the default build path?!?)

Got it. I will do that in the future.

> Can you post your systemd unit file to this elog, others may find it useful.

[Unit]
Description=MIDAS data acquisition system
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=3
User=neo
ExecStart=/opt/midas/bin/mhttpd -e WAGASCI --http 8081 --https 8444
Environment="MIDASSYS=/opt/midas" "MIDAS_EXPTAB=/home/neo/Code/WAGASCI/MIDAS/online/exptab" "MIDAS_EXPT_NAME=WAGASCI" 
"SVN_EDITOR=emacs -nw" "GIT_EDITOR=emacs -nw"
PassEnvironment=MIDASSYS MIDAS_EXPTAB MIDAS_EXPT_NAME SVN_EDITOR GIT_EDITOR

[Install]
WantedBy=multi-user.target

#!/bin/sh

[ ! "$MIDASSYS" ] && MIDASSYS=/opt/MIDAS.PSI/Version/Current
[ ! "$HTTPPORT" ] && HTTPPORT=8080
[ ! "$SRVHOST" ]  && SRVHOST=localhost
LOGGER=${MIDASSYS}/bin/mlogger

EXPPATH=/home/das/online/test
CODEPATH=${EXPPATH}/code
LOGGER=${MIDASSYS}/bin/mlogger

PROG_FE=${CODEPATH}/frontend
PROG_ANA=${CODEPATH}/analyzer

if [ ! "$MIDAS_EXPTAB" ]; then
	MIDAS_DIR=${EXPPATH}
else
	MIDAS_EXPT_NAME="test"
fi

> ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.

Usually we expect that it is "safe" to update to the latest version of MIDAS.

In the sense that we do not have to track down every single frontend
and rebuild it. We have several experiments where tracking down the source code
and rebuilding a frontend takes more than 5 seconds. In many cases these are
10 year old executables that worked just fine through many updates of MIDAS
without having to rebuild them.

So any binary incompatible change is best avoided and must be clearly announced.

The present binary-incompatible change is this commit:
https://bitbucket.org/tmidas/midas/commits/5899f1657ba31121c9f420a824b3c6c13b173988

I tagged the last commit before this change as: midas-2024-12-a

K.O.

> > ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.
> 
> Usually we expect that it is "safe" to update to the latest version of MIDAS.
> 
> In the sense that we do not have to track down every single frontend
> and rebuild it. We have several experiments where tracking down the source code
> and rebuilding a frontend takes more than 5 seconds. In many cases these are
> 10 year old executables that worked just fine through many updates of MIDAS
> without having to rebuild them.
> 
> So any binary incompatible change is best avoided and must be clearly announced.
> 
> The present binary-incompatible change is this commit:
> https://bitbucket.org/tmidas/midas/commits/5899f1657ba31121c9f420a824b3c6c13b173988
> 
> I tagged the last commit before this change as: midas-2024-12-a

Here are my replies:

- this is not a binary incompatibility, but a incompatibility of the /Alarm record which got two more variables. Old 
frontends will complain during their structure check and remove the two variables, new frontend will then complain as 
well and add the two variables. This will go in circles, that why all frontends need to be recompiled if the new code is 
used

- I did clearly announce this change in the forum. Is there another location where I should communicate that?

- The extension is not there "just for fun" but needed by several experiments which experience spurious alarms 
triggered by some noisy signals. Requiring an value to be above a limit for a certain minimum time fixes many issues in 
many experiments here at PSI, this is why it has been implemented, even if it causes work for everybody with re-
compilation.

- If there are frontend programs which have not been re-compiled for ten years, I think it is very unlikely that these 
experiments need the latest cutting edge version of midas. The can safely stick to your tag midas-2024-12-a and not 
experience the issue of different /Alarm trees.

Stefan

> > > ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.
>
> - I did clearly announce this change in the forum.
>

I missed the announcement and I was surprised to discover this change.

This is why I changed the title from "useful improvement" to "ACHTUNG!!!"

> - The extension is not there "just for fun" but needed by several experiments which experience spurious alarms 
> triggered by some noisy signals.

Agreed, a useful improvement.

To prevent this kind of trouble in the future, I think I should finish my crusade against db_get_record().

> - If there are frontend programs which have not been re-compiled for ten years, I think it is very unlikely that these 
> experiments need the latest cutting edge version of midas. The can safely stick to your tag midas-2024-12-a and not 
> experience the issue of different /Alarm trees.

Yes, now that I added the tag, posted a more visible notice and people who have to run frontends build against older midas 
(because reasons) can breathe out.

Also please note that "Konstantin != all experiments at TRIUMF".

K.O.

Hello everyone,

It seems that if a frontend is started automatically by using Program->Auto start then the status page does not show it as started. This is since the FE name has a number after the name. If I stop and start manually then the status page shows the correct state of the FE. Am I doing something wrong or is this a bug somewhere?

thanks,
Zaher

> It seems that if a frontend is started automatically by using Program->Auto start then the status page does not show it as started. This is since the FE name has a number after the name. If I stop and start manually then the status page shows the correct state of the FE. Am I doing something wrong or is this a bug somewhere?

Zaher, please read https://daq00.triumf.ca/elog-midas/Midas/919

K.O.

In one of the experiments, we hit a long time bug in mdump - there was an array of 32 equipments and if 
there were more than 32 entries under /equipment, it would overrun and corrupt memory. Somehow this 
only showed up after mdump was switched to c++. The solution was to use std::vector instead of fixed 
size array.

Just in case, I checked other midas programs for fixed size arrays (other than fixed size strings) and found 
none. (in midas.c, there is a fixed size array of TR_FIFO[10], but code inspection shows that it cannot 
overrun).

I used this script. It can be modified to also identify any strange sized string arrays.

K.O.

#!/usr/bin/perl -w

while (1) {
  my $in = <STDIN>;
  last unless $in;
  #print $in;

  $in =~ s/^\s+//;

  next if $in =~ /^char/;
  next if $in =~ /^static char/;

  my $a = $in =~ /(.*)[(\d+)\]/;

  next unless $a;

  my $a1 = $1;
  my $a2 = $2;

  next if $a2 == 0;
  next if $a2 == 1;
  next if $a2 == 2;
  next if $a2 == 3;

  #print "[$a] [$a1] [$a2]\n";
  print "-> $a1[$a2]\n";
}

# end

I always got a 503 server error while I tried to connect this log book the latest 
weeks. I don't know why. I hope it is not due to the network censorship because 
of the coming National Day of China. Anyway, good luck to me when I want to paste 
something here.