No, I'm not aware of this problem, but I suspect that your events somehow got corrupted. You can try the mdump utility
or the "Event Dump" web page to peek into your events, maybe you see an issue there. To give you more detailed information,
I would have to reproduce your problem, which is probably hard without your hardware.

Stefan

I cannot confirm that. I just tried myself with examples/experiment/frontend.cxx, removed the RO_ODB, and the trigger events did NOT get copied to the ODB.

Actually you can debug the code yourself. The relevant line is in mfe.cxx:2075:

   /* send event to ODB */
   if (pevent->data_size && (eq_info->read_on & RO_ODB)) {
      if (actual_millitime - eq->last_called > ODB_UPDATE_TIME) {
         eq->last_called = actual_millitime;
         update_odb(pevent, eq->hkey_variables, eq->format);
         eq->odb_out++;
      }
   }

so if read_on is equal 1, the function update_odb should never be called.

So the problem must be on your side.

Best,
Stefan

> - multiple subdetectors are taking test data during early commissioning 
> - a postgres db is a single sorce of run numbers.
> - test runs taken by different subsystems are assigned different [unique] run numbers and 
>   the data taken by the subsystem are identified not by the run number/dataset name , but 
>   by the run type, different for different susbsystems.

For that purpose I would not "mis-use" run numbers. Run number are meant to be incremented 
sequentially, like if you have a time-stamp in seconds since 1.1.1970 (Unix time). Intead, I 
would add additional attributes under /Experiment/Run Parameters like "Subsystem type", "Run 
mode (production/commissioning)" etc. You have much more freedom in choosing any number of 
attributes there. Then, send this attributes to your postgred db via "/Logger/Runlog/SQL/Links 
BOR". Then you can query your database to give you all runs of a certain subtype or mode.

See https://daq00.triumf.ca/MidasWiki/index.php/Logging_to_a_mySQL_database

Stefan

> Stefan, I don't think we're talking 'mis-use' - rather different subdetectors being commisisoned 
> at different locations, on an uncorrelated schedule, using independent run control (RC) instances. 
> At this point in time, we can't use a common RC instance. 
> The collected data, however, are written back into a common storage, and we need to avoid two 
> subdetectors  using the same run number. As all RC instances can connect to the same database and request a 
> run number from there, an external DB serving run numbers to multiple clients looks as a reasonable solution, 
> which provides unique run numbers for everyone. Of course, the run number gets incremented (although on the DB 
> server side), and of course different susbystems are assigned different subsystem types. 
> 
> So, in essense, it is about _where_ the run number is incremented - the RC vs the DB. 
> If there were a good strategy to implement a DB-based solution that w/o violating 
> first principles of Midas:), I'd be happy to contribute. It looks like a legitimate use case.

Ok, maybe attitude comes from the fact that I never used such a scheme in the last 30 years with midas.

If you go in this direction, there is an alternative to what Ben wrote: Use the sequencer to start a run.
The sequencer script can obtain a new run number from a central instance (e.g. by calling a shell script 
like 'curl ...' to obtain the new run number, then put it into /Runinfo/Run number as Ben wrote. This has
the advantage that the run is _started_ already with the correct number, so the history system is fine.

The script can then wait for n events, then stop the run etc. A sequencer script will also be necessary if
you want to configure your electronics (see next answer...)

Stefan

> The MIDAS state machine doesn't seem to have a state in between 'initialized' and 
> 'running'.
> In a larger detectors with multiple subsystems, the DAQ systems often have one more state:
> after ending a previous run and before starting a new one from the 'stopped' state, 
> one needs to make sure that all subdetectors are ready, or 'configured' for the new run. 
> So theat calls for a 'configure' step during which the detector (all subsystems in 
> parallel, to save the time) transitions from 'initialized'/'stopped' to 'configured' state, 
> from which it transitions to the 'running' state.
> 
> If one of the subdetectors fails to get configured, it could be excluded from the run 
> configuration and another attempt to reconfigure the system could be made without 
> starting a new run. Or an attempt could be made to troubleshoot and configure the 
> failed subsytem individually , with the rest subsystems waiting in a 'configured' state.
> 
> How does the logic of configuring the detector for the new run is implemented in MIDAS? 
> - it is a fairly common operational procedure, so I'm sure there should be a way 
> of doing that.

We have a similar requirement in our MEG experiment. Configuring your subdetectors can
be quite complex and therefore it's almost impossible to define a 'configure' step in
the run transition system to accommodate all corner cases.

Instead of a new state, we do everything through the sequencer:

- To start a run, we start a special sequencer script. We have different scripts for
calibration runs, data runs, special runs.

- When the user starts the script, they are asked for certain parameters, like number
of events, number of runs to take, how to configure the subdetectors, which subdetectors
to read out etc.

- The script then configures the whole experiment by setting everything in the ODB for
each equipment.

- The frontends connected to their equipment get a hotlink from the ODB and start the
configuration of the trigger etc. based on the parameters from the ODB

- The progress of the configuration is indicated by the frontend by writing back the
progress (like 0...100) into the ODB

- The script now waits for the progress to reach 100. It shows the current progress
on the sequencer page, so you see exactly where we are.

- If we have several subdetectors, each of them can publish a progress, and the script
can wait for an AND of all progress, or exclude one if it fails etc. Any logic is
possible there.

- Once all progresses are at 100, the run is finally started.

- If the mechanics of configuration become more elaborate, one can 'hide' it in
sub-modules of the script.

This scheme allows us to configure very different run modes, we use it in MEG since
many years (about 0.5M runs) and it works very nice.

Attached is our main script to start a full data run. You don't have to understand
all details, but it can give you a glimpse of what it's possible with the sequencer.
The function "ApplySettings" is the one waiting for the configuration flag in the ODB
(we simply use a boolean flag there). The code is:

SUBROUTINE ApplySettings
  ODBSET "/Equipment/Trigger/Settings/Reload all", y, 1
  WAIT seconds, 2
  WAIT ODBValue, "/Equipment/Trigger/Variables/Config busy", ==, y
ENDSUBROUTINE

Best,
Stefan

> - how does one communicate with an external shell script from MSL ? I looked at the MIDAS Sequencer page 
>   
>   https://daq00.triumf.ca/MidasWiki/index.php/Sequencer
> 
>   and didn't find an immediately obvious candidate among the MSL commands. 
>   The closest seems to be 
> 
>   'SCRIPT script [, a, b, c, ...]' 
> 
>   but I couldn't easily figure how to propagate the output of the script back to MIDAS. 
>   Let say, the script creates an ASCII file with the next run number. What is the easiest 
>   way to import the run number into ODB? - Should an external script spawn a [short-lived] 
>   MIDAS client ?  - That would work, but I'm almost sure there is a more straightforward solution. 

The output of the SCRTIP command is stored in the variable $SCRIPT_RESULT. Please pull midas to get this
new functionality.

Stefan

>    [that was not obvious from the documentation on MIDAS wiki, and adding a couple of clarifying 
>     sentences there would go long ways]

I added a sentence there. Please have a look. If you like more info, please write it yourself and send it to me. 
It's always better if that comes from users than from me.

 
> 2. if a script produces an ascii file with a known name, for example, 'a.odb', with the following two lines:

Use $SCRIPT_RESULT as described before.

Best,
Stefan

Dear beloved MIDAS users,

I'm happy to announce a "holiday update" for MIDAS. In countless hours, Zaher from 
PSI worked hard to introduce syntax highlighting in the midas script editor. In 
addition, there are additional features like a cleaner user interface, the option 
to see all variables also in non-debug mode and more. Have a look at the picture 
below, doesn't it beginning to look a lot like Christmas?

We have tested this quite a bit and went through many iterations, but no guarantee 
that it's flawless. So please report any issue here.

I wish you all a happy holiday season,
Stefan

> If I understand the msequencer code correctly:
> Under '/Sequencer/State/Path' the path can be defined from where the msequencer gets the files, generates the xml, etc.
> However, the new javascript code reads/writes the files to '<exp>/userfiles/sequencer/'
> 
> If the path in the ODB is different to '<exp>/userfiles/sequencer/', it leads to quite some unexpected behavior. If '<exp>/userfiles/sequencer/' is the place where things should go, the ODB entry of the msequencer and the internal handling should probably adopted, no?

Indeed there is a change in philosophy. Previously, /Sequencer/State/Path could point anywhere in the file system. This was considered a security problem, since one could access system files under /etc for example via the midas interface. When the new file API was 
introduced recently, it has therefor been decided that all files accessible remotely should reside under <exp>/userfiles. If an experiment needs some files outside of that directory, the experiment could define some symbolic link, but that's then in the responsibility of 
the experiment. 

To resolve now the issue between the sequencer path and the userfiles, we have different options, and I would like to get some feedback from the community, since *all experiments* have to do that change.

1) Leave thins as they are, but explain that everybody should modify /Sequencer/Stat/Path to some subdirectory of <exp>/userfiles/sequencer

2) Drop /Sequencer/State/Path completely and "hard-wire" it to <exp>/usefiles/sequencer

3) Make /Sequencer/State/Path relative to <exp>/userfiles. Like if /Sequencer/State/Path=test would then result to a final directory <exp>/userfiles/sequencer/test

I'm kind of tempted to go with 3), since this allows the experiment to define different subdirectories under <exp>/userfiles/sequencer/... depending on the situation of the experiment.

Best,
Stefan

> to plot time dependencies of the monitored detector parameters, say, voltages or temperatures,
> one needs to convert the coresponging ADC readings into floats.
> 
> One could think of two ways of doing that: 
> 
> - one can perform the ADC-->T or ADC-->V conversion in the MIDAS frontend, 
>   store their [float] values in the data bank, and plot precalculated parameters vs time
> 
> - one can also store in the data bank the ADC readings which typically are short's 
>   and convert them into floats (V's or T's) at the plotting time 
> 
> The first approach doubles the storage space requirements, and I couldn't find the place where 
> one would do the conversion, if stored were the 16-bit ADC readings.
> 
> I'm sure this issue has been thought about, so what is the "recommended MIDAS way" of performing 
> the ADC -> monitored_number conversion when making MIDAS history plots ?

Most experiment go with the second method. The front-end program converts all ADC reading into physicsl 
units, i.e. not only Volt, but even Degrees Centigrade or Tesla or whatever. The slow control part of 
midas then puts these number into /Equipment/<name>/Variables as "float", and the history system picks 
them up from there. This way your history is shown in physical units and not ADC count. Actually the 
recommended slow control framework (check the examples direcotory) does not rely on data banks, but 
puts values directly into the ODB. This is typically done faster, like once per second if a value 
changes, rather than slow control events which are generated maybe once per 10 seconds or once per 
minute. Usually the slow control values are only few compared with trigger data, so a factor of two 
there does not really matter. In the MEG experiment, we have like 400 GB of slow control data per year, 
but 400 TB of trigger data per year.

Best,
Stefan

First of all it's important to understand that the slow control system has nothing to do
with events. So if you look at event statistics, these are the events with the slow control
data sent to the midas data file, not the history database. So the logging period (the one you 
decreased from 60s to 10s to 6s) only affect the generation of events. 

What is important in your case is the number of events sent to the ODB. You see these in the
screen output of the slow control frontend (see attachment). This number show increase every
second.

I tried your modification (change nulldev with a sine function), and left the sc_fe.cxx
otherwise untouched. I then started with a fresh ODB ("rm /"). Started logger, mhttpd, sc_fe
and started a run. In the attachments is what I see. So I don't understand what your problem
is.

> > 3) Make /Sequencer/State/Path relative to <exp>/userfiles. Like if /Sequencer/State/Path=test would then result to a final directory <exp>/userfiles/sequencer/test
> > 
> > I'm kind of tempted to go with 3), since this allows the experiment to define different subdirectories under <exp>/userfiles/sequencer/... depending on the situation of the experiment.
> > 
> > Best,
> > Stefan
> 
> For me the option 3) seems the most coherent one.
> Andreas

Ok, I implemented option 3) above. This means everybody using the midas sequencer has to change /Sequencer/State/Path to an empty string and move the 
sequencer files under <exp>/userfiles/sequencer as a starting point. I tested most thing, including the INCLUDE statements, but there could still be
a bug here or there, so please give it a try and report any issue to me.

Best,
Stefan

> Also, I think that having a sine wave displayed by midas/examples/slowcont/scfe.cxx 
> would make this example even more helpful.

Indeed. I reworked the example to have a out-of-the-box sine wave plotter, including the
automatic creation of a history panel. Thanks for the hint.

Best,
Stefan

Custom plots including scatter, histogram and color map plots have been 
implemented. This lets you plot graphs of X/Y data or histogram data stored in the 
ODB on a custom page. For some examples and documentation please go to

https://daq00.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

Enjoy!
Stefan

> the DNS alias for midas.triumf.ca moved from old ladd00.triumf.ca to new 
> daq00.triumf.ca. same as before it redirects to the MidasWiki and to the midas 
> forum (elog) that moved from ladd00 to daq00 quite some time ago. if you see any 
> anomalies in accessing them (broken links, bad https certificates), please report 
> them to this forum or to me directly at olchansk@triumf.ca. K.O.

I found the first issue: The link to

https://midas.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

does not work any more. The link

https://daq00.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

however does work. Same with

https://midas.triumf.ca/MidasWiki/index.php/Sequencer

and

https://daq00.triumf.ca/MidasWiki/index.php/Sequencer

I have a few cases in mhttpd where I link directly to our documentation. I prefer
to have those link with "midas.triumf.ca" instead of "daq00.triumf.ca" in case you
change the machine again in the future.

Best,
Stefan

> > git pull
> > git submodule update
> 
> confirmed. just run into this myself. I think "make" should warn about out of 
> date git modules. Also check that the build git version is tagged with "-dirty".
> 
> K.O.

The submodule business becomes kind of annoying. I updated the documentation at

https://daq00.triumf.ca/MidasWiki/index.php/Quickstart_Linux#MIDAS_Package_Installatio
n

to tell people to use 

1) "git clone ... --recurse-submodules" during the first clone
2) "git submodule update --init --recursive" in case they forgot 1)
3) "git pull --recurse-submodules" for each update or to use
4) "git config submodule.recurse true" to make the --recurse-submodules the default

I use 4) since a while and it works nicely, so one does not have to remember to pull 
recursively each time.

Stefan

Put a 

  ss_sleep(10);

into your frontend_loop(), then you should be fine.

The event loop runs as fast as possible in order not to miss any (triggered) event, so no seep in the 
event loop, because this would limit the (triggered) event rate to 100 Hz (minimum sleep is 10 ms). 
Therefore, you have to slow down the event loop manually with the method described above.

Best,
Stefan

> Hi Stefan, thanks a lot !
> 
> I just thought that for the EQ_SLOW type equipment calls to sleep() could be hidden in mfe.cxx 
> and handled based on the requested frequency of the history updates.

Most people combine EQ_SLOW with EQ_POLLED, so they want to read out as quickly as possible. Since 
the framework cannot "guess" what the users want there, I removed all sleep() in the framework.



> Doing the same in the user side is straighforward - the important part is to know where the 
> responsibility line goes (: smile :) 


Pushing this to the user gives you more freedom. Like you can add sleep() for some frontends, but not 
for others, only when the run is stopped and more.

Stefan

Have you updated to the current midas version? This issue has been fixed a while ago. Below 
you see a screenshot of a long list scrolled all the way to the bottom.

Revision: Thu Dec 7 14:26:37 2023 +0100 - midas-2022-05-c-762-g1eb9f627-dirty on branch 
develop

Chrome on MacOSX 14.2.1

The fix is actually in "controls.js", so make sure your browser does not cache an old 
version of that file. I usually have to clear my browser history to get the new file from 
mhttpd.

Best regards,
Stefan

> 1) In the attached snapshot you see that the values shown from our vacuum Pirani and Penning cells are all zero, which of course is not true.
> It would be nice to have under the equipment settings some formatting options, like the possibility to add units.

You have a 

  /Equipment/LEMVAC/Settings/Format Input

array where you can specify the format for every value. Default is "%f2" for two digits after the period. For vacuum levels you might want to 
consider "%e3" which give you exponential format with three significant digits. The "format" setting is described at

  https://daq00.triumf.ca/MidasWiki/index.php//Equipment_ODB_tree#Format_%3Cvariable%3E

and the details are at

  https://daq00.triumf.ca/MidasWiki/index.php/Custom_Page#Formatting

The was a bug with the format handling, so please pull the current develop branch.


> 2) If one of the number evaluates to Infinity, the table is not shown properly anymore.

I fixed that as well in the current version.

Best,
Stefan