> Stefan, I don't think we're talking 'mis-use' - rather different subdetectors being commisisoned 
> at different locations, on an uncorrelated schedule, using independent run control (RC) instances. 
> At this point in time, we can't use a common RC instance. 
> The collected data, however, are written back into a common storage, and we need to avoid two 
> subdetectors  using the same run number. As all RC instances can connect to the same database and request a 
> run number from there, an external DB serving run numbers to multiple clients looks as a reasonable solution, 
> which provides unique run numbers for everyone. Of course, the run number gets incremented (although on the DB 
> server side), and of course different susbystems are assigned different subsystem types. 
> 
> So, in essense, it is about _where_ the run number is incremented - the RC vs the DB. 
> If there were a good strategy to implement a DB-based solution that w/o violating 
> first principles of Midas:), I'd be happy to contribute. It looks like a legitimate use case.

Ok, maybe attitude comes from the fact that I never used such a scheme in the last 30 years with midas.

If you go in this direction, there is an alternative to what Ben wrote: Use the sequencer to start a run.
The sequencer script can obtain a new run number from a central instance (e.g. by calling a shell script 
like 'curl ...' to obtain the new run number, then put it into /Runinfo/Run number as Ben wrote. This has
the advantage that the run is _started_ already with the correct number, so the history system is fine.

The script can then wait for n events, then stop the run etc. A sequencer script will also be necessary if
you want to configure your electronics (see next answer...)

Stefan

I have one more question, though I understand that it could be somewhat border-line.

The MIDAS state machine doesn't seem to have a state in between 'initialized' and 
'running'.
In a larger detectors with multiple subsystems, the DAQ systems often have one more state:
after ending a previous run and before starting a new one from the 'stopped' state, 
one needs to make sure that all subdetectors are ready, or 'configured' for the new run. 
So theat calls for a 'configure' step during which the detector (all subsystems in 
parallel, to save the time) transitions from 'initialized'/'stopped' to 'configured' state, 
from which it transitions to the 'running' state.

If one of the subdetectors fails to get configured, it could be excluded from the run 
configuration and another attempt to reconfigure the system could be made without 
starting a new run. Or an attempt could be made to troubleshoot and configure the 
failed subsytem individually , with the rest subsystems waiting in a 'configured' state.

How does the logic of configuring the detector for the new run is implemented in MIDAS? 
- it is a fairly common operational procedure, so I'm sure there should be a way 
of doing that.

-- thanks again, Pasha

> > - multiple subdetectors are taking test data during early commissioning 
> > - a postgres db is a single sorce of run numbers.
> > - test runs taken by different subsystems are assigned different [unique] run numbers and 
> >   the data taken by the subsystem are identified not by the run number/dataset name , but 
> >   by the run type, different for different susbsystems.
> 
> For that purpose I would not "mis-use" run numbers. Run number are meant to be incremented 
> sequentially, like if you have a time-stamp in seconds since 1.1.1970 (Unix time). Intead, I 
> would add additional attributes under /Experiment/Run Parameters like "Subsystem type", "Run 
> mode (production/commissioning)" etc. You have much more freedom in choosing any number of 
> attributes there. Then, send this attributes to your postgred db via "/Logger/Runlog/SQL/Links 
> BOR". Then you can query your database to give you all runs of a certain subtype or mode.
> 
> See https://daq00.triumf.ca/MidasWiki/index.php/Logging_to_a_mySQL_database
> 
> Stefan

Ben, Stefan - thanks much for your suggestions!(and apologies for the thanks being delayed) 

Stefan, I don't think we're talking 'mis-use' - rather different subdetectors being commisisoned 
at different locations, on an uncorrelated schedule, using independent run control (RC) instances. 
At this point in time, we can't use a common RC instance. 
The collected data, however, are written back into a common storage, and we need to avoid two 
subdetectors  using the same run number. As all RC instances can connect to the same database and request a 
run number from there, an external DB serving run numbers to multiple clients looks as a reasonable solution, 
which provides unique run numbers for everyone. Of course, the run number gets incremented (although on the DB 
server side), and of course different susbystems are assigned different subsystem types. 

So, in essense, it is about _where_ the run number is incremented - the RC vs the DB. 
If there were a good strategy to implement a DB-based solution that w/o violating 
first principles of Midas:), I'd be happy to contribute. It looks like a legitimate use case.

-- let me know, regards, Pasha 

> - multiple subdetectors are taking test data during early commissioning 
> - a postgres db is a single sorce of run numbers.
> - test runs taken by different subsystems are assigned different [unique] run numbers and 
>   the data taken by the subsystem are identified not by the run number/dataset name , but 
>   by the run type, different for different susbsystems.

For that purpose I would not "mis-use" run numbers. Run number are meant to be incremented 
sequentially, like if you have a time-stamp in seconds since 1.1.1970 (Unix time). Intead, I 
would add additional attributes under /Experiment/Run Parameters like "Subsystem type", "Run 
mode (production/commissioning)" etc. You have much more freedom in choosing any number of 
attributes there. Then, send this attributes to your postgred db via "/Logger/Runlog/SQL/Links 
BOR". Then you can query your database to give you all runs of a certain subtype or mode.

See https://daq00.triumf.ca/MidasWiki/index.php/Logging_to_a_mySQL_database

Stefan

> I wonder if there is a non-intrusive way to have an external (wrt MIDAS)*SQL database 
> serving as a primary source of the run number information for a MIDAS-based DAQ system? 
> - like a plugin with a getNextRunNumber() function, for example, or a special client?

One of my experiments has special rules for run numbering as well. I created a client that registers a begin-of-run transition handler with sequence 1 (so it's the first client to handle the begin-of-run transition). That client updates "/Runinfo/Run number" in the ODB. 

This mostly works. mlogger will create .mid files based on the new run number, the ODB dumps within those files show the new run number etc.

But there are 2 quirks. Let's say your client changed the number from 11 to 400. The message log will say "Run #11 started" and "Run #400 stopped". And the history system will record the start/stop times the same way. That only matters for when you're viewing history plots on the webpage and zoom in far enough to see the run transitions (represented by green and red vertical dashed lines) - the green line will be labelled 11 and the red line 400.

Depending on the exact logic you need, you may be able to avoid these quirks by also recomputing the run number before the user even tries to start a run (e.g. after the end of the previous run, or when the user changes an important setting in the ODB). If you're changing the run number between runs, make sure to set it to "desired number - 1", as midas will increment the run number automatically before handling the next start run request.

Dear MIDAQ developers,

I wonder if there is a non-intrusive way to have an external (wrt MIDAS)*SQL database 
serving as a primary source of the run number information for a MIDAS-based DAQ system? 
- like a plugin with a getNextRunNumber() function, for example, or a special client?

Here is the use case: 

- multiple subdetectors are taking test data during early commissioning 
- a postgres db is a single sorce of run numbers.
- test runs taken by different subsystems are assigned different [unique] run numbers and 
  the data taken by the subsystem are identified not by the run number/dataset name , but 
  by the run type, different for different susbsystems.

-- many thanks, regards, Pasha

I cannot confirm that. I just tried myself with examples/experiment/frontend.cxx, removed the RO_ODB, and the trigger events did NOT get copied to the ODB.

Actually you can debug the code yourself. The relevant line is in mfe.cxx:2075:

   /* send event to ODB */
   if (pevent->data_size && (eq_info->read_on & RO_ODB)) {
      if (actual_millitime - eq->last_called > ODB_UPDATE_TIME) {
         eq->last_called = actual_millitime;
         update_odb(pevent, eq->hkey_variables, eq->format);
         eq->odb_out++;
      }
   }

so if read_on is equal 1, the function update_odb should never be called.

So the problem must be on your side.

Best,
Stefan

Good morning, 

In our setup, we have a neutron detector that creates up to 16 MB of polled (EQ_POLLED) data in one event (event limit = 1) that we do not want to have saved into the ODB. Nevertheless, I cannot disable it. The equipment has only the read-on-flag RO_RUNNING, and the read-on value in the ODB is 1. The data are also not saved to the history. I also tried with a minimal example frontend with the same settings, but also those "data" get written to the ODB. For now, I increased the size of the ODB to 40 MB (50% for keys and 50% for data is automatic), but in principle, I do not want it to be saved to the ODB at all. Is there something I am missing?

Thanks in advance for your advice. 

Cheers,
Ivo

> No, I'm not aware of this problem, but I suspect that your events somehow got corrupted. You can try the mdump utility
> or the "Event Dump" web page to peek into your events, maybe you see an issue there. To give you more detailed information,
> I would have to reproduce your problem, which is probably hard without your hardware.
> 
> Stefan

Hi Stefan,

So I did a few things:
- I checked with mdump online, the data stream looks good, and I can see the bank name properly
- I checked with mdump offline the .mid files, the banks are there, and the data look good
- I removed the creating of the bank MSRD in the class driver. This stopped the writing of the data in the midas/root file but kept the stream to the history files. In principle, this is a quick and dirty fix because we still have all the data in the history files. Do you see any bigger problem with that solution?
- I tried to run the multi-threaded slow-control frontend with the generic class driver (generic.cxx) and the nulldev device driver (nulldev.cxx). This produces the DMND and MSRD bank with and also produces the error in the with the logger when trying to save in the root format (received unknown bank "DMND" in event #8). This means it is not related to the devices (maybe some other part of my user code of course). 

Cheers,
Ivo

No, I'm not aware of this problem, but I suspect that your events somehow got corrupted. You can try the mdump utility
or the "Event Dump" web page to peek into your events, maybe you see an issue there. To give you more detailed information,
I would have to reproduce your problem, which is probably hard without your hardware.

Stefan

> Stefan is right. I forgot this. As solution to our troubles, mlogger is built without root support. use rmlogger instead.
> 
> K.O.

Thanks, Stefan and Konstantin, for your feedback. 

So I checked the cmake file, and already the existence of the rmlogger shows that HAVE_ROOT was set. It was really only the problem of not being aware of rmlogger. This now works, and it produces root files that are readable. 

However, we encountered a new problem not that it does not find a bank that is produced by a multi-threaded slow-control frontend. The logger triggers the error "mlogger.cxx:3328:root_book_bank,ERROR] received unknown bank 'MSRD' in event #8". After this, we get a segmentation violation, but I guess this is then coming from the error. If we run only the polled FE, it works fine. If we run the polled and the multi-threaded FE with only the logger saving mid files, it works fine as well. Are you aware of issues with multi-threaded slow-control frontends and saving their banks in root format?

Cheers,
Ivo

> Finally, make sure you start "rmlogger" and not "mlogger". Only "rmlogger" contains the ROOT binding.

Stefan is right. I forgot this. As solution to our troubles, mlogger is built without root support. use rmlogger instead.

K.O.

When you do "cmake .." in the build directory, you will see

-- MIDAS: Found ROOT version xxx in yyy

which will tell you that ROOT has been found. Then you should check if it has been turned off manually by doing\

ccmake ..

in the build directory. You will then see all the control variables. Make sure NO_ROOT is turned OFF (meaning ROOT is enabled).

Finally, make sure you start "rmlogger" and not "mlogger". Only "rmlogger" contains the ROOT binding.

Stefan

> I am setting up Midas (v2.1) for a new experiment. We want to save the data in the ROOT format. We installed ROOT from source (v6.28/06), and ROOTSYS is set. When we compile Midas, it says that it found ROOT. We set up a second logger channel where we set the filename to run%05d.root, the format to ROOT, and the output to ROOT. Nevertheless, when starting a run, the logger writes the error that "channel '1' requested ROOT output, but mlogger is built without HAVE_ROOT". From the CMake file, I would assume that it is set automatically if ROOT is found. Do you have any idea why the mlogger does not find ROOT or save the data in the ROOT format?

when you build midas using "make cmake", it prints information about packages that it finds (or does not). please post this here. it would be even more helpful if you post the whole output of "make cmake" (make cmake >& make.log, post make.log here as attachment).

historically, this problem has been a major annoyance over the years, mlogger would not find ROOT when needed, will find the wrong ROOT when not needed or ROOT at run time will be different from ROOT at build time. "cmake" has been of no help in improving on this, only made all debugging more difficult.

K.O.

Good evening, 

I am setting up Midas (v2.1) for a new experiment. We want to save the data in the ROOT format. We installed ROOT from source (v6.28/06), and ROOTSYS is set. When we compile Midas, it says that it found ROOT. We set up a second logger channel where we set the filename to run%05d.root, the format to ROOT, and the output to ROOT. Nevertheless, when starting a run, the logger writes the error that "channel '1' requested ROOT output, but mlogger is built without HAVE_ROOT". From the CMake file, I would assume that it is set automatically if ROOT is found. Do you have any idea why the mlogger does not find ROOT or save the data in the ROOT format?

Thanks in advance for your ideas, input, and help. 

Cheers,
Ivo

The "multi" class driver takes care of that. It properly calls the SET and GET functions 
with the correct index. The code for that is in multi.cxx:105:

 device_driver(m_info->driver_input[i], CMD_GET,
               i - m_info->channel_offset_input[i],
               &m_info->var_input[i]);

The "channel_offset_input" and "channel_offset_output" store the first index of the 
channel in the overall ODB array (where inputs and outputs are staggered together), so 
the device_driver is always called with an index 0...n each for input and output, but 
with different commands CMD_GET and CMD_SET. You can take the mscbdev.cxx device driver 
as a working example.

Stefan

Dear all,
       I'm writing a very simple device driver starting from the nulldev.cxx 
example. 

I define an equipment as reported at the end of this message then, if all 
variables are Input variables, I define them with:

  mdevice device("myEquimpent", "Input", DF_INPUT | DF_MULTITHREAD, mydevice);
  device.define_var("Var1", 0.1);
  device.define_var("Var2", 0.1);
  ...

If all variables are output variables, I define them with:

  mdevice device("myEquipment", "Output", DF_OUTPUT | DF_MULTITHREAD, mydevice);
  device.define_var("Var1", 0.1);
  device.define_var("Var2", 0.1);

But I don't know what to do if I have mixed input and output variables in the same 
device. I think I can do:

  mdevice device_in("myEquipment", "Input", DF_INPUT | DF_MULTITHREAD, mydevice);
  device.define_var("Var1", 0.1);

  mdevice device_out("myEquipment", "Output", DF_OUTPUT | DF_MULTITHREAD, 
mydevice);
  device.define_var("Var2", 0.1);

but in this case, inside mydevice.cxx, I don't know how to distinguish Var1 and 
Var2, because they are both identified as channel 0.

Do you have any suggestion?

Thank you,
      Francesco




-------------------------------------------------------------------



   {"SourceMotor",                       /* equipment name */
    {7, 0,                       /* event ID, trigger mask */
     "SYSTEM",                  /* event buffer */
     EQ_SLOW,                   /* equipment type */
     0,                         /* event source */
     "MIDAS",                   /* format */
     TRUE,                      /* enabled */
     RO_ALWAYS,        /* read when running and on transitions */
     60000,                     /* read every 60 sec */
     0,                         /* stop run after this event limit */
     0,                         /* number of sub events */
     1,                         /* log history every event */
     "", "", ""} ,
    cd_multi_read,                 /* readout routine */
    cd_multi,                      /* class driver main routine */
   },

Thanks for the bug report Gennaro! 

I've fixed the code so that we'll now find the end-of-run ODB dump even if the user is already at the end of the file when they call get_eor_odb_dump().

Ben

An additional functionality has been implemented on the equipment table:

You can now select several elements by Ctrl/Shift-Click on their names, then change the 
first one. After a confirmation dialog, all selected variables are then set to the new 
value. This way one can very easily change all values to zero etc.

Stefan

Since a long time we tried to convert all "static" mhttpd-generated pages to 
dynamic JavaScript. With the new history panel editor we were almost there. Now I 
committed the last missing piece - the equipment display. This is shown when you 
click on some equipment on the main status page, or if you define some Alias with 

?cmd=eqtable&eq=Trigger

This is now a dynamic display, so the values change if they change in the ODB. The 
also flash briefly in yellow to visually highlight any change. In addition, these 
pages have a unit display, and some values can be edited. This is controlled by 
following settings:

/Equipment/<name>/settings/Unit <variable>

where <name> is the name of the equipment and <variable> the variable array name 
under /Equipment/<name>/Variables/<variable>

If the unit setting is not present, just a blank column is shown.

The other setting is

/Equipment/<name>/settings/Editable 

which may contain a comma-separated string of variables which can be editied on 
the equipment page.

In addition, one can save/export the equipment in a json file, which is the same 
as a ODB save of that branch. A load or import however only loads values into the 
ODB which are under the "Editable" setting above. This allows a simple editor for 
HV values etc.

Stefan