Thanks for reporting this problem. It has been fixed today, so the start command is only written if it's emtpy.

Stefan

> When making a link in the odb (web interface) a nice browser window pop's up. There is however not scrolling possible in the window. As a result, you can not reach a odb key if it is nested to deeply. 
> 
> Trying to type out the Link target in the field only allows for 32 characters

Thanks for reporting the bug with the pop-up not being able to scroll, I fixed that and committed the change.

I do however not understand the issue with 32 characters. The link NAME should not be more than 32 chars (which applies to all ODB keys). 
But if I try I can write more than 32 chars in the link target.

Stefan

The JavaScript function

mjsonrpc_db_save / mjonrpc_db_load

have been dropped from the API because they were not considered safe. Users 
should use now the new function

file_save_ascii()

and

file_load_ascii()

These function have the additional advantage that the file is not loaded 
directly into the ODB but goes into the JavaScript code in the browser, which 
can check or modify it before sending it to the ODB via mjsonrpc_db_paste(). 

Access of these functions is limited to <experiment>/userfiles/* where 
<experiment> is the normal MIDAS experiment directory defined by "exptab" or 
"MIDAS_DIR". This ensures that there is no access to e.g. system-level files. If 
you need to access a directory not under "userfile", us symbolic links.

These files can be combined with file_picker(), which lets you select files on 
the server interactively.

Stefan

Thanks for reporting this bug, I fixed it in the last commit.

Best,
Stefan

Since a long time we tried to convert all "static" mhttpd-generated pages to 
dynamic JavaScript. With the new history panel editor we were almost there. Now I 
committed the last missing piece - the equipment display. This is shown when you 
click on some equipment on the main status page, or if you define some Alias with 

?cmd=eqtable&eq=Trigger

This is now a dynamic display, so the values change if they change in the ODB. The 
also flash briefly in yellow to visually highlight any change. In addition, these 
pages have a unit display, and some values can be edited. This is controlled by 
following settings:

/Equipment/<name>/settings/Unit <variable>

where <name> is the name of the equipment and <variable> the variable array name 
under /Equipment/<name>/Variables/<variable>

If the unit setting is not present, just a blank column is shown.

The other setting is

/Equipment/<name>/settings/Editable 

which may contain a comma-separated string of variables which can be editied on 
the equipment page.

In addition, one can save/export the equipment in a json file, which is the same 
as a ODB save of that branch. A load or import however only loads values into the 
ODB which are under the "Editable" setting above. This allows a simple editor for 
HV values etc.

Stefan

An additional functionality has been implemented on the equipment table:

You can now select several elements by Ctrl/Shift-Click on their names, then change the 
first one. After a confirmation dialog, all selected variables are then set to the new 
value. This way one can very easily change all values to zero etc.

Stefan

The "multi" class driver takes care of that. It properly calls the SET and GET functions 
with the correct index. The code for that is in multi.cxx:105:

 device_driver(m_info->driver_input[i], CMD_GET,
               i - m_info->channel_offset_input[i],
               &m_info->var_input[i]);

The "channel_offset_input" and "channel_offset_output" store the first index of the 
channel in the overall ODB array (where inputs and outputs are staggered together), so 
the device_driver is always called with an index 0...n each for input and output, but 
with different commands CMD_GET and CMD_SET. You can take the mscbdev.cxx device driver 
as a working example.

Stefan

When you do "cmake .." in the build directory, you will see

-- MIDAS: Found ROOT version xxx in yyy

which will tell you that ROOT has been found. Then you should check if it has been turned off manually by doing\

ccmake ..

in the build directory. You will then see all the control variables. Make sure NO_ROOT is turned OFF (meaning ROOT is enabled).

Finally, make sure you start "rmlogger" and not "mlogger". Only "rmlogger" contains the ROOT binding.

Stefan

No, I'm not aware of this problem, but I suspect that your events somehow got corrupted. You can try the mdump utility
or the "Event Dump" web page to peek into your events, maybe you see an issue there. To give you more detailed information,
I would have to reproduce your problem, which is probably hard without your hardware.

Stefan

I cannot confirm that. I just tried myself with examples/experiment/frontend.cxx, removed the RO_ODB, and the trigger events did NOT get copied to the ODB.

Actually you can debug the code yourself. The relevant line is in mfe.cxx:2075:

   /* send event to ODB */
   if (pevent->data_size && (eq_info->read_on & RO_ODB)) {
      if (actual_millitime - eq->last_called > ODB_UPDATE_TIME) {
         eq->last_called = actual_millitime;
         update_odb(pevent, eq->hkey_variables, eq->format);
         eq->odb_out++;
      }
   }

so if read_on is equal 1, the function update_odb should never be called.

So the problem must be on your side.

Best,
Stefan

> - multiple subdetectors are taking test data during early commissioning 
> - a postgres db is a single sorce of run numbers.
> - test runs taken by different subsystems are assigned different [unique] run numbers and 
>   the data taken by the subsystem are identified not by the run number/dataset name , but 
>   by the run type, different for different susbsystems.

For that purpose I would not "mis-use" run numbers. Run number are meant to be incremented 
sequentially, like if you have a time-stamp in seconds since 1.1.1970 (Unix time). Intead, I 
would add additional attributes under /Experiment/Run Parameters like "Subsystem type", "Run 
mode (production/commissioning)" etc. You have much more freedom in choosing any number of 
attributes there. Then, send this attributes to your postgred db via "/Logger/Runlog/SQL/Links 
BOR". Then you can query your database to give you all runs of a certain subtype or mode.

See https://daq00.triumf.ca/MidasWiki/index.php/Logging_to_a_mySQL_database

Stefan

> Stefan, I don't think we're talking 'mis-use' - rather different subdetectors being commisisoned 
> at different locations, on an uncorrelated schedule, using independent run control (RC) instances. 
> At this point in time, we can't use a common RC instance. 
> The collected data, however, are written back into a common storage, and we need to avoid two 
> subdetectors  using the same run number. As all RC instances can connect to the same database and request a 
> run number from there, an external DB serving run numbers to multiple clients looks as a reasonable solution, 
> which provides unique run numbers for everyone. Of course, the run number gets incremented (although on the DB 
> server side), and of course different susbystems are assigned different subsystem types. 
> 
> So, in essense, it is about _where_ the run number is incremented - the RC vs the DB. 
> If there were a good strategy to implement a DB-based solution that w/o violating 
> first principles of Midas:), I'd be happy to contribute. It looks like a legitimate use case.

Ok, maybe attitude comes from the fact that I never used such a scheme in the last 30 years with midas.

If you go in this direction, there is an alternative to what Ben wrote: Use the sequencer to start a run.
The sequencer script can obtain a new run number from a central instance (e.g. by calling a shell script 
like 'curl ...' to obtain the new run number, then put it into /Runinfo/Run number as Ben wrote. This has
the advantage that the run is _started_ already with the correct number, so the history system is fine.

The script can then wait for n events, then stop the run etc. A sequencer script will also be necessary if
you want to configure your electronics (see next answer...)

Stefan

> The MIDAS state machine doesn't seem to have a state in between 'initialized' and 
> 'running'.
> In a larger detectors with multiple subsystems, the DAQ systems often have one more state:
> after ending a previous run and before starting a new one from the 'stopped' state, 
> one needs to make sure that all subdetectors are ready, or 'configured' for the new run. 
> So theat calls for a 'configure' step during which the detector (all subsystems in 
> parallel, to save the time) transitions from 'initialized'/'stopped' to 'configured' state, 
> from which it transitions to the 'running' state.
> 
> If one of the subdetectors fails to get configured, it could be excluded from the run 
> configuration and another attempt to reconfigure the system could be made without 
> starting a new run. Or an attempt could be made to troubleshoot and configure the 
> failed subsytem individually , with the rest subsystems waiting in a 'configured' state.
> 
> How does the logic of configuring the detector for the new run is implemented in MIDAS? 
> - it is a fairly common operational procedure, so I'm sure there should be a way 
> of doing that.

We have a similar requirement in our MEG experiment. Configuring your subdetectors can
be quite complex and therefore it's almost impossible to define a 'configure' step in
the run transition system to accommodate all corner cases.

Instead of a new state, we do everything through the sequencer:

- To start a run, we start a special sequencer script. We have different scripts for
calibration runs, data runs, special runs.

- When the user starts the script, they are asked for certain parameters, like number
of events, number of runs to take, how to configure the subdetectors, which subdetectors
to read out etc.

- The script then configures the whole experiment by setting everything in the ODB for
each equipment.

- The frontends connected to their equipment get a hotlink from the ODB and start the
configuration of the trigger etc. based on the parameters from the ODB

- The progress of the configuration is indicated by the frontend by writing back the
progress (like 0...100) into the ODB

- The script now waits for the progress to reach 100. It shows the current progress
on the sequencer page, so you see exactly where we are.

- If we have several subdetectors, each of them can publish a progress, and the script
can wait for an AND of all progress, or exclude one if it fails etc. Any logic is
possible there.

- Once all progresses are at 100, the run is finally started.

- If the mechanics of configuration become more elaborate, one can 'hide' it in
sub-modules of the script.

This scheme allows us to configure very different run modes, we use it in MEG since
many years (about 0.5M runs) and it works very nice.

Attached is our main script to start a full data run. You don't have to understand
all details, but it can give you a glimpse of what it's possible with the sequencer.
The function "ApplySettings" is the one waiting for the configuration flag in the ODB
(we simply use a boolean flag there). The code is:

SUBROUTINE ApplySettings
  ODBSET "/Equipment/Trigger/Settings/Reload all", y, 1
  WAIT seconds, 2
  WAIT ODBValue, "/Equipment/Trigger/Variables/Config busy", ==, y
ENDSUBROUTINE

Best,
Stefan

> - how does one communicate with an external shell script from MSL ? I looked at the MIDAS Sequencer page 
>   
>   https://daq00.triumf.ca/MidasWiki/index.php/Sequencer
> 
>   and didn't find an immediately obvious candidate among the MSL commands. 
>   The closest seems to be 
> 
>   'SCRIPT script [, a, b, c, ...]' 
> 
>   but I couldn't easily figure how to propagate the output of the script back to MIDAS. 
>   Let say, the script creates an ASCII file with the next run number. What is the easiest 
>   way to import the run number into ODB? - Should an external script spawn a [short-lived] 
>   MIDAS client ?  - That would work, but I'm almost sure there is a more straightforward solution. 

The output of the SCRTIP command is stored in the variable $SCRIPT_RESULT. Please pull midas to get this
new functionality.

Stefan

>    [that was not obvious from the documentation on MIDAS wiki, and adding a couple of clarifying 
>     sentences there would go long ways]

I added a sentence there. Please have a look. If you like more info, please write it yourself and send it to me. 
It's always better if that comes from users than from me.

 
> 2. if a script produces an ascii file with a known name, for example, 'a.odb', with the following two lines:

Use $SCRIPT_RESULT as described before.

Best,
Stefan

Dear beloved MIDAS users,

I'm happy to announce a "holiday update" for MIDAS. In countless hours, Zaher from 
PSI worked hard to introduce syntax highlighting in the midas script editor. In 
addition, there are additional features like a cleaner user interface, the option 
to see all variables also in non-debug mode and more. Have a look at the picture 
below, doesn't it beginning to look a lot like Christmas?

We have tested this quite a bit and went through many iterations, but no guarantee 
that it's flawless. So please report any issue here.

I wish you all a happy holiday season,
Stefan

> If I understand the msequencer code correctly:
> Under '/Sequencer/State/Path' the path can be defined from where the msequencer gets the files, generates the xml, etc.
> However, the new javascript code reads/writes the files to '<exp>/userfiles/sequencer/'
> 
> If the path in the ODB is different to '<exp>/userfiles/sequencer/', it leads to quite some unexpected behavior. If '<exp>/userfiles/sequencer/' is the place where things should go, the ODB entry of the msequencer and the internal handling should probably adopted, no?

Indeed there is a change in philosophy. Previously, /Sequencer/State/Path could point anywhere in the file system. This was considered a security problem, since one could access system files under /etc for example via the midas interface. When the new file API was 
introduced recently, it has therefor been decided that all files accessible remotely should reside under <exp>/userfiles. If an experiment needs some files outside of that directory, the experiment could define some symbolic link, but that's then in the responsibility of 
the experiment. 

To resolve now the issue between the sequencer path and the userfiles, we have different options, and I would like to get some feedback from the community, since *all experiments* have to do that change.

1) Leave thins as they are, but explain that everybody should modify /Sequencer/Stat/Path to some subdirectory of <exp>/userfiles/sequencer

2) Drop /Sequencer/State/Path completely and "hard-wire" it to <exp>/usefiles/sequencer

3) Make /Sequencer/State/Path relative to <exp>/userfiles. Like if /Sequencer/State/Path=test would then result to a final directory <exp>/userfiles/sequencer/test

I'm kind of tempted to go with 3), since this allows the experiment to define different subdirectories under <exp>/userfiles/sequencer/... depending on the situation of the experiment.

Best,
Stefan

> to plot time dependencies of the monitored detector parameters, say, voltages or temperatures,
> one needs to convert the coresponging ADC readings into floats.
> 
> One could think of two ways of doing that: 
> 
> - one can perform the ADC-->T or ADC-->V conversion in the MIDAS frontend, 
>   store their [float] values in the data bank, and plot precalculated parameters vs time
> 
> - one can also store in the data bank the ADC readings which typically are short's 
>   and convert them into floats (V's or T's) at the plotting time 
> 
> The first approach doubles the storage space requirements, and I couldn't find the place where 
> one would do the conversion, if stored were the 16-bit ADC readings.
> 
> I'm sure this issue has been thought about, so what is the "recommended MIDAS way" of performing 
> the ADC -> monitored_number conversion when making MIDAS history plots ?

Most experiment go with the second method. The front-end program converts all ADC reading into physicsl 
units, i.e. not only Volt, but even Degrees Centigrade or Tesla or whatever. The slow control part of 
midas then puts these number into /Equipment/<name>/Variables as "float", and the history system picks 
them up from there. This way your history is shown in physical units and not ADC count. Actually the 
recommended slow control framework (check the examples direcotory) does not rely on data banks, but 
puts values directly into the ODB. This is typically done faster, like once per second if a value 
changes, rather than slow control events which are generated maybe once per 10 seconds or once per 
minute. Usually the slow control values are only few compared with trigger data, so a factor of two 
there does not really matter. In the MEG experiment, we have like 400 GB of slow control data per year, 
but 400 TB of trigger data per year.

Best,
Stefan

First of all it's important to understand that the slow control system has nothing to do
with events. So if you look at event statistics, these are the events with the slow control
data sent to the midas data file, not the history database. So the logging period (the one you 
decreased from 60s to 10s to 6s) only affect the generation of events. 

What is important in your case is the number of events sent to the ODB. You see these in the
screen output of the slow control frontend (see attachment). This number show increase every
second.

I tried your modification (change nulldev with a sine function), and left the sc_fe.cxx
otherwise untouched. I then started with a fresh ODB ("rm /"). Started logger, mhttpd, sc_fe
and started a run. In the attachments is what I see. So I don't understand what your problem
is.

> > 3) Make /Sequencer/State/Path relative to <exp>/userfiles. Like if /Sequencer/State/Path=test would then result to a final directory <exp>/userfiles/sequencer/test
> > 
> > I'm kind of tempted to go with 3), since this allows the experiment to define different subdirectories under <exp>/userfiles/sequencer/... depending on the situation of the experiment.
> > 
> > Best,
> > Stefan
> 
> For me the option 3) seems the most coherent one.
> Andreas

Ok, I implemented option 3) above. This means everybody using the midas sequencer has to change /Sequencer/State/Path to an empty string and move the 
sequencer files under <exp>/userfiles/sequencer as a starting point. I tested most thing, including the INCLUDE statements, but there could still be
a bug here or there, so please give it a try and report any issue to me.

Best,
Stefan