Back Midas Rome Roody Rootana
  Midas DAQ System, Page 38 of 136  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
ID Date Author Topic Subjectup
  1908   13 May 2020 Stefan RittForumList of sequencer files
If you load a file into the sequencer from the web interface, you get a list of all files in that directory. 
This basically gives you a list of possible sequencer files. It's even more powerful, since you can 
create subdirectories and thus group the sequencer files. Attached an example from our 
experiment.

Stefan
Attachment 1: Screenshot_2020-05-13_at_9.11.55_.png
Screenshot_2020-05-13_at_9.11.55_.png
  1909   18 May 2020 Ruslan PodviianiukForumList of sequencer files
> If you load a file into the sequencer from the web interface, you get a list of all files in that directory. 
> This basically gives you a list of possible sequencer files. It's even more powerful, since you can 
> create subdirectories and thus group the sequencer files. Attached an example from our 
> experiment.
> 
> Stefan


Dear Stefan,

Thank you for the explanation.

Ruslan
  1910   19 May 2020 Ruslan PodviianiukForumList of sequencer files
> If you load a file into the sequencer from the web interface, you get a list of all files in that directory. 
> This basically gives you a list of possible sequencer files. It's even more powerful, since you can 
> create subdirectories and thus group the sequencer files. Attached an example from our 
> experiment.
> 
> Stefan

Dear Stefan,

Could you please answer one more question:

We have a custom webpage and trying to get list of files from the custom webpage and need jrpc command to show it 
in custom page. Is there a jrpc command to get this file list?

Thanks,
  1912   20 May 2020 Konstantin OlchanskiForumList of sequencer files
> 
> We have a custom webpage and trying to get list of files from the custom webpage and need jrpc command to show it 
> in custom page. Is there a jrpc command to get this file list?
> 

The rpc method used by sequencer web pages is "seq_list_files". How to use it, see resources/load_script.html.

To see list of all rpc methods implemented by your mhttpd, see "help"->"json-rpc schema, text format".

As general explanation, so far we have successfully resisted the desire to turn mhttpd into a generic NFS file
server - if we automatically give all web pages access to all files accessible to the midas user account, it is easy
to lose control of system security (i.e. bad things will happen if web pages can read the ssh private keys ~/.ssh/id_rsa and
modify ~/.ssh/authorized_keys). Generally it is impossible to come up with a whitelist or blacklist of "secrets" that
need to be "hidden" from web pages. But we did implement methods to access files from specific subdirectory trees
defined in ODB which hopefully do not contain any "secrets".

K.O.
  2294   25 Oct 2021 Francesco RengaForumLogger crash
Hello,
     I'm experiencing crashes of the mlogger program on the time scale of a couple 
of days. The only messages from MIDAS are:

05:34:47.336 2021/10/24 [mhttpd,INFO] Client 'Logger' (PID 14281) on database 
'ODB' removed by db_cleanup called by cm_periodic_tasks (idle 10.2s,TO 10s)
05:34:47.335 2021/10/24 [mhttpd,INFO] Client 'Logger' on buffer 'SYSMSG' removed 
by cm_periodic_tasks (idle 10.2s, timeout 10s)

Any suggestion to further investigate this issue?

Thank you very much,
         Francesco
  2295   25 Oct 2021 Stefan RittForumLogger crash
The short term solution would be to increase the logger timeout in the ODB under

/Programs/Logger/Watchdog timeout

and set it to 6000 (one minute). But that is curing just the symptoms. It would be 
interesting to understand the cause of this error. Probably the logger takes more than 10 
seconds to start or stop the run. The reason could be that the history grow too big (what 
we have right now in MEG II), or some disk problems. But that needs detailed debugging on 
the logger side.

Stefan
  2064   05 Jan 2021 Isaac Labrie BoulayBug ReportLogger: Disk nearly full.
Hi all,

I've ran into a problem where my experiment gets interrupted with a message from 
the logger saying that my disk is nearly full. This does not make sense to me 
because I have deleted almost all the data files from my data directory. I'm 
guessing that somewhere the ODB perceives that the directory is full when in 
reality its not.

Here is the exact message:

[ODBEdit,INFO] Run #252 stopped

09:22:19 [Logger,TALK] disk nearly full, stopping the run

09:22:19 [Logger,ERROR] [mlogger.cxx:4475:log_write,ERROR] Disk '/home/caendaq/A       
NIS/data/run00252.mid.lz4' is almost full: 81 MiBytes free out of 922497 MiBytes       
, stopping the run

Does any body have a solution for this? Thanks so much.

Isaac
  2066   06 Jan 2021 Stefan RittBug ReportLogger: Disk nearly full.
The logger simple requests the disk free space level from the operating system in the same 
way as the "df" command does. Can you do a "df" on your system? I have seen that some file 
systems free up space not immediately if you delete files, but some times later (like 24h).

Stefan
  2068   06 Jan 2021 Isaac Labrie BoulayBug ReportLogger: Disk nearly full.
> The logger simple requests the disk free space level from the operating system in the same 
> way as the "df" command does. Can you do a "df" on your system? I have seen that some file 
> systems free up space not immediately if you delete files, but some times later (like 24h).
> 
> Stefan

Thanks Stefan. Yes the files were still held open by some processes. It's solved now.

Cheers.

Isaac
  1077   24 Jul 2015 Konstantin OlchanskiInfoMAX_EVENT_SIZE removed
The define for MAX_EVENT_SIZE was removed from midas.h.

Replacing it is DEFAULT_MAX_EVENT_SIZE set to 4 MiBytes and DEFAULT_BUFFER_SIZE 
set to 32 MiBytes.

For a long time now MIDAS does not have hardcoded maximum event size and buffer size 
and this change merely renames the define to reflect it's current function.

The actual maximum event size is set by ODB /Experiment/MAX_EVENT_SIZE.

The actual event buffer sizes are set by ODB "/Experiment/Buffer sizes/SYSTEM" & co

K.O.
  1271   15 Apr 2017 Konstantin OlchanskiBug ReportMAX_STRING_LENGTH, stop form odbedit broken
> 
> I shall check on the use of MAX_STRING_LENGTH at least in odb itself...
> 

Ok, I looked at the use of MAX_STRING_LENGTH in ODB (odb.c):

a) it is not used in any critical places for the database itself, so it is not a limit on maximum length of TID_STRING data. good.
b) it is used in the code for saving/loading odb from .odb files (old format), not sure how it works against overlong strings, but probably 
truncates/corrupts/crashes.
c) it is used in the code for saving odb to odb.xml files. Overlong strings are truncated (I added a message about it).
d) code for loading/saving to json files handles overlong strings okey.
e) odbedit "ls" truncates overlong strings, mhttpd has some oddities against overlong strings.
f) db_sprintf() truncates string text to MAX_STRING_LENGTH to avoid output buffer overflow (should use db_snprintf() instead).

Conclusion, overlong strings should be okey, but do not use the old .odb and .xml save files. (mlogger saves odb to output .mid file in xml 
format, we should switch it to use json format).

> > CSS File = STRING : [1024] mhttpd.css
> > Sqlite dir = STRING : [1024]
> > History dir = STRING : [1024]
> > Sound = STRING : [1000] alarm.mp3
> > are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)

So these should not cause any corruption or problem unless actual content length exceeds 255 bytes,
even then they are okey if odb is only saved and loaded into json files.

> > 1) I get the error message that some strings are too long (exceeding
> > MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
> > variables this is.

this is in db_check_record(), where it compares odb content with user-supplied data descriptions (there is no system-supplied
data descriptions with strings longer than MAX_STRING_LENGTH).

so I think what happened is you created a data structure with overlong strings, passed it to db_paste() or something,
db_check_record() complained about it, and db_paste() corrupted memory.

> > 
> > 2) After this reload, essentially nothing is working anymore. Any client I tried to start just crashed.
> > 

Somebody corrupted some shared memory, most likely it was db_paste() corrupted odb shared memory.

K.O.
  1272   15 Apr 2017 Konstantin OlchanskiBug ReportMAX_STRING_LENGTH, stop form odbedit broken
> > 
> > I shall check on the use of MAX_STRING_LENGTH at least in odb itself...
> > 
> 
> Ok, I looked at the use of MAX_STRING_LENGTH in ODB (odb.c):
> 

Fixed a small buglet, now saving and reloading odb in the old ".odb" format will silently truncate all overlong strings to 256 bytes. (I think it always did that).

K.O.
  1274   19 Apr 2017 Stefan RittBug ReportMAX_STRING_LENGTH, stop form odbedit broken
ODB name lengths (the name of a key) are limited to 256 characters, the length of strings in the ODB should NOT be limited. At some point we wanted to have complete web pages inside the ODB, 
which for sure are longer than 256 characters. While this was the idea, I see now that db_paste & co. is hopelessly broken. To fix it, everything should be changed to std::string which is in my opinion 
the only 'clean' solution. That would also remove the cumbersome strlcpy and strlcat.

But looking at odb.c, replacing everything with std::string would probably take a brave programmer a couple of weeks. Not sure if we should dive into that adventure right now. The quick fix would be:

a) The strings "CSS File", "Sqlite dir" etc. reported below get reduced to 256 characters (MAX_STRING_LENGTH). The value of 256 characters came from the file system limitation in linux (some many 
years ago), where a full path of a file could not exceed 256 characters. Not sure if this limit is still valid today, but having all file names in the ODB limited to 256 characters is maybe not a bad idea 
anyhow (who wants to type in file names with more than 256 characters ???).

b) Change the max string length in db_paste to 1024 to cover the few exceptions above.


If we go with a), KO has to change his ODB file names, in case of b) I can do the change.

So what is your opinion?

Best regards,
Stefan

> > 
> > I shall check on the use of MAX_STRING_LENGTH at least in odb itself...
> > 
> 
> Ok, I looked at the use of MAX_STRING_LENGTH in ODB (odb.c):
> 
> a) it is not used in any critical places for the database itself, so it is not a limit on maximum length of TID_STRING data. good.
> b) it is used in the code for saving/loading odb from .odb files (old format), not sure how it works against overlong strings, but probably 
> truncates/corrupts/crashes.
> c) it is used in the code for saving odb to odb.xml files. Overlong strings are truncated (I added a message about it).
> d) code for loading/saving to json files handles overlong strings okey.
> e) odbedit "ls" truncates overlong strings, mhttpd has some oddities against overlong strings.
> f) db_sprintf() truncates string text to MAX_STRING_LENGTH to avoid output buffer overflow (should use db_snprintf() instead).
> 
> Conclusion, overlong strings should be okey, but do not use the old .odb and .xml save files. (mlogger saves odb to output .mid file in xml 
> format, we should switch it to use json format).
> 
> > > CSS File = STRING : [1024] mhttpd.css
> > > Sqlite dir = STRING : [1024]
> > > History dir = STRING : [1024]
> > > Sound = STRING : [1000] alarm.mp3
> > > are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)
> 
> So these should not cause any corruption or problem unless actual content length exceeds 255 bytes,
> even then they are okey if odb is only saved and loaded into json files.
> 
> > > 1) I get the error message that some strings are too long (exceeding
> > > MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
> > > variables this is.
> 
> this is in db_check_record(), where it compares odb content with user-supplied data descriptions (there is no system-supplied
> data descriptions with strings longer than MAX_STRING_LENGTH).
> 
> so I think what happened is you created a data structure with overlong strings, passed it to db_paste() or something,
> db_check_record() complained about it, and db_paste() corrupted memory.
> 
> > > 
> > > 2) After this reload, essentially nothing is working anymore. Any client I tried to start just crashed.
> > > 
> 
> Somebody corrupted some shared memory, most likely it was db_paste() corrupted odb shared memory.
> 
> K.O.
  1275   19 Apr 2017 Stefan RittBug ReportMAX_STRING_LENGTH, stop form odbedit broken
> Fixed a small buglet, now saving and reloading odb in the old ".odb" format will silently truncate all overlong strings to 256 bytes. (I think it always did that).

Not sure that we want that. There might be cases where people want to store long strings. I would remove the truncation completely when saving .odb or .xml files, and fix the load routines to 
deal with overlong strings.

Stefan
  1276   22 Apr 2017 Konstantin OlchanskiBug ReportMAX_STRING_LENGTH, stop form odbedit broken
> > Fixed a small buglet, now saving and reloading odb in the old ".odb" format will silently truncate all overlong strings to 256 bytes. (I think it always did that).
> 
> Not sure that we want that. There might be cases where people want to store long strings. I would remove the truncation completely when saving .odb or .xml files, and fix the load routines to 
> deal with overlong strings.
> 

Since I just looked at the code for reading/writing .odb format, I see that it uses fixed size buffer for reading lines from a file,
currently 2*MAX_STRING_LENGTH). I am not in the mood to rewrite and retest all that code. Never looked at the xml reader,
probably has same problem (xml writer truncates long strings via truncation in db_sprintf()).

Since we already have the json odb reader/writer that handles unlimited string length correctly (also handles unicode and
unusual odb names), perhaps we should make json as the default and be done with it.

K.O.
  1277   22 Apr 2017 Konstantin OlchanskiBug ReportMAX_STRING_LENGTH, stop form odbedit broken
> ODB name lengths (the name of a key) are limited to 256 characters, the length of strings in the ODB should NOT be limited.

Right, I was not ever aware of such limitation until I just now looked at the .odb and .xml writing code. Definitely string length
is truncated to MAX_STRING_LENGTH on writing, chokes or truncates on reading.

The new json reader/writer handles overlength strings correctly. I would say we should deprecate the old formats and go forward
with json. Most current software can work with json data much easier than xml or custom .odb.

> I see now that db_paste & co. is hopelessly broken. To fix it, everything should be changed to std::string which is in my opinion 
> the only 'clean' solution. That would also remove the cumbersome strlcpy and strlcat.

Yes, that's the code for reading .odb format.

>
> But looking at odb.c, replacing everything with std::string would probably take a brave programmer a couple of weeks. Not sure if we should dive into that adventure right now.
>

I agree. Too much of an adventure.

Simpler solution could be add a db_get_data(), db_get_value() that allocates a data buffer of correct size (user has to remember to free it).

> a) The strings "CSS File", "Sqlite dir" etc. reported below get reduced to 256 characters (MAX_STRING_LENGTH).

We should fix the inconsistency, my vote is it should be either MAX_STRING_LENGTH or PATH_MAX (from limits.h).

K.O.
  1299   06 Jun 2017 Konstantin OlchanskiBug ReportMAX_STRING_LENGTH, stop form odbedit broken
> ... the xml reader, probably has same problem
> ... xml writer truncates long strings via truncation in db_sprintf()

Removed truncation of overlong strings in the xml writer and confirmed that xml reader handles them correctly (always loaded overlong strings correctly).

Both JSON and XML odb dumps now handle strings of unlimited size correctly.

K.O.
  901   13 Sep 2013 Carl BlaksleyForumMIDAS CITATION
Dear MIDAS programmers,

I have been using your software in my lab (APC, Paris)
to run our data acqusition system. It is very robust and flexible.s

I would like to give you the large amount of credit which you are due.
How should I cite both MIDAS and ROODY? I have not been able to find any
information in the usual places.

Cheers, and thanks for the great program!
-Carl
  902   13 Sep 2013 Konstantin OlchanskiForumMIDAS CITATION
> 
> I have been using your software in my lab (APC, Paris)
> to run our data acqusition system. It is very robust and flexible.s
> 
> I would like to give you the large amount of credit which you are due.
> How should I cite both MIDAS and ROODY? I have not been able to find any
> information in the usual places.
> 

Good to hear from a happy user.

I think the best way to give us credit is to recommend MIDAS to 10 of your friends.

For MIDAS citations, I think Pierre and Stefan have a standard one somewhere, we should have it linked from 
midas.triumf.ca.

For ROODY citations, I am not sure we have one. The idea behind it was to make a ROOT version of PAW++. 
Main authors are Greg King (UBC COOP student, where is he now?), Joe Chuma (also author of PHYSICA, 
R.I.P.), Pierre Amaudruz, myself and a few others.

K.O.
  903   13 Sep 2013 Stefan RittForumMIDAS CITATION
> Dear MIDAS programmers,
> 
> I have been using your software in my lab (APC, Paris)
> to run our data acqusition system. It is very robust and flexible.s
> 
> I would like to give you the large amount of credit which you are due.
> How should I cite both MIDAS and ROODY? I have not been able to find any
> information in the usual places.
> 
> Cheers, and thanks for the great program!
> -Carl

The standard citation for midas is a link to

http://midas.psi.ch

At the moment this points automatically to http://midas.triumf.ca, so both institutes are credited.

/Stefan
ELOG V3.1.4-2e1708b5