> We're trying to install the SuperCDMS version of MIDAS on a Rocky 9.4 Virtual 
> Machine and are getting a persistent error when we run mserver.  As far as I 
> know there are minimal changes between this and the MIDAS branch, but Ben Smith 
> may have more to say on this.

For reference, "the SuperCDMS version of MIDAS" is just a fork that no longer has any meaningful differences vs the main MIDAS repo, but we only pull updates infrequently after testing a bunch. We last pulled from the develop branch in November 2023. But that should be irrelevant here as semaphore code hasn't been touched for a very long time.

We're running Alma 9.4 on a machine at TRIUMF and the same version of midas works fine there (Amy, you may already have access to scdms-zeus). I believe Alma and Rocky should be basically identical for this.

So the questions are:
* Have you tried other midas programs, or only mserver? E.g. did odbedit and mhttpd work?
* If other programs work, have you been running them all as the same user? In particular, if you ran one program as root and another as an unprivileged user, then you will likely get odd permissions issues.
* What do you see if you run `ls -l /dev/shm` and `ls -l ~/packages/SuperCDMS_DAQ/MidasDAQ/online/.*SHM`? (Or wherever your online dir is for the 2nd one).
* Did you follow the full instructions for recovering from a corrupt ODB? https://daq00.triumf.ca/MidasWiki/index.php/FAQ#How_to_recover_from_a_corrupted_ODB   In particular the bit about running odbinit with the --cleanup flag?

> I'm using MIDAS javascript interface (mjsonrpc) to communicate with a frontend from a custom web page 
> and observe that the if the frontend's response exceeds certain number of bytes, it is always truncated. 
> 
> MIDAS C/C++ RPC interface allows users to specify the max response length :
> 
> https://daq00.triumf.ca/MidasWiki/index.php/Remote_Procedure_Calls_(RPC)#C++_2
> 
> How would one do the same from with mjsonrpc ? 

I just documented the max_reply_length (javascript) and max_len (python) parameters on that page. Both are optional and default to 1024 bytes.

I also added a link to the full mjsonrpc schema https://daq00.triumf.ca/MidasWiki/index.php/Mjsonrpc#Schema_(List_of_all_RPC_methods) . You can also find the auto-generated schema on any midas installation by going to the "Help" webpage served by mhttpd and clicking the "JSON-RPC schema" > "text table format" link.

> Valid events are:
> Enter event name:
> 
> was printed out, which signified that no events were found. No errors were displayed.

I can't reproduce this. I made a brand new experiment, started mlogger/mhttpd/fetest, then ran the same program. I get:

```
$ python basic_hist_script.py
Valid events are:
* Run transitions
* test_slow/data
Enter event name: 
```

Are you sure you ran the python program after running mlogger and not before? Can you try again after restarting mlogger? And can you verify that your python is connecting to the correct experiment if you have multiple experiments defined?

I tested with python 3.12.8 and 3.13.1, and am on MacOS 14.5, but I can't imagine those differences matter.

The python interface is a trivial wrapper around the C++ function, so the only python-specific thing that would result in an empty list is extracting an integer from a ctypes reference. If that's broken in your version then I don't think any of the midas python code would be working.

Unfortunately I again cannot reproduce this:

$ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
Valid events are:
* Run transitions
* test_slow/data
Enter event name: test_slow/data
Valid tags for test_slow/data are:
* data
Enter tag name: data
Event/tag test_slow/data/data has 1 elements
How many hours: 1
Interval in seconds: 1
78 entries found
2025/03/17 17:00:56 => 98.097391
2025/03/17 17:00:57 => 98.982151
2025/03/17 17:00:58 => 99.589187
2025/03/17 17:00:59 => 99.926821
2025/03/17 17:01:00 => 99.989878
2025/03/17 17:01:01 => 99.778216
2025/03/17 17:01:02 => 99.292485
.......


I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.



1) Run code directly in the python interpreter: 

Can you run the following and send the output please?

```
import midas.client
c = midas.client.MidasClient("history_test")
data = c.hist_get_recent_data(1,1,"test_slow/data","data")
print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
```

For me, I get:
event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441



2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?



3) Debug the python/C conversions.

In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():

```
        print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
```

Then run the following and send the output:

```
import midas.client
c = midas.client.MidasClient("history_test")
data = c.hist_get_recent_data(1,1,"test_slow/data","data")
```

For me, I get:
c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.


Thanks,
Ben

Hi,

I have just installed Midas and set-up the ODB for a SuperCDMS test-facility (on
a SL6.7 machine). All works fine except that I receive the following error message:

[mserver,ERROR] [odb.c:944:db_validate_db,ERROR] Warning: database data area is
100% full

Which is puzzling for the following reason:

-> I have created the ODB with: odbedit -s 4194304
-> Checking the size of the .ODB.SHM it says: 4.2M
-> When I save the ODB as .xml and check the file's size it says: 1.1M
-> When I start odbedit and check the memory usage issuing 'mem', it says: 
...
Free Key area: 1982136 bytes out of 2097152 bytes
...
Free Data area: 2020072 bytes out of 2097152 bytes
Free: 1982136 (94.5%) keylist, 2020072 (96.3%) data

So it seems like nearly all memory is still free. As a test I created more
instances of one of our front-ends and checked 'mem' again. As expected the free
memory was decreasing. I did this ten times in fact, reaching

...
Free Key area: 1440976 bytes out of 2097152 bytes
...
Free Data area: 1861264 bytes out of 2097152 bytes
Free: 1440976 (68.7%) keylist, 1861264 (88.8%) data

So I could use another >20% of the database data area, which is according to the
error message 100% (resp. >95%) full. Am I misunderstanding the error message?
I'd appreciate any comments or ideas on that subject!

Thanks, Belina

I am having difficulty getting the custom scripts to work within the updated MIDAS. Before the 
update I was using something like : 

<input type=submit name=customscript value="test">

on my custom page to run a script under /CustomScript/test, however, with the update to 
MIDAS this is no longer working. I can't find any information about this functionality being 
updated in the latest version - has this changed? Or should it still work? 

Thanks,
Becky (g-2 DAQ)

I want to report a bug in the ROOT build process that might be relevant to the midas implementation. I had an annoying failure to build root 6.18 (current pro version) with a misleading error message about a fault in the root code. It turned out this was a cmake problem, and the error was from my cmake version being older than 3.14, which is quite recent. Took a bit of searching to find this.

I recommend when the cmake version is distributed that the instructions include the required cmake version. Developers are generally working well ahead of what is available in the older OS's.

I am working for the SuperCDMS collaboration on some DAQ issues for our upcoming
SNOLAB installation.  So far, the MIDAS sequencer seems to be a good paradigm
for us to do procedural tasks for our detectors and data running interspersed
with other protocols.  

In our testing we've found that the sequencer works very well for this kind of
activity, although it would be useful to have a kind of scripted "abort" for
when something goes wrong -- especially if the user selects to abort a run
sequence. 

Because the sequencer is setting various detector parameters to a certain value
before performing the tasks, the values will never be restored if the user
aborts the sequence.  Instead, perhaps there can be a portion of a MIDAS
sequence script which is instructed to happen on an abort.  Perhaps something
like all commands after a given tag like:

ON ABORT:

get run on a user-initiated abort?  

The current versions of mlogger SVN 4215 is flooding our message system with
stuff like

> Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events

This is fatal to us and blowing up the midas.log like hell. I would prefer if
one could flag these kind of messages (ODB /Logger/..), i.e. enable and disable
it. At the moment I have to comment it out in the source code since we cannot
work with it.

Cheers,
  Andreas 

There is an obvious "unwanted feature" in this version of the mhttpd. It writes the
"plot time" into the gif (mhttpd, if-statement starting in line 8853). 

Please check this obvious things more carefully in the future before submitting code. ;-)

> mhttpd and mlogger have been updated with potentially troublesome changes.
> Before using these latest versions, please make a backup of your ODB. This is
> svn revisions 4434 (mhttpd.c) and 4435 (mlogger.c).
> 
> These new features are now available:
> - a "feature complete" implementation of "history in an SQL database". We use
> this new code to write history data from the T2K test setup in the TRIUMF M11
> beam line to a MySQL database (mlogger) and to make history plots directly from
> this database (mhttpd). We still write normal midas history files and we have a
> utility to import midas .hst files into an SQL database (utils/mh2sql). The code
> is functional, but incomplete. For best SQL database data layout, you should
> enable the "per variable history" (but backup your ODB before you do this!). All
> are welcome to try it, kick the tires, report any problems. Documentation TBW.
> - experimental implementation of "ODBRpc" added to the midas javascript library
> (ODBSet, ODBGet & co). This permits buttons on midas "custom" web pages to
> invoke RPC calls directly into user frontend programs, for example to turn
> things on or off. Documentation TBW.
> - the mlogger/mhttpd implementation of /History/Tags has proved troublesome and
> we are moving away from it. The SQL database history implementation already does
> not use it. During the present transition period:
> - mlogger and mhttpd will now work without /History/Tags. This implementation
> reads history tags directly from the history files themselves. Two downsides to
> this: it is slower and tags become non-persistent: if some frontends have not
> been running for a while, their variables may vanish from the history panel
> editor. To run in this mode, set "/History/DisableTags" to "y". Existing
> /History/Tags will be automatically deleted.
> - for the above 2 reasons, I still recommend using /History/Tags, but the format
> of the tags is now changed to simplify management and reduce odb size. mlogger
> will automatically convert the tags to this new format (this is why you should
> make a backup of your ODB).
> - using old mlogger with new mhttpd is okey: new mhttpd understands both formats
> of /History/Tags.
> - using old mhttpd with new mlogger is okey: please set ODB
> "/History/CreateOldTags" to "y" (type TID_BOOL/"boolean") before starting mlogger.
> 
> K.O.

I have the following problems after updating to midas SVN 4936: the history 
system (web-page via mhttpd) seems to stop working. I checked the history files 
themself and they are indeed written, except that the events ID's are not the 
same anymore (I mean the ones defined under /Equipment/XXX/Common/Event ID), 
rather the mlogger seems to choose an ID by itself.
Currently the only way to get things working again was to recompile midas with 
adding -DOLD_HISTORY to the CFLAGS which is troublesome since it is likely to be 
forgotton with the next SVN update. When looking into the SVN I have the 
impression there is something going on concerning the history system, however
I couldn't find any documentation.
What is the best practice for the future, in order not to run into any problems 
but still being able to look at the old history (also from within the web-page 
via mhttpd)?

1. If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd. Wouldn't it be beneficial to have ODB entries where to get is from? This way people could change the look and feel more freely.
   
2. Especially the look and feel of the runlog is unsatisfactorily from my point of view. See . The old style was much more readable. I could recover the old style look and feel by slightly changing the mhttpd.cxx where I changed in show_rawfile(const char*) "dialogTable" to "runlogTable" in the table class. This way I could tinker around with the mhttpd.css by adding the following stuff there:
   
   • adding .runlogTable in line 289 :
     
   • adding some style information for the runlogTable :
     

Understand me right, I mostly like the new style, except the runlog as reported.
Attached you will find the diff's you were asking for. But as pointed out, I
haven't worked so far on CSS and hence this should be checked!!

I understand that the mhttpd.js needs to be the default one, however, mhttpd.css
might be left to the end-user to adopt to their specific needs. I shortly
checked in the mhttpd demon. It checks for the resources path in the ODB. If it
also would check for a CSS name, mhttpd.css could be changed/adopted by the
end-users without breaking things (at least it would then be their one business).

> > If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd.
> 
> mhttpd.css is served from $MIDASSYS/resources/mhttpd.css. The actual path is
reported on the mhttpd 
> "help" page.
> 
> (I think the internal mhttpd.css and mhttpd.js should be removed as no longer
useful - nothing will work 
> right if the real mhttpd.js and mhttpd.css cannot be served).
> 
> > Especially the look and feel of the runlog is unsatisfactorily from my point
of view.
> 
> persons in charge of implementing the CSS stuff failed to convert quite a few
pages, for example, the elog 
> and the history editor pages were left completely broken. (mostly fixed now).
> 
> so thank you for reporting the runlog breakage, I hope Stefan & co can fix it
quickly. (I cannot do - I have 
> have no runlog pages on any of my test experiments).
> 
> > the old style was much more readable.
> 
> I think the new style is not too bad, except for a few visual artefacts here
and there, the general comment 
> that CSS is too complicated and hard to debug and the fact that over-subtle
colouring yields inconsistent 
> visuals between different monitors and ambient lighting conditions. (persons
who select the colours always 
> respond that "but to me, it looks just fine on my laptop", making it hard to
resolve any issues).
> 
> > I could recover the old style look and feel by slightly changing the mhttpd.cxx
> 
> If you post the patches that fix it for you, I can commit them to midas. (git
diff | mail olchansk@triumf.ca).
> 
> K.O.

16:07:06 [Logger,INFO] Program Logger on host lem00 started
[local:nemu:S]/>q
[nemu@lem00 2014]$ odbedit -e nemu
odbedit: src/odb.c:753: db_update_open_record: Assertion `xkey->notify_count == pkey->notify_count' failed.
Aborted