> equipment names are 'mu2edaq09:DTC0' and 'mu2edaq09:DTC1'

I think all names permitted for ODB keys are allowed as equipment names, any valid UTF-8,
forbidden chars are "/" (ODB path separator) and "\0" (C string terminator). Maximum length
is 31 byte (plus "\0" string terminator). (Fixed length 32-byte names with implied terminator
are no longer permitted).

The ":" character is used in history plot definitions and we are likely eventually change that,
history event names used to be pairs of "equipment_name:tag_name" but these days with per-variable
history, they are triplets "equipment_name,variable_name,tag_name". The history plot editor
and the corresponding ODB entries need to be updated for this. Then, ":" will again be a valid
equipment name.

I think if you disable the history for your equipments, MIDAS will stop complaining about ":" in the name.

K.O.

> > > For ODB keys of type TID_KEY, the value num_values IS the number of subkeys. 
> > 
> > this logic makes sense, however it doesn't seem to be consistent with the printout of the test example
> > at the end of https://daq00.triumf.ca/elog-midas/Midas/240203_095803/a.cc . The printout reports 
> > 
> > key.num_values = 1, but the actual number of subkeys = 6, and all subkeys being of TID_KEY type
> > 
> > I'm certain that the ODB subtree in question was not accessed concurrently during the test.
> 
> You are right, num_values is always 1 for TID_KEYS. The number of subkeys is stored in 
> 
>   ((KEYLIST *) ((char *)pheader + pkey->data))->num_keys
> 
> Maybe we should add a function to return this. But so far db_enum_key() was enough.
> 
> Stefan

I would rather add a function that atomically returns an std::vector<KEY>. number of entries
is vector size, entry names are in key.name. If you need to do something with an entry,
like iterate a subdirectory, you have to go by name (not by HNDLE), and if somebody deleted
it, you get an error "entry deleted, tough!", (HNDLE becomes invalid without any error message about it, 
subsequent db_get_data() likely returns gibberish, subsequent db_set_data() likely corrupts ODB).

K.O.

> > You are right, num_values is always 1 for TID_KEYS. The number of subkeys is stored in 
> >   ((KEYLIST *) ((char *)pheader + pkey->data))->num_keys
> > Maybe we should add a function to return this. But so far db_enum_key() was enough.

Hmm... is there any use case where you want to know the number of directory entries, but you will not iterate 
over them later?

K.O.

> Hmm... is there any use case where you want to know the number of directory entries, but you will not iterate 
> over them later?

I agree. 

One more way to iterate over subkeys by name is by using the new odbxx API:


   midas::odb tree("/Test/Settings");
   for (midas::odb& key : tree)
      std::cout << key.get_name() << std::endl;


Stefan

> $ cat mhf_1697445335_20231016_run_transitions.dat
> event name: [Run transitions], time [1697445335]
> tag: tag: /DWORD 1 4 /timestamp
> tag: tag: UINT32 1 4 State
> tag: tag: UINT32 1 4 Run number
> record size: 12, data offset: 1024
> ...
> 
> data is in fixed-length record format. from the file header, you read "record size" is 12 and data starts at offset 1024.
> 
> the 12 bytes of the data record are described by the tags:
> 4 bytes of timestamp (DWORD, unix time)
> 4 bytes of State (UINT32)
> 4 bytes of "Run number" (UINT32)
> 
> endianess is "local endian", which means "little endian" as we have no big-endian hardware anymore to test endian conversions.
> 
> file format is designed for reading using read() or mmap().
> 
> and you are right mhdump, does not work on these files, I guess I can write another utility that does what I just described and spews the numbers to stdout.
> 
> K.O.


Thanks for the answer. As this FILE system is advertised as the new default (eog:2617), this format does merit some more WIKI info.

> > Hmm... is there any use case where you want to know the number of directory entries, but you will not iterate 
> > over them later?
> 
> I agree. 

here comes the use case: 

I have a slow control frontend which monitors several DAQ components - software processes. 
The components are listed in the system configuration stored in ODB, a subkey per component.

Each component has its own driver, so the length of the driver list, defined by the number of components, 
needs to be determined at run time.

I calculate the number of components by iterating over the list of component subkeys in the system configuration, 
allocate space for the driver list, and store the pointer to the driver list in the equipment record.

The approach works, but it does require pre-calculating the number of subkeys of a given key.

-- regards, Pasha

> Hi,
> 
> I have just installed Midas and set-up the ODB for a SuperCDMS test-facility (on
> a SL6.7 machine). All works fine except that I receive the following error message:
> 
> [mserver,ERROR] [odb.c:944:db_validate_db,ERROR] Warning: database data area is
> 100% full
> 
> Which is puzzling for the following reason:
> 
> -> I have created the ODB with: odbedit -s 4194304
> -> Checking the size of the .ODB.SHM it says: 4.2M
> -> When I save the ODB as .xml and check the file's size it says: 1.1M
> -> When I start odbedit and check the memory usage issuing 'mem', it says: 
> ...
> Free Key area: 1982136 bytes out of 2097152 bytes
> ...
> Free Data area: 2020072 bytes out of 2097152 bytes
> Free: 1982136 (94.5%) keylist, 2020072 (96.3%) data
> 
> So it seems like nearly all memory is still free. As a test I created more
> instances of one of our front-ends and checked 'mem' again. As expected the free
> memory was decreasing. I did this ten times in fact, reaching
> 
> ...
> Free Key area: 1440976 bytes out of 2097152 bytes
> ...
> Free Data area: 1861264 bytes out of 2097152 bytes
> Free: 1440976 (68.7%) keylist, 1861264 (88.8%) data
> 
> So I could use another >20% of the database data area, which is according to the
> error message 100% (resp. >95%) full. Am I misunderstanding the error message?
> I'd appreciate any comments or ideas on that subject!
> 
> Thanks, Belina

This is an old post, but I encountered the same error message recently and was looking for a 
solution here. Here's how I solved it, for anyone else who finds this: 
The size of .ODB.SHM was bigger than the maximum ODB size (4.2M > 4194304 in Belina's case). For us, 
the very large odb size was in error and I suspect it happened because we forgot to shut down midas 
cleanly before shutting the computer down. Using odbedit to load a previously saved copy of the ODB 
did not help me to get .ODB.SHM back to a normal size. Following the instructions on the wiki for 
recovery from a corrupted odb, 
https://daq00.triumf.ca/MidasWiki/index.php/FAQ#How_to_recover_from_a_corrupted_ODB, (odbinit with --cleanup option) should 
work, but didn't for me. Unfortunately I didn't save the output to figure out why. My solution was to manually delete/move/hide 
the .ODB.SHM file, and an equally large file called .ODB.SHM.1701109528, then run odbedit again and reload that same saved copy of my ODB. 
Manually changing files used by mserver is risky - for anyone who has the same problem, I suggest trying odbinit --cleanup -s 
<yoursize> first.

Dear MIDAS Experts,

I'm having an odd problem when trying to display an integer stored in ODB on a custom 
web page:  the hex specifier, "%x", displays integers as if it were "%d" .

- attachment 1 shows the layout and the contents of the ODB sub-tree in question
- attachment 2 shows the web page as it is displayed
- attachment 3 shows the snippet of html/js producing the web page

I bet I'm missing smth trivial - an advice is greatly appreciated! 

Also, is there an equivalent of a "0x%04x" specifier to have the output formatted 
into a fixed length string ?  

-- thanks, regards, Pasha  

Thanks for reporting that bug. I fixed it and committed the change to the develop branch.

Stefan

Hi Stefan (and Ben),

thanks for reacting so promptly - your commits on Bitbucket fixed the problem.

For those of us who knows little about how the web browsers work: 

- picking up the fix required flushing the cache of the MIDAS client web browser - apparently the web browser 
  I'm using - Firefox 115.6 - cached the old version of midas.js but wouldn't report it cached and wouldn't load 
  the updated file on its own.

-- thanks again, regards, Pasha

Hello there,

I am working on a task that involves some ODB changes that happen within 20-500 
ms. The wait command for Midas Sequencer only works for > 1 second. As a 
workaround, I tried calling a python script that has a time.sleep() command, but 
the sequencer doesn't wait for the python script to terminate before moving to the 
next command. Obviously, I could just move the entire script to python, but that 
would cause further issues to us. Is there a way to have a wait that has precision 
in order of milliseconds from within the Midas Sequencer? If there is no Midas-
native methods for doing this, what workaround will you suggest to get this to 
work?

I guess the simplest way to do that without breaking with existing code is to change the 
second number to a float. So a

WAIT SECONDS, 1

will still work, and you can then write

WAIT SECONDS, 0.01

to get a 10 ms wait. Would that work for you?

Stefan

> I guess the simplest way to do that without breaking with existing code is to change the 
> second number to a float. So a
> 
> WAIT SECONDS, 1
> 
> will still work, and you can then write
> 
> WAIT SECONDS, 0.01
> 
> to get a 10 ms wait. Would that work for you?

This would work fine in principle, but isn't implemented in the current MIDAS sequencer as we understand it.  (We tried!) Is your proposal to rewrite the sequencer 
to allow fractional waits?  Right now the code seems to store the start_time as a DWORD and uses atoi to parse the wait time, and uses ss_time (which seems only get
the time to the nearest second) to fetch the time.

> This would work fine in principle, but isn't implemented in the current MIDAS sequencer as we understand it.  (We tried!) Is your proposal to rewrite the sequencer 
> to allow fractional waits?  Right now the code seems to store the start_time as a DWORD and uses atoi to parse the wait time, and uses ss_time (which seems only get
> the time to the nearest second) to fetch the time.

No it's not implemented, was just my idea. If it would work for you, I can implement it in the next couple of days.

Stefan

> > This would work fine in principle, but isn't implemented in the current MIDAS sequencer as we understand it.  (We tried!) Is your proposal to rewrite the sequencer 
> > to allow fractional waits?  Right now the code seems to store the start_time as a DWORD and uses atoi to parse the wait time, and uses ss_time (which seems only get
> > the time to the nearest second) to fetch the time.
> 
> No it's not implemented, was just my idea. If it would work for you, I can implement it in the next couple of days.
> 
> Stefan

Yes, please!  Something like WAIT seconds, 0.01 would be perfect.

While I will do it, i'm not sure if this is what you want. If I understand correctly, some process gets triggered and then writes some values to the ODB, then the sequencer 
should continue. Putting a wait there is dangerous. Maybe your process always takes like 10-20 ms, so you put a wait of let's say 100ms, and things are fine with you. Your 
script runs many days, but then once in a while your machine is on heavy load because someone starts a web browser, and your process takes 110ms, and you script crashes.

I would rather go following path: put a "done" flag in the ODB, which is the last one which gets set by your process. Then the sequencer does a 

WAIT ODBvalue, /path/value, =, 1

which will work always, independend of the delay of your process.

Stefan

> While I will do it, i'm not sure if this is what you want. If I understand correctly, some process gets triggered and then writes some values to the ODB, then the sequencer 
> should continue. Putting a wait there is dangerous. Maybe your process always takes like 10-20 ms, so you put a wait of let's say 100ms, and things are fine with you. Your 
> script runs many days, but then once in a while your machine is on heavy load because someone starts a web browser, and your process takes 110ms, and you script crashes.
> 
> I would rather go following path: put a "done" flag in the ODB, which is the last one which gets set by your process. Then the sequencer does a 
> 
> WAIT ODBvalue, /path/value, =, 1
> 
> which will work always, independend of the delay of your process.
> 
> Stefan

Our use case is pretty simple and I don't think is affected by the scenario you envision.  We want to turn on a setting in our equipment, and turn it off again some 0.2 s later.  We don't need msec timing.  So something like:

ODBSET /somekey 1   # this will cause a front-end to flip a bit in our hardware     
WAIT seconds, 0.2
ODBSET /somekey 0   # this will cause a front-end to reset a bit in our hardware 

It is true that if the load is high there could be a little delay, and the time that the bit is set will not be 0.2 seconds, but on average it should work, 
and it should be good enough we think.

Yes, we could also check an ODB key to see that something is done, but we'd still need the ability to wait for time intervals less than 1 second, which
right now doesn't exist.

Ok, I implemented the float second wait function. Internally it works in ms, so the maximum resolution is 0.001 s.

Best,
Stefan

> Ok, I implemented the float second wait function. Internally it works in ms, so the maximum resolution is 0.001 s.
> 
> Best,
> Stefan

Thank you, we will test this soon and let you know if we see any issues (but we're not expecting any).

> > Ok, I implemented the float second wait function. Internally it works in ms, so the maximum resolution is 0.001 s.
> > 
> > Best,
> > Stefan
> 
> Thank you, we will test this soon and let you know if we see any issues (but we're not expecting any).

Hello Stefan,

Thank you for the help you provided for us so far. I tried your code changes on our midas fork. Now, I notice that any 
wait command takes at least 0.2 seconds to run. 

For example, when I use the following script:
SCRIPT source scripts/time_print.sh
WAIT Seconds, 0.1
SCRIPT source scripts/time_print.sh
WAIT Seconds, 0.1
SCRIPT source scripts/time_print.sh

The time_print.sh script prints time segments separated by almost exactly 0.2 seconds. Same goes for when I use 0.01 
second waits.

However, when I use 0.2 seconds wait, then I get time segments separated by 0.3 seconds. I also tried something like 
this:
SCRIPT source scripts/time_print.sh
WAIT Seconds, 0.2
WAIT Seconds, 0.2
SCRIPT source scripts/time_print.sh
WAIT Seconds, 0.2
WAIT Seconds, 0.2
SCRIPT source scripts/time_print.sh

This script results in time segements of 0.6 seconds difference. It is not immidiately clear to me from the sequencer 
code what causes this effect. The code as it stands is a lot better than what we had before the changes, but I am 
wondering if this can be reduced to the order of 1ms or 10ms.

Best regards,
Musaab Faozi