The current ODB code has several structural problems and I think I now figured out how to straighten them out.

Here is the problems:

a) nested (recursive) odb locks
b) no clear separation between read-only access and read-write access
c) no clear separation between odb validation and repair functions
d) cm_msg() is called while holding a database lock

Discussion:

a) odb locks are nested because most functions lock the database, then call other functions that lock the database again. Most locking primitives - SystemV 
semaphores, POSIX semaphores and mutexes - usually do not permit nested (recursive) locking.

For locking the odb shared memory we use a SystemV semaphore with recursion implemented "by hand" in ss_semaphore_wait_for(). This works ok.

For making odb thread-safe, we use POSIX mutexes, and we rely on an optional feature (PTHREAD_MUTEX_RECURSIVE) which seems to work on most OSes, but 
is not required to exist and work by any standard. For example, recursive mutexes do not work in uclinux (linux for machines without an MMU).

I looked at implementing recursive mutexes "by hand", same as we have the recursive semaphores, and realized that it is quite complicated and computationally 
expensive (read: inefficient). (Also I think nested and recursive locks is "not mainstream" and should rather be avoided). As an example you can see full 
complexity of a nested lock as recent implementation in ROOT. (good luck finding it).

A solution for this problem is well known. All functions are separated into "unlocked" user-callable functions and "locked" internal functions. Nested locking is 
naturally eliminated.

Call sequences:
db_get_key() -> db_find_key() // odb is locked twice
become
db_get_key() -> db_get_key_locked() -> db_find_key_locked() // odb is locked once

Actual implementation of this scheme turns out to be a very clean and mechanical refactoring (moving the code without changing what it does).

As a try, I refactored db_find_key() and db_get_key() and I like the result. Locking is now obvious - obscure error paths with hidden "unlock before return"  - are all 
gone. Extra conversions between hDB and pheader are gone.

b) in this refactoring, functions that do not (should not) modify odb become easy to identify - the pheader argument is tagged "const".

This simplifies the implementation of "write-protected" odb - instead of ad-hoc db_allow_write_locked() sprinkled everywhere, one can have obvious calls to 
"db_lock_read_only()" and "db_lock_read_write()".

Separation of locks into "read" and "write" locks, in turn, improves locking behaviour - helps against problems like lock starvation - which we did see with MIDAS - 
as "read" locks are much more efficient - all readers can read the data at the same time, locking is only done when somebody need to "write".

c) some db_validate() functions also try to do repair. this cannot work if validation is called from "read-only" functions like db_find_key(). I now think the "repair" 
functions should be separate from "validate" functions. validate functions should detect problems, repair functions would repair them. The question remains - 
when is good time to run a full repair. (probably at the time when we connect to the database - this way, simply starting "odbedit" will force a database check and 
repair).

d) calls to cm_msg() when odb is locked has been a problem for a long time. because cm_msg() itself calls odb and because it also calls event buffer code 
(SYSMSG buffer) which in turn call odb functions, there was trouble with deadlocks between ODB and event buffer semaphores, trouble with recursive use of 
ODB, etc.

Right now we have all this partially papered over by having cm_msg() put messages into a memory buffer that we periodically flush, but I was never super happy 
with that solution. For example, if we crash before the message buffer is flushed, all error messages are lost, they do not go into midas.log, they are not printed on 
the screen, they are not accessible in the core dump.

To resolve this problem, I have all "locked" functions call db_msg() instead of cm_msg(). db_msg() saves the messages in a linked list which is flushed into 
cm_msg() immediately after we unlock odb.

If we crash after generating an error message but before it is flushed to cm_msg(), we can still access it through the linked list inside the core dump. This is an 
improvement over what we have now. Ideally, all messages should be printed to the terminal and saved to midas.log and pushed into SYSMSG, but most of this is 
impractical at a moment when odb is locked - as we already know it leads to deadlocks and other trouble...

Bottom line, I now have a path to improve the odb code and to resolve some of the long standing structural problems.

K.O.

All makes sense to me. I agree to proceed with the refactoring.

One additional comment: In the 90's when I developed this code, locking was expensive. On a decent computer you could do a couple of thousand lock operations per second before you hit the 100% 
CPU limit. Therefore I tried to reduce the number of lock operation as much as possible. Like a db_find_key locks the ODB once and then goes through all keys before it unlocks again. If I would lock for 
every key and have an ODB with ten thousands of keys, that would have taken very long in the old days. 

Now the world has changed, we can do almost a million locks a second. So a db_get_record() does not have to obtain a whole directory in one go, but can get each value separately, and if necessary lock 
the ODB on each key access. This would be slower, but only a negligible amount these days. So in the spirit of making midas more robust, we can even go a step beyond simple refactoring and change the 
locking scheme if it becomes more transparent and stable.

Best,
Stefan

> One additional comment: In the 90's when I developed this code, locking was expensive.
> Now the world has changed, we can do almost a million locks a second.

I am not sure this is quite true. The CPU can execute 3000 million operations per second (3GHz CPU, assuming 1 op/Hz),
so 1 lock operation is worth 3000 normal operations. Of course cache misses and branch mispredictions mess up
this simple arithmetic...

But I think cost of mutex lock/unlock can be easily measured. (hmm... now I am curious).

Bigger question is architectural, nested/recursive locks is definitely a bad thing to do (not just my opinion).

But closer to home, as I implemented "write protected" ODB, lock/unlock suddenly has to do MMU operations
(map unmap memory) and this is *very* expensive.

Also as we start doing more multithreading, lock contention is becoming a problem, and the standard solution
is to implement read-locks and write-locks. (everybody holding a read-lock can read ODB at the same time
without waiting).

So, moving in the direction of separate read and write locks and write-protected (and/or read-protected) ODB shared memory,
all points in the direction of reworking of ODB locks in the direction of removing the need for nested/recursive locks.

I think me and Stefan are in agreement here.

K.O.

> I am not sure this is quite true. The CPU can execute 3000 million operations per second (3GHz CPU, assuming 1 op/Hz),
> so 1 lock operation is worth 3000 normal operations. Of course cache misses and branch mispredictions mess up
> this simple arithmetic...

You can try that with "t1" in odbedit. This times the number of db_get_data() calls midas can do per second. On my MacBook Pro I get 470'000 
accesses per second.

Good afternoon,

After updating Midas to the latest develop commit 
(0f5436d901a1dfaf6da2b94e2d87f870e3611cf1) we found out a bug when starting 
sequencer. If we have a simple loop from start value to stop value and step 
size, just printing the value at each iteration, we see everything good (see 
first attachment). Then we included another script though, which contains 
several subroutines we defined for our detector, and we try to run the same 
script. Unfortunately after this the parameters seem uninitialized, and the 
value at each loop does not make sense (see second attachment). Also, sometimes 
when pressing run the set parameter window would pop-up, but sometimes not.

The script is this one:

>>>
COMMENT Test script to check for a specific bug

INCLUDE global_basic_functions

#CALL setup_paths
#CALL generate_DUT_params

PARAM lv_start, "Start of LV", 1.8
PARAM lv_stop, "Stop of LV", 2.1
PARAM lv_step, "Step of LV", 0.02

n_iterations = (($lv_stop - $lv_start)/$lv_step)

MSG "Parameters:"
MSG $lv_start
MSG $lv_stop
MSG $lv_step
MSG $n_iterations

MSG "Start of looping"

LOOP n, $n_iterations
   lv_now = $lv_start + $n * $lv_step
   MSG $lv_now
   WAIT SECONDS, 1
ENDLOOP
<<<

and the only difference comes from commenting the line:

>>>
INCLUDE global_basic_functions
<<<

as global_basic_functions is defined as a LIBRARY and it includes 75 (!) 
subroutines...

Is it possible that when loading a large script it messes up the loading of 
parameters?

Thank you very much,
Regards,
Luigi.

Could you please export and send me the /Sequencer ODB tree (or just /Sequencer/Param and /Sequencer/Variables) in both cases while the sequence is running. 

thanks,
Zaher


> Good afternoon,
> 
> After updating Midas to the latest develop commit 
> (0f5436d901a1dfaf6da2b94e2d87f870e3611cf1) we found out a bug when starting 
> sequencer. If we have a simple loop from start value to stop value and step 
> size, just printing the value at each iteration, we see everything good (see 
> first attachment). Then we included another script though, which contains 
> several subroutines we defined for our detector, and we try to run the same 
> script. Unfortunately after this the parameters seem uninitialized, and the 
> value at each loop does not make sense (see second attachment). Also, sometimes 
> when pressing run the set parameter window would pop-up, but sometimes not.
> 
> The script is this one:
> 
> >>>
> COMMENT Test script to check for a specific bug
> 
> INCLUDE global_basic_functions
> 
> #CALL setup_paths
> #CALL generate_DUT_params
> 
> PARAM lv_start, "Start of LV", 1.8
> PARAM lv_stop, "Stop of LV", 2.1
> PARAM lv_step, "Step of LV", 0.02
> 
> n_iterations = (($lv_stop - $lv_start)/$lv_step)
> 
> MSG "Parameters:"
> MSG $lv_start
> MSG $lv_stop
> MSG $lv_step
> MSG $n_iterations
> 
> MSG "Start of looping"
> 
> LOOP n, $n_iterations
>    lv_now = $lv_start + $n * $lv_step
>    MSG $lv_now
>    WAIT SECONDS, 1
> ENDLOOP
> <<<
> 
> and the only difference comes from commenting the line:
> 
> >>>
> INCLUDE global_basic_functions
> <<<
> 
> as global_basic_functions is defined as a LIBRARY and it includes 75 (!) 
> subroutines...
> 
> Is it possible that when loading a large script it messes up the loading of 
> parameters?
> 
> Thank you very much,
> Regards,
> Luigi.

Ok, here is the complete code to reproduce the problem. Load parameter_test.msl which includes functions.msl. From the screenshot you see the variables containing 
garbage, and you also see that from the ODB screenshot. For completeness, I added Sequencer.json which contains the whole sequencer tree.

The interesting thing is that this works sometimes, and sometimes not. I'm not sure if this in the GUI or in the sequencer program, so we have to sort out who can 
fix it ;-)

Best,
Stefan

It is pretty much the same as Stefan, I attach here the screenshots. Also in my case it works sometimes, and sometimes partially (one or 2 params, like in 
attachment 3).

> Could you please export and send me the /Sequencer ODB tree (or just /Sequencer/Param and /Sequencer/Variables) in both cases while the sequence is running. 
> 
> thanks,
> Zaher
> 
> 
> > Good afternoon,
> > 
> > After updating Midas to the latest develop commit 
> > (0f5436d901a1dfaf6da2b94e2d87f870e3611cf1) we found out a bug when starting 
> > sequencer. If we have a simple loop from start value to stop value and step 
> > size, just printing the value at each iteration, we see everything good (see 
> > first attachment). Then we included another script though, which contains 
> > several subroutines we defined for our detector, and we try to run the same 
> > script. Unfortunately after this the parameters seem uninitialized, and the 
> > value at each loop does not make sense (see second attachment). Also, sometimes 
> > when pressing run the set parameter window would pop-up, but sometimes not.
> > 
> > The script is this one:
> > 
> > >>>
> > COMMENT Test script to check for a specific bug
> > 
> > INCLUDE global_basic_functions
> > 
> > #CALL setup_paths
> > #CALL generate_DUT_params
> > 
> > PARAM lv_start, "Start of LV", 1.8
> > PARAM lv_stop, "Stop of LV", 2.1
> > PARAM lv_step, "Step of LV", 0.02
> > 
> > n_iterations = (($lv_stop - $lv_start)/$lv_step)
> > 
> > MSG "Parameters:"
> > MSG $lv_start
> > MSG $lv_stop
> > MSG $lv_step
> > MSG $n_iterations
> > 
> > MSG "Start of looping"
> > 
> > LOOP n, $n_iterations
> >    lv_now = $lv_start + $n * $lv_step
> >    MSG $lv_now
> >    WAIT SECONDS, 1
> > ENDLOOP
> > <<<
> > 
> > and the only difference comes from commenting the line:
> > 
> > >>>
> > INCLUDE global_basic_functions
> > <<<
> > 
> > as global_basic_functions is defined as a LIBRARY and it includes 75 (!) 
> > subroutines...
> > 
> > Is it possible that when loading a large script it messes up the loading of 
> > parameters?
> > 
> > Thank you very much,
> > Regards,
> > Luigi.

I have been able to reproduce the problem only once. From what I see, it seems that the Variables ODB tree is not initialized properly from the Param tree. Below are the messages from the failed run compared to a successful one. As far as I could see, the javascript code does not change anything in the Variables ODB tree (only monitors it). The actual changes are done by the sequencer program, or am I wrong?

Failed run:
16:14:25.849 2024/05/03 [Sequencer,INFO]  + 3 * 
16:14:24.722 2024/05/03 [Sequencer,INFO]  + 2 * 
16:14:23.594 2024/05/03 [Sequencer,INFO]  + 1 * 
16:14:23.592 2024/05/03 [Sequencer,INFO] Start of looping
16:14:23.591 2024/05/03 [Sequencer,INFO] (( - )/)
16:14:23.591 2024/05/03 [Sequencer,INFO] 
16:14:23.590 2024/05/03 [Sequencer,INFO] 
16:14:23.590 2024/05/03 [Sequencer,INFO] 
16:14:23.589 2024/05/03 [Sequencer,INFO] Parameters:
16:14:23.562 2024/05/03 [Sequencer,TALK] Sequencer started with script "testpars.msl".


Successful run:
16:15:37.472 2024/05/03 [Sequencer,INFO] 1.820000
16:15:37.471 2024/05/03 [Sequencer,INFO] Start of looping
16:15:37.471 2024/05/03 [Sequencer,INFO] 15
16:15:37.470 2024/05/03 [Sequencer,INFO] 0.020000
16:15:37.470 2024/05/03 [Sequencer,INFO] 2.100000
16:15:37.469 2024/05/03 [Sequencer,INFO] 1.800000
16:15:37.469 2024/05/03 [Sequencer,INFO] Parameters:
16:15:37.450 2024/05/03 [Sequencer,TALK] Sequencer started with script "testpars.msl".

Ahh, that rings a bell:

1) JS opens start dialog box
2) User enters parameters and presses start
3) JS writes parameters
4) JS starts sequencer
5) Sequencer copies parameters to variables

Now how do you handle 3) and 4). Just issue two mjsonrpc commands together? What then could happen is that 4) is executed before 3) and we get the garbage.
You have to do 3) and WAIT for the return ("then" in the JS promise), and only then issue 4) from there.

Stefan

Thanks for the hint Stefan. I pushed a possible fix but I cannot test it since I cannot reproduce the issue.

> Ahh, that rings a bell:
> 
> 1) JS opens start dialog box
> 2) User enters parameters and presses start
> 3) JS writes parameters
> 4) JS starts sequencer
> 5) Sequencer copies parameters to variables
> 
> Now how do you handle 3) and 4). Just issue two mjsonrpc commands together? What then could happen is that 4) is executed before 3) and we get the garbage.
> You have to do 3) and WAIT for the return ("then" in the JS promise), and only then issue 4) from there.
> 
> Stefan

Seems to me like the problem happens less frequently, but I still see it (1 out of 5 or so). The fact that /Sequencer/Params/Value is empty tells me that the GUI 
has the problem and not the sequencer side.

Stefan

[quote="Zaher Salman"]I think that I finally managed to fix the problem. The default values of the parameters are now written first in one go, then the sequencer waits for confirmation that everything is completed before proceeding. Please test and let me know if there are still any issues. Zaher [/quote]