> 
> The issue occurs when e.g. one channel can not be turned on and ramp for some temp/specific 
> reason, and someone else is working on the daq and reloads the odb for e.g. 1h ago.  
> 

So you want to ensure that some HV channels are turned off and stay turned off. Yes?

Most effective solution will depend on the consequences of unwanted turning-on of your channels:

- if hardware is destroyed if turned on - I think you should have a hardware lock-out. (unplug the HV cable)
- if hardware malfunctions and will degrade if left turned on for long time (i.e. a hot phototube or sparking wire chamber) - your data 
monitoring software should detect the anomaly (it will show up as a hot channel, dead channel, etc) and the people running the 
experiment will realize the mistake and turn the channel back off. also hardware monitoring (HV currents, etc) should detect this, with 
same effect.
- if collected data becomes useless (the turned-off channel make big noise in all other channels), then same thing, your data 
monitoring should catch it.

The next consideration is what are you protecting against:

a) one person flags channel defective, turns it off, next person knows nothing, turns it back on - you need to work on documentation, 
shift hand-off and other human-level procedures
b) people running experiment load random odb files - same thing, from human-level procedures and documentation it should be made 
clear which odb files are correct and which should not be used
c) software malfunction (not human person) causes data change in odb causes turned-off channel to turn back on
d) hardware malfunction causes turned-off channel to turn back on (HV power supply hardware or firmware malfunctions and decides 
that all channels should be turned on at maximum high voltage)

In the experiments I am most familiar with, problem (b) is avoided by never loading/reloading odb files directly, most/all interaction
with the experiment is done through web pages, and these web pages are carefully coded to be safe against most user mistakes.

Cases (a), (b) and (c) you can protect against by changing the frontend code to refuse to turn on some channels:

int set_hv(int channel, int voltage) {
   if (channel == 35) return COMMAND_REFUSED;
   write_to_hardware(channel, voltage);
   return COMMAND_SUCCESS;
}

But in reality this solution only creates problem (e):

e) people running the experiment start random versions of the frontend program, make random changes to the frontend source code, 
multiple people working on the frontend have their own personal versions/copies of the source code, etc.

This is the worst-case scenario, meaning the experiment lost control of software configuration, and even basic software version 
control tools (like svn or git) are not being used. If your experiment gets that chaotic, all protections are likely to be ineffective - 
documentation will not work (people will ignore post-it notes "do not turn on!"), hardware protections will not work (unplugged cable 
labeled "do not plug in!" will be plugged back in and powered), etc. good luck, then.

K.O.

There is a not-so-well-known function in the ODB to write protect some keys. You can do

odbedit> chmod 1 /Equipment/HV/Demand

which will write protect your Demand values. You see that by doing

odbedit> ls -ls

where you only see a "R" at the right end instead a "RWD". I haven't tried it yet (so better do a dry run yourself), but that should prevent an odb load to overwrite your demand values. To change the values, put some logic on a custom page to unprotect the 
values, change them, and then protect them again.

Stefan

There were some complaints that the MIDAS sequencer was slow.  Specifically, the
complaint was that even lines in the sequence that didn't do any (like COMMENT
commands) tooks > 100ms to execute.  These slow sequencer steps could be a
little annoying if a script had to change a large number of ODB variables before
starting.

I tested this a little using a trivial sequence; note that I did all tests using
mhttpd with mongoose enabled on a newer macbook pro.  I found that with the
mongoose server each line in a sequencer script was taking ~100ms.  This is
consistent with the loop in the main thread, which is only doing a cm_yield and
a sleep:

   while (!_abort) {
      status = ss_mutex_wait_for(request_mutex, 0);
      status = cm_yield(0);
      if (status == RPC_SHUTDOWN)
         break;
      sequencer();
      status = ss_mutex_release(request_mutex);
      ss_sleep(100);
   }

I tested reducing the sleep to 20ms.  As expected, this made the sequencer more
zippy, able to execute ~50 commands per second.

I tried to think what would be downsides to making this change.  I think that
the main web communication should not be affected, because that communication is
all handled by the separate mongoose thread.

I checked how much extra CPU was used if the sleep was reduced from 100ms to
20ms.  I found that when a sequence was not running the CPU increased from 0% to
0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
4% with my change.  4% is a little high, though I'd say still reasonable.  I
found that most of the CPU usage was occuring because every call to
'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
guess that making that call 50 times causes the somewhat heavy CPU usage.

I would argue that it would still be worth making that change, so that the
sequencer can be more zippy.

> There were some complaints that the MIDAS sequencer was slow.  Specifically, the
> complaint was that even lines in the sequence that didn't do any (like COMMENT
> commands) tooks > 100ms to execute.  These slow sequencer steps could be a
> little annoying if a script had to change a large number of ODB variables before
> starting.
> ...
> I checked how much extra CPU was used if the sleep was reduced from 100ms to
> 20ms.  I found that when a sequence was not running the CPU increased from 0% to
> 0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
> 4% with my change.  4% is a little high, though I'd say still reasonable.  I
> found that most of the CPU usage was occuring because every call to
> 'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
> guess that making that call 50 times causes the somewhat heavy CPU usage.

One additional point: I think that it would be reasonably simple to reduce this CPU
usage even while a sequence was going on.  I would guess that for many sequences a
lot of time was spent in a 'WAIT SECONDS' command, since you would presumably want
to wait while data was being taken or conditions stabilizing.  I think that if you
are in a 'WAIT SECONDS' command that hasn't been satisfied then there probably isn't
any reason to do the db_set_record at the end of the sequencer() method.

> There were some complaints that the MIDAS sequencer was slow.  Specifically, the
> complaint was that even lines in the sequence that didn't do any (like COMMENT
> commands) tooks > 100ms to execute.  These slow sequencer steps could be a
> little annoying if a script had to change a large number of ODB variables before
> starting.
> 
> I tested this a little using a trivial sequence; note that I did all tests using
> mhttpd with mongoose enabled on a newer macbook pro.  I found that with the
> mongoose server each line in a sequencer script was taking ~100ms.  This is
> consistent with the loop in the main thread, which is only doing a cm_yield and
> a sleep:
> 
>    while (!_abort) {
>       status = ss_mutex_wait_for(request_mutex, 0);
>       status = cm_yield(0);
>       if (status == RPC_SHUTDOWN)
>          break;
>       sequencer();
>       status = ss_mutex_release(request_mutex);
>       ss_sleep(100);
>    }
> 
> I tested reducing the sleep to 20ms.  As expected, this made the sequencer more
> zippy, able to execute ~50 commands per second.
> 
> I tried to think what would be downsides to making this change.  I think that
> the main web communication should not be affected, because that communication is
> all handled by the separate mongoose thread.
> 
> I checked how much extra CPU was used if the sleep was reduced from 100ms to
> 20ms.  I found that when a sequence was not running the CPU increased from 0% to
> 0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
> 4% with my change.  4% is a little high, though I'd say still reasonable.  I
> found that most of the CPU usage was occuring because every call to
> 'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
> guess that making that call 50 times causes the somewhat heavy CPU usage.
> 
> I would argue that it would still be worth making that change, so that the
> sequencer can be more zippy.

The minimal time slice on most systems is 10 ms, and nothing prevents us from switching to
that. The original 100 ms was more for the fact that you can see the sequencer statements
executed one after the other (with the color bar). But this is more a "debugging" feature which 
we not really need. 

To do it "right" the sequencer would have to _return_ a sleep time. Like if it is in a wait loop (as
most of the time), the sleep time could be close to 1 second, to correctly update the wait
progress bar. If the sequencer executes ODB set statements, the wait time could be zero, so
thousands of statements can be executed in one second. The problem we will then have of course
that the sequencer will block the "request_mutex" almost always, which would prevent the
mongoose server from serving anything. So this should be carefully tested. It could be (on most OS)
that releasing the mutex by the main loop immediately switches to the mongoose thread, which would
make the web server still quite responsive, but I'm not sure about that. So as a first change making
the sleep time 10ms should be fine.

Stefan

> > I checked how much extra CPU was used if the sleep was reduced from 100ms to
> > 20ms.  I found that when a sequence was not running the CPU increased from 0% to
> > 0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
> > 4% with my change.  4% is a little high, though I'd say still reasonable.  I
> > found that most of the CPU usage was occuring because every call to
> > 'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
> > guess that making that call 50 times causes the somewhat heavy CPU usage.
> > 
> > I would argue that it would still be worth making that change, so that the
> > sequencer can be more zippy.
> 
> The minimal time slice on most systems is 10 ms, and nothing prevents us from switching to
> that. The original 100 ms was more for the fact that you can see the sequencer statements
> executed one after the other (with the color bar). But this is more a "debugging" feature which 
> we not really need. 

OK, I made this change; sleep is now 10ms on main thread.  Seems to work fine on SL6 and MacOS.

> To do it "right" the sequencer would have to _return_ a sleep time. Like if it is in a wait loop (as
> most of the time), the sleep time could be close to 1 second, to correctly update the wait
> progress bar. If the sequencer executes ODB set statements, the wait time could be zero, so
> thousands of statements can be executed in one second. The problem we will then have of course
> that the sequencer will block the "request_mutex" almost always, which would prevent the
> mongoose server from serving anything. So this should be carefully tested. It could be (on most OS)
> that releasing the mutex by the main loop immediately switches to the mongoose thread, which would
> make the web server still quite responsive, but I'm not sure about that. So as a first change making
> the sleep time 10ms should be fine.

Hmm, yeah, I'm not sure about how to handle reducing the wait time to zero after ODB set commands.

But it does seem like it would be straight-forward to increase the sleep time for waits; I'll look into
a clean way of doing that.

> Hmm, yeah, I'm not sure about how to handle reducing the wait time to zero after ODB set commands.
> 
> But it does seem like it would be straight-forward to increase the sleep time for waits; I'll look into
> a clean way of doing that.

Let's see how your 10 ms work in real life. If we need variable wait times, I can implement this for your without much effort.

Stefan

cm_watchdog() has been removed from the latest midas sources. The watchdog functions performed by cm_watchdog() were 
moved to cm_yield() - those are - maintaining odb and event buffer "last active" timestamps and checking for and removing of 
timed-out clients.

Those who write midas programs should ensure that they call cm_yield() at least every 1-2 seconds (for the normal 10 second 
timeout settings). As always, before calling potentially time consuming operations, such as accessing slow responding 
hardware, one should increase or disable their watchdog timeout.

Those who write midas programs that do not use cm_yield() (i.e. the mserver), should call cm_periodic_tasks() instead with 
the same period as described for cm_yield() above.

Removal of cm_watchdog() solves many problems in the midas code base:
- firing of cm_watchdog() at random times in random places makes it difficult to do static code path analysis (call paths, etc) - 
this is needed to ensure correct multithread locking, etc
- ditto caused trouble with multithread locking when cm_watchdog() fires *inside* the pthread mutex locking library itself 
causing mutexes to be in an inconsistent state. This had to be kludged against in the ODB multithread locks - now this kludge 
can be removed.
- many non-midas codes in experiment frontends, etc was not expecting and did not correctly handle firing of the 
cm_watchdog() SIGALARM at random times (i.e. select() with timeout returned too soon, etc).
- today, UNIX signals are pretty obscure, best avoided. (i.e. interaction of signals and threads is not super will defined, etc).

commits up to (and including) plus merge of branch feature/remove_cm_watchdog
https://bitbucket.org/tmidas/midas/commits/9f1775d2fc75d0de0b9d4ef1abc7b2fb9bacca28

K.O.

> cm_watchdog() has been removed from the latest midas sources
> Removal of cm_watchdog() solves many problems in the midas code base:

Removal of cm_watchdog() creates new problems:

a) the bm_send_event(BM_WAIT) and bm_receive_event(BM_WAIT) wait for free space and wait for new event do not update the timeouts (need to add a call 
to cm_periodic_tasks())
b) frontends that talk to slow external equipment now die unless they have their timeout adjusted to be longer than the longest equipment operation (they 
were already supposed to do this, but...)
c) mhttpd sometimes dies from from an odb timeout (with the default 10 sec timeout).

As one solution, we may bring an automatic cm_watchdog() back, but running from a thread instead of from SIGALARM.

K.O.

> > cm_watchdog() has been removed from the latest midas sources
> > Removal of cm_watchdog() solves many problems in the midas code base:
> Removal of cm_watchdog() creates new problems:
> a) the bm_send_event(BM_WAIT) and bm_receive_event(BM_WAIT)
> b) frontends that talk to slow external equipment
> c) mhttpd sometimes dies from from an odb timeout (with the default 10 sec timeout).

The watchdog is back, in a "light" form. Added:

- cm_watchdog_thread() - runs every 2 seconds and updates the timestamps on ODB and all open event buffers (SYSMSG, SYSTEM, etc).
- cm_start_watchdog_thread() - added to mfe.c and mhttpd - so user frontends work the same as before cm_watchdog() removal
- cm_stop_watchdog_thread() - added to cm_disconnect_experiment() to avoid leaving the thread running after we closed odb and all event buffers.

As before, the watchdog only runs on locally attached midas programs. For programs attached remotely via the mserver, the mserver handles the watchdog functions.

This new light-weight watchdog thread only updates the timestamps, it does not check and remove dead clients, it does not check the alarms. These functions are now performed 
by cm_yield() and cm_periodic_tasks(). At least some program in an experiment should call them periodically. (normally, at least mlogger and mhttpd will do that).

Programs that accidentally relied on SIGALRM firing at 1Hz may still be affected - i.e. with the old cm_watchdog(), ::sleep(1000) will only sleep for 1 second (interrupted by 
SIGALARM), now it will sleep for the full 1000 seconds. Other syscalls, i.e. select(), are similarly affected.

For now, I think only mfe.c frontends and mhttpd need the watchdog thread. With luck all the other midas programs (mlogger, mdump, etc) will run fine without it.

K.O.

Our VxWorks C compiler (gcc-2.8-something) does not like the "//" comments. Luckily, on VxWorks, we 
only compile a small subset of midas, so there is no point in banning all "//" comments. But I did have to 
convert a couple of them to /* commens */ in odb.c to make it compile. Changes to odb.c commited. K.O.

The modbset() function in mhttpd.js is not used anywhere in midas and it misleads midas users into thinking that it works like the old ODBSet() function, when 
it can not and it does not.

To explain the difference:

1) ODBSet() used synchronous RPC requests, which have been deprecated by the powers that be. Read more here:
https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests
https://x443.wordpress.com/2012/12/01/why-you-should-use-xmlhttprequest-asynchronously/

2) in midas, we followed these instructions and developed an asynchronous RPC mechanism for calling midas functions from javascript. (we use the Promise 
construct, but the underlying JSON-RPC compatible communications can be used directly, without it).

3) using the asynchronous RPC is not as easy as the old ODBSet() & co - instead of just making a call "to write to ODB", one has to create a chain of nested 
event handler functions and one has to do at least some error handling.

4) this makes it impossible to program midas custom pages in javascript as if it were C/C++. (Please direct your complaints to the "web" and "javascript" 
powers that be).

5) to help writing midas custom pages, we have a good number of examples. For example, example.html has example
code for calling pretty much every midas json rpc function.

5a) to see the complete list of all rpc functions available in your copy of midas, follow the link to "json-rpc schema, text format" on the midas "help" page.

6) if you are writing a new custom page we suggest you start with one of the example templates in .../resources, a_example, a_template.

7) if you are updating an existing custom page, good luck. synchronous rpc seems to still work in most browsers, so the old OSBSet() & co should continue to 
work for now. For new code you should use the async rpc (with Promises, like we do for all midas pages). In practice this means a complete rewrite of each 
custom page (welcome to the 21st century).

Note that we have two separate js files in midas:

- midas.js is intended as a general purpose library for writing midas custom pages
- mhttpd.js is not intended for general use and contains javascript code used by mhttpd internally

The function itself is here, in case somebody needs it:

-function modbset(path, value)
-/* shortcut for mjsonrpc_db_paste() with standard error handling */
-{
-   if (Array.isArray(path)) {
-      mjsonrpc_db_paste(path,value).then(function(rpc) {}).catch(function(error) {
-         mjsonrpc_error_alert(error); });
-   } else {
-      mjsonrpc_db_paste([path],[value]).then(function(rpc) {}).catch(function(error) {
-         mjsonrpc_error_alert(error); });
-   }
-}
-

K.O.

I disagree. The modbset() function is used in many custom pages at PSI because people are tired of typing mjsonrpc_db_paste([path],[value]) vs. modbset(path, value). We need to keep 
modbset() which is well documented at 

https://midas.triumf.ca/MidasWiki/index.php/Custom_Page#modbset

Since modbset() does call the underlying mjsonrpc_db_paste(), it is as good or bad as that function. Plus it adds standard error handling to avoid the need of catching errors for each and 
every mjsonrpc_db_paste() call. If it is believed that modbset() has a problem, then this should be fixed in the source code of modbset(). Removing that function is not an option.

Stefan

If it's a function intended for general use, it should be in midas.js.

The documentation for such a function should be made very clear that:

a) it does not actually write to ODB, instead it queues a request for writing, this request is executed at a later (undefined) time.

b) the following javascript code results in undefined behaviour:

modbset("/foo/bar", 1); // queue rpc request
modbset("/foo/bar", 2); // queue rpc request
// is ODB /foo/bar set to 1 or 2?

Why? The best I know javascript does not require for RPCs to execute in-order, so the second RPC may be issued
before the first one. More likely both RPCs are started roughly at the same time (i.e. in different RPC worker
threads), in which case we do not know in which order they will be processed by mhttpd, which is also
multithreaded and does not necessarily execute requests in the same order as (i.e.) they connect to the rpc port 8080.

To answer the question "1 or 2", the answer is neither, as at that point in the code, the RPC requests
probably have not started executing yet, and even if they did, mhttpd most likely did not write anything
into odb yet, as processing RPC requests takes much longer than executing a few lines of javascript.

So to ensure correct sequence of writes and to ensure that something was actually written to odb,
one has to roll out the full ladder of promise event handlers.

Is correct sequence of writes important? Maybe yes, maybe no.

But if you use modbset() without being aware of these issues, you will write code
for ramping high voltage like this:

modbset("/eq/hv/set/voltage", 0); // set voltage to zero
modbset("/eq/hv/set/hv_enable", 1); // enable high voltage
modbset("/eq/hv/set/voltage", 50); // start slow
modbset("/eq/hv/set/voltage", 1000); // ramp to half-way
modbset("/eq/hv/set/voltage", 1900); // stop a bit before the final voltage
modbset("/eq/hv/set/voltage", 2000); // !!! ramping from 0 to 2000 should never be done in one step !!!

And as the author of all this RPC code, I promise that some day you will see the voltage
on the detector go directly from 0 to 2000 (then up and down to 50, 1000 and 1900).

To me, this makes helpful helper functions actually dangerous to use.

Now that I have experience with sync RPC (from C/C++) and async RPC (with Promises in javascript),
I would say that synchronous C/C++ style programming is much easier
and much less verbose and much easier to read and to modify/adjust compared to the javascript-style
ladder of nested event handlers.

I now see that synchronous requests are again permitted if one uses the "Web Worker API". Maybe we should revisit
the MIDAS javascript RPC code and see if we can use this web worker stuff to officially support synchronous RPC requests,
again.

About the sync rpc deprecation, read more here:
https://stackoverflow.com/questions/30876093/will-chrome-and-other-browsers-drop-support-for-synchronous-xmlhttprequest

K.O.


> I disagree. The modbset() function is used in many custom pages at PSI because people are tired of typing mjsonrpc_db_paste([path],[value]) vs. modbset(path, value). We need to keep 
> modbset() which is well documented at 
> 
> https://midas.triumf.ca/MidasWiki/index.php/Custom_Page#modbset
> 
> Since modbset() does call the underlying mjsonrpc_db_paste(), it is as good or bad as that function. Plus it adds standard error handling to avoid the need of catching errors for each and 
> every mjsonrpc_db_paste() call. If it is believed that modbset() has a problem, then this should be fixed in the source code of modbset(). Removing that function is not an option.
> 
> Stefan

A ladder of promise event handlers is certainly one possibility to enforce the order of ODB writes, but I wonder if we could so something simpler:

- modbset creates an object remembering the status of the RPC request. Initially, this object receives the status "open request"
- when the rpc call got executed successfully, the callback sets the state of the above object to "request succeeded" or "request failed" (in case of error)
- if a new modbset comes BEFORE the previous one has completed, the function queues the new request in a data field of the above object
- if a rpc call finishes, and a queued new rpc request is present, it gets executed

This would be relatively easy to be implemented and keep the order of the rpc calls. Does that make sense?

Best,
Stefan

> A ladder of promise event handlers is certainly one possibility to enforce the order of ODB writes, but I wonder if we could so something simpler:
> 
> - modbset creates an object remembering the status of the RPC request. Initially, this object receives the status "open request"
> - when the rpc call got executed successfully, the callback sets the state of the above object to "request succeeded" or "request failed" (in case of error)
> - if a new modbset comes BEFORE the previous one has completed, the function queues the new request in a data field of the above object
> - if a rpc call finishes, and a queued new rpc request is present, it gets executed
> 
> This would be relatively easy to be implemented and keep the order of the rpc calls. Does that make sense?
> 

Yes, this is a neat idea, I am really happy with how a complete rpc request can be held by one object, and we can make queues of them, etc.

Anyhow here is the proof of the pudding. I added a test to example.html, there are two buttons, one makes 5 modbset() calls, second has a ladder of 5 db_paste calls. Then I watch 
the result in odbedit. 1, 2, 3, 4, 5 is the modbset(), 6, 7, 8, 9, 10 is the ladder of db_paste calls:

$ odbedit
[local:javascript1:S]/>watch Example/int
Watch key "/Example/int" to be modified, abort with any key

/Example/int = 1
/Example/int = 2
/Example/int = 3
/Example/int = 4
/Example/int = 5

/Example/int = 1
/Example/int = 5 <== fault
/Example/int = 5
/Example/int = 5
/Example/int = 5

/Example/int = 1
/Example/int = 2
/Example/int = 3
/Example/int = 5 <== 4 and 5 reversed
/Example/int = 4 <== 4 and 5 reversed

/Example/int = 6
/Example/int = 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

/Example/int = 6
/Example/int = 8 <== should be 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

/Example/int = 6
/Example/int = 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

I immediately notice that we have a race condition between the RPCs, db_watch notifications and db_get_value() in the watch handler:
there are 5 rpcs, 5 watch notifications, 5 calls to db_get_value() in the watch handler, but sometimes the handler is too slow
and the data in odb changes before it reads it, thus duplicate values (missing "7" above). (The old db_open_record() had a "hidden"
db_get_value() inside it, while db_watch() requires an explicit db_get_value() call, making it obvious why we get
the wrong (newer) data sometimes).

Possible fixes for this is to slow down the RPCs (the race condition is still there, probability is reduced) or send the changed
data as part of the notification. If this were C/C++, a "sleep(1)" between modbset() calls would have fixed it,
but there is no sleeping and waiting in javascript. (I guess one could use a ladder of timers).

Other than that, I am surprised how easy it was to see that indeed out-of-order RPCs can happen, see the case
of out-of-order 4 and 5 above. It only took me maybe 5-10 clicks on the button to see that. I expected that I would
need to try several browsers or use a slow network connection, but here it is, on my home mac, localhost network,
google chrome browser.

Below is the test code. I do NOT vote that everybody should use ladders of db_paste calls.

function test_modbset() {
           modbset("/example/int", 1);
           modbset("/example/int", 2);
           modbset("/example/int", 3);
           modbset("/example/int", 4);
           modbset("/example/int", 5);
        }

        function test_chained_db_paste() {
           var paths = [ "/example/int" ];
           mjsonrpc_db_paste(paths,[6]).then(function(rpc) {
              mjsonrpc_db_paste(paths,[7]).then(function(rpc) {
                 mjsonrpc_db_paste(paths,[8]).then(function(rpc) {
                    mjsonrpc_db_paste(paths,[9]).then(function(rpc) {
                       mjsonrpc_db_paste(paths,[10]).then(function(rpc) {
                          // nothing
                       }).catch(function(error){mjsonrpc_error_alert(error);});
                    }).catch(function(error){mjsonrpc_error_alert(error);});
                 }).catch(function(error){mjsonrpc_error_alert(error);});
              }).catch(function(error){mjsonrpc_error_alert(error);});
           }).catch(function(error){mjsonrpc_error_alert(error);});
        }

        </script>

        <input type=button value='test modbset()' onClick='test_modbset();'></input>
        <input type=button value='test chained db_paste()' onClick='test_chained_db_paste();'></input>

K.O.

Just to make this point clear: The "write-to-odb-read-via-hotlink" was never meant to guarantee the receiving side to see each change. If changes happen too often, updates might get lost. If one relies on the 
sequence of updates, one should use direct RPC calls to the frontend or use a midas buffer and encode updates in events.

Stefan

> Just to make this point clear: The "write-to-odb-read-via-hotlink" was never meant to guarantee the receiving side to see each change. If changes happen too often, updates might get lost. If one relies on the 
> sequence of updates, one should use direct RPC calls to the frontend or use a midas buffer and encode updates in events.

I recommend that people use the jrpc mechanism that does an RPC directly from javascript into the frontend.
It passes 2 strings as arguments (command and data value). Arbitrary objects can be passed by encoding
the data in json (use mjson.h to decode it in the frontend). A string is returned back to javascript (again, encode
arbitrary data as json, use the mjson.h library).

Call sequence:
javascript -> (http) -> mhttpd -> (MIDAS RPC call) -> frontend -> (write, read, frob hardware) -> frontend -> (MIDAS RPC reply) -> mhttpd -> (http reply) -> javascript

Example of all this is in example.html and fetest.cxx:

javascript side code: mjsonrpc_call("jrpc", { "client_name":"fetest", "cmd":"xxx", "args":"xxx" })

frontend side code:

INT rpc_callback(INT index, void *prpc_param[])
{
   const char* cmd  = CSTRING(0);
   const char* args = CSTRING(1);
   char* return_buf = CSTRING(2);
   int   return_max_length = CINT(3);
   cm_msg(MINFO, "rpc_callback", "--------> rpc_callback: index %d, max_length %d, cmd [%s], args [%s]", index, return_max_length, cmd, args);
   ... do stuff ... put result into string "tmp"
   strlcpy(return_buf, tmp, return_max_length);
   return RPC_SUCCESS;
}

... somewhere in frontend_init(), register the RPC:

#ifdef RPC_JRPC
   status = cm_register_function(RPC_JRPC, rpc_callback);
   assert(status == SUCCESS);
#endif

K.O.

Request:

Please use "make cmake" ("make cmake3") to report build problems.

cmake with VERBOSE=1 prints way too much stuff and I find it is impossible to use.

"make cmake" filters out all the gunk to make it easy to see the relevant information:
- the compiler command lines (CFLAGS, library paths, etc)
- the compiler output (error messages, etc).

The best I can tell, cmake "VERBOSE=1" in only useful for cmake developers to debug cmake itself.

K.O.

> I tried on my Mac (macOS 10.14.6, Xcode 11.3.1, current develop branch, openssl 1.1.1d) and it woks fine. Below is the transcript. I 
> see that your cmake output is much shorter (no C compiler listed etc.). Did you remove some lines? For such comparisons, it's 
> always good to start with an empty build directory.
> 
> OpenSSL libraries are the same (1.1.1d). Just for comparison, I run the build process with "make VERBOSE=1" and extract the line 
> which fails for you when linking mhttp, so maybe you can compare.
> 
> Linking mhttpd
> ---------------
> 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -O2 -g -DNDEBUG -isysroot 
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-
> min=10.14 -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o 
> CMakeFiles/mhttpd.dir/mongoose6.cxx.o CMakeFiles/mhttpd.dir/mgd.cxx.o CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o 
> mhttpd  -L/opt/local/lib/mysql57/mysql -lmysqlclient -lz ../libmidas.a /opt/local/lib/libssl.dylib /opt/local/lib/libcrypto.dylib -lz -
> L/opt/local/lib/mysql57/mysql -lmysqlclient -lz 
> 
> 
> Here is the full transcript
> -------------------------
> 
> /midas/build$ cmake ..
> -- MIDAS: cmake version: 3.16.3
> -- The C compiler identification is AppleClang 11.0.0.11000033
> -- The CXX compiler identification is AppleClang 11.0.0.11000033
> -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
> -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- 
> works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Detecting C compile features
> -- Detecting C compile features - done
> -- Check for working CXX compiler: 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
> -- Check for working CXX compiler: 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Detecting CXX compile features
> -- Detecting CXX compile features - done
> -- MIDAS: CMAKE_INSTALL_PREFIX: /midas
> -- MIDAS: Found ROOT version 6.16/00
> -- Found ZLIB: /opt/local/lib/libz.dylib (found version "1.2.11") 
> -- MIDAS: Found ZLIB version 1.2.11
> -- Found OpenSSL: /opt/local/lib/libcrypto.dylib (found version "1.1.1d")  
> -- MIDAS: Found OpenSSL version 1.1.1d
> -- MIDAS: Found MySQL version 5.7.26
> -- MIDAS: ODBC not found
> -- MIDAS: SQLITE not found
> -- MIDAS: nvidia-smi not found
> -- Setting default build type to "RelWithDebInfo"
> -- Found Git: /usr/bin/git (found version "2.21.1 (Apple Git-122.3)") 
> -- MIDAS example/experiment: MIDAS in-tree-build
> -- MIDAS: Found ZLIB version 1.2.11
> -- MIDAS example/experiment: Found ROOT version 6.16/00
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /midas/build
> /midas/build$ make
> Scanning dependencies of target rmana
> Scanning dependencies of target git_revision_h
> Scanning dependencies of target mfe
> Scanning dependencies of target manao
> Scanning dependencies of target mfeo
> Scanning dependencies of target objlib-c-compat
> Scanning dependencies of target mana
> Scanning dependencies of target rmanao
> [  0%] Building CXX object CMakeFiles/objlib-c-compat.dir/src/midas_c_compat.cxx.o
> [  2%] Building CXX object CMakeFiles/mfe.dir/src/mfe.cxx.o
> [  2%] Building CXX object CMakeFiles/mfeo.dir/src/mfe.cxx.o
> [  3%] Building CXX object CMakeFiles/mana.dir/src/mana.cxx.o
> [  4%] Building CXX object CMakeFiles/manao.dir/src/mana.cxx.o
> [  5%] Building CXX object CMakeFiles/rmanao.dir/src/mana.cxx.o
> [  6%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
> [  6%] Built target git_revision_h
> Scanning dependencies of target objlib
> [  7%] Building CXX object CMakeFiles/objlib.dir/src/midas.cxx.o
> [  7%] Built target objlib-c-compat
> [  8%] Building CXX object CMakeFiles/objlib.dir/src/midas_cxx.cxx.o
> [  8%] Linking CXX static library libmana.a
> [  8%] Built target manao
> [  8%] Building CXX object CMakeFiles/objlib.dir/src/odb.cxx.o
> [  8%] Built target mana
> [  9%] Building CXX object CMakeFiles/objlib.dir/src/device_driver.cxx.o
> [  9%] Built target mfeo
> [ 10%] Linking CXX static library libmfe.a
> [ 11%] Building CXX object CMakeFiles/objlib.dir/src/system.cxx.o
> [ 11%] Built target mfe
> [ 11%] Building CXX object CMakeFiles/objlib.dir/src/alarm.cxx.o
> [ 12%] Building CXX object CMakeFiles/objlib.dir/src/elog.cxx.o
> [ 13%] Building CXX object CMakeFiles/objlib.dir/src/mrpc.cxx.o
> [ 13%] Building CXX object CMakeFiles/objlib.dir/src/mjson.cxx.o
> [ 14%] Building CXX object CMakeFiles/objlib.dir/src/tmfe.cxx.o
> [ 15%] Building CXX object CMakeFiles/objlib.dir/src/mvodb.cxx.o
> [ 15%] Built target rmanao
> [ 15%] Linking CXX static library librmana.a
> [ 16%] Building CXX object CMakeFiles/objlib.dir/src/nullodb.cxx.o
> [ 16%] Building CXX object CMakeFiles/objlib.dir/src/midasodb.cxx.o
> [ 16%] Built target rmana
> [ 17%] Building CXX object CMakeFiles/objlib.dir/src/mxmlodb.cxx.o
> [ 18%] Building CXX object CMakeFiles/objlib.dir/src/mjsonodb.cxx.o
> [ 18%] Building CXX object CMakeFiles/objlib.dir/src/json_paste.cxx.o
> [ 19%] Building CXX object CMakeFiles/objlib.dir/src/mjsonrpc.cxx.o
> [ 20%] Building CXX object CMakeFiles/objlib.dir/src/mjsonrpc_user.cxx.o
> [ 21%] Building CXX object CMakeFiles/objlib.dir/src/history.cxx.o
> [ 21%] Building CXX object CMakeFiles/objlib.dir/src/history_common.cxx.o
> [ 22%] Building CXX object CMakeFiles/objlib.dir/src/history_odbc.cxx.o
> [ 23%] Building CXX object CMakeFiles/objlib.dir/src/history_schema.cxx.o
> [ 23%] Building CXX object CMakeFiles/objlib.dir/src/lz4.cxx.o
> [ 24%] Building CXX object CMakeFiles/objlib.dir/src/lz4frame.cxx.o
> [ 25%] Building CXX object CMakeFiles/objlib.dir/src/lz4hc.cxx.o
> [ 26%] Building CXX object CMakeFiles/objlib.dir/src/xxhash.cxx.o
> [ 26%] Building CXX object CMakeFiles/objlib.dir/src/crc32c.cxx.o
> [ 27%] Building CXX object CMakeFiles/objlib.dir/src/sha256.cxx.o
> [ 28%] Building CXX object CMakeFiles/objlib.dir/src/sha512.cxx.o
> [ 28%] Building CXX object CMakeFiles/objlib.dir/src/ftplib.cxx.o
> [ 29%] Building CXX object CMakeFiles/objlib.dir/src/mdsupport.cxx.o
> [ 30%] Building CXX object CMakeFiles/objlib.dir/mxml/mxml.cxx.o
> [ 31%] Building CXX object CMakeFiles/objlib.dir/mxml/strlcpy.cxx.o
> [ 31%] Built target objlib
> Scanning dependencies of target midas-shared
> Scanning dependencies of target midas
> [ 33%] Linking CXX shared library libmidas-shared.dylib
> [ 33%] Linking CXX static library libmidas.a
> [ 33%] Built target midas-shared
> [ 33%] Built target midas
> Scanning dependencies of target mhttpd
> Scanning dependencies of target feudp
> Scanning dependencies of target odb_lock_test
> Scanning dependencies of target rmlogger
> Scanning dependencies of target mchart
> Scanning dependencies of target msysmon
> Scanning dependencies of target midas-c-compat
> Scanning dependencies of target mfe_link_test
> [ 33%] Building CXX object progs/CMakeFiles/mfe_link_test.dir/mfe_link_test.cxx.o
> [ 33%] Linking CXX shared library libmidas-c-compat.dylib
> [ 34%] Building CXX object progs/CMakeFiles/feudp.dir/feudp.cxx.o
> [ 35%] Building CXX object progs/CMakeFiles/msysmon.dir/msysmon.cxx.o
> [ 36%] Building CXX object progs/CMakeFiles/mchart.dir/mchart.cxx.o
> [ 37%] Building CXX object progs/CMakeFiles/odb_lock_test.dir/odb_lock_test.cxx.o
> [ 38%] Building CXX object progs/CMakeFiles/rmlogger.dir/mlogger.cxx.o
> [ 39%] Building CXX object progs/CMakeFiles/mhttpd.dir/mhttpd.cxx.o
> [ 39%] Built target midas-c-compat
> [ 40%] Building CXX object progs/CMakeFiles/mhttpd.dir/mongoose6.cxx.o
> [ 41%] Linking CXX executable mfe_link_test
> [ 42%] Linking CXX executable odb_lock_test
> [ 42%] Built target mfe_link_test
> Scanning dependencies of target mjson_test
> [ 43%] Building CXX object progs/CMakeFiles/mjson_test.dir/mjson_test.cxx.o
> [ 43%] Linking CXX executable mchart
> [ 43%] Built target odb_lock_test
> Scanning dependencies of target odbinit
> [ 44%] Building CXX object progs/CMakeFiles/odbinit.dir/odbinit.cxx.o
> [ 45%] Linking CXX executable feudp
> [ 45%] Built target mchart
> [ 46%] Building CXX object progs/CMakeFiles/mhttpd.dir/mgd.cxx.o
> [ 46%] Built target feudp
> [ 46%] Building CXX object progs/CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o
> [ 47%] Linking CXX executable mjson_test
> [ 47%] Built target mjson_test
> Scanning dependencies of target fetest_tmfe_thread
> [ 47%] Building CXX object progs/CMakeFiles/fetest_tmfe_thread.dir/fetest_tmfe_thread.cxx.o
> [ 47%] Linking CXX executable msysmon
> [ 47%] Built target msysmon
> [ 48%] Linking CXX executable odbinit
> Scanning dependencies of target fetest_tmfe
> [ 49%] Building CXX object progs/CMakeFiles/fetest_tmfe.dir/fetest_tmfe.cxx.o
> [ 49%] Built target odbinit
> [ 50%] Linking CXX executable fetest_tmfe_thread
> Scanning dependencies of target mh2sql
> [ 51%] Building CXX object progs/CMakeFiles/mh2sql.dir/mh2sql.cxx.o
> Scanning dependencies of target odbhist
> [ 51%] Building CXX object progs/CMakeFiles/odbhist.dir/odbhist.cxx.o
> [ 51%] Built target fetest_tmfe_thread
> Scanning dependencies of target melog
> [ 51%] Building CXX object progs/CMakeFiles/melog.dir/melog.cxx.o
> [ 52%] Linking CXX executable odbhist
> [ 53%] Built target odbhist
> [ 53%] Linking CXX executable melog
> [ 54%] Linking CXX executable fetest_tmfe
> Scanning dependencies of target mfe_link_test_cxx
> [ 55%] Building CXX object progs/CMakeFiles/mfe_link_test_cxx.dir/mfe_link_test_cxx.cxx.o
> [ 55%] Built target melog
> Scanning dependencies of target crc32c_sum
> [ 55%] Built target fetest_tmfe
> Scanning dependencies of target odbedit
> [ 56%] Building CXX object progs/CMakeFiles/crc32c_sum.dir/crc32c_sum.cxx.o
> [ 56%] Building CXX object progs/CMakeFiles/odbedit.dir/odbedit.cxx.o
> [ 57%] Linking CXX executable crc32c_sum
> [ 58%] Linking CXX executable mh2sql
> [ 58%] Built target crc32c_sum
> [ 59%] Building CXX object progs/CMakeFiles/odbedit.dir/cmdedit.cxx.o
> [ 59%] Linking CXX executable mfe_link_test_cxx
> Scanning dependencies of target mdump
> [ 59%] Built target mh2sql
> Scanning dependencies of target mhdump
> [ 60%] Building CXX object progs/CMakeFiles/mdump.dir/mdump.cxx.o
> [ 60%] Building CXX object progs/CMakeFiles/mhdump.dir/mhdump.cxx.o
> [ 60%] Built target mfe_link_test_cxx
> Scanning dependencies of target lazylogger
> [ 61%] Building CXX object progs/CMakeFiles/lazylogger.dir/lazylogger.cxx.o
> Scanning dependencies of target mtransition
> [ 61%] Building CXX object progs/CMakeFiles/mtransition.dir/mtransition.cxx.o
> [ 62%] Linking CXX executable mdump
> [ 63%] Linking CXX executable rmlogger
> [ 63%] Built target mdump
> Scanning dependencies of target mserver
> [ 64%] Building CXX object progs/CMakeFiles/mserver.dir/mserver.cxx.o
> [ 65%] Linking CXX executable mtransition
> [ 65%] Built target rmlogger
> Scanning dependencies of target mhist
> [ 66%] Building CXX object progs/CMakeFiles/mhist.dir/mhist.cxx.o
> [ 66%] Built target mtransition
> Scanning dependencies of target get_record_test
> [ 67%] Building CXX object progs/CMakeFiles/get_record_test.dir/get_record_test.cxx.o
> [ 68%] Linking CXX executable mhdump
> Scanning dependencies of target msequencer
> [ 68%] Building CXX object progs/CMakeFiles/msequencer.dir/msequencer.cxx.o
> [ 68%] Built target mhdump
> Scanning dependencies of target fetest
> [ 69%] Linking CXX executable odbedit
> [ 70%] Building CXX object progs/CMakeFiles/fetest.dir/fetest.cxx.o
> [ 70%] Built target odbedit
> [ 70%] Linking CXX executable mserver
> Scanning dependencies of target mstat
> [ 70%] Linking CXX executable get_record_test
> [ 71%] Building CXX object progs/CMakeFiles/mstat.dir/mstat.cxx.o
> [ 71%] Built target mserver
> Scanning dependencies of target mlogger
> [ 71%] Built target get_record_test
> [ 71%] Building CXX object progs/CMakeFiles/mlogger.dir/mlogger.cxx.o
> Scanning dependencies of target analyzer
> [ 71%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/analyzer.cxx.o
> [ 71%] Linking CXX executable mhist
> [ 71%] Linking CXX executable fetest
> [ 72%] Linking CXX executable lazylogger
> [ 72%] Built target mhist
> [ 72%] Built target fetest
> [ 73%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/adccalib.cxx.o
> [ 74%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/adcsum.cxx.o
> [ 74%] Built target lazylogger
> Scanning dependencies of target frontend
> [ 74%] Building CXX object examples/experiment/CMakeFiles/frontend.dir/frontend.cxx.o
> [ 75%] Linking CXX executable mstat
> Scanning dependencies of target mscb_fe
> [ 76%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/mscb_fe.cxx.o
> [ 76%] Built target mstat
> [ 76%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/scaler.cxx.o
> [ 77%] Linking CXX executable frontend
> Scanning dependencies of target scfe
> [ 77%] Built target frontend
> [ 77%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/class/hv.cxx.o
> [ 78%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/class/multi.cxx.o
> [ 79%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/scfe.cxx.o
> Scanning dependencies of target mtfe
> [ 80%] Building CXX object examples/mtfe/CMakeFiles/mtfe.dir/mtfe.cxx.o
> [ 81%] Linking CXX executable analyzer
> [ 81%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/class/hv.cxx.o
> [ 82%] Linking CXX executable mtfe
> [ 82%] Built target analyzer
> [ 83%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/device/nulldev.cxx.o
> Scanning dependencies of target rpc_srvr
> [ 83%] Built target mtfe
> [ 84%] Building CXX object examples/lowlevel/CMakeFiles/rpc_srvr.dir/rpc_srvr.cxx.o
> [ 85%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/class/multi.cxx.o
> Scanning dependencies of target rpc_clnt
> [ 85%] Building CXX object examples/lowlevel/CMakeFiles/rpc_clnt.dir/rpc_clnt.cxx.o
> [ 85%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/bus/null.cxx.o
> [ 85%] Linking CXX executable rpc_srvr
> [ 85%] Built target rpc_srvr
> [ 86%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/device/nulldev.cxx.o
> [ 87%] Linking CXX executable rpc_clnt
> [ 88%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/device/mscbdev.cxx.o
> Scanning dependencies of target rpc_test
> [ 89%] Building CXX object examples/lowlevel/CMakeFiles/rpc_test.dir/rpc_test.cxx.o
> [ 89%] Built target rpc_clnt
> [ 90%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/mscb/src/mscb.cxx.o
> [ 90%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/bus/null.cxx.o
> [ 91%] Linking CXX executable mlogger
> Scanning dependencies of target consume
> [ 92%] Building CXX object examples/lowlevel/CMakeFiles/consume.dir/consume.cxx.o
> [ 93%] Linking CXX executable rpc_test
> [ 94%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/device/mscbdev.cxx.o
> [ 94%] Built target mlogger
> Scanning dependencies of target produce
> [ 94%] Built target rpc_test
> [ 94%] Building CXX object examples/lowlevel/CMakeFiles/produce.dir/produce.cxx.o
> [ 95%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/mscb/src/mscb.cxx.o
> [ 96%] Linking CXX executable msequencer
> [ 96%] Built target msequencer
> [ 96%] Linking CXX executable consume
> [ 96%] Built target consume
> [ 97%] Linking CXX executable produce
> [ 97%] Built target produce
> [ 98%] Linking CXX executable mscb_fe
> [ 98%] Built target mscb_fe
> [ 99%] Linking CXX executable scfe
> [ 99%] Built target scfe
> [100%] Linking CXX executable mhttpd
> [100%] Built target mhttpd
> /midas/build$ 

> url contains midas.js -> send_resource("midas.js")

mhttpd looks for resource files in these directories in this order:

(ODB /experiment/Resources)/filename ### this ODB entry is not created automatically (hidden)
./filename                       ### for testing custom files, start mhttpd in the directory with the test files
./resources/filename     ### ditto
$MIDAS_DIR/filename   ### per experiment custom files or overwrite of midas standard files
$MIDAS_DIR/resources/filename
$MIDASSYS/resources/filename ### standard midas resource files live here: midas.js, midas.css, etc

K.O.