> Ubuntu LTS 18.04 does not run a restrictive firewall and access to mhttpd ports 8080 and 8443 is not 
blocked.

Clearly, the present defaults settings of mhttpd are out of date.

The best I remember our internal discussions, we have converged on the following new default settings:

- mhttpd only listens on the localhost interface
- only accepts http (not https)
- password protection is off

These settings allow one to easily test midas on a laptop or on a single-user computer.

They also happen to be the correct settings when using an https proxy (i.e. apache httpd).

If the https proxy cannot be on the same computer, (i.e. ALPHA at CERN):

- one would enable mhttpd to listen on the external network interface
- this will enable the mhttpd access controls (ODB /expt/security/mhttpd hosts/allowed hosts)
- one would allow the https proxy machine access to mhttpd by adding it's hostname to "allowed hosts".

In the case where a separate https proxy cannot be used:

- one would enable https on the external network interface
- one would have to obtain an https certificate (there is possibility of adding certbot integration to mhttpd, 
if there is demand for this)
- this will activate the mhttpd password protection, so one would have to define a username and password 
in the .htdigest file (this is done by the mongoose web server library).

I was planning to implement these changes when I update the mongoose web server library to the latest 
version (fixes a memory leak and improves/simplifies multithreading).

But maybe I should implement them sooner.

I am also thinking of adding a proxy function to mhttps (same as "ProxyPass" in apache httpd), set ODB 
/Proxy/webcam to "http://webcam_on_private_network/magic_webcam_url", and access to 
https://midas/webcam will return the data from the webcam without having to set this up in apache httpd 
(requires root access, etc).

K.O.

> - this will activate the mhttpd password protection, so one would have to define a username and password 
> in the .htdigest file (this is done by the mongoose web server library).

Actually I'm thinking since a while to have user-level access to mhttpd, similarly to elog. Each user has to log in with a unique username/password. After some time of inactivity, you're logged out. This would have 
the advantage that one knows who is active where, like when using the chat functionality in mhttpd. Or who started/stopped a run etc. This might not be necessary for simple local installations, but if you have 20 
people controlling an experiment from three different continents simultaneously, this could be beneficial. Using the elog authentication libraries, one could even forward the login process to LDAP or KERBEROS, 
so you could log in with out institutional account, and don't have to remember an additional password.

Just some food for thought.

Stefan

> > - this will activate the mhttpd password protection, so one would have to define a username and password 
> > in the .htdigest file (this is done by the mongoose web server library).
> 
> Actually I'm thinking since a while to have user-level access to mhttpd, similarly to elog.
>

With per-user login, we have the possibility to add better permissions/access controls. In past
discussions we talked about 3 levels of user access:

- read-only user: can look, but cannot affect anything
- operator: same as read-only user, but can start/stop runs, can clear alarms, can push buttons on custom pages, can cause predefined scripts to run, etc.
- root user: can do everything

Technically, this is easy to implement in the mjsonrpc library: each username will be mapped to a privilege level,
and each rpc request handler will specify minimum required privilege: odb write rpc would require root level,
run start would require operator level, odb read permitted for everybody. This will be enforced inside mhttpd.

>
> Each user has to log in with a unique username/password. After some time of inactivity, you're logged out.
>

For now, we use the password protection built into the apache httpd web server.

It is known to be secure, but it does not have the "advanced" user management functions
that we take for granted with the elog, with wiki pages, with github, etc. Missing are self-registration
with approval, password reset and recovery and so forth.

On the other hand, apache httpd is supposed to be easy to integrate with "enterprise" user management
systems, like the CERN single-sign-on system. (We did not look yet at the integration with the TRIUMF
single-sign-on system, based on Microsoft AD).

(I see the nginx web server is gaining in popularity, but I do not know what features it has
for user and password management).

The elog software does have very good user and password management, and we could bring it into midas,
if we figure out how to ensure that it is actually secure. I know a professional security audit was done
for the elog software and I know that mhttpd will not pass such an audit.

But with some extra work it is possible.

>
> This would have the advantage that one knows who is active where, like when using the chat functionality in mhttpd. Or who started/stopped a run etc. This might not be necessary for simple local installations, but if you have 20 
> people controlling an experiment from three different continents simultaneously, this could be beneficial. Using the elog authentication libraries, one could even forward the login process to LDAP or KERBEROS, 
> so you could log in with out institutional account, and don't have to remember an additional password.
> 
> Just some food for thought.
> 

Some of this food looks very good, indeed.

K.O.

> The modbset() function is used in many custom pages at PSI ...

I restored this function in midas.js with a documentation blurb warning about it's asynchronous nature and about the possibility of out-of-order writes.

The more I think about this, the more it looks to me that we should look at this web worker api business to support
synchronous communications for MIDAS web pages.

K.O.

I'd like to use the mjsonrpc db_get_values method, but (as indicated in the
documentation) it returns all ODB keys as lowercase.

This breaks quite a lot of my code - it was written with the old AJAX commands,
and these did respect the capitalization of the ODB keys.

Would it be possible to add a capitalization-preserve option to db_get_values? 

It appears I needed to read the documentation more closely - the method db_save
does respect key-name capitalization and solves my problem.

Is db_save considered a deprecated method?  If so, I'd reiterate my suggestion for
a capitalize-preserve option for db_get_values.

Otherwise, I'll plan on using db_save.

> I'd like to use the mjsonrpc db_get_values method, but (as indicated in the
> documentation) it returns all ODB keys as lowercase.
> 
> This breaks quite a lot of my code - it was written with the old AJAX commands,
> and these did respect the capitalization of the ODB keys.
> 
> Would it be possible to add a capitalization-preserve option to db_get_values? 

Hi Amy,

Let me start by explaining the reasoning for the default behavior of db_get_values.  I think it was mentioned elsewhere, but is worth repeating.

The ODB is case-insensitive.  So the ODB key name /Equipment/dcrc01 is equivalent to /equipment/Dcrc01; you could rename the variable like that and your 
frontend programs would still work fine.  Javascript, of course, is case sensitive.  However, we want our default MIDAS webpages to work no matter what the 
capitalization is for a particular ODB; so, for instance, the main status.html page should work whether the ODB key is called /Runinfo or /rUnInFo, since both 
of these are equivalent from the point of the ODB (and the rest of MIDAS).

The solution was to have the db_get_values method convert all key names to lower case and consistently use the lower case spelling when writing the main 
MIDAS webpages; this makes us insensitive to ODB capitalization (and hence makes the MIDAS pages behaviour match the previous mhttpd behaviour).

That being said, I agree that it is sometimes counter-intuitive to use lower case key names with db_get_values, particularly if you are directly creating ODB 
keys and writing the javascript at the same time.  So we have added the option 'preserve_case' to db_get_values, which preserves the ODB key name 
capitalization (the default behaviour is still to make key names lower case).

This option should not be used for writing any standard MIDAS webpages (ie, webpages that will be used across multiple experiments), since standard MIDAS 
webpages should not break when ODB key name capitalization changes.  For the same reason you should use caution with this option for custom pages as 
well.

With regards to your second question: the db_save method is not deprecated and you could use that method instead.  The use-case for the db_save method 
is different; db_save is used to make dumps of the ODB.  In that case it seems best that key name capitalization is preserved.  Otherwise if you dumped your 
whole ODB and then reloaded it from the dump the new ODB would be different (in key capitalization) from the old ODB; different in a way that shouldn't 
matter but still probably not the behaviour that people expect.

Admittedly this means that the mjsonrpc API is not always intuitive; but I think is the best we can do, given the underlying case-insensitivity of the ODB.

Thomas


> It appears I needed to read the documentation more closely - the method db_save
> does respect key-name capitalization and solves my problem.
> 
> Is db_save considered a deprecated method?  If so, I'd reiterate my suggestion for
> a capitalize-preserve option for db_get_values.
> 
> Otherwise, I'll plan on using db_save.
> 
> > I'd like to use the mjsonrpc db_get_values method, but (as indicated in the
> > documentation) it returns all ODB keys as lowercase.
> > 
> > This breaks quite a lot of my code - it was written with the old AJAX commands,
> > and these did respect the capitalization of the ODB keys.
> > 
> > Would it be possible to add a capitalization-preserve option to db_get_values? 

> url contains midas.js -> send_resource("midas.js")

mhttpd looks for resource files in these directories in this order:

(ODB /experiment/Resources)/filename ### this ODB entry is not created automatically (hidden)
./filename                       ### for testing custom files, start mhttpd in the directory with the test files
./resources/filename     ### ditto
$MIDAS_DIR/filename   ### per experiment custom files or overwrite of midas standard files
$MIDAS_DIR/resources/filename
$MIDASSYS/resources/filename ### standard midas resource files live here: midas.js, midas.css, etc

K.O.

Request:

Please use "make cmake" ("make cmake3") to report build problems.

cmake with VERBOSE=1 prints way too much stuff and I find it is impossible to use.

"make cmake" filters out all the gunk to make it easy to see the relevant information:
- the compiler command lines (CFLAGS, library paths, etc)
- the compiler output (error messages, etc).

The best I can tell, cmake "VERBOSE=1" in only useful for cmake developers to debug cmake itself.

K.O.

> I tried on my Mac (macOS 10.14.6, Xcode 11.3.1, current develop branch, openssl 1.1.1d) and it woks fine. Below is the transcript. I 
> see that your cmake output is much shorter (no C compiler listed etc.). Did you remove some lines? For such comparisons, it's 
> always good to start with an empty build directory.
> 
> OpenSSL libraries are the same (1.1.1d). Just for comparison, I run the build process with "make VERBOSE=1" and extract the line 
> which fails for you when linking mhttp, so maybe you can compare.
> 
> Linking mhttpd
> ---------------
> 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -O2 -g -DNDEBUG -isysroot 
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-
> min=10.14 -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o 
> CMakeFiles/mhttpd.dir/mongoose6.cxx.o CMakeFiles/mhttpd.dir/mgd.cxx.o CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o 
> mhttpd  -L/opt/local/lib/mysql57/mysql -lmysqlclient -lz ../libmidas.a /opt/local/lib/libssl.dylib /opt/local/lib/libcrypto.dylib -lz -
> L/opt/local/lib/mysql57/mysql -lmysqlclient -lz 
> 
> 
> Here is the full transcript
> -------------------------
> 
> /midas/build$ cmake ..
> -- MIDAS: cmake version: 3.16.3
> -- The C compiler identification is AppleClang 11.0.0.11000033
> -- The CXX compiler identification is AppleClang 11.0.0.11000033
> -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
> -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- 
> works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Detecting C compile features
> -- Detecting C compile features - done
> -- Check for working CXX compiler: 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
> -- Check for working CXX compiler: 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Detecting CXX compile features
> -- Detecting CXX compile features - done
> -- MIDAS: CMAKE_INSTALL_PREFIX: /midas
> -- MIDAS: Found ROOT version 6.16/00
> -- Found ZLIB: /opt/local/lib/libz.dylib (found version "1.2.11") 
> -- MIDAS: Found ZLIB version 1.2.11
> -- Found OpenSSL: /opt/local/lib/libcrypto.dylib (found version "1.1.1d")  
> -- MIDAS: Found OpenSSL version 1.1.1d
> -- MIDAS: Found MySQL version 5.7.26
> -- MIDAS: ODBC not found
> -- MIDAS: SQLITE not found
> -- MIDAS: nvidia-smi not found
> -- Setting default build type to "RelWithDebInfo"
> -- Found Git: /usr/bin/git (found version "2.21.1 (Apple Git-122.3)") 
> -- MIDAS example/experiment: MIDAS in-tree-build
> -- MIDAS: Found ZLIB version 1.2.11
> -- MIDAS example/experiment: Found ROOT version 6.16/00
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /midas/build
> /midas/build$ make
> Scanning dependencies of target rmana
> Scanning dependencies of target git_revision_h
> Scanning dependencies of target mfe
> Scanning dependencies of target manao
> Scanning dependencies of target mfeo
> Scanning dependencies of target objlib-c-compat
> Scanning dependencies of target mana
> Scanning dependencies of target rmanao
> [  0%] Building CXX object CMakeFiles/objlib-c-compat.dir/src/midas_c_compat.cxx.o
> [  2%] Building CXX object CMakeFiles/mfe.dir/src/mfe.cxx.o
> [  2%] Building CXX object CMakeFiles/mfeo.dir/src/mfe.cxx.o
> [  3%] Building CXX object CMakeFiles/mana.dir/src/mana.cxx.o
> [  4%] Building CXX object CMakeFiles/manao.dir/src/mana.cxx.o
> [  5%] Building CXX object CMakeFiles/rmanao.dir/src/mana.cxx.o
> [  6%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
> [  6%] Built target git_revision_h
> Scanning dependencies of target objlib
> [  7%] Building CXX object CMakeFiles/objlib.dir/src/midas.cxx.o
> [  7%] Built target objlib-c-compat
> [  8%] Building CXX object CMakeFiles/objlib.dir/src/midas_cxx.cxx.o
> [  8%] Linking CXX static library libmana.a
> [  8%] Built target manao
> [  8%] Building CXX object CMakeFiles/objlib.dir/src/odb.cxx.o
> [  8%] Built target mana
> [  9%] Building CXX object CMakeFiles/objlib.dir/src/device_driver.cxx.o
> [  9%] Built target mfeo
> [ 10%] Linking CXX static library libmfe.a
> [ 11%] Building CXX object CMakeFiles/objlib.dir/src/system.cxx.o
> [ 11%] Built target mfe
> [ 11%] Building CXX object CMakeFiles/objlib.dir/src/alarm.cxx.o
> [ 12%] Building CXX object CMakeFiles/objlib.dir/src/elog.cxx.o
> [ 13%] Building CXX object CMakeFiles/objlib.dir/src/mrpc.cxx.o
> [ 13%] Building CXX object CMakeFiles/objlib.dir/src/mjson.cxx.o
> [ 14%] Building CXX object CMakeFiles/objlib.dir/src/tmfe.cxx.o
> [ 15%] Building CXX object CMakeFiles/objlib.dir/src/mvodb.cxx.o
> [ 15%] Built target rmanao
> [ 15%] Linking CXX static library librmana.a
> [ 16%] Building CXX object CMakeFiles/objlib.dir/src/nullodb.cxx.o
> [ 16%] Building CXX object CMakeFiles/objlib.dir/src/midasodb.cxx.o
> [ 16%] Built target rmana
> [ 17%] Building CXX object CMakeFiles/objlib.dir/src/mxmlodb.cxx.o
> [ 18%] Building CXX object CMakeFiles/objlib.dir/src/mjsonodb.cxx.o
> [ 18%] Building CXX object CMakeFiles/objlib.dir/src/json_paste.cxx.o
> [ 19%] Building CXX object CMakeFiles/objlib.dir/src/mjsonrpc.cxx.o
> [ 20%] Building CXX object CMakeFiles/objlib.dir/src/mjsonrpc_user.cxx.o
> [ 21%] Building CXX object CMakeFiles/objlib.dir/src/history.cxx.o
> [ 21%] Building CXX object CMakeFiles/objlib.dir/src/history_common.cxx.o
> [ 22%] Building CXX object CMakeFiles/objlib.dir/src/history_odbc.cxx.o
> [ 23%] Building CXX object CMakeFiles/objlib.dir/src/history_schema.cxx.o
> [ 23%] Building CXX object CMakeFiles/objlib.dir/src/lz4.cxx.o
> [ 24%] Building CXX object CMakeFiles/objlib.dir/src/lz4frame.cxx.o
> [ 25%] Building CXX object CMakeFiles/objlib.dir/src/lz4hc.cxx.o
> [ 26%] Building CXX object CMakeFiles/objlib.dir/src/xxhash.cxx.o
> [ 26%] Building CXX object CMakeFiles/objlib.dir/src/crc32c.cxx.o
> [ 27%] Building CXX object CMakeFiles/objlib.dir/src/sha256.cxx.o
> [ 28%] Building CXX object CMakeFiles/objlib.dir/src/sha512.cxx.o
> [ 28%] Building CXX object CMakeFiles/objlib.dir/src/ftplib.cxx.o
> [ 29%] Building CXX object CMakeFiles/objlib.dir/src/mdsupport.cxx.o
> [ 30%] Building CXX object CMakeFiles/objlib.dir/mxml/mxml.cxx.o
> [ 31%] Building CXX object CMakeFiles/objlib.dir/mxml/strlcpy.cxx.o
> [ 31%] Built target objlib
> Scanning dependencies of target midas-shared
> Scanning dependencies of target midas
> [ 33%] Linking CXX shared library libmidas-shared.dylib
> [ 33%] Linking CXX static library libmidas.a
> [ 33%] Built target midas-shared
> [ 33%] Built target midas
> Scanning dependencies of target mhttpd
> Scanning dependencies of target feudp
> Scanning dependencies of target odb_lock_test
> Scanning dependencies of target rmlogger
> Scanning dependencies of target mchart
> Scanning dependencies of target msysmon
> Scanning dependencies of target midas-c-compat
> Scanning dependencies of target mfe_link_test
> [ 33%] Building CXX object progs/CMakeFiles/mfe_link_test.dir/mfe_link_test.cxx.o
> [ 33%] Linking CXX shared library libmidas-c-compat.dylib
> [ 34%] Building CXX object progs/CMakeFiles/feudp.dir/feudp.cxx.o
> [ 35%] Building CXX object progs/CMakeFiles/msysmon.dir/msysmon.cxx.o
> [ 36%] Building CXX object progs/CMakeFiles/mchart.dir/mchart.cxx.o
> [ 37%] Building CXX object progs/CMakeFiles/odb_lock_test.dir/odb_lock_test.cxx.o
> [ 38%] Building CXX object progs/CMakeFiles/rmlogger.dir/mlogger.cxx.o
> [ 39%] Building CXX object progs/CMakeFiles/mhttpd.dir/mhttpd.cxx.o
> [ 39%] Built target midas-c-compat
> [ 40%] Building CXX object progs/CMakeFiles/mhttpd.dir/mongoose6.cxx.o
> [ 41%] Linking CXX executable mfe_link_test
> [ 42%] Linking CXX executable odb_lock_test
> [ 42%] Built target mfe_link_test
> Scanning dependencies of target mjson_test
> [ 43%] Building CXX object progs/CMakeFiles/mjson_test.dir/mjson_test.cxx.o
> [ 43%] Linking CXX executable mchart
> [ 43%] Built target odb_lock_test
> Scanning dependencies of target odbinit
> [ 44%] Building CXX object progs/CMakeFiles/odbinit.dir/odbinit.cxx.o
> [ 45%] Linking CXX executable feudp
> [ 45%] Built target mchart
> [ 46%] Building CXX object progs/CMakeFiles/mhttpd.dir/mgd.cxx.o
> [ 46%] Built target feudp
> [ 46%] Building CXX object progs/CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o
> [ 47%] Linking CXX executable mjson_test
> [ 47%] Built target mjson_test
> Scanning dependencies of target fetest_tmfe_thread
> [ 47%] Building CXX object progs/CMakeFiles/fetest_tmfe_thread.dir/fetest_tmfe_thread.cxx.o
> [ 47%] Linking CXX executable msysmon
> [ 47%] Built target msysmon
> [ 48%] Linking CXX executable odbinit
> Scanning dependencies of target fetest_tmfe
> [ 49%] Building CXX object progs/CMakeFiles/fetest_tmfe.dir/fetest_tmfe.cxx.o
> [ 49%] Built target odbinit
> [ 50%] Linking CXX executable fetest_tmfe_thread
> Scanning dependencies of target mh2sql
> [ 51%] Building CXX object progs/CMakeFiles/mh2sql.dir/mh2sql.cxx.o
> Scanning dependencies of target odbhist
> [ 51%] Building CXX object progs/CMakeFiles/odbhist.dir/odbhist.cxx.o
> [ 51%] Built target fetest_tmfe_thread
> Scanning dependencies of target melog
> [ 51%] Building CXX object progs/CMakeFiles/melog.dir/melog.cxx.o
> [ 52%] Linking CXX executable odbhist
> [ 53%] Built target odbhist
> [ 53%] Linking CXX executable melog
> [ 54%] Linking CXX executable fetest_tmfe
> Scanning dependencies of target mfe_link_test_cxx
> [ 55%] Building CXX object progs/CMakeFiles/mfe_link_test_cxx.dir/mfe_link_test_cxx.cxx.o
> [ 55%] Built target melog
> Scanning dependencies of target crc32c_sum
> [ 55%] Built target fetest_tmfe
> Scanning dependencies of target odbedit
> [ 56%] Building CXX object progs/CMakeFiles/crc32c_sum.dir/crc32c_sum.cxx.o
> [ 56%] Building CXX object progs/CMakeFiles/odbedit.dir/odbedit.cxx.o
> [ 57%] Linking CXX executable crc32c_sum
> [ 58%] Linking CXX executable mh2sql
> [ 58%] Built target crc32c_sum
> [ 59%] Building CXX object progs/CMakeFiles/odbedit.dir/cmdedit.cxx.o
> [ 59%] Linking CXX executable mfe_link_test_cxx
> Scanning dependencies of target mdump
> [ 59%] Built target mh2sql
> Scanning dependencies of target mhdump
> [ 60%] Building CXX object progs/CMakeFiles/mdump.dir/mdump.cxx.o
> [ 60%] Building CXX object progs/CMakeFiles/mhdump.dir/mhdump.cxx.o
> [ 60%] Built target mfe_link_test_cxx
> Scanning dependencies of target lazylogger
> [ 61%] Building CXX object progs/CMakeFiles/lazylogger.dir/lazylogger.cxx.o
> Scanning dependencies of target mtransition
> [ 61%] Building CXX object progs/CMakeFiles/mtransition.dir/mtransition.cxx.o
> [ 62%] Linking CXX executable mdump
> [ 63%] Linking CXX executable rmlogger
> [ 63%] Built target mdump
> Scanning dependencies of target mserver
> [ 64%] Building CXX object progs/CMakeFiles/mserver.dir/mserver.cxx.o
> [ 65%] Linking CXX executable mtransition
> [ 65%] Built target rmlogger
> Scanning dependencies of target mhist
> [ 66%] Building CXX object progs/CMakeFiles/mhist.dir/mhist.cxx.o
> [ 66%] Built target mtransition
> Scanning dependencies of target get_record_test
> [ 67%] Building CXX object progs/CMakeFiles/get_record_test.dir/get_record_test.cxx.o
> [ 68%] Linking CXX executable mhdump
> Scanning dependencies of target msequencer
> [ 68%] Building CXX object progs/CMakeFiles/msequencer.dir/msequencer.cxx.o
> [ 68%] Built target mhdump
> Scanning dependencies of target fetest
> [ 69%] Linking CXX executable odbedit
> [ 70%] Building CXX object progs/CMakeFiles/fetest.dir/fetest.cxx.o
> [ 70%] Built target odbedit
> [ 70%] Linking CXX executable mserver
> Scanning dependencies of target mstat
> [ 70%] Linking CXX executable get_record_test
> [ 71%] Building CXX object progs/CMakeFiles/mstat.dir/mstat.cxx.o
> [ 71%] Built target mserver
> Scanning dependencies of target mlogger
> [ 71%] Built target get_record_test
> [ 71%] Building CXX object progs/CMakeFiles/mlogger.dir/mlogger.cxx.o
> Scanning dependencies of target analyzer
> [ 71%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/analyzer.cxx.o
> [ 71%] Linking CXX executable mhist
> [ 71%] Linking CXX executable fetest
> [ 72%] Linking CXX executable lazylogger
> [ 72%] Built target mhist
> [ 72%] Built target fetest
> [ 73%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/adccalib.cxx.o
> [ 74%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/adcsum.cxx.o
> [ 74%] Built target lazylogger
> Scanning dependencies of target frontend
> [ 74%] Building CXX object examples/experiment/CMakeFiles/frontend.dir/frontend.cxx.o
> [ 75%] Linking CXX executable mstat
> Scanning dependencies of target mscb_fe
> [ 76%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/mscb_fe.cxx.o
> [ 76%] Built target mstat
> [ 76%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/scaler.cxx.o
> [ 77%] Linking CXX executable frontend
> Scanning dependencies of target scfe
> [ 77%] Built target frontend
> [ 77%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/class/hv.cxx.o
> [ 78%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/class/multi.cxx.o
> [ 79%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/scfe.cxx.o
> Scanning dependencies of target mtfe
> [ 80%] Building CXX object examples/mtfe/CMakeFiles/mtfe.dir/mtfe.cxx.o
> [ 81%] Linking CXX executable analyzer
> [ 81%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/class/hv.cxx.o
> [ 82%] Linking CXX executable mtfe
> [ 82%] Built target analyzer
> [ 83%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/device/nulldev.cxx.o
> Scanning dependencies of target rpc_srvr
> [ 83%] Built target mtfe
> [ 84%] Building CXX object examples/lowlevel/CMakeFiles/rpc_srvr.dir/rpc_srvr.cxx.o
> [ 85%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/class/multi.cxx.o
> Scanning dependencies of target rpc_clnt
> [ 85%] Building CXX object examples/lowlevel/CMakeFiles/rpc_clnt.dir/rpc_clnt.cxx.o
> [ 85%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/bus/null.cxx.o
> [ 85%] Linking CXX executable rpc_srvr
> [ 85%] Built target rpc_srvr
> [ 86%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/device/nulldev.cxx.o
> [ 87%] Linking CXX executable rpc_clnt
> [ 88%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/device/mscbdev.cxx.o
> Scanning dependencies of target rpc_test
> [ 89%] Building CXX object examples/lowlevel/CMakeFiles/rpc_test.dir/rpc_test.cxx.o
> [ 89%] Built target rpc_clnt
> [ 90%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/mscb/src/mscb.cxx.o
> [ 90%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/bus/null.cxx.o
> [ 91%] Linking CXX executable mlogger
> Scanning dependencies of target consume
> [ 92%] Building CXX object examples/lowlevel/CMakeFiles/consume.dir/consume.cxx.o
> [ 93%] Linking CXX executable rpc_test
> [ 94%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/device/mscbdev.cxx.o
> [ 94%] Built target mlogger
> Scanning dependencies of target produce
> [ 94%] Built target rpc_test
> [ 94%] Building CXX object examples/lowlevel/CMakeFiles/produce.dir/produce.cxx.o
> [ 95%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/mscb/src/mscb.cxx.o
> [ 96%] Linking CXX executable msequencer
> [ 96%] Built target msequencer
> [ 96%] Linking CXX executable consume
> [ 96%] Built target consume
> [ 97%] Linking CXX executable produce
> [ 97%] Built target produce
> [ 98%] Linking CXX executable mscb_fe
> [ 98%] Built target mscb_fe
> [ 99%] Linking CXX executable scfe
> [ 99%] Built target scfe
> [100%] Linking CXX executable mhttpd
> [100%] Built target mhttpd
> /midas/build$ 

The modbset() function in mhttpd.js is not used anywhere in midas and it misleads midas users into thinking that it works like the old ODBSet() function, when 
it can not and it does not.

To explain the difference:

1) ODBSet() used synchronous RPC requests, which have been deprecated by the powers that be. Read more here:
https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests
https://x443.wordpress.com/2012/12/01/why-you-should-use-xmlhttprequest-asynchronously/

2) in midas, we followed these instructions and developed an asynchronous RPC mechanism for calling midas functions from javascript. (we use the Promise 
construct, but the underlying JSON-RPC compatible communications can be used directly, without it).

3) using the asynchronous RPC is not as easy as the old ODBSet() & co - instead of just making a call "to write to ODB", one has to create a chain of nested 
event handler functions and one has to do at least some error handling.

4) this makes it impossible to program midas custom pages in javascript as if it were C/C++. (Please direct your complaints to the "web" and "javascript" 
powers that be).

5) to help writing midas custom pages, we have a good number of examples. For example, example.html has example
code for calling pretty much every midas json rpc function.

5a) to see the complete list of all rpc functions available in your copy of midas, follow the link to "json-rpc schema, text format" on the midas "help" page.

6) if you are writing a new custom page we suggest you start with one of the example templates in .../resources, a_example, a_template.

7) if you are updating an existing custom page, good luck. synchronous rpc seems to still work in most browsers, so the old OSBSet() & co should continue to 
work for now. For new code you should use the async rpc (with Promises, like we do for all midas pages). In practice this means a complete rewrite of each 
custom page (welcome to the 21st century).

Note that we have two separate js files in midas:

- midas.js is intended as a general purpose library for writing midas custom pages
- mhttpd.js is not intended for general use and contains javascript code used by mhttpd internally

The function itself is here, in case somebody needs it:

-function modbset(path, value)
-/* shortcut for mjsonrpc_db_paste() with standard error handling */
-{
-   if (Array.isArray(path)) {
-      mjsonrpc_db_paste(path,value).then(function(rpc) {}).catch(function(error) {
-         mjsonrpc_error_alert(error); });
-   } else {
-      mjsonrpc_db_paste([path],[value]).then(function(rpc) {}).catch(function(error) {
-         mjsonrpc_error_alert(error); });
-   }
-}
-

K.O.

I disagree. The modbset() function is used in many custom pages at PSI because people are tired of typing mjsonrpc_db_paste([path],[value]) vs. modbset(path, value). We need to keep 
modbset() which is well documented at 

https://midas.triumf.ca/MidasWiki/index.php/Custom_Page#modbset

Since modbset() does call the underlying mjsonrpc_db_paste(), it is as good or bad as that function. Plus it adds standard error handling to avoid the need of catching errors for each and 
every mjsonrpc_db_paste() call. If it is believed that modbset() has a problem, then this should be fixed in the source code of modbset(). Removing that function is not an option.

Stefan

If it's a function intended for general use, it should be in midas.js.

The documentation for such a function should be made very clear that:

a) it does not actually write to ODB, instead it queues a request for writing, this request is executed at a later (undefined) time.

b) the following javascript code results in undefined behaviour:

modbset("/foo/bar", 1); // queue rpc request
modbset("/foo/bar", 2); // queue rpc request
// is ODB /foo/bar set to 1 or 2?

Why? The best I know javascript does not require for RPCs to execute in-order, so the second RPC may be issued
before the first one. More likely both RPCs are started roughly at the same time (i.e. in different RPC worker
threads), in which case we do not know in which order they will be processed by mhttpd, which is also
multithreaded and does not necessarily execute requests in the same order as (i.e.) they connect to the rpc port 8080.

To answer the question "1 or 2", the answer is neither, as at that point in the code, the RPC requests
probably have not started executing yet, and even if they did, mhttpd most likely did not write anything
into odb yet, as processing RPC requests takes much longer than executing a few lines of javascript.

So to ensure correct sequence of writes and to ensure that something was actually written to odb,
one has to roll out the full ladder of promise event handlers.

Is correct sequence of writes important? Maybe yes, maybe no.

But if you use modbset() without being aware of these issues, you will write code
for ramping high voltage like this:

modbset("/eq/hv/set/voltage", 0); // set voltage to zero
modbset("/eq/hv/set/hv_enable", 1); // enable high voltage
modbset("/eq/hv/set/voltage", 50); // start slow
modbset("/eq/hv/set/voltage", 1000); // ramp to half-way
modbset("/eq/hv/set/voltage", 1900); // stop a bit before the final voltage
modbset("/eq/hv/set/voltage", 2000); // !!! ramping from 0 to 2000 should never be done in one step !!!

And as the author of all this RPC code, I promise that some day you will see the voltage
on the detector go directly from 0 to 2000 (then up and down to 50, 1000 and 1900).

To me, this makes helpful helper functions actually dangerous to use.

Now that I have experience with sync RPC (from C/C++) and async RPC (with Promises in javascript),
I would say that synchronous C/C++ style programming is much easier
and much less verbose and much easier to read and to modify/adjust compared to the javascript-style
ladder of nested event handlers.

I now see that synchronous requests are again permitted if one uses the "Web Worker API". Maybe we should revisit
the MIDAS javascript RPC code and see if we can use this web worker stuff to officially support synchronous RPC requests,
again.

About the sync rpc deprecation, read more here:
https://stackoverflow.com/questions/30876093/will-chrome-and-other-browsers-drop-support-for-synchronous-xmlhttprequest

K.O.


> I disagree. The modbset() function is used in many custom pages at PSI because people are tired of typing mjsonrpc_db_paste([path],[value]) vs. modbset(path, value). We need to keep 
> modbset() which is well documented at 
> 
> https://midas.triumf.ca/MidasWiki/index.php/Custom_Page#modbset
> 
> Since modbset() does call the underlying mjsonrpc_db_paste(), it is as good or bad as that function. Plus it adds standard error handling to avoid the need of catching errors for each and 
> every mjsonrpc_db_paste() call. If it is believed that modbset() has a problem, then this should be fixed in the source code of modbset(). Removing that function is not an option.
> 
> Stefan

A ladder of promise event handlers is certainly one possibility to enforce the order of ODB writes, but I wonder if we could so something simpler:

- modbset creates an object remembering the status of the RPC request. Initially, this object receives the status "open request"
- when the rpc call got executed successfully, the callback sets the state of the above object to "request succeeded" or "request failed" (in case of error)
- if a new modbset comes BEFORE the previous one has completed, the function queues the new request in a data field of the above object
- if a rpc call finishes, and a queued new rpc request is present, it gets executed

This would be relatively easy to be implemented and keep the order of the rpc calls. Does that make sense?

Best,
Stefan

> A ladder of promise event handlers is certainly one possibility to enforce the order of ODB writes, but I wonder if we could so something simpler:
> 
> - modbset creates an object remembering the status of the RPC request. Initially, this object receives the status "open request"
> - when the rpc call got executed successfully, the callback sets the state of the above object to "request succeeded" or "request failed" (in case of error)
> - if a new modbset comes BEFORE the previous one has completed, the function queues the new request in a data field of the above object
> - if a rpc call finishes, and a queued new rpc request is present, it gets executed
> 
> This would be relatively easy to be implemented and keep the order of the rpc calls. Does that make sense?
> 

Yes, this is a neat idea, I am really happy with how a complete rpc request can be held by one object, and we can make queues of them, etc.

Anyhow here is the proof of the pudding. I added a test to example.html, there are two buttons, one makes 5 modbset() calls, second has a ladder of 5 db_paste calls. Then I watch 
the result in odbedit. 1, 2, 3, 4, 5 is the modbset(), 6, 7, 8, 9, 10 is the ladder of db_paste calls:

$ odbedit
[local:javascript1:S]/>watch Example/int
Watch key "/Example/int" to be modified, abort with any key

/Example/int = 1
/Example/int = 2
/Example/int = 3
/Example/int = 4
/Example/int = 5

/Example/int = 1
/Example/int = 5 <== fault
/Example/int = 5
/Example/int = 5
/Example/int = 5

/Example/int = 1
/Example/int = 2
/Example/int = 3
/Example/int = 5 <== 4 and 5 reversed
/Example/int = 4 <== 4 and 5 reversed

/Example/int = 6
/Example/int = 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

/Example/int = 6
/Example/int = 8 <== should be 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

/Example/int = 6
/Example/int = 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

I immediately notice that we have a race condition between the RPCs, db_watch notifications and db_get_value() in the watch handler:
there are 5 rpcs, 5 watch notifications, 5 calls to db_get_value() in the watch handler, but sometimes the handler is too slow
and the data in odb changes before it reads it, thus duplicate values (missing "7" above). (The old db_open_record() had a "hidden"
db_get_value() inside it, while db_watch() requires an explicit db_get_value() call, making it obvious why we get
the wrong (newer) data sometimes).

Possible fixes for this is to slow down the RPCs (the race condition is still there, probability is reduced) or send the changed
data as part of the notification. If this were C/C++, a "sleep(1)" between modbset() calls would have fixed it,
but there is no sleeping and waiting in javascript. (I guess one could use a ladder of timers).

Other than that, I am surprised how easy it was to see that indeed out-of-order RPCs can happen, see the case
of out-of-order 4 and 5 above. It only took me maybe 5-10 clicks on the button to see that. I expected that I would
need to try several browsers or use a slow network connection, but here it is, on my home mac, localhost network,
google chrome browser.

Below is the test code. I do NOT vote that everybody should use ladders of db_paste calls.

function test_modbset() {
           modbset("/example/int", 1);
           modbset("/example/int", 2);
           modbset("/example/int", 3);
           modbset("/example/int", 4);
           modbset("/example/int", 5);
        }

        function test_chained_db_paste() {
           var paths = [ "/example/int" ];
           mjsonrpc_db_paste(paths,[6]).then(function(rpc) {
              mjsonrpc_db_paste(paths,[7]).then(function(rpc) {
                 mjsonrpc_db_paste(paths,[8]).then(function(rpc) {
                    mjsonrpc_db_paste(paths,[9]).then(function(rpc) {
                       mjsonrpc_db_paste(paths,[10]).then(function(rpc) {
                          // nothing
                       }).catch(function(error){mjsonrpc_error_alert(error);});
                    }).catch(function(error){mjsonrpc_error_alert(error);});
                 }).catch(function(error){mjsonrpc_error_alert(error);});
              }).catch(function(error){mjsonrpc_error_alert(error);});
           }).catch(function(error){mjsonrpc_error_alert(error);});
        }

        </script>

        <input type=button value='test modbset()' onClick='test_modbset();'></input>
        <input type=button value='test chained db_paste()' onClick='test_chained_db_paste();'></input>

K.O.

Just to make this point clear: The "write-to-odb-read-via-hotlink" was never meant to guarantee the receiving side to see each change. If changes happen too often, updates might get lost. If one relies on the 
sequence of updates, one should use direct RPC calls to the frontend or use a midas buffer and encode updates in events.

Stefan

> Just to make this point clear: The "write-to-odb-read-via-hotlink" was never meant to guarantee the receiving side to see each change. If changes happen too often, updates might get lost. If one relies on the 
> sequence of updates, one should use direct RPC calls to the frontend or use a midas buffer and encode updates in events.

I recommend that people use the jrpc mechanism that does an RPC directly from javascript into the frontend.
It passes 2 strings as arguments (command and data value). Arbitrary objects can be passed by encoding
the data in json (use mjson.h to decode it in the frontend). A string is returned back to javascript (again, encode
arbitrary data as json, use the mjson.h library).

Call sequence:
javascript -> (http) -> mhttpd -> (MIDAS RPC call) -> frontend -> (write, read, frob hardware) -> frontend -> (MIDAS RPC reply) -> mhttpd -> (http reply) -> javascript

Example of all this is in example.html and fetest.cxx:

javascript side code: mjsonrpc_call("jrpc", { "client_name":"fetest", "cmd":"xxx", "args":"xxx" })

frontend side code:

INT rpc_callback(INT index, void *prpc_param[])
{
   const char* cmd  = CSTRING(0);
   const char* args = CSTRING(1);
   char* return_buf = CSTRING(2);
   int   return_max_length = CINT(3);
   cm_msg(MINFO, "rpc_callback", "--------> rpc_callback: index %d, max_length %d, cmd [%s], args [%s]", index, return_max_length, cmd, args);
   ... do stuff ... put result into string "tmp"
   strlcpy(return_buf, tmp, return_max_length);
   return RPC_SUCCESS;
}

... somewhere in frontend_init(), register the RPC:

#ifdef RPC_JRPC
   status = cm_register_function(RPC_JRPC, rpc_callback);
   assert(status == SUCCESS);
#endif

K.O.

Our VxWorks C compiler (gcc-2.8-something) does not like the "//" comments. Luckily, on VxWorks, we 
only compile a small subset of midas, so there is no point in banning all "//" comments. But I did have to 
convert a couple of them to /* commens */ in odb.c to make it compile. Changes to odb.c commited. K.O.

cm_watchdog() has been removed from the latest midas sources. The watchdog functions performed by cm_watchdog() were 
moved to cm_yield() - those are - maintaining odb and event buffer "last active" timestamps and checking for and removing of 
timed-out clients.

Those who write midas programs should ensure that they call cm_yield() at least every 1-2 seconds (for the normal 10 second 
timeout settings). As always, before calling potentially time consuming operations, such as accessing slow responding 
hardware, one should increase or disable their watchdog timeout.

Those who write midas programs that do not use cm_yield() (i.e. the mserver), should call cm_periodic_tasks() instead with 
the same period as described for cm_yield() above.

Removal of cm_watchdog() solves many problems in the midas code base:
- firing of cm_watchdog() at random times in random places makes it difficult to do static code path analysis (call paths, etc) - 
this is needed to ensure correct multithread locking, etc
- ditto caused trouble with multithread locking when cm_watchdog() fires *inside* the pthread mutex locking library itself 
causing mutexes to be in an inconsistent state. This had to be kludged against in the ODB multithread locks - now this kludge 
can be removed.
- many non-midas codes in experiment frontends, etc was not expecting and did not correctly handle firing of the 
cm_watchdog() SIGALARM at random times (i.e. select() with timeout returned too soon, etc).
- today, UNIX signals are pretty obscure, best avoided. (i.e. interaction of signals and threads is not super will defined, etc).

commits up to (and including) plus merge of branch feature/remove_cm_watchdog
https://bitbucket.org/tmidas/midas/commits/9f1775d2fc75d0de0b9d4ef1abc7b2fb9bacca28

K.O.

> cm_watchdog() has been removed from the latest midas sources
> Removal of cm_watchdog() solves many problems in the midas code base:

Removal of cm_watchdog() creates new problems:

a) the bm_send_event(BM_WAIT) and bm_receive_event(BM_WAIT) wait for free space and wait for new event do not update the timeouts (need to add a call 
to cm_periodic_tasks())
b) frontends that talk to slow external equipment now die unless they have their timeout adjusted to be longer than the longest equipment operation (they 
were already supposed to do this, but...)
c) mhttpd sometimes dies from from an odb timeout (with the default 10 sec timeout).

As one solution, we may bring an automatic cm_watchdog() back, but running from a thread instead of from SIGALARM.

K.O.

> > cm_watchdog() has been removed from the latest midas sources
> > Removal of cm_watchdog() solves many problems in the midas code base:
> Removal of cm_watchdog() creates new problems:
> a) the bm_send_event(BM_WAIT) and bm_receive_event(BM_WAIT)
> b) frontends that talk to slow external equipment
> c) mhttpd sometimes dies from from an odb timeout (with the default 10 sec timeout).

The watchdog is back, in a "light" form. Added:

- cm_watchdog_thread() - runs every 2 seconds and updates the timestamps on ODB and all open event buffers (SYSMSG, SYSTEM, etc).
- cm_start_watchdog_thread() - added to mfe.c and mhttpd - so user frontends work the same as before cm_watchdog() removal
- cm_stop_watchdog_thread() - added to cm_disconnect_experiment() to avoid leaving the thread running after we closed odb and all event buffers.

As before, the watchdog only runs on locally attached midas programs. For programs attached remotely via the mserver, the mserver handles the watchdog functions.

This new light-weight watchdog thread only updates the timestamps, it does not check and remove dead clients, it does not check the alarms. These functions are now performed 
by cm_yield() and cm_periodic_tasks(). At least some program in an experiment should call them periodically. (normally, at least mlogger and mhttpd will do that).

Programs that accidentally relied on SIGALRM firing at 1Hz may still be affected - i.e. with the old cm_watchdog(), ::sleep(1000) will only sleep for 1 second (interrupted by 
SIGALARM), now it will sleep for the full 1000 seconds. Other syscalls, i.e. select(), are similarly affected.

For now, I think only mfe.c frontends and mhttpd need the watchdog thread. With luck all the other midas programs (mlogger, mdump, etc) will run fine without it.

K.O.