> Hi again, we now have Midas installed on the Rpi (remote frontend machine) and
> have managed to run Fetest on it. Now we are at a stage where we want to send
> the Fetest data over to the Data Acquisition machine ...
> 
> Does the mserver have any role in this?
> 

Yes. mserver runs on your daq machine and handles connections from frontends running on frontend machines. It needs to be configured 
correctly before it will work: in odb on your daq machine, non-local rpc has to be enabled and the frontend machine has to be added to the 
midas rpc access control list.

Read this:
https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Running_with_one_or_more_REMOTE_frontends

And this:
https://midas.triumf.ca/MidasWiki/index.php/Security#MIDAS_programs_on_remote_machines

K.O.

Hi again, we now have Midas installed on the Rpi (remote frontend machine) and
have managed to run Fetest on it. Now we are at a stage where we want to send
the Fetest data over to the Data Acquisition machine, which also has Midas
installed. We want this data to be read into the Webserver Status page. We have
tried commands such as: (but Fetest then doesn't run)

./fetest -h DAQ-system-ip-address
./fetest -e sampleexpt -h DAQ-system-Ip-address
./fetest -e sampleexpt -h DAQ-system-Ip-address-with-webserver-port

our experiment name is sampleexpt on Rpi and DAQ machine in their respective
exptab files. Maybe the Rpi is getting confused as to whether it should be
running the experiment on Rpi or the DAQ. We need it to run on the DAQ.

Does the mserver have any role in this?

Thanks you for your kind help (we summer interns are really stuck!)

Thanks for your advice. We now have Midas installed on both our machines (remote machine-Rpi &
hostmachine-Centos). 

=========================================================================================================
One the host machine:
[hh19285@it038146 bin]$ uname -a
Linux it038146.users.bris.ac.uk 3.10.0-957.21.2.el7.x86_64 #1 SMP Wed Jun 5 14:26:44 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
[hh19285@it038146 bin]$ uname -a
Linux it038146.users.bris.ac.uk 3.10.0-957.21.2.el7.x86_64 #1 SMP Wed Jun 5 14:26:44 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
[hh19285@it038146 bin]$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array
--disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install
--with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install
--enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) 
[hh19285@it038146 bin]$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.6.1810 (Core) 
Release:	7.6.1810
Codename:	Core

===========================================================================================================
On remote machine:
pi@raspberrypi:~/packages/midas/bin $ uname -a
Linux raspberrypi 4.19.42-v7+ #1219 SMP Tue May 14 21:20:58 BST 2019 armv7l GNU/Linux
pi@raspberrypi:~/packages/midas/bin $ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/6/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 6.3.0-18+rpi1+deb9u1'
--with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs
--enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6
--program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib
--without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/
--enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new
--enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib
--disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-armhf/jre --enable-java-home
--with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-armhf
--with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-armhf --with-arch-directory=arm
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --with-target-system-zlib --enable-objc-gc=auto
--enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard
--enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 6.3.0 20170516 (Raspbian 6.3.0-18+rpi1+deb9u1) 
pi@raspberrypi:~/packages/midas/bin $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 9.9 (stretch)
Release:	9.9
Codename:	stretch







> Yikes, the error is in the CRC library. The assembly-optimized crc32c function fails to build, and the 
> error does not look familiar to me. I do not see this error here. What is your host system ("uname -
> a") and what is your gcc ("gcc -v")?
> 
> BTW, "make linux32" will build an Intel 32-bit version (see "-m32" in "man gcc"). For ARM 32-bit 
> you need a different switch, I think, also depending how you are cross-compiling it.
> 
> For straight cross-compilation, look at the Makefile target "make linuxarm" (you will need to change 
> the location of your ARM gcc cross-compiler).
> 
> For running MIDAS frontend on the Raspberry Pi 3, I build MIDAS on the Pi3 itself, the machine is big 
> enough to run CentOS7 linux and gcc to build the full MIDAS.
> 
> But if you have a different cross-compilation scheme, I am happy to help you and to add your 
> scheme to the MIDAS Makefile. We can start by looking at "uname -a" and "gcc -v" and "lsb_release 
> -a" (if you have it).
> 
> K.O.
> 
> 
> > Hi, we are part of the Mu3e research based at University of Bristol. We have a 
> > remote 32 bit frontend (raspberry pi) connected to a 64 bit Data Acquisition 
> > system.we are following the instructions at installation/quickstart linux/Build 
> > 32-bit MIDAS libraries. when we execute the commands: 
> > [mhostpc] cd /home/packages/midas
> > [mhostpc] make linux32             
> > 
> > we get an error:
> > 
> > make NO_ROOT=1 NO_MYSQL=1 NO_ODBC=1 NO_SQLITE=1 OS_DIR=linux-m32 USERFLAGS=-
> m32
> > make[1]: Entering directory `/home/hh19285/packages/midas'
> > g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
> > Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
> > DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -
> DMG_ENABLE_SSL 
> > -DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
> > src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
> > src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
> >                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
> >                                                                   ^
> > src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
> >                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
> >                                                                   ^
> > src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
> > src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
> >                  : "%ebx", "%edx"); \
> >                                   ^
> > src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
> >      SSE42(sse42);
> >      ^
> > make[1]: *** [lib/crc32c.o] Error 1
> > make[1]: Leaving directory `/home/hh19285/packages/midas'
> > make: *** [linux32] Error 2
> > 
> > Could you please help with getting past this? otherwise we may need to change 
> > our whole experimental setup.
> > 
> > Thank you in advance

> --> In the midas back-end I never created an array. I created an INT in the ODB with db_create_key(hDB, 0,
> "Equipment/Switching/Variables/DATA_WRITE", TID_INT). By using modset in javascript and parsing the string
> "/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]" I call it like an array and it shows up like an
> array in the ODB.

I think you are good. In ODB, a TID_INT is actually an array of size 1. Writing to an array index automatically
extends the array. I am not sure where this is written down, but this is how most ODB array index access functions
have always worked.

You do have a performance bug with your loop, though, by writing a[1], a[2], a[3], you cause
the array to grow from size 1 to size 2. Then grow it from size 2 to size 3, etc. Of course
this only happens the first time you run the thing. Afterwards, the array has the correct size
and does not need to be grown.

K.O.

Yikes, the error is in the CRC library. The assembly-optimized crc32c function fails to build, and the 
error does not look familiar to me. I do not see this error here. What is your host system ("uname -
a") and what is your gcc ("gcc -v")?

BTW, "make linux32" will build an Intel 32-bit version (see "-m32" in "man gcc"). For ARM 32-bit 
you need a different switch, I think, also depending how you are cross-compiling it.

For straight cross-compilation, look at the Makefile target "make linuxarm" (you will need to change 
the location of your ARM gcc cross-compiler).

For running MIDAS frontend on the Raspberry Pi 3, I build MIDAS on the Pi3 itself, the machine is big 
enough to run CentOS7 linux and gcc to build the full MIDAS.

But if you have a different cross-compilation scheme, I am happy to help you and to add your 
scheme to the MIDAS Makefile. We can start by looking at "uname -a" and "gcc -v" and "lsb_release 
-a" (if you have it).

K.O.


> Hi, we are part of the Mu3e research based at University of Bristol. We have a 
> remote 32 bit frontend (raspberry pi) connected to a 64 bit Data Acquisition 
> system.we are following the instructions at installation/quickstart linux/Build 
> 32-bit MIDAS libraries. when we execute the commands: 
> [mhostpc] cd /home/packages/midas
> [mhostpc] make linux32             
> 
> we get an error:
> 
> make NO_ROOT=1 NO_MYSQL=1 NO_ODBC=1 NO_SQLITE=1 OS_DIR=linux-m32 USERFLAGS=-
m32
> make[1]: Entering directory `/home/hh19285/packages/midas'
> g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
> Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
> DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -
DMG_ENABLE_SSL 
> -DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
> src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
> src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
>                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
>                                                                   ^
> src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
>                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
>                                                                   ^
> src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
> src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
>                  : "%ebx", "%edx"); \
>                                   ^
> src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
>      SSE42(sse42);
>      ^
> make[1]: *** [lib/crc32c.o] Error 1
> make[1]: Leaving directory `/home/hh19285/packages/midas'
> make: *** [linux32] Error 2
> 
> Could you please help with getting past this? otherwise we may need to change 
> our whole experimental setup.
> 
> Thank you in advance

Update: "make" instead of "make linux32" should also work. I believe the "linux32" target came 
from some special case at TRIUMF for some FPGA embedded linux, which is not applicable for 
the Raspberry Pi.

Note that the build process has to be initiated on the Raspberry Pi, NOT a host PC.

Stefan

> > for (i = 0; i < lines.length; i++) {
> >         modbset("/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]", parseInt(lines[i]));
> > }
> 
> this is wrong.
> 
> a) you are programming javascript as if it were C/C++. You think this code wrote lines.length() values 
> to ODB, when what the code actually did is queued lines.length() RPC requests for later execution. 
> Eventually some time later, each RPC request will open a connection to mhttpd, send a request, wait 
> for mhtttpd to process it, etc. Where do you wait for the completion of all these RPCs before 
> proceeding as if all the data has been successfully written to ODB? (answer: you cannot, javascript 
> cannot "wait for things", instead you have to make chains of event handlers. javascript != C/C++. 
> They are completely different).

--> Following your discussion about async. functions I will change this part of the code and make chains of
event handlers.

> b)  you should write the whole array in one operation instead of looping over each element. see 
> mjsonrpc_db_paste() and example.html.

--> In the midas back-end I never created an array. I created an INT in the ODB with db_create_key(hDB, 0,
"Equipment/Switching/Variables/DATA_WRITE", TID_INT). By using modset in javascript and parsing the string
"/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]" I call it like an array and it shows up like an
array in the ODB. So for explaining it a bit better how the value changes in the ODB take this pseudo code
example:

// midas part //
> int a = 1; // this is more or less what I think db_create_key is doing in the ODB
// midas part //

// ODB //
> print(a) // this prints me 1 and this is also the value what I see in the ODB
// ODB //

// javascript part //
> for int i in [1,2,3,4] do 
> modset(a[i], i) // for simplification I don't use event handlers here
> end for
// javascript part //

// ODB //
> print(a) // now I see [1,2,3,4]
// ODB //

This example violates type safety. I know that javascript is not type safe. According to this I would like
to know if this behavior is wanted or why there is no bounds checking?

> I do not understand your question about "calling an INT like an array".
--> Here I mean that I call the variable in the ODB via string passing, like I would call a variable, which is
an array. I don't speak about function calls.

> parseInt() (defined where?)
--> This is a global JavaScript function (https://www.w3schools.com/jsref/jsref_obj_global.asp)

Cheers,
Marius

Why don't your try the (yet undocumented) new installation procedure:

$ git clone https://bitbucket.com/tmidas/midas --recursive
$ cd midas
$ mkdir build
$ cd build
$ cmake ..
$ make
$ make install

In case your RPi does not have cmake pre-installed, you need

$ sudo apt-get install cmake.

Works for my RPi.

Best,
Stefan

Hi, we are part of the Mu3e research based at University of Bristol. We have a 
remote 32 bit frontend (raspberry pi) connected to a 64 bit Data Acquisition 
system.we are following the instructions at installation/quickstart linux/Build 
32-bit MIDAS libraries. when we execute the commands: 
[mhostpc] cd /home/packages/midas
[mhostpc] make linux32             

we get an error:

make NO_ROOT=1 NO_MYSQL=1 NO_ODBC=1 NO_SQLITE=1 OS_DIR=linux-m32 USERFLAGS=-m32
make[1]: Entering directory `/home/hh19285/packages/midas'
g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -DMG_ENABLE_SSL 
-DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
                     : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
                                                                  ^
src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
                     : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
                                                                  ^
src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
                 : "%ebx", "%edx"); \
                                  ^
src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
     SSE42(sse42);
     ^
make[1]: *** [lib/crc32c.o] Error 1
make[1]: Leaving directory `/home/hh19285/packages/midas'
make: *** [linux32] Error 2

Could you please help with getting past this? otherwise we may need to change 
our whole experimental setup.

Thank you in advance

We are happy to the midas release "midas-2019-06" with the build system implemented in cmake and the midas, mxml and mscb 
projects switched to C++.

Changes since midas-2019-03:

minor bug fixes
switch of midas build to c++ with c++ linkage (no "extern C")
switch of midas build to cmake
removal of $(OS_DIR) from the midas library and bin paths (use $MIDASSYS/lib instead of $MIDASSYS/linux/lib)
mxml and mscb are implemented as git submodules

Please review the following guide to update midas from previous release midas-2019-03 or older.

Update the code:

git checkout develop
git pull
git checkout feature/midas-2019-06
git pull
git submodule update --init # this will checkout correct versions of mxml and mscb
make clean
make cclean
rm -rf linux/bin
rm -rf linux/lib
rmdir linux
make cmake3 # or "make cmake" on ubuntu and macos
ls -l bin/odbedit bin/mlogger

Update experiment environment:

- change PATH from $MIDASSYS/linux/bin to $MIDASSYS/bin

Cleanup unneeded stuff:

- remove $HOME/packages/mxml (new location $MIDASSYS/mxml)
- remove $HOME/packages/mscb (new location $MIDASSYS/mscb)

Update experiment frontend build:

- change Makefile to remove $(OS_DIR) from library search path ($MIDASSYS/linux/lib becomes $MIDASSYS/lib)
- change Makefile to set mxml include path from $MIDASSYS/../mxml to $MIDASSYS/mxml (to avoid including the wrong 
version of mxml/strlcpy.h)
- update frontend code to use mfe.h and build as C++, see https://midas.triumf.ca/elog/Midas/1526

K.O.

> Just to make this point clear: The "write-to-odb-read-via-hotlink" was never meant to guarantee the receiving side to see each change. If changes happen too often, updates might get lost. If one relies on the 
> sequence of updates, one should use direct RPC calls to the frontend or use a midas buffer and encode updates in events.

I recommend that people use the jrpc mechanism that does an RPC directly from javascript into the frontend.
It passes 2 strings as arguments (command and data value). Arbitrary objects can be passed by encoding
the data in json (use mjson.h to decode it in the frontend). A string is returned back to javascript (again, encode
arbitrary data as json, use the mjson.h library).

Call sequence:
javascript -> (http) -> mhttpd -> (MIDAS RPC call) -> frontend -> (write, read, frob hardware) -> frontend -> (MIDAS RPC reply) -> mhttpd -> (http reply) -> javascript

Example of all this is in example.html and fetest.cxx:

javascript side code: mjsonrpc_call("jrpc", { "client_name":"fetest", "cmd":"xxx", "args":"xxx" })

frontend side code:

INT rpc_callback(INT index, void *prpc_param[])
{
   const char* cmd  = CSTRING(0);
   const char* args = CSTRING(1);
   char* return_buf = CSTRING(2);
   int   return_max_length = CINT(3);
   cm_msg(MINFO, "rpc_callback", "--------> rpc_callback: index %d, max_length %d, cmd [%s], args [%s]", index, return_max_length, cmd, args);
   ... do stuff ... put result into string "tmp"
   strlcpy(return_buf, tmp, return_max_length);
   return RPC_SUCCESS;
}

... somewhere in frontend_init(), register the RPC:

#ifdef RPC_JRPC
   status = cm_register_function(RPC_JRPC, rpc_callback);
   assert(status == SUCCESS);
#endif

K.O.

> target_compile_options(rmlogger PRIVATE -DHAVE_ROOT)

Got it. Now I can build the duplets of mana.o and rmana.o (and .a) - mana always without ROOT, rmana with ROOT if available. This is the same as 
the old Makefile, the best I can tell.

With the fix to rmlogger, all known problems with the cmake build seem to be fixed.

K.O.

Just to make this point clear: The "write-to-odb-read-via-hotlink" was never meant to guarantee the receiving side to see each change. If changes happen too often, updates might get lost. If one relies on the 
sequence of updates, one should use direct RPC calls to the frontend or use a midas buffer and encode updates in events.

Stefan

> A ladder of promise event handlers is certainly one possibility to enforce the order of ODB writes, but I wonder if we could so something simpler:
> 
> - modbset creates an object remembering the status of the RPC request. Initially, this object receives the status "open request"
> - when the rpc call got executed successfully, the callback sets the state of the above object to "request succeeded" or "request failed" (in case of error)
> - if a new modbset comes BEFORE the previous one has completed, the function queues the new request in a data field of the above object
> - if a rpc call finishes, and a queued new rpc request is present, it gets executed
> 
> This would be relatively easy to be implemented and keep the order of the rpc calls. Does that make sense?
> 

Yes, this is a neat idea, I am really happy with how a complete rpc request can be held by one object, and we can make queues of them, etc.

Anyhow here is the proof of the pudding. I added a test to example.html, there are two buttons, one makes 5 modbset() calls, second has a ladder of 5 db_paste calls. Then I watch 
the result in odbedit. 1, 2, 3, 4, 5 is the modbset(), 6, 7, 8, 9, 10 is the ladder of db_paste calls:

$ odbedit
[local:javascript1:S]/>watch Example/int
Watch key "/Example/int" to be modified, abort with any key

/Example/int = 1
/Example/int = 2
/Example/int = 3
/Example/int = 4
/Example/int = 5

/Example/int = 1
/Example/int = 5 <== fault
/Example/int = 5
/Example/int = 5
/Example/int = 5

/Example/int = 1
/Example/int = 2
/Example/int = 3
/Example/int = 5 <== 4 and 5 reversed
/Example/int = 4 <== 4 and 5 reversed

/Example/int = 6
/Example/int = 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

/Example/int = 6
/Example/int = 8 <== should be 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

/Example/int = 6
/Example/int = 7
/Example/int = 8
/Example/int = 9
/Example/int = 10

I immediately notice that we have a race condition between the RPCs, db_watch notifications and db_get_value() in the watch handler:
there are 5 rpcs, 5 watch notifications, 5 calls to db_get_value() in the watch handler, but sometimes the handler is too slow
and the data in odb changes before it reads it, thus duplicate values (missing "7" above). (The old db_open_record() had a "hidden"
db_get_value() inside it, while db_watch() requires an explicit db_get_value() call, making it obvious why we get
the wrong (newer) data sometimes).

Possible fixes for this is to slow down the RPCs (the race condition is still there, probability is reduced) or send the changed
data as part of the notification. If this were C/C++, a "sleep(1)" between modbset() calls would have fixed it,
but there is no sleeping and waiting in javascript. (I guess one could use a ladder of timers).

Other than that, I am surprised how easy it was to see that indeed out-of-order RPCs can happen, see the case
of out-of-order 4 and 5 above. It only took me maybe 5-10 clicks on the button to see that. I expected that I would
need to try several browsers or use a slow network connection, but here it is, on my home mac, localhost network,
google chrome browser.

Below is the test code. I do NOT vote that everybody should use ladders of db_paste calls.

function test_modbset() {
           modbset("/example/int", 1);
           modbset("/example/int", 2);
           modbset("/example/int", 3);
           modbset("/example/int", 4);
           modbset("/example/int", 5);
        }

        function test_chained_db_paste() {
           var paths = [ "/example/int" ];
           mjsonrpc_db_paste(paths,[6]).then(function(rpc) {
              mjsonrpc_db_paste(paths,[7]).then(function(rpc) {
                 mjsonrpc_db_paste(paths,[8]).then(function(rpc) {
                    mjsonrpc_db_paste(paths,[9]).then(function(rpc) {
                       mjsonrpc_db_paste(paths,[10]).then(function(rpc) {
                          // nothing
                       }).catch(function(error){mjsonrpc_error_alert(error);});
                    }).catch(function(error){mjsonrpc_error_alert(error);});
                 }).catch(function(error){mjsonrpc_error_alert(error);});
              }).catch(function(error){mjsonrpc_error_alert(error);});
           }).catch(function(error){mjsonrpc_error_alert(error);});
        }

        </script>

        <input type=button value='test modbset()' onClick='test_modbset();'></input>
        <input type=button value='test chained db_paste()' onClick='test_chained_db_paste();'></input>

K.O.

A ladder of promise event handlers is certainly one possibility to enforce the order of ODB writes, but I wonder if we could so something simpler:

- modbset creates an object remembering the status of the RPC request. Initially, this object receives the status "open request"
- when the rpc call got executed successfully, the callback sets the state of the above object to "request succeeded" or "request failed" (in case of error)
- if a new modbset comes BEFORE the previous one has completed, the function queues the new request in a data field of the above object
- if a rpc call finishes, and a queued new rpc request is present, it gets executed

This would be relatively easy to be implemented and keep the order of the rpc calls. Does that make sense?

Best,
Stefan

> - it looks like cmake does not like building variant executables and object files, i.e. "with ROOT" and "without ROOT".
> 
> I need to set "-DHAVE_ROOT" for building "with ROOT" and unset it via remove_definitions() for building "without ROOT",
> but remove_definitions() and add_definitions() do not work on a per-target basis, instead they operate
> per-directory and per-project.

You should not use per-directory and per-project definitions, but per-target definitions, such as

target_compile_options(mhttpd PRIVATE -DMG_ENABLE_SSL)

> In midas, we build mlogger without ROOT (to avoid tangling it with the ROOT RPATH and ROOT shared libraries),
> but if ROOT is present, we build rmlogger "with ROOT support". Same for the analyzer (mana.o and rmana.o).
> 
> For now we have this:
> - mana.o is built with ROOT if ROOT is detected
> - rmana.o is not built
> - rmlogger is not built (not clear why)

I added rmlogger to the install instructions. I believe it was always built, but just not installed into the /bin directory.

Stefan

> > Status update on the cmake conversion:

After the latest updates from Stefan & co, it looks like the cmake builds are working correctly,
there is only one bug remaining (rmlogger is not built). (rmana.o is also not built, but I think only 0 people use it).

I will test this in a couple of our test experiments, write the instructions for migrating from the old midas and tag a new release (midas-2019-06)

K.O.

> The modbset() function is used in many custom pages at PSI ...

I restored this function in midas.js with a documentation blurb warning about it's asynchronous nature and about the possibility of out-of-order writes.

The more I think about this, the more it looks to me that we should look at this web worker api business to support
synchronous communications for MIDAS web pages.

K.O.

If it's a function intended for general use, it should be in midas.js.

The documentation for such a function should be made very clear that:

a) it does not actually write to ODB, instead it queues a request for writing, this request is executed at a later (undefined) time.

b) the following javascript code results in undefined behaviour:

modbset("/foo/bar", 1); // queue rpc request
modbset("/foo/bar", 2); // queue rpc request
// is ODB /foo/bar set to 1 or 2?

Why? The best I know javascript does not require for RPCs to execute in-order, so the second RPC may be issued
before the first one. More likely both RPCs are started roughly at the same time (i.e. in different RPC worker
threads), in which case we do not know in which order they will be processed by mhttpd, which is also
multithreaded and does not necessarily execute requests in the same order as (i.e.) they connect to the rpc port 8080.

To answer the question "1 or 2", the answer is neither, as at that point in the code, the RPC requests
probably have not started executing yet, and even if they did, mhttpd most likely did not write anything
into odb yet, as processing RPC requests takes much longer than executing a few lines of javascript.

So to ensure correct sequence of writes and to ensure that something was actually written to odb,
one has to roll out the full ladder of promise event handlers.

Is correct sequence of writes important? Maybe yes, maybe no.

But if you use modbset() without being aware of these issues, you will write code
for ramping high voltage like this:

modbset("/eq/hv/set/voltage", 0); // set voltage to zero
modbset("/eq/hv/set/hv_enable", 1); // enable high voltage
modbset("/eq/hv/set/voltage", 50); // start slow
modbset("/eq/hv/set/voltage", 1000); // ramp to half-way
modbset("/eq/hv/set/voltage", 1900); // stop a bit before the final voltage
modbset("/eq/hv/set/voltage", 2000); // !!! ramping from 0 to 2000 should never be done in one step !!!

And as the author of all this RPC code, I promise that some day you will see the voltage
on the detector go directly from 0 to 2000 (then up and down to 50, 1000 and 1900).

To me, this makes helpful helper functions actually dangerous to use.

Now that I have experience with sync RPC (from C/C++) and async RPC (with Promises in javascript),
I would say that synchronous C/C++ style programming is much easier
and much less verbose and much easier to read and to modify/adjust compared to the javascript-style
ladder of nested event handlers.

I now see that synchronous requests are again permitted if one uses the "Web Worker API". Maybe we should revisit
the MIDAS javascript RPC code and see if we can use this web worker stuff to officially support synchronous RPC requests,
again.

About the sync rpc deprecation, read more here:
https://stackoverflow.com/questions/30876093/will-chrome-and-other-browsers-drop-support-for-synchronous-xmlhttprequest

K.O.


> I disagree. The modbset() function is used in many custom pages at PSI because people are tired of typing mjsonrpc_db_paste([path],[value]) vs. modbset(path, value). We need to keep 
> modbset() which is well documented at 
> 
> https://midas.triumf.ca/MidasWiki/index.php/Custom_Page#modbset
> 
> Since modbset() does call the underlying mjsonrpc_db_paste(), it is as good or bad as that function. Plus it adds standard error handling to avoid the need of catching errors for each and 
> every mjsonrpc_db_paste() call. If it is believed that modbset() has a problem, then this should be fixed in the source code of modbset(). Removing that function is not an option.
> 
> Stefan

I disagree. The modbset() function is used in many custom pages at PSI because people are tired of typing mjsonrpc_db_paste([path],[value]) vs. modbset(path, value). We need to keep 
modbset() which is well documented at 

https://midas.triumf.ca/MidasWiki/index.php/Custom_Page#modbset

Since modbset() does call the underlying mjsonrpc_db_paste(), it is as good or bad as that function. Plus it adds standard error handling to avoid the need of catching errors for each and 
every mjsonrpc_db_paste() call. If it is believed that modbset() has a problem, then this should be fixed in the source code of modbset(). Removing that function is not an option.

Stefan