notes from converting ALPHA-g chronobox frontend fechrono to tmfe c++ framework.

the chronobox device is a timestamp/low resolution tdc/scaler/generic TTL and ECL io
mainboard with an altera DE10_NANO plugin board. it has a cyclone-5 FPGA SOC running Raspbian linux.
FPGA communication is done by avalon-bus memory mapped registers, main data readout
is PIO from an FPGA 32-bit wide FIFO (no DMA yet).

- login to main computer (daq16)
- cd packages
- git clone https://bitbucket.org/tmidas/midas midas-develop
- cd midas-develop
- make mini ### creates linux-x86_64/{bin,lib}
- ssh agdaq@cb02 ### private network
- cd ~/packages/midas-develop
- make mini ### creates linux-linux-armv7l/{bin/lib}
- cd ~/online/chronobox_software
- cat fechrono.cxx ~/packages/midas-develop/progs/tmfe_example_everything.cxx > fechrono_tmfe.cxx
- edit fechrono_tmfe.cxx:

- rename "FeEverything" to "FeChrono"
- copy contents of frontend_init() to HandleFrontendInit()
- copy contents of frontend_exit() to HandleFrontendExit()
- replace get_frontend_index() with fFe->fFeIndex
- replace "return SUCCESS" with return TMFeOk()
- replace "return !SUCCESS" with return TMFeErrorMessage("boo!!!")
- this frontend has 3 indexed equipments, copy EqEverything 3 times, rename EqEverything to EqCbHist, EqCbms, EqCbFlow
- copy contents of begin_of_run() to EqCbHist::HandleBeginRun()
- copy contents of end_of_run() to EqCbHist::HandleEndRun()
- pause_run(), resume_run() are empty, delete all HandlePauseRun() and all HandleResumeRun()
- frontend_loop() is empty, delete
- poll_event() and interrupt_configure() are empty, delete
- delete all HandleStartAbortRun(), delete all calls to RegisterTransitionStartAbort();
- examine equipment[]:
- "cbhist%02d" - periodic, copy contents of read_cbhist() to EqCbHist::HandlePeriodic()
- "cbms%02d" - polled, copy contents of read_cbms_fifo() to EqCbms::HandlePollRead()
- "cbflow%02d" - periodic, copy contents of read_flow() to EqCbFlow::HandlePeriodic()
- delete unused HandlePoll(), HandlePollRead() and HandlePeriodic()
- replace bk_init32() with "size_t event_size = 100*1024; char* event = (char*)malloc(event_size); ComposeEvent(event, 
event_size); BkInit(event, event_size);"
- replace bk_create(pevent) with BkOpen(event)
- replace bk_close(pevent, ...) with BkClose(event, ...)
- replace "return bk_size(pevent)" with "EqSendEvent(event); free(event);"
- remove unused example SendData()
- if there linker complains about references to "hDB", add "HNDLE hDB" is global scope, add "hDB = fMfe->fDB"
- replace set_equipment_status() with EqSetStatus()
- move equipment configuration from the equipment[] array to the equipment constructors
- remove unused HandleRpc()
- remove unused HandleBeginRun() and unused HandleEndRun()
- remove all example code from HandleInit(), breakup frontend_init() code into per-equipment HandleInit() functions
- EqCbms::HandlePoll() replace all example code with "return true"
- if desired, replace ODB functions from utils.cxx with MVOdb RI(), RD(), etc
- if desired, replace cm_msg() with Msg() and delete "const char* frontend_name"
- update FeChrono() constructor:
      FeSetName("fechrono%02d");
      FeAddEquipment(new EqCbHist("cbhist%02d", __FILE__));
      FeAddEquipment(new EqCbms("cbms%02d", __FILE__));
      FeAddEquipment(new EqCbFlow("cbflow%02d", __FILE__));
- build:
g++ -std=c++11 -Wall -Wuninitialized -g -Ialtera -Dsoc_cv_av -I/home/agdaq/packages/midas-develop/include -
I/home/agdaq/packages/midas-develop/mvodb -c fechrono_tmfe.cxx
g++ -o fechrono_tmfe.exe -std=c++11 -Wall -Wuninitialized -g -Ialtera -Dsoc_cv_av -I/home/agdaq/packages/midas-develop/include 
-I/home/agdaq/packages/midas-develop/mvodb fechrono_tmfe.o utils.o cb.o /home/agdaq/packages/midas-develop/linux-
armv7l/lib/libmidas.a -lm -lz -lutil -lnsl -lpthread -lrt
- run:
- bombs on bm_set_cache_size(), reduce default cache size, old mserver cannot deal with the new default size, set 
fEqConfWriteCacheSize = 100*1024;
- run:
- prints too many messages, comment out print "HandlePollRead!"
- run:
- good now!

success, was not too bad.

also:
- replace gHaveRun with fMfe->fStateRunning
- replace gRunNumber with fMfe->fRunNumber

see tmfe.md section "variables provided by the framework"

K.O.

Result is here:
https://bitbucket.org/expalpha/chronobox_software/src/master/fechrono_tmfe.cxx

Original code is in fechrono.cxx. Not super pretty, but representative of most mfe-based frontends
we see around here. A good example of why the old mfe.c structure no longer works so well for us.

After conversion to tmfe, we do not win a beauty contest yet, but the path for further
clean up and refactoring into better c++ is quite clear. (And it is very obvious where
the missing "event object" wants to be here)

K.O.

Now we are converting the alpha-g event builder from rev0 tmfe (midas-2020-xx) to the new tmfe c++
framework in midas-develop. Earlier, I followed the steps outlined in this blog
to convert this event builder from mfe.c framework to rev0 tmfe.

- get latest midas-develop
- examine progs/tmfe_example_everything.cxx
- open feevb.cxx
- comment-out existing main() function
- from tmfe_example_everything.cxx, copy class FeEverything and main() to the bottom of feevb.cxx
- comment-out old main()
- make sure we include the correct #include "tmfe.h"
- rename example frontend class FeEverything to FeEvb
- rename feevb's "rpc handler" and "periodic handler" class EvbEq to EqEvb
- update class declaration and constructor of EqEvb from EqEverything in example_everything: EqEvb extends TMFeEquipment, 
EqEvb constructor calls constructor of base class (c++ bogosity), keep the bits of the example that initialize the 
equipment "common"
- in EqEvb, remove data members fMfe and fEq: fMfe is now inherited from the base class, fEq is now "this"
- in FeEvb constructor, wire-in the EqEvb constructor: FeSetName("feevb") and FeAddEquipment(new EqEvb("EVB",__FILE__))
- migrate function names:
- fEq->SendEvent() with EqSendEvent()
- fEq->SetStatus() with EqSetStatus()
- fEq->ZeroStatistics() with EqZeroStatistics() -- can be removed, taken care of in the framework
- fEq->WriteStatistics() with EqWriteStatistics() -- can be removed, taken care of in the framework
- (my feevb.o now compiles, but will not work, yet, keep going:)
- EqEvb - update prototypes of all HandleFoo() methods per example_everything.cxx or per tmfe.h: otherwise the framework 
will not call them. c++ compiler will not warn about this!
- migrate old main():
- restore initialization of "common" and other things done in the old main():
- TMFeCommon was merged into TMFeEquipment, move common->Foo = ... to the EqEvb constructor, consult tmfe.h and tmfe.md 
for current variable names.
- consider adding "fEqConfReadConfigFromOdb = false;" (see tmfe.md)
- if EqEvb has a method Init() called from old main(), change it's name to HandleInit() with correct arguments.
- split EqEvb constructor: leave initialization of "common" in the constructor, move all functions, etc into HandleInit()
- move fMfe->SetTransitionSequenceFoo() calls to HandleFrontendInit()
- move fMfe->DeregisterTransition{Pause,Resume}() to HandleFrontendInit()
- old main should be empty now
- remove linking tmfe_rev0.o from feevb Makefile, now it builds!
- try to run it!
- it works!
- done.

K.O.

I truncated the bitbucket build to only build on ubuntu LTS 20.04.

Somehow all the other build targets - centos-7, centos-8, ubuntu-18 - have
an obsolete version of cmake. I do not know where the bitbucket os images
get these obsolete versions of cmake - my centos-7 and centos-8 have much
more recent versions of cmake.

If somebody has time to figure it out, please go at it, I would like very
much to have centos-7 and centos-8 builds restored (with ROOT), also
to have a ubuntu LTS 20.04 build with ROOT. (For me, debugging bitbucket
builds is extremely time consuming).

Right now many midas cmake files require cmake 3.12 (released in late 2018).

I do not know why that particular version of cmake (I took the number
from the tutorials I used).

I do not know what is the actual version of cmake that MIDAS (and ROOTANA) 
require/depend on.

I wish there were a tool that would look at a cmake file, examine all the 
features it uses and report the lowest version of cmake that supports them.

K.O.

OS: OSX 10.14.6 Mojave
MIDAS: Downloaded from repo on April 2021.

I have a slow control frontend doing the command/readout of a MPOD HV/LV. Since I am reading out the current that are in nA (after updating snmp), I wanted to multiply the number by 1e9.

I noticed the new "Formula" field (introduced in 2019 it seems) instead of the "Factor/Offset" I was used to. None of my entries seems to be accepted (after hitting save, when coming back thee field is empty).

Looking in ODB in "/History/Display/MPOD/HV (Current)/", the field "Formula" is a string of size 32 (even if I have multiple plots in that display). I noticed that the fields "Factor" and "Offset" are still existing and they are arrays with the correct size. However, changing the values does not seem to do anything.

Deleting "Formula" by hand and creating a new field as an array of string (of correct length) seems to do the trick: the formula is displayed in the History display config, and correctly used.

> OS: OSX 10.14.6 Mojave
> MIDAS: Downloaded from repo on April 2021.
> 
> I have a slow control frontend doing the command/readout of a MPOD HV/LV. Since I am reading out the current that are in nA (after updating snmp), I wanted to multiply the number by 1e9.
> 
> I noticed the new "Formula" field (introduced in 2019 it seems) instead of the "Factor/Offset" I was used to. None of my entries seems to be accepted (after hitting save, when coming back thee field is empty).
> 
> Looking in ODB in "/History/Display/MPOD/HV (Current)/", the field "Formula" is a string of size 32 (even if I have multiple plots in that display). I noticed that the fields "Factor" and "Offset" are still existing and they are arrays with the correct size. However, changing the values does not seem to do anything.
> 
> Deleting "Formula" by hand and creating a new field as an array of string (of correct length) seems to do the trick: the formula is displayed in the History display config, and correctly used.

I see this, too. Problem is that the history plot code must be compatible with both
the old scheme (factor/offset) and the new scheme (formula). But something goes wrong somewhere.

https://bitbucket.org/tmidas/midas/issues/307/history-plot-config-incorrect-in-odb

Why?

- new code cannot to "3 year" plots, old code has no problem with it
- old experiments (alpha1, etc) have only the old-style history plot definitions,
and both old and new plotting code should be able to show them (there is nobody
to convert this old stuff to the "new way", but we still desire to be able to look at it!)

K.O.

Hi,

I am running a simple FE executable that is supposed to define a PRAW DWORD bank.
The issue is that, right after the start of the run, the logger crashes without messages.
Then the FE reports this error, which is rather confusing.
```
12:59:29.140 2021/05/24 [feTestDatastruct,ERROR] [odb.cxx:6986:db_set_data1,ERROR] "/Equipment/Trigger/Variables/PRAW" is of type UINT32, not UINT32
```

> Hi,
> 
> I am running a simple FE executable that is supposed to define a PRAW DWORD bank.
> The issue is that, right after the start of the run, the logger crashes without messages.
> Then the FE reports this error, which is rather confusing.
> ```
> 12:59:29.140 2021/05/24 [feTestDatastruct,ERROR] [odb.cxx:6986:db_set_data1,ERROR] "/Equipment/Trigger/Variables/PRAW" is of type UINT32, not UINT32
> ```

I think this is fixed in latest midas. There was a typo in this message, the same tid was printed twice,
with result you report "mismatch UINT32 and UINT32", instead of "mismatch of UINT32 vs what is actually there".

This fixes the message, after that you have to manually fix the mismatch in the data type in ODB (delete old one, I guess).

K.O.

Dear all,
is there any way to order the labels in the history plot legend? In the old 
system there was the “order” column in the config panel, but I can not find it 
in the new system. Thanks in advance for the support.

Best regards,
Marco Chiappini

> is there any way to order the labels in the history plot legend? In the old 
> system there was the “order” column in the config panel, but I can not find it 
> in the new system. Thanks in advance for the support.

correct, for reasons unknown, the function to reorder and to delete individual 
entries was removed from the history panel editor.

K.O.

> > is there any way to order the labels in the history plot legend? In the old 
> > system there was the “order” column in the config panel, but I can not find it 
> > in the new system. Thanks in advance for the support.
> 
> correct, for reasons unknown, the function to reorder and to delete individual 
> entries was removed from the history panel editor.
> 
> K.O.

https://bitbucket.org/tmidas/midas/issues/284/history-panel-editor-reordering-of

K.O.

Hi,
with the recent fix of the CMakeLists.txt, it seems like another bug surfaced. 
In midas/progs/mlogger.cxx:48/49, the mysql header files are included without a 
prefix. However, mysql.h and mysqld_error.h are in a subdirectory, so for our 
systems, the lines should be
  48 #include <mysql/mysql.h>
  49 #include <mysql/mysqld_error.h>
This is the case with MariaDB 10.5.5 on OpenSuse Leap 15.2, MariaDB 10.5.5 on 
Fedora Workstation 34 and MySQL 5.5.60 on Raspbian 10.

If this problem occurs for other Linux/MySQL versions as well, it should be 
fixed in mlogger.cxx and midas/src/history_schema.cxx.
If this problem only occurs on some distributions or MySQL versions, it needs 
some more differentiation than #ifdef OS_UNIX. 

Also, this somehow seems familiar, wasn't there such a problem in the past?

Hi,

> with the recent fix of the CMakeLists.txt, it seems like another bug 
surfaced. 
> In midas/progs/mlogger.cxx:48/49, the mysql header files are included without 
a 
> prefix. However, mysql.h and mysqld_error.h are in a subdirectory, so for our 
> systems, the lines should be
>   48 #include <mysql/mysql.h>
>   49 #include <mysql/mysqld_error.h>
> This is the case with MariaDB 10.5.5 on OpenSuse Leap 15.2, MariaDB 10.5.5 on 
> Fedora Workstation 34 and MySQL 5.5.60 on Raspbian 10.
> 
> If this problem occurs for other Linux/MySQL versions as well, it should be 
> fixed in mlogger.cxx and midas/src/history_schema.cxx.
> If this problem only occurs on some distributions or MySQL versions, it needs 
> some more differentiation than #ifdef OS_UNIX.

What does "mariadb_config --cflags" or "mysql_config --cflags" return on 
these systems? For mariadb 10.3.27 on Debian 10 it returns both paths:

% mariadb_config --cflags
-I/usr/include/mariadb -I/usr/include/mariadb/mysql

Note also that mysql.h and mysqld_error.h reside in /usr/include/mariadb *not* 
/usr/include/mariadb/mysql so using "#include <mysql/mysql.h>" would not work.

On CentOS 7 with mariadb  5.5.68:

%  mysql_config --include
-I/usr/include/mysql
% ls -l /usr/include/mysql/mysql*.h
-rw-r--r--. 1 root root 38516 May  6  2020 /usr/include/mysql/mysql.h
-r--r--r--. 1 root root 76949 Oct  2  2020 /usr/include/mysql/mysqld_ername.h
-r--r--r--. 1 root root 28805 Oct  2  2020 /usr/include/mysql/mysqld_error.h
-rw-r--r--. 1 root root 24717 May  6  2020 /usr/include/mysql/mysql_com.h
-rw-r--r--. 1 root root  1167 May  6  2020 /usr/include/mysql/mysql_embed.h
-rw-r--r--. 1 root root  2143 May  6  2020 /usr/include/mysql/mysql_time.h
-r--r--r--. 1 root root   938 Oct  2  2020 /usr/include/mysql/mysql_version.h

So this seems to be the correct setup for both Debian and RHEL. If this is to 
be worked around in Midas I would think it would be better to do it at the 
cmake level than by putting another #ifdef in the code.

Cheers,

Nick.

> % mariadb_config --cflags
> -I/usr/include/mariadb -I/usr/include/mariadb/mysql

I get similar, both .../include and .../include/mysql are in my include path,
so both #include "mysql/mysql.h" and #include "mysql.h" work.

I added a message to cmake to report the MySQL CFLAGS and libraries, so next time
this is a problem, we can see what happened from the cmake output:

4ed0:midas olchansk$ make cmake | grep MySQL
...
-- MIDAS: Found MySQL version 10.4.16
-- MIDAS: MySQL CFLAGS: -I/opt/local/include/mariadb-10.4/mysql;-I/opt/local/include/mariadb-
10.4/mysql/mysql and libs: -L/opt/local/lib/mariadb-10.4/mysql/ -lmariadb

K.O.

I have created a simple program that parses the message buffer in MIDAS and 
sends notifications by webhook to Discord, Slack and or Mattermost.

Active pull request can be found here:

https://bitbucket.org/tmidas/midas/pull-requests/21


Its written in python and CMake will install it in bin (if the Python3 binary 
is found by cmake). The only dependency outside of the MIDAS python library is 
'requests', full documentation are in the mmessenger.md 

A simple program to forward MIDAS messages to Discord, Slack and or Mattermost

(Python 3 required)

Pull request accepted! Documentation can be found on the wiki

https://midas.triumf.ca/MidasWiki/index.php/Mmessenger

> A simple program to forward MIDAS messages to Discord, Slack and or Mattermost
> 
> (Python 3 required)
> 
> Pull request accepted! Documentation can be found on the wiki
> 
> https://midas.triumf.ca/MidasWiki/index.php/Mmessenger

This sounds like a very useful and welcome addition to MIDAS.

But from documentation provided, I have clue how to activate it.

Perhaps it would help if you could write up the basic steps on how to go about it, i.e.
- go to discord
- push these buttons
- cut and paste this thingy from the web page to ODB

K.O.

Dear all,
      I'm trying to use the logging on a mysql DB. Following the instructions on 
the Wiki, I recompiled MIDAS after installing mysql, and cmake with NEED_MYSQL=1 
can find it:

-- MIDAS: Found MySQL version 8.0.23

Then, I compiled my frontend (cmake with no options + make) and run it, but in the 
ODB I cannot find the tree for mySQL. I have only:

Logger/Runlog/ASCII

while I would expect also:

Logger/Runlog/SQL

What could be missing? Maybe should I add something in the CMakeList file or run 
cmake with some option?

Thank you,
      Francesco

I solved this, it was a failed "make clean" before recompiling. Now it works.

Sorry for the noise.

Francesco

> Dear all,
>       I'm trying to use the logging on a mysql DB. Following the instructions on 
> the Wiki, I recompiled MIDAS after installing mysql, and cmake with NEED_MYSQL=1 
> can find it:
> 
> -- MIDAS: Found MySQL version 8.0.23
> 
> Then, I compiled my frontend (cmake with no options + make) and run it, but in the 
> ODB I cannot find the tree for mySQL. I have only:
> 
> Logger/Runlog/ASCII
> 
> while I would expect also:
> 
> Logger/Runlog/SQL
> 
> What could be missing? Maybe should I add something in the CMakeList file or run 
> cmake with some option?
> 
> Thank you,
>       Francesco

a big update to the event buffer code was merged today.

two important bug fixes:

- a logic error in bm_receive_event() (actually bm_fill_read_cache_locked()) 
caused use of uninitialized variable to increment the read pointer and crash 
with error "read pointed points to an invalid event")
- missing bm_unlock() in bm_flush_cache() caused double-locking of event buffer 
caused a hang and a subsequent crash via the watchdog timeout.

several improvements:

- bm_receive_event_vec(std::vector<char>) with automatic memory allocation, one 
does not need to worry about providing a large event buffer to receive event 
data. For local connections MAX_EVENT_SIZE is no longer used, for remote 
connections, a buffer of MAX_EVENT_SIZE is allocated automatically, this is a 
limitation of the MIDAS RPC layer (it does not know how to allocate memory to 
receive arbitrary large data)

(MAX_EVENT_SIZE is now only used in bm_receive_event_rpc()).

- rpc_send_event_sg() - thread safe method to send events to the mserver. it 
takes an array of scatter-gather buffers, so a midas event does not have to be 
in one continious buffer.

- bm_send_event_sg() - same for local connections.

- on top of bm_send_event_sg() we now have bm_send_event_vec(std::vector<char>) 
and bm_send_event_vec(std::vector<vector<char>>). now we can move forward with 
implementing a new "event object" (the TMEvent event object from midasio.h 
already works with these new methods).

- remote connected bm_send_event() & co now always send events to the mserver 
using the event socket. (before, bm_send_event() used RPC_BM_SEND_EVENT and 
suffered from the RPC layer encoding/decoding overhead. mfe.c used 
rpc_send_event() for remote connections)

- bm_send_event(), bm_receive_event() & co now take a timeout value (in 
milliseconds) instead of an async_flag. The old async_flag values BM_WAIT and 
BM_NO_WAIT continue working as expected (wait forever and do not wait at all, 
respectively).

- following improvements are only for remote connections:

- in the case of event buffer congestion (event readers are slow, event buffers 
are close to 100% full), the bm_flush_cache() RPC will no longer timeout due to 
mserver being stuck waiting for free buffer space. (RPC is called with a 1000 
msec timeout, infinite loop waiting for flush is done on the frontend side, the 
RPC timeout will never fire)

- in the case of event buffer congestion, ODB RPC will no longer timeout. 
(previously mserver was stuck waiting for free buffer space and did not process 
any RPCs).

- at the end of run, last few events could be stuck in the event socket. now, 
frontends can flush it using bm_flush_cache(0,BM_WAIT) (use zero for the buffer 
handle). correct run transition should stop the trigger, stop generating new 
events, call bm_flush_cache(0,BM_WAIT), call bm_flush_cache("SYSTEM",BM_WAIT) 
and return success. (TMFE frontend already does this). Note that 
bm_flush_cache(BM_WAIT) can be stuck for very long time waiting for the event 
buffers to empty-out, so run transition RPC timeout is still possible.

K.O.

It seems that a modbselect triggers a "change" in an ODB which has a hot link. This happens onload (or whenever the custom page is reloaded) and otherwise it behaves as expected, i.e. no change unless the modbselect is actually changed. Is this the intended behaviour? can this be modified?

Thanks for reporting that bug, I fixed it in the last commit.

Stefan

For those unfamiliar, odbxx is the interface that looks like a C++ map, but automatically syncs with the ODB - https://midas.triumf.ca/MidasWiki/index.php/Odbxx.

I've added a new feature that is similar to the existing odb::connect() function, but works like the old db_check_record(). The new odb::connect_and_fix_structure() function:
- keeps the existing value of any keys that are in both the ODB and your code
- creates any keys that are in your code but not yet in the ODB
- deletes any keys that are in the ODB but not your code
- updates the order of keys in the ODB to match your code

This will hopefully make it easier to automate ODB structure changes when you add/remove keys from a frontend.

The new feature is currently in the develop branch, and should be included in the next release.

Hello,

I see this mhttpd error starting MSL-script: 
Uncaught (in promise) ReferenceError: m is not defined
at mhttpd_message (VM2848 mhttpd.js:2304)
at VM2848 mhttpd.js:2122

As I can see it does not affect work of MSL script but shows ReferenceError in 
Midas sequencer (see picture).

Could please point me how to fix this error?

Thanks.
Ruslan

> I see this mhttpd error starting MSL-script: 
> Uncaught (in promise) ReferenceError: m is not defined
> at mhttpd_message (VM2848 mhttpd.js:2304)
> at VM2848 mhttpd.js:2122

your line numbers do not line up with my copy of mhttpd.js. what version of midas 
do you run?

please give me the output of odbedit "ver" command (GIT revision, looks like this: 
IT revision:       Wed Feb 3 11:47:02 2021 -0800 - midas-2020-08-a-84-g78d18b1c on 
branch feature/midas-2020-12).

same info is in the midas "help" page (GIT revision).

to decipher the git revision string:

midas-2020-08-a-84-g78d18b1c means:
is commit 78d18b1c
which is 84 commits after git tag midas-2020-08-a

"on branch feature/midas-2020-12" confirms that I have the midas-2020-12 pre-
release version without having to do all the decoding above.

if you also have "-dirty" it means you changed something in the source code 
 and warranty is voided. (just joking! we can debug even modified midas source 
code)

K.O.

We had the same issue here, which comes from mhttpd.js line 2395 on the current git version. This seems to happen mostly when there is an alarm triggered or when there is an error message.

Anyway, the easiest solution for us was to define m at the beginning of mhttpd_message function 

let m;

and replace line 2395 with

if (m !== undefined) {


> > I see this mhttpd error starting MSL-script: 
> > Uncaught (in promise) ReferenceError: m is not defined
> > at mhttpd_message (VM2848 mhttpd.js:2304)
> > at VM2848 mhttpd.js:2122
> 
> your line numbers do not line up with my copy of mhttpd.js. what version of midas 
> do you run?
> 
> please give me the output of odbedit "ver" command (GIT revision, looks like this: 
> IT revision:       Wed Feb 3 11:47:02 2021 -0800 - midas-2020-08-a-84-g78d18b1c on 
> branch feature/midas-2020-12).
> 
> same info is in the midas "help" page (GIT revision).
> 
> to decipher the git revision string:
> 
> midas-2020-08-a-84-g78d18b1c means:
> is commit 78d18b1c
> which is 84 commits after git tag midas-2020-08-a
> 
> "on branch feature/midas-2020-12" confirms that I have the midas-2020-12 pre-
> release version without having to do all the decoding above.
> 
> if you also have "-dirty" it means you changed something in the source code 
>  and warranty is voided. (just joking! we can debug even modified midas source 
> code)
> 
> K.O.

Thanks for reporting and pointing to the right location.

I fixed and committed it.

Best,
Stefan

The new history as well as the clock in the web page header show the local time 
of the user's computer running the browser.
Would it be possible to make it either always use the time zone of the Midas 
server, or make it selectable from the config page?
It's not ideal trying to relate error messages from the midas.log to history 
plots if the time stamps don't match.

> The new history as well as the clock in the web page header show the local time 
> of the user's computer running the browser.
> Would it be possible to make it either always use the time zone of the Midas 
> server, or make it selectable from the config page?
> It's not ideal trying to relate error messages from the midas.log to history 
> plots if the time stamps don't match.

I implemented a new row in the config page to select the time zone. 

"Local": Time zone where the browser runs
"Server": Time zone where the midas server runs (you have to update mhttpd for that)
"UTC+X": Any other time zone

The setting affects both the status header and the history display.

I spent quite some time with "named" time zones like "PST" "EST" "CEST", but the 
support for that is not that great in JavaScript, so I decided to go with simple 
UTC+X. Hope that's ok.

Please give it a try and let me know if it's working for you.

Best,
Stefan

> > The new history as well as the clock in the web page header show the local time 
> > of the user's computer running the browser.
> > Would it be possible to make it either always use the time zone of the Midas 
> > server, or make it selectable from the config page?
> > It's not ideal trying to relate error messages from the midas.log to history 
> > plots if the time stamps don't match.
> 
> I implemented a new row in the config page to select the time zone. 
> 
> "Local": Time zone where the browser runs
> "Server": Time zone where the midas server runs (you have to update mhttpd for that)
> "UTC+X": Any other time zone
> 
> The setting affects both the status header and the history display.
> 
> I spent quite some time with "named" time zones like "PST" "EST" "CEST", but the 
> support for that is not that great in JavaScript, so I decided to go with simple 
> UTC+X. Hope that's ok.
> 
> Please give it a try and let me know if it's working for you.
> 
> Best,
> Stefan

Hi Stefan,

This is great, the UTC+x is perfect, thank you.
PAA

Is it possible to embed modbvalue in an SVG for use within a custom page?

thanks.

You can't really embed it, but you can overlay it. You tag the SVG with a 
"relative" position and then move the modbvalue with an "absolute" position over 
it:

<svg style="position:relative" width="400" height="100">
  <rect width="300" height="100" style="fill:rgb(255,0,0);stroke-width:3;stroke:rgb(0,0,0)" />
  <div class="modbvalue" style="position:absolute;top:50px;left:50px" data-odb-path="/Runinfo/Run number"></div>
</svg>

I found a way to embed modbvalue into a SVG:

<text x="100" y="100" font-size="30rem">
Run=<tspan class="modbvalue" data-odb-path="/Runinfo/Run number"></tspan>
</text>

This seems to behave better that the suggestion below.

> You can't really embed it, but you can overlay it. You tag the SVG with a 
> "relative" position and then move the modbvalue with an "absolute" position over 
> it:
> 
> <svg style="position:relative" width="400" height="100">
>   <rect width="300" height="100" style="fill:rgb(255,0,0);stroke-width:3;stroke:rgb(0,0,0)" />
>   <div class="modbvalue" style="position:absolute;top:50px;left:50px" data-odb-path="/Runinfo/Run number"></div>
> </svg>

Version: release/midas-2020-12

In the new history display, the checkbox "Change all time axes together" works 
well with the mouse-based zoom, but does not apply to the +- buttons.

> Version: release/midas-2020-12
> 
> In the new history display, the checkbox "Change all time axes together" works 
> well with the mouse-based zoom, but does not apply to the +- buttons.

Fixed in current commit.

Stefan

Version: release/midas-2020-12

I'm exporting the history data shown in elog:2132/1 to CSV, but when I look at the 
CSV data, the step no longer occurs at the same time in both data sets (elog:2132/2)

History is from two separate equipments/frontends, but both have "Log history" set to 1.

Tried with export of two different time ranges, and the shift appears to remain the same, 
about 4040 rows.

I confirm there is a problem. If variables are from the same equipment, they have the same 
time stamps, like

t1 v1(t1) v2(t1)
t2 v1(t2) v2(t2)
t3 v1(t3) v2(t3)

when they are from different equipments, they have however different time stamps

t1 v1(t1)
t2    v2(t2)
t3 v1(t3)
t4    v2(t4)

The bug in the current code is that all variables use the time stamps of the first variable, 
which is wrong in the case of different equipments, like

t1 v1(t1) v2(*t2*)
t3 v1(t3) v2(*t4*)

So I can change the code, but I'm not sure what would be the bast way. The easiest would be to 
export one array per variable, like

t1 v1(t1)
t2 v1(t2)
...
t3 v2(t3)
t4 v2(t4)
...

Putting that into a single array would leave gaps, like

t1 v1(t1) [gap]
t2 [gap]  v2(t2)
t3 v1(t3) [gap]
t4 [ga]]  v2(t4)

plus this is programmatically more complicated, since I have to merge two arrays. So which 
export format would you prefer?

Stefan

I think from my perspective the separate files are fine. I personally don't really like the format 
with the gaps, so don't see an advantage in putting in the extra work.
I'm surprised the shift is this big, though, it was more than a whole hour in my case, is it the 
time difference between when the frontends were started?

I finally found some time to fix this issue in the latest commit. Please update and check if it's 
working for you.

Stefan

We have to request of a 64-bit integer data type to be included in MIDAS banks.
Since 64-bit integers are on some systems "long" and on other systems "long long",
I decided to create the two new data types

TID_INT64
TID_UINT64

which follows more the standard C++ tradition:

https://en.cppreference.com/w/cpp/types/integer

To be consistent, I renamed the old types:

TID_BYTE       -> TID_UINT8
TID_SBYTE      -> TID_INT8
TID_WORD       -> TID_UINT16
TID_SHORT      -> TID_INT16
TID_DWORD      -> TID_UINT32
TID_INT        -> TID_INT32

I left the old definitions in midas.h, so old code will still compile fine and be binary
compatible. But if you write new code, the new types are recommended.

If you save the ODB in ASCII format, the new types are used as stings as well, like

[/Experiment]
ODB timeout = INT32 : 10000

but the old types are still understood when you load an old ODB file.

I hope I didn't break anything, please report if you have any issue.

Stefan

> We have to request of a 64-bit integer data type to be included in MIDAS banks.
> Since 64-bit integers are on some systems "long" and on other systems "long long",
> I decided to create the two new data types
> 
> TID_INT64
> TID_UINT64
> 

These 64-bit data types do not work with ODB and they do not work with the MIDAS history.

As of commits on 30 March 2021, mlogger will refuse to write them to the history and 
db_create_key() will refuse to create them in ODB.

Why these limitations:

a1) all reading of history is done using the "double" data type, IEEE-754 double precision 
floating point numbers have around 53 bits of precision and are too small to represent all 
possible values of 64-bit integers.
a2) SQL, SQLite and FILE history know nothing about reading and writing 64-bit integer data 
types (this should be easy to fix, as long as MySQL/MariaDB and PostgresQL support it)

b1) in ODB, odbedit and mhttd web pages do not display INT64/UINT64/QWORD data
b2) ODB save and restore from odb, xml and json formats most likely does not work for these 
data types

Fixing all this is possible, with a medium amount of work. As long as somebody needs it. 
Display of INT64/UINT64/QWORD on history plots will probably forever be truncated to 
"double" precision.

K.O.

> These 64-bit data types do not work with ODB and they do not work with the MIDAS history.

They were never meant to work with the history. They were primarily implemented to put large 64-
bit data words into midas banks. We did not yet have a request to put these values into the ODB. 
Once such a request comes, we can address this.

Stefan

> 
> To be consistent, I renamed the old types:
> 
> TID_DWORD      -> TID_UINT32
> TID_INT        -> TID_INT32
> 

this created an incompatibility with old XML save files,
old versions of midas cannot load new XML save files,
variable types have changed i.e. from "INT" to "INT32".

it would have been better if XML save files kept using the old names.

now packages that read midas XML files also need updating.

specifically, in ROOTANA:
- the old TVirtualOdb/XmlOdb.cxx (no longer used, deleted),
- mvodb/mxmlodb.cxx

K.O.

Hi all,

I'm running into a curious problem when I try to run a program using my custom 
script button. I have been using a script button to start my DAQ, this button 
has always worked. It starts by exporting an absolute path to scripts and then 
runs scripts, my frontend, my analyzer, and mlogger relative to this path.

I recently added a line of code to run a new script "logic_controller". If I run 
the script_daq from my terminal (./start_daq), mhttpd accepts the client and the 
program works as intended. But, if I use the script button, the logic_controller 
program is immediately deleted by MIDAS. It can be seen appearing in the status 
page clients list and then immediately gets deleted. This is a client that runs 
on the local experiment host.

What might be the issue? What is the difference between running the script 
through the terminal as opposed to running it through the mhttpd button?

I have added a picture of my simple script and the logic_controller code.

Any help would be greatly appreciated.

Cheers.

Isaac

> if I use the script button, the logic_controller program is immediately deleted by MIDAS.

This is indeed very curious, and I can't reproduce it on my test experiment. Can you redirect stdout and stderr from the logic_controller program into a file, to see how far the program gets? If it gets to the while loop at the end, then it would be useful to add some debug statements to see what condition causes it to exit the loop.

Are there any relevant messages in the midas message log about the program being killed? What's the value of "/Programs/logic_controller/Watchdog timeout"? 

> > if I use the script button, the logic_controller program is immediately deleted by MIDAS.
> 
> This is indeed very curious, and I can't reproduce it on my test experiment. Can you redirect stdout and stderr from the logic_controller program into a file, to see how far the program gets? If it gets to the while loop at the end, then it would be useful to add some debug statements to see what condition causes it to exit the loop.

I have redirected stdout and stderr into a text file and I have attached it to this entry. From what the stdout says, it seems that the lambda
function gets called 4 times before the program disconnects from the experiment. Somehow the status must become SS_ABORT or RPC_SHUTDOWN.

> Are there any relevant messages in the midas message log about the program being killed? What's the value of "/Programs/logic_controller/Watchdog timeout"? 

There are no interesting messages in the midas.log and "/Programs/logic_controller/Watchdog timeout" is 10000 when I run the command from the terminal window.
What happens when you run it on your test experiment?

I'll try some more debugging.

Thanks for helping me out! Cheers.

Isaac

I think it would be useful to find the minimal example that exhibits this behaviour.

What happens if your logic controller code is simply the 17 lines below? What happens if you create another script button that only starts the logic controller, not any of the other programs? etc. Gradually re-add features until you hit the problem (or scream in horror if it breaks with 17 lines of C++ and a 1 line shell script).



#include "midas.h"
#include "stdio.h"

int main() {
   cm_connect_experiment("", "", "logic_controller", NULL);

   do {
     int status = cm_yield(100);
     printf("cm_yield returned %d\n", status);
     if (status == SS_ABORT || status == RPC_SHUTDOWN)
       break;
   } while (!ss_kbhit());

   cm_disconnect_experiment();

   return 0;
}

> I think it would be useful to find the minimal example that exhibits this behaviour.
> 
> What happens if your logic controller code is simply the 17 lines below? What happens if you create another script button that only starts the logic controller, not any of the other programs? etc. Gradually re-add features until you hit the problem (or scream in horror if it breaks with 17 lines of C++ and a 1 line shell script).
> 

Hi Ben,

I have followed your suggestions and the program still stops immediately. My status as returned from "cm_yield(100)" is always 412 (SS_TIMEOUT) which is fine. 
The issue is that, when run with the script button, the do-wile loop stops immediately because the !ss_kbhit() always evaluates to FALSE.

My temporary solution has been to let the loop run forever :)

Let me know what think. Thanks again!

Isaac

> 
> 
> #include "midas.h"
> #include "stdio.h"
> 
> int main() {
>    cm_connect_experiment("", "", "logic_controller", NULL);
> 
>    do {
>      int status = cm_yield(100);
>      printf("cm_yield returned %d\n", status);
>      if (status == SS_ABORT || status == RPC_SHUTDOWN)
>        break;
>    } while (!ss_kbhit());
> 
>    cm_disconnect_experiment();
> 
>    return 0;
> }

> I have followed your suggestions and the program still stops immediately. My status as returned from "cm_yield(100)" is always 412 (SS_TIMEOUT) which is fine. 
> The issue is that, when run with the script button, the do-wile loop stops immediately because the !ss_kbhit() always evaluates to FALSE.
> 
> My temporary solution has been to let the loop run forever :)

Ahh, could be that ss_kbhit() misbehaves if there is no keyboard, meaning that it is started in the background as a script. 
We never had the issue before, since all "standard" midas programs like mlogger, mhttpd etc. also use ss_kbhit() and they 
can be started in the background via the "-D" flag, but maybe the stdin is then handled differentlhy. 

So just remove the ss_kbhit(), but keep the break, so that you can stop your program via the web page, like

#include "midas.h"
#include "stdio.h"

int main() {
  cm_connect_experiment("", "", "logic_controller", NULL);

  do {
    int status = cm_yield(100);
    printf("cm_yield returned %d\n", status);
    if (status == SS_ABORT || status == RPC_SHUTDOWN)
      break;
  } while (TRUE);

  cm_disconnect_experiment();

  return 0;
}

In April 4th 2020 Stefan added a new data format that fixes the well known problem with alternating banks being 
misaligned against 64-bit addresses. (cannot find announcement on this forum. midas commit 
https://bitbucket.org/tmidas/midas/commits/541732ea265edba63f18367c7c9b8c02abbfc96e)

This brings the number of midas data formats to 3:

bk_init: bank_header_flags set to 0x0000001 (BANK_FORMAT_VERSION)
bk_init32: bank_header_flags set to 0x0000011 (BANK_FORMAT_VERSION | BANK_FORMAT_32BIT)
bk_init32a: bank_header_flags set to 0x0000031 (BANK_FORMAT_VERSION | BANK_FORMAT_32BIT | BANK_FORMAT_64BIT_ALIGNED;

TMEvent (midasio and manalyzer) support for "bk_init32a" format added today (commit 
https://bitbucket.org/tmidas/midasio/commits/61b7f07bc412ea45ed974bead8b6f1a9f2f90868)

TMidasEvent (rootana) support for "bk_init32a" format added today (commit 
https://bitbucket.org/tmidas/rootana/commits/3f43e6d30daf3323106a707f6a7ca2c8efb8859f)

ROOTANA should be able to handle bk_init32a() data now.

TMFE MIDAS c++ frontend switched from bk_init32() to bk_init32a() format (midas commit 
https://bitbucket.org/tmidas/midas/commits/982c9c2f8b1e329891a782bcc061d4c819266fcc)

K.O.

Until commit a4043ceacdf241a2a98aeca5edf40613a6c0f575 today, mdump mostly did not work with bank32a data.
K.O.


> In April 4th 2020 Stefan added a new data format that fixes the well known problem with alternating banks being 
> misaligned against 64-bit addresses. (cannot find announcement on this forum. midas commit 
> https://bitbucket.org/tmidas/midas/commits/541732ea265edba63f18367c7c9b8c02abbfc96e)
> 
> This brings the number of midas data formats to 3:
> 
> bk_init: bank_header_flags set to 0x0000001 (BANK_FORMAT_VERSION)
> bk_init32: bank_header_flags set to 0x0000011 (BANK_FORMAT_VERSION | BANK_FORMAT_32BIT)
> bk_init32a: bank_header_flags set to 0x0000031 (BANK_FORMAT_VERSION | BANK_FORMAT_32BIT | BANK_FORMAT_64BIT_ALIGNED;
> 
> TMEvent (midasio and manalyzer) support for "bk_init32a" format added today (commit 
> https://bitbucket.org/tmidas/midasio/commits/61b7f07bc412ea45ed974bead8b6f1a9f2f90868)
> 
> TMidasEvent (rootana) support for "bk_init32a" format added today (commit 
> https://bitbucket.org/tmidas/rootana/commits/3f43e6d30daf3323106a707f6a7ca2c8efb8859f)
> 
> ROOTANA should be able to handle bk_init32a() data now.
> 
> TMFE MIDAS c++ frontend switched from bk_init32() to bk_init32a() format (midas commit 
> https://bitbucket.org/tmidas/midas/commits/982c9c2f8b1e329891a782bcc061d4c819266fcc)
> 
> K.O.

For my analyzer I generate the experim.h file from the odb.

Before midas commit 13c3b2b this generates structs with INT data types. compiles fine with my analysis code (using the old mana.cpp)

newer midas versions generate INT32, ... types. I get a 

‘INT32’ does not name a type   

although I include midas.h 

how to fix this?

> For my analyzer I generate the experim.h file from the odb.
> 
> Before midas commit 13c3b2b this generates structs with INT data types. compiles fine with my analysis code (using the old mana.cpp)
> 
> newer midas versions generate INT32, ... types. I get a 
> 
> ‘INT32’ does not name a type   
> 
> although I include midas.h 
> 
> how to fix this?

You could run experim.h through "sed" to replace the "wrong" data types with the correct data types.

You can also #define the "wrong" data types before doing #include experim.h.

I put your bug report into our bug tracker, but for myself I am very busy
with the alpha-g experiment and cannot promise to fix this quickly.

https://bitbucket.org/tmidas/midas/issues/289/int32-types-in-experimh

Here is an example to substitute things using "sed" (it can also do "in-place" editing, "man sed" and google sed examples)
sed "sZshm_unlink(.*)Zshm_unlink(SHM)Zg"

K.O.

> > For my analyzer I generate the experim.h file from the odb.
This issue is still open. Shouldn't midas.h provide the 'new' data types as typedefs like  

typedef int INT32;

etc. Of course you would need to deal with all the supported targets and wrap it accordingly.

A.S.

> > 
> > Before midas commit 13c3b2b this generates structs with INT data types. compiles fine with my analysis code (using the old mana.cpp)
> > 
> > newer midas versions generate INT32, ... types. I get a 
> > 
> > ‘INT32’ does not name a type   
> > 
> > although I include midas.h 
> > 
> > how to fix this?
> 
> You could run experim.h through "sed" to replace the "wrong" data types with the correct data types.
> 
> You can also #define the "wrong" data types before doing #include experim.h.
> 
> I put your bug report into our bug tracker, but for myself I am very busy
> with the alpha-g experiment and cannot promise to fix this quickly.
> 
> https://bitbucket.org/tmidas/midas/issues/289/int32-types-in-experimh
> 
> Here is an example to substitute things using "sed" (it can also do "in-place" editing, "man sed" and google sed examples)
> sed "sZshm_unlink(.*)Zshm_unlink(SHM)Zg"
> 
> K.O.

Ok, I added

/* define integer types with explicit widths */
#ifndef NO_INT_TYPES_DEFINE
typedef unsigned char      UINT8;
typedef char               INT8;
typedef unsigned short     UINT16;
typedef short              INT16;
typedef unsigned int       UINT32;
typedef int                INT32;
typedef unsigned long long UINT64;
typedef long long          INT64;
#endif

to cover all new types. If there is a collision with user defined types, compile your program with -DNO_INT_TYPES_DEFINE and you remove the 
above definition. I hope there are no other conflicts.

Stefan

works!

> Ok, I added
> 
> /* define integer types with explicit widths */
> #ifndef NO_INT_TYPES_DEFINE
> typedef unsigned char      UINT8;
> typedef char               INT8;
> typedef unsigned short     UINT16;
> typedef short              INT16;
> typedef unsigned int       UINT32;
> typedef int                INT32;
> typedef unsigned long long UINT64;
> typedef long long          INT64;
> #endif
> 
> to cover all new types. If there is a collision with user defined types, compile your program with -DNO_INT_TYPES_DEFINE and you remove the 
> above definition. I hope there are no other conflicts.
> 
> Stefan

> > 
> > /* define integer types with explicit widths */
> > #ifndef NO_INT_TYPES_DEFINE
> > typedef unsigned char      UINT8;
> > typedef char               INT8;
> > typedef unsigned short     UINT16;
> > typedef short              INT16;
> > typedef unsigned int       UINT32;
> > typedef int                INT32;
> > typedef unsigned long long UINT64;
> > typedef long long          INT64;
> > #endif
> > 

NIH at work. In C and C++ the standard fixed bit length data types are available
in #include <stdint.h> as uint8_t, uint16_t, uint32_t, uint64_t & co.

BTW, the definition of UINT32 as "unsigned int" is technically incorrect, on 16-bit machines
"int" is 16-bit wide and on some 64-bit machines "int" is 64-bit wide.

K.O.