I am using the Midas sequencer to run subsequent measurements in a loop, without 
knowing how many iterations in advance. Therefore, I am using the "infinity" 
option. Since I have other commands after the loop, it would be nice to have the 
possibility to break the loop, but let the sequencer then finish the rest of the 
commands. 
Cheers,
Ivo

> I am using the Midas sequencer to run subsequent measurements in a loop, without 
> knowing how many iterations in advance. Therefore, I am using the "infinity" 
> option. Since I have other commands after the loop, it would be nice to have the 
> possibility to break the loop, but let the sequencer then finish the rest of the 
> commands. 
> Cheers,
> Ivo

You can do that with the "GOTO" statement, jumping to the first line after the loop.

Here is a working example:


LOOP runs, 5
     WAIT Seconds 3
     IF $runs > 2
         GOTO 7
     ENDIF
ENDLOOP
MESSAGE "Finished", 1

Best,
Stefan

> You can do that with the "GOTO" statement, jumping to the first line after the loop.
> 
> Here is a working example:
> 
> 
> LOOP runs, 5
>      WAIT Seconds 3
>      IF $runs > 2
>          GOTO 7
>      ENDIF
> ENDLOOP
> MESSAGE "Finished", 1
> 
> Best,
> Stefan

Hoi Stefan

Thanks for your answer. As I understand it, this has to be in the sequence script before 
running. So, in the end, it is not different than just saying "LOOP runs, 2" and 
therefore the number of runs has do be known in advance as well. Or is there an option to 
change the script on runtime? What I would like, is to start a sequence with "LOOP runs, 
infinite" and when I come back to the experiment after falling asleep being able to break 
the loop after the next iteration, but still execute everything after ENDLOOP, i.e. the 
MESSAGE statement in your example. Because if I do a "Stop after current run", this seems 
not to happen. 

Best, Ivo

> > You can do that with the "GOTO" statement, jumping to the first line after the loop.
> > 
> > Here is a working example:
> > 
> > 
> > LOOP runs, 5
> >      WAIT Seconds 3
> >      IF $runs > 2
> >          GOTO 7
> >      ENDIF
> > ENDLOOP
> > MESSAGE "Finished", 1
> > 
> > Best,
> > Stefan
> 
> Hoi Stefan
> 
> Thanks for your answer. As I understand it, this has to be in the sequence script before 
> running. So, in the end, it is not different than just saying "LOOP runs, 2" and 
> therefore the number of runs has do be known in advance as well. Or is there an option to 
> change the script on runtime? What I would like, is to start a sequence with "LOOP runs, 
> infinite" and when I come back to the experiment after falling asleep being able to break 
> the loop after the next iteration, but still execute everything after ENDLOOP, i.e. the 
> MESSAGE statement in your example. Because if I do a "Stop after current run", this seems 
> not to happen. 
> 
> Best, Ivo

First, you have the sequencer button "Stop after current run", but that does of course ot
execute anything after ENDLOOP. 

Second, you can put anything in the IF statement. Like create a variable on the ODB like
/Experiment/Run parameters/Stop loop and set this to zero. Then put in your script:

...
ODBGET /Experiment/Run parameters/Stop loop, flag
IF $flag == 1
   GOTO 7
...

So once you want to stop the loop, set the flag in the ODB to one.

Best,
Stefan

> LOOP runs, 5
> ...
> ENDLOOP

Classical loop constructs usually include "break" (exit the loop) and "continue" (loop again!) 
constructs. I would say it is an unfortunate omission if they are not present in the midas sequencer.

But Stefan is right, of course, both commands are just funny names for "goto".

K.O.

the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas. If you do not want to use this 
updated code or if it causes problems, please use the mhttpd6 executable or midas from the midas-2020-03 release branch.

new features:

- IPv6 support
- built-in http proxy
- fine grain locking - serving "resource" files (html, css, etc) and serving json-rpc requests no longer takes the global lock
- reduced number of DNS queries when checking host list access (DNS replies are cached)
- (I decided to not implement caching of password requests and dynamic reload of password file - it is too hard).

internal changes:

Recent versions of the mongoose web server library have removed all their internal multithreading,
leaving the library fully single-threaded. This resulted in major simplification of many things. An improvement.
(the civetweb fork of mongoose retains the old multithreading code, that model seems to work better
which used inside ROOT). As implemented in mhttpd, all network connections are handled by the main thread,
all midas http requests are handled by worker threads that are started on the as-needed basis.

The old mongoose 6.4 based mhttpd code survived almost without changes - as a compile-time
option - so now I build 2 mhttpd executables: mhttpd with the new code and mhttpd6 with the old code
so people have something to run in case the new code bombs.

http proxy:

Experiments that use private networks usually configure the apache httpd as a web proxy to allow
access from the outside to the web-controlled devices on the private network. Making changes
to this proxy requires root access, requires restarting httpd, etc. To make things simpler, mhttpd now
includes a web proxy (almost the complete implementation is provided by the mongoose library). Configuration
is done from ODB, restarting mhttpd is not needed.

improved multithreading:

Since most of the MIDAS library is now thread-safe, mhttpd no longer needs to take the "big midas lock"
to service most web requests. Access to files, access to ODB, etc is now fully threaded. Some parts
of MIDAS are not thread-safe, i.e. access to history and log files, so a flag was added to the mjsonrpc library
to mark which RPC methods are not thread-safe.

Note that despite these improvements, mhttpd still suffers from "http head-of-queue blocking"
https://en.wikipedia.org/wiki/Head-of-line_blocking
because (i.e. the google chrome web browser) tends to use just 1 TCP connection for all JSONRPC requests,
after a request for a history read (can take a long time), all subsequent requests for web page updates, etc
will have to wait until it completes, causing unresponsive user experience. (it looks as if mhttpd is single-threaded!).

A solution for this problem is HTTP/2, which is not yet implemented by mongoose and is not quite yet available
for apache httpd.

More later...
K.O.

> the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.

The new code implements 3 http ports:

- localhost port 8080 - enabled by default - suitable for "I want to test midas on my laptop" and for connecting from the apache httpd
https password protected gateway.

- insecure http port 8081 - disabled by default - with optional password protection (HTTP Digest auth), and optional hostlist access
control - for the case when the https gateway is running on a different computer (i.e. ALPHA at CERN).

(My reading of "internet opinions" about HTTP Digest authentication over unencrypted HTTP is
that while considered very obsolete, there are no specific security problems and exploits
against it - other than the usual - man-in-the-middle and "steal the password file" attacks.
So while I do not recommend using it, I do not feel justified to remove/disable it on security grounds.
It provides an alternative password protection when use of SSL/HTTPS is too difficult).

- https port 8443 - disabled by default - also with optional password protection (HTTP Digest auth), and optional hostlist access
control. HTTP Digest password protection over HTTPS is deemed as secure at "HTTP Basic" password protection over HTTPS and
that is what is used by apache httpd password protection.

(The main problem with mhttpd support of HTTPS is obtaining an https certificate. Right now mhttpd
instructs the user to generate a self-signed certificate. But there is 2 problems: modern browsers dislike self-signed
certificates (even when explicitely marked "trust it!") and there is no check for certificate expiration.
I guess one could try to integrate mhttpd with certbot and the let's-encrypt system, but there
is problems, i.e. the certificate files live in readable-only-by-root directories, etc. I would rather
wait until mongoose implement certbot integration in their code).

More later...
K.O.

> > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.

Configuration is done by ODB /WebServer:

---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
mime.types                      DIR
Enable localhost port           BOOL    1     4     2h   0   RWD  y
localhost port                  INT     1     4     2h   0   RWD  8080
localhost port passwords        BOOL    1     4     2h   0   RWD  n
Enable insecure port            BOOL    1     4     12h  0   RWD  n
insecure port                   INT     1     4     2h   0   RWD  8081
insecure port passwords         BOOL    1     4     2h   0   RWD  y
insecure port host list         BOOL    1     4     2h   0   RWD  y
Enable https port               BOOL    1     4     12h  0   RWD  n
https port                      INT     1     4     2h   0   RWD  8443
https port passwords            BOOL    1     4     2h   0   RWD  y
https port host list            BOOL    1     4     2h   0   RWD  y
Host list                       STRING  10    32    2h   0   RWD
                                        [0]             localhost
                                        [1]
                                        [2]
                                        [3]
                                        [4]
                                        [5]
                                        [6]
                                        [7]
                                        [8]
                                        [9]
Enable IPv6                     BOOL    1     4     2h   0   RWD  y
Proxy                           DIR
---------------------------------------------------------------------------

Most entries are self-obvious, but note:

- mime.types contains the mapping of file extensions of file content-type telling browser what to do:

---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l mime.types/
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
.HTML                           STRING  1     10    2h   0   RWD  text/html
.HTM                            STRING  1     10    2h   0   RWD  text/html
.CSS                            STRING  1     9     2h   0   RWD  text/css
---------------------------------------------------------------------------

- Proxy directory configures the http proxy (as implemented by mongoose, I am
not sure if I understand all limitations):

---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l Proxy/
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
example                         STRING  1     27    17h  0   RWD  #http://localhost:8080
---------------------------------------------------------------------------

("#" means - commented-out)

http://localhost:8080/proxy/example/foo/bar/baz proxies to http://localhost:8080/foo/bar/baz

- "Enable IPv6" tells mhttpd to also listen on the IPv6 ports. The best I can tell IPv6 works on the Mac,
and with luck will get some testing at CERN where IPv6 is in use.

Documentation on the midas wiki still needs to be updated for this.
K.O.

> > > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.

current code looks for the mbedtls library in ../mbedtls (next to midas)

if cmake misdetects it, turn it off by setting NO_MBEDTLS (same as NO_ROOT & co)

if you do want to build mhttpd with mbedtls, do this:

cd .../midas
cd ../
git clone https://github.com/ARMmbed/mbedtls.git
cd mbedtls
git submodule update --init ### this will populate the "crypto" directory
make ### if "python2" is missing, building of test suite programs will fail, but the libraries needed for midas will be built

cd ../midas
make cmake...

K.O.

I had some quick look at the new mongoose code and didn't find anything I dislike. Did a quick test of the proxy which worked and is nice to have. 
Agree with all KO said about authentication.

So if there are no complaints, I would suggest that we move the summary of this thread into the official documentation.

Stefan

Dear All,

At our experiment we write ROOT files. When starting/stopping runs we get the following error messages:

[Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8000

[Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8001

Looking into the source code I found that log_write (line 4248) sends these Midas System Events (BOR,EOR) to root_write without filtering them. root_write() checks in a first step if it gets such Midas System Events and if yes, moans.

Wouldn't it be better just to filter these events in log_write, before calling root_write, avoiding unnecessary error messages?

Is there something I miss?

Thanks,
  Andreas

> [Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8000

Hi, Andreas, please open a bug report for this problem on bitbucket, there is now at least 2 bugs against
the ROOT writer (some events are written in duplicate sometimes), and I hope to fix this next time i review
the mlogger (RSN!). Biggest problem is that I do not use the ROOT output myself, so I have no way
to know if ROOT files produced by mlogger are correct or make sense. (without setting up some kind
of test environment with a ROOT file reader.

Thank you for reporting this problem here, so more people know about it.

If somebody has a patch to fix this, please send it in!

K.O.

Dear simplest solution seems to me to just remove the error message generation and silently ignore the BOE EOR events. 

Committed that change.

Stefan

Hi Stefan,

I think this only partially resolves the issue, in log_write:

#ifdef HAVE_ROOT
   } else if (log_chn->format == FORMAT_ROOT) {
      status = root_write(log_chn, pevent, pevent->data_size + sizeof(EVENT_HEADER));
#endif
   }

   actual_time = ss_millitime();
   if ((int) actual_time - (int) start_time > 3000)
      cm_msg(MINFO, "log_write", "Write operation on \'%s\' took %d ms", log_chn->path.c_str(), actual_time - start_time);

   if (status != SS_SUCCESS && !stop_requested) {
      cm_msg(MTALK, "log_write", "Error writing output file, stopping run");
      cm_msg(MERROR, "log_write", "Cannot write \'%s\', error %d, stopping run", log_chn->path.c_str(), status);
      stop_the_run(0);

      return status;
   }

In your solution root_write returns quietly but status == SS_INVALID_FORMAT (not SS_SUCCESS) and hence I get another misleading error message "Error writing output file, stopping run".

In order to prevent this you also would need to change the return value to SS_SUCCESS.

Ok, changed.

Stefan

Dear all
I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 
Thanks in advance,
Ivo

> I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 

Hi, Ivo, good to hear from a midas user in these difficult times.

We do not use FTP at TRIUMF, but Stefan asked us to keep FTP alive and working, so we should be able
to get you going. I will try to find the FTP instructions for you, I am pretty sure I have them somewhere.

In the mean time, I am very curious why you are using a FTP to record data, is it some kind
of data appliance where simplest input for data is FTP? Using NFS does not work or is too hard?

Also for example at CERN, we write data to Castor and EOS, for this mlogger writes data to local disk,
then the lazylogger runs a script to move the data to Castor and EOS. The example lazylogger
scripts for this are in the MIDAS "progs" directory. But maybe you do not have a local disk and this would
not work for you.

In other news, I hope to work on mlogger and lazylogger support for cloud storage (swift and s3 apis?),
would that be useful as replacement for FTP?

K.O.

> > I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 
> 
> Hi, Ivo, good to hear from a midas user in these difficult times.
> 
> We do not use FTP at TRIUMF, but Stefan asked us to keep FTP alive and working, so we should be able
> to get you going. I will try to find the FTP instructions for you, I am pretty sure I have them somewhere.
> 
> In the mean time, I am very curious why you are using a FTP to record data, is it some kind
> of data appliance where simplest input for data is FTP? Using NFS does not work or is too hard?
> 
> Also for example at CERN, we write data to Castor and EOS, for this mlogger writes data to local disk,
> then the lazylogger runs a script to move the data to Castor and EOS. The example lazylogger
> scripts for this are in the MIDAS "progs" directory. But maybe you do not have a local disk and this would
> not work for you.
> 
> In other news, I hope to work on mlogger and lazylogger support for cloud storage (swift and s3 apis?),
> would that be useful as replacement for FTP?
> 
> K.O.
>
Good Morning Konstantin

Thanks for the fast reply.  Yes, it is, Midas is one of the things we can at least improve from home. 

Our experiment is planned to measure (soon) at ILL. Now since we don't use the equipment/detector from the 
beamline but our own, all the data from Midas is saved on the local drive. This is fine in the first instance
but then we also need proper backup. Since our experiment is quite small, the easiest solution I came up with
is to copy all of our data to the ILL storage which has enough space and is properly backed up. The ILL data 
storage allows only SFTP connections, nothing else. Since Midas has the FTP feature, having a separate FTP 
logger channel seemed the easiest way to go. 

Thanks for your input, I will look into how to mount SFTP and then this would also be a solution. 

Since ILL only provides access via SFTP and everything else is not existent or blocked (not even ssh is possible),
this is the only thing we can work with by now. 

Best regards,
Ivo

Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain. 
What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the 
local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.

Best,
Stefan

> Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain. 
> What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the 
> local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.
> 
> Best,
> Stefan

Yes, see this now too. I will, therefore, try to set up the lazylogger properly. 

> 
> Since ILL only provides access via SFTP and everything else is not existent or blocked (not even ssh is possible),
> this is the only thing we can work with by now. 
> 

Oops. SFTP != FTP.

SFTP uses SSH for data transport, so we cannot do it directly from C++ code in MIDAS. (we could use libssh, etc, but...)

I suggest you use lazylogger with the lazy_dache script, replace "dccp" with "sftp", replace "nsls" with an sftp "ls" command.

If you get it working, please consider contributing your lazylogger script to midas. (and does not have to be written in perl, python should work equally well).

For setting up lazylogger with the script method, I am pretty sure I posted the instructions to the forum (ages ago),
let me know if you cannot find them.

Good luck.

K.O.

After much discussion, and following the MIDAS workshop at TRIUMF, we made the decision to use C++11 in MIDAS.

There are many benefits, and only one drawback - no c++11 compilers in the default OS install on older computers (i.e. 
RHEL/SL/CentOS before el7). (the same applies to our use of cmake).

Specifically for el6, the solution is to use c++11 compatible gcc-8 from devtoolset-8, see 
https://midas.triumf.ca/elog/Midas/1649

The c++11 features we most welcome - initialization of class members at declaration time (no more forgetting to add initialization to 
each and every constructor), c++ threads and mutexes, lambdas and "auto".

K.O.

> After much discussion, and following the MIDAS workshop at TRIUMF, we made the decision to use C++11 in MIDAS.
> 
> There are many benefits, and only one drawback - no c++11 compilers in the default OS install on older computers (i.e. 
> RHEL/SL/CentOS before el7). (the same applies to our use of cmake).
>

It turns out that support for the c++11 "regex" feature is missing on el7 (CentOS-7, our most common platform at TRIUMF).

According to https://stackoverflow.com/questions/12530406/is-gcc-4-8-or-earlier-buggy-about-regular-expressions
gcc 4.9.0 is the first one to implement c++11 regular expressions. el7 comes with gcc-4.8.5 and I confirm
that examples of using std::regex_replace() do not compile. I was looking to use std::regex_replace to implement URL rewriting
in the reverse proxy code in mhttpd.

I do not need this feature immediately, but I am surprised that such a thing can happen, thought others should know.

K.O.

About the boost library, that is exactly 
what I did for a project of mine (the 
calibration software for the WAGASCI 
experiment). It turned out not so easy to 
mantain because different Linux distros 
package different versions of boost.

The reason I went down the "c++11 plus 
boost" road is that the official T2K OS 
is CentOS7 as well.

Looking back I think that using c++17 and 
requiring a more recent version of the 
compiler is much easier to maintain than 
the combo c++11 + boost. In CentOS is 
just a matter of installing a recent 
devtool package ...

Another solution might be too repackage 
boost into MIDAS so you have full control 
of the environment.

> > After much discussion, and following 
the MIDAS workshop at TRIUMF, we made the 
decision to use C++11 in MIDAS.
> > 
> > There are many benefits, and only one 
drawback - no c++11 compilers in the 
default OS install on older computers 
(i.e. 
> > RHEL/SL/CentOS before el7). (the same 
applies to our use of cmake).
> >
> 
> It turns out that support for the c++11 
"regex" feature is missing on el7 
(CentOS-7, our most common platform at 
TRIUMF).
> 
> According to 
https://stackoverflow.com/questions/12530
406/is-gcc-4-8-or-earlier-buggy-about-
regular-expressions
> gcc 4.9.0 is the first one to implement 
c++11 regular expressions. el7 comes with 
gcc-4.8.5 and I confirm
> that examples of using 
std::regex_replace() do not compile. I 
was looking to use std::regex_replace to 
implement URL rewriting
> in the reverse proxy code in mhttpd.
> 
> I do not need this feature immediately, 
but I am surprised that such a thing can 
happen, thought others should know.
> 
> K.O.

Just FYI, I am looking at the ROOT web programming component JSROOT and I notice that the RPC mechanism quite different from the JSON-
RPC I implemented for MIDAS.

https://github.com/root-project/jsroot/blob/master/docs/HttpServer.md (explanation of JSROOT RPC and server side machinery)
https://github.com/root-project/jsroot/blob/master/docs/JSROOT.md (explanation of JSROOT javascript library)

Then I looked at the dates:
MIDAS mjsonrpc was done at the end of 2013
JSROOT main development started at the end of 2014.

The web server component in both projects is (almost) the same - vanilla mongoose in mhttpd
and civetweb, a fork of an older version of mongoose, in ROOT/JSROOT.

The web server in both projects is partially multithreaded:
- ROOT THttpServer/TCivetWeb uses multiple threads to handle the network connections and some file access,
but interaction with ROOT is done in the main thread of ROOT. (The main thread must periodically call ProcessRequests()).
- mhttpd uses a single thread to multiplex the network connections (it is a change from old mongoose/civetweb to current mongoose 6.16),
but all requests are farmed to a pool of threads and execute in parallel (unless not thread-safe, i.e. accessing history files).

Both implementations suffer from "head of queue" blocking, a "slow" request i.e. a slow file read, will
delay subsequent quick requests, see https://en.wikipedia.org/wiki/Head-of-line_blocking#In_HTTP
Solution for this problem is to use HTTP/2 when it becomes supported in mongoose/civetweb/apache httpd (in el7).

It will be interesting to see which on of the two systems works better for building "user facing" web pages... especially
hybrid pages that have to pull data both from midas (using mjsonrpc) and from online ROOT analyzers (using jsroot).

K.O.

I ported a bunch of frontends to C++ and now I'm occasionally getting this RPC 
error message:

http error: readyState: 4, HTTP status: 502 (Proxy Error), batch request: method: 
"db_get_values", params: [object Object], id: 1583456958869 method: "get_alarms", 
params: null, id: 1583456958869 method: "cm_msg_retrieve", params: [object 
Object], id: 1583456958869 method: "cm_msg_retrieve", params: [object Object], 
id: 1583456958869

I'm assuming I'm doing wrong something somewhere, but does this message contain 
information where to look? Does the ID mean something?

I do not see this error, but there was one more report (they did not clearly say what http errors 
they see) https://bitbucket.org/tmidas/midas/issues/209/get-rid-of-mjsonrpc-dialogs-put-it-to-
the

To debug this, I need to know: what version of MIDAS, what version of what web browser, what 
computer is mhttpd running on? (so I can go look at the log files).

Also can you say more when you see these errors? Every time from every midas web page, or only 
some pages or only when you do something specific (push some button, etc?).

> I ported a bunch of frontends to C++ and now I'm occasionally getting this RPC 
> error message:
> 
> http error: readyState: 4, HTTP status: 502 (Proxy Error), batch request: method: 
> "db_get_values", params: [object Object], id: 1583456958869 method: "get_alarms", 
> params: null, id: 1583456958869 method: "cm_msg_retrieve", params: [object 
> Object], id: 1583456958869 method: "cm_msg_retrieve", params: [object Object], 
> id: 1583456958869
> 
> I'm assuming I'm doing wrong something somewhere, but does this message contain 
> information where to look? Does the ID mean something?

It is unlikely that this error has anything to do with the frontends: usually web page interaction 
goes through: web browser - network - apache httpd - localhost - mhttpd - midas odb.

http error 502 is very generic, does not tell us much about what happened, there may be more 
information in the httpd log files.

the json-rpc request "id" is generated by midas code in the web browser and it currently is a 
timestamp. it is not used for anything. but it is required by the json-rpc standard.

K.O.


K.O.

Hi all, 

I have updated our daq computer to the latest OS 10.15 with the idea that then I will get all our daq 
programs running and will not need to update again in years to come. 
But now I am running unto even more problems trying to compile Midas. 
OS 10.15.3, Xcode 11.3 with commad line tools 

Darrens-Mac-mini:~ betacage$ cd packages/midas/build/
Darrens-Mac-mini:build betacage$ cmake .. 
-- MIDAS: cmake version: 3.16.3
-- The C compiler identification is AppleClang 11.0.0.11000033
-- The CXX compiler identification is AppleClang 11.0.0.11000033
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
-- MIDAS: Found ROOT version 6.18/04
-- Found ZLIB: /usr/lib/libz.dylib  
-- MIDAS: Found ZLIB version 
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: MySQL not found
-- MIDAS: ODBC not found
-- MIDAS: SQLITE not found
-- MIDAS: nvidia-smi not found
-- Found Git: /usr/bin/git (found version "2.21.0 (Apple Git-122.2)") 
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 
-- MIDAS example/experiment: Found ROOT version 6.18/04
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/betacage/packages/midas/build
Darrens-Mac-mini:build betacage$ make install
[  1%] Building CXX object CMakeFiles/mfeo.dir/src/mfe.cxx.o
In file included from /Users/betacage/packages/midas/src/mfe.cxx:15:
In file included from /Users/betacage/packages/midas/include/mfe.h:13:
/Users/betacage/packages/midas/include/midas.h:159:10: fatal error: 'pthread.h' file not found
#include <pthread.h>
         ^~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/mfeo.dir/src/mfe.cxx.o] Error 1
make[1]: *** [CMakeFiles/mfeo.dir/all] Error 2


I guess that I am maybe the first one trying to install MIDAS in this OS, so I am willing to help as much as I 
can with getting this going. I have found a solution to this by downgrading Xcode to version 10 
https://stackoverflow.com/questions/58524715/failing-to-compile-a-c-application-under-macos-catalina-10-15
If you don't have other solutions I will try to do that tomorrow.

Thanks 

Berta 

> I have updated our daq computer to the latest OS 10.15 ...

FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?

> In file included from /Users/betacage/packages/midas/include/mfe.h:13:
> /Users/betacage/packages/midas/include/midas.h:159:10: fatal error: 'pthread.h' file not found
> #include <pthread.h>
>          ^~~~~~~~~~~

Hmm... pthread.h should be there, I do not see any notice of it's removal in macos.

So I suspect a mis-installation of c++ compilers. In the past we had problems with this,
in addition to installing Xcode from the app store, one most install some missing stuff
manually ("XCode command line tools"), if I have it right, the command is "xcode-select --install".

If you figure out how to build it, I think midas will work just fine on the latest macos.

K.O.

> > I have updated our daq computer to the latest OS 10.15 ...
> 
> FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?

No, I'm stuck with 10.14 due to the current lack of 64-bit support of some programs.

I would try KOs proposal to do a "xcode-select --install".

Stefan

> > > I have updated our daq computer to the latest OS 10.15 ...
> > 
> > FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?
> 
> No, I'm stuck with 10.14 due to the current lack of 64-bit support of some programs.
> 

Ok, in this case, I will update my office mac mini to 10.15.

K.O.

> 
> Ok, in this case, I will update my office mac mini to 10.15.
> 
> K.O.


Hi Konstantin, 

Any luck with Midas in OS 10.15? 
I have downgraded to Xcode 10.2.1 as suggested in the post linked in my first post, without any luck, I keep getting the same pthead.h error. 
MY gcc uses the 10.15 sdk despite having v 10.2.1 for Xcode and command line tools 

Darrens-Mac-mini:~ betacage$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-
dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin19.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin


and that is where I suspect the problem is. Will keep investigating next week. Thanks for your help.

Berta 

> Any luck with Midas in OS 10.15? 

Best I can tell, the problem is not in midas: pthread.h should be there, somewhere.

K.O.

> > Any luck with Midas in OS 10.15? 
> 
> Best I can tell, the problem is not in midas: pthread.h should be there, somewhere.
> 
> K.O.


HI Konstain, 

Thanks for your reply. I have been investigating this issue of pthread in OS 10.15.  Found this forum post 
https://github.com/wjakob/nori/issues/16

that seem to suggest that pthread.h is in a different directory from OS 10.14 and on. And the end of the post the 
developer suggests that he has found a way to fix the Cmake to work so I have checked his code updates in here 
https://github.com/wjakob/nori/commit/be46cccc01a75b21dad1a3f61baa108fe644fc4b

I have added his lines 


if(APPLE)
   # Try to auto-detect a suitable SDK
   execute_process(COMMAND bash -c "xcodebuild -version -sdk | grep MacOSX | grep Path | head -n 1 | cut -f 2 -
d ' '" OUTPUT_VARIABLE CMAKE_OSX_SYSROOT)
   string(REGEX REPLACE "(\r?\n)+$" "" CMAKE_OSX_SYSROOT "${CMAKE_OSX_SYSROOT}")
   string(REGEX REPLACE "^.*X([0-9.]*).sdk$" "\\1" CMAKE_OSX_DEPLOYMENT_TARGET 
"${CMAKE_OSX_SYSROOT}")
 endif()

to my local pakages/midas/CMakeLists.txt and run cmake .., make install again but now I get a different error 

[  1%] Building CXX object CMakeFiles/mfeo.dir/src/mfe.cxx.o
clang: error: invalid version number in '-mmacosx-version-
min=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/DriverKit19.0.sdk'
clang: warning: using sysroot for 'DriverKit' but targeting 'MacOSX' [-Wincompatible-sysroot]


I will continue investigating this issue tomorrow, but please let me know if you have any suggestions.

Also my version of cmake is 3.16.3

Thanks 

Berta 

For your reference, here on my MacOSX 10.14.6 with XCode 11.3.1 the pthread.h file is present in locations listed below.

Did you execute "xcode-select --install" ?


~$ locate pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h

> For your reference, here on my MacOSX 10.14.6 with XCode 11.3.1 the pthread.h file is present in locations listed below.
> 
> Did you execute "xcode-select --install" ?
> 
> 
> ~$ locate pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread.h
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h

Hi Stefan,

Thanks for looking into this. Yes, I have done  "xcode-select --install"

This is my output when I try to locate pthread.h

Darrens-Mac-mini:~ betacage$ locate pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/pthread/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/pthread/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/pthread.h

and this is what I see in my SDKs folder 

Darrens-Mac-mini:SDKs betacage$ pwd
/Library/Developer/CommandLineTools/SDKs
Darrens-Mac-mini:SDKs betacage$ ls
MacOSX.sdk	MacOSX10.14.sdk	MacOSX10.15.sdk

So it looks like in this computer one needs to read the version of the SDK as pthread.h is not in MacOSX.sdk but in MacOSX10.15.sdk

Thanks 

Berta 

Another thought: Can you delete the midas build directory and run cmake again? Like

$ cd midas/build
$ rm -rf *
$ cmake ..
$ make VERBOSE=1

If you have the old build cache and upgraded your OS in meantime, the cache needs to be rebuild. The VERBOSE 
flag tells you the compiler options, and you see where the compiler looks for the SDK directory.

Cheers,
Stefan

> Another thought: Can you delete the midas build directory and run cmake again? Like
> 
> $ cd midas/build
> $ rm -rf *
> $ cmake ..
> $ make VERBOSE=1
> 
> If you have the old build cache and upgraded your OS in meantime, the cache needs to be rebuild. The VERBOSE 
> flag tells you the compiler options, and you see where the compiler looks for the SDK directory.
> 
> Cheers,
> Stefan

Hi Stefan, 

Thanks again for your reply. Yes! the suggestion of rebuilding the cache was the right one, this is the output of cmake ..

Darrens-Mac-mini:~ betacage$ cd packages/midas/build/
Darrens-Mac-mini:build betacage$ rm -rf *
Darrens-Mac-mini:build betacage$ cmake ..
-- MIDAS: cmake version: 3.16.3
-- The C compiler identification is AppleClang 11.0.0.11000033
-- The CXX compiler identification is AppleClang 11.0.0.11000033
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
-- MIDAS: Found ROOT version 6.18/04
-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.11") 
-- MIDAS: Found ZLIB version 1.2.11
-- Found OpenSSL: /usr/lib/libcrypto.dylib (found version "1.1.1d")  
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: MySQL not found
-- MIDAS: ODBC not found
-- MIDAS: SQLITE not found
-- MIDAS: nvidia-smi not found
-- Setting default build type to "RelWithDebInfo"
-- Found Git: /usr/bin/git (found version "2.21.1 (Apple Git-122.3)") 
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS example/experiment: Found ROOT version 6.18/04
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/betacage/packages/midas/build


Unfortunately running make VERBOSE=1 the compilation crashes, with the following error 

[ 39%] Linking CXX executable mhttpd
cd /Users/betacage/packages/midas/build/progs && /Applications/CMake.app/Contents/bin/cmake -E cmake_link_script CMakeFiles/mhttpd.dir/link.txt 
--verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -O2 -g -DNDEBUG -isysroot 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -Wl,-search_paths_first -Wl,-
headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o CMakeFiles/mhttpd.dir/mongoose6.cxx.o CMakeFiles/mhttpd.dir/mgd.cxx.o 
CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o mhttpd  ../libmidas.a /usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz 
ld: cannot link directly with dylib/framework, your binary is not an allowed client of /usr/lib/libcrypto.dylib for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [progs/mhttpd] Error 1
make[1]: *** [progs/CMakeFiles/mhttpd.dir/all] Error 2
make: *** [all] Error 2


Libcrypto seems to be part of the OpenSSL, so I am back to the original problem again. I did install OpenSSL via MacPorts. 

I have found this tread regarding this problem. But have to go and pick up my kids from school, so I don't have time to investigate today.
https://stackoverflow.com/questions/58446253/xcode-11-ld-error-your-binary-is-not-an-allowed-client-of-usr-lib-libcrypto-dy

Thanks again for staying with me and for responding so promptly. 

Berta 

Now you are stuck with openssl, which is optional for mhttpd. If you only use mhttpd locally, you maybe don't need SSL support. In that case you can jus do

[midas/build] $ cmake -D NO_SSL=1 ..

To disable that. If you do need SSL, maybe you can try to install openssl11 via mac ports.

Stefan

> Now you are stuck with openssl, which is optional for mhttpd. If you only use mhttpd locally, you maybe don't need SSL support. In that case you can jus do
> 
> [midas/build] $ cmake -D NO_SSL=1 ..
> 
> To disable that. If you do need SSL, maybe you can try to install openssl11 via mac ports.
> 
> Stefan

Thanks Stefan. 
If I run the compilation with the flag NO_SSL it works just fine. Still I think that Mhttpd is going to be important for us in the future as we may want to control the experiment remotely, so I will keep trying. But at least I 
can get started.

Thanks 

Berta 

> > [midas/build] $ cmake -D NO_SSL=1 ..
> If I run the compilation with the flag NO_SSL it works just fine. ...

FYI, the mongoose616 branch now has support for using the mbedtls https library,
this library seems to build easily from sources and removes our dependency
on where/how/which openssl library is installed. I hope to have this included
in the next release of midas.

K.O.

Thanks Konstantin,

I will keep an eye for the next release so that I can update my Midas to include ssl libraries.

Thanks 



> > > [midas/build] $ cmake -D NO_SSL=1 ..
> > If I run the compilation with the flag NO_SSL it works just fine. ...
> 
> FYI, the mongoose616 branch now has support for using the mbedtls https library,
> this library seems to build easily from sources and removes our dependency
> on where/how/which openssl library is installed. I hope to have this included
> in the next release of midas.
> 
> K.O.

Dear all,

we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:

// event 1
dma_buf[0] = 0x00000001; // Trigger and Event ID
dma_buf[1] = 0x00000001; // Serial number
dma_buf[2] = TIME; // time
dma_buf[3] = 18*4-4*4; // event size
dma_buf[4] = 18*4-6*4; // all bank size
dma_buf[5] = 0x11; // flags
// bank 0
dma_buf[6] = 0x46454230; // bank name
dma_buf[7] = 0x6; // bank type TID_DWORD
dma_buf[8] = 0x3*4; // data size
dma_buf[9] = 0xAFFEAFFE; // data
dma_buf[10] = 0xAFFEAFFE; // data
dma_buf[11] = 0xAFFEAFFE; // data
// bank 1
dma_buf[12] = 0x1; // bank name
dma_buf[12] = 0x46454231; // bank name
dma_buf[13] = 0x6; // bank type TID_DWORD
dma_buf[14] = 0x3*4; // data size
dma_buf[15] = 0xAFFEAFFE; // data
dma_buf[16] = 0xAFFEAFFE; // data
dma_buf[17] = 0xAFFEAFFE; // data

// event 2
.....

dma_buf[fpga_pointer] = 0xXXXXXXXX;


And we do something like:

while{true}
   // obtain buffer space
   status = rb_get_wp(rbh, (void **)&pdata, 10);
   fpga_pointer = fpga.read_last_data_add();

   wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
   copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
   rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte

   last_fpga_pointer = fpga_pointer;

Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 

The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.

Remark: dma_buf is volatile so memcpy is not possible here.

Cheers,
Marius

The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:

1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. It's your responsibility to not write more bytes into pdata then 
what you specified as max event size in rb_create()

2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I don't see this behaviour in your code. You would need something 
like

do {
   status = rb_get_wp(rbh, (void **)&pdata, 10);
   if (status == DB_TIMEOUT)
      ss_sleep(10);
   } while (status == DB_TIMEOUT);

Best,
Stefan


> Dear all,
> 
> we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:
> 
> // event 1
> dma_buf[0] = 0x00000001; // Trigger and Event ID
> dma_buf[1] = 0x00000001; // Serial number
> dma_buf[2] = TIME; // time
> dma_buf[3] = 18*4-4*4; // event size
> dma_buf[4] = 18*4-6*4; // all bank size
> dma_buf[5] = 0x11; // flags
> // bank 0
> dma_buf[6] = 0x46454230; // bank name
> dma_buf[7] = 0x6; // bank type TID_DWORD
> dma_buf[8] = 0x3*4; // data size
> dma_buf[9] = 0xAFFEAFFE; // data
> dma_buf[10] = 0xAFFEAFFE; // data
> dma_buf[11] = 0xAFFEAFFE; // data
> // bank 1
> dma_buf[12] = 0x1; // bank name
> dma_buf[12] = 0x46454231; // bank name
> dma_buf[13] = 0x6; // bank type TID_DWORD
> dma_buf[14] = 0x3*4; // data size
> dma_buf[15] = 0xAFFEAFFE; // data
> dma_buf[16] = 0xAFFEAFFE; // data
> dma_buf[17] = 0xAFFEAFFE; // data
> 
> // event 2
> .....
> 
> dma_buf[fpga_pointer] = 0xXXXXXXXX;
> 
> 
> And we do something like:
> 
> while{true}
>    // obtain buffer space
>    status = rb_get_wp(rbh, (void **)&pdata, 10);
>    fpga_pointer = fpga.read_last_data_add();
> 
>    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
>    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
>    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> 
>    last_fpga_pointer = fpga_pointer;
> 
> Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> 
> The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> 
> Remark: dma_buf is volatile so memcpy is not possible here.
> 
> Cheers,
> Marius

Hello Stefan,
is there a difference for the later data processing (after writing the ring buffer blocks)
if we write single events or multiple in one rb_get_wp - memcopy - rb_increment_wp cycle?
Both Marius and me have seen some inconsistencies in the number of events produced that is reported in the status page when writing multiple events in one go,
so I was wondering if this is due to us treating the buffer badly or the way midas handles the events after that.

Given that we produce the full event in our (FPGA) domain, an option would be to always copy one event from the dma to the midas-system buffer in a loop.
The question is if there is a difference (for midas) between
[pseudo code, much simplified]

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
    dma_read_index+=event_size;
    continue;   
  }
  copy_n(dma_buffer, pdata, event_size);
  rb_increment_wp(event_size);
  dma_read_index+=event_size;
} 

and

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
     ...
  };
  total_size=max_n_events_that_fit_in_rb_block();
  copy_n(dma_buffer, pdata, total_size);
  rb_increment_wp(total_size);
  dma_read_index+=total_size;
}

Cheers,
Konrad

> The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:
> 
> 1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. It's your responsibility to not write more bytes into pdata then 
> what you specified as max event size in rb_create()
> 
> 2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
> function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I don't see this behaviour in your code. You would need something 
> like
> 
> do {
>    status = rb_get_wp(rbh, (void **)&pdata, 10);
>    if (status == DB_TIMEOUT)
>       ss_sleep(10);
>    } while (status == DB_TIMEOUT);
> 
> Best,
> Stefan
> 
> 
> > Dear all,
> > 
> > we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:
> > 
> > // event 1
> > dma_buf[0] = 0x00000001; // Trigger and Event ID
> > dma_buf[1] = 0x00000001; // Serial number
> > dma_buf[2] = TIME; // time
> > dma_buf[3] = 18*4-4*4; // event size
> > dma_buf[4] = 18*4-6*4; // all bank size
> > dma_buf[5] = 0x11; // flags
> > // bank 0
> > dma_buf[6] = 0x46454230; // bank name
> > dma_buf[7] = 0x6; // bank type TID_DWORD
> > dma_buf[8] = 0x3*4; // data size
> > dma_buf[9] = 0xAFFEAFFE; // data
> > dma_buf[10] = 0xAFFEAFFE; // data
> > dma_buf[11] = 0xAFFEAFFE; // data
> > // bank 1
> > dma_buf[12] = 0x1; // bank name
> > dma_buf[12] = 0x46454231; // bank name
> > dma_buf[13] = 0x6; // bank type TID_DWORD
> > dma_buf[14] = 0x3*4; // data size
> > dma_buf[15] = 0xAFFEAFFE; // data
> > dma_buf[16] = 0xAFFEAFFE; // data
> > dma_buf[17] = 0xAFFEAFFE; // data
> > 
> > // event 2
> > .....
> > 
> > dma_buf[fpga_pointer] = 0xXXXXXXXX;
> > 
> > 
> > And we do something like:
> > 
> > while{true}
> >    // obtain buffer space
> >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> >    fpga_pointer = fpga.read_last_data_add();
> > 
> >    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
> >    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
> >    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> > 
> >    last_fpga_pointer = fpga_pointer;
> > 
> > Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> > 
> > The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> > 
> > Remark: dma_buf is volatile so memcpy is not possible here.
> > 
> > Cheers,
> > Marius

rb_xxx functions are midas event agnostic. The receiving side in mfe.cxx (lines 1418 in receive_trigger_event) however pulls one event at a time. If you 
have some inconsistency I would put some debugging code there.

Stefan

> Hello Stefan,
> is there a difference for the later data processing (after writing the ring buffer blocks)
> if we write single events or multiple in one rb_get_wp - memcopy - rb_increment_wp cycle?
> Both Marius and me have seen some inconsistencies in the number of events produced that is reported in the status page when writing multiple 
events in one go,
> so I was wondering if this is due to us treating the buffer badly or the way midas handles the events after that.
> 
> Given that we produce the full event in our (FPGA) domain, an option would be to always copy one event from the dma to the midas-system buffer 
in a loop.
> The question is if there is a difference (for midas) between
> [pseudo code, much simplified]
> 
> while(dma_read_index < last_dma_write_index){
>   if(rb_get_wp(pdata)!=SUCCESS){
>     dma_read_index+=event_size;
>     continue;   
>   }
>   copy_n(dma_buffer, pdata, event_size);
>   rb_increment_wp(event_size);
>   dma_read_index+=event_size;
> } 
> 
> and
> 
> while(dma_read_index < last_dma_write_index){
>   if(rb_get_wp(pdata)!=SUCCESS){
>      ...
>   };
>   total_size=max_n_events_that_fit_in_rb_block();
>   copy_n(dma_buffer, pdata, total_size);
>   rb_increment_wp(total_size);
>   dma_read_index+=total_size;
> }
> 
> Cheers,
> Konrad
> 
> > The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:
> > 
> > 1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection 
any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. 
It's your responsibility to not write more bytes into pdata then 
> > what you specified as max event size in rb_create()
> > 
> > 2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the 
producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. 
If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
> > function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. 
Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I 
don't see this behaviour in your code. You would need something 
> > like
> > 
> > do {
> >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> >    if (status == DB_TIMEOUT)
> >       ss_sleep(10);
> >    } while (status == DB_TIMEOUT);
> > 
> > Best,
> > Stefan
> > 
> > 
> > > Dear all,
> > > 
> > > we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA 
sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The 
buffer looks something like:
> > > 
> > > // event 1
> > > dma_buf[0] = 0x00000001; // Trigger and Event ID
> > > dma_buf[1] = 0x00000001; // Serial number
> > > dma_buf[2] = TIME; // time
> > > dma_buf[3] = 18*4-4*4; // event size
> > > dma_buf[4] = 18*4-6*4; // all bank size
> > > dma_buf[5] = 0x11; // flags
> > > // bank 0
> > > dma_buf[6] = 0x46454230; // bank name
> > > dma_buf[7] = 0x6; // bank type TID_DWORD
> > > dma_buf[8] = 0x3*4; // data size
> > > dma_buf[9] = 0xAFFEAFFE; // data
> > > dma_buf[10] = 0xAFFEAFFE; // data
> > > dma_buf[11] = 0xAFFEAFFE; // data
> > > // bank 1
> > > dma_buf[12] = 0x1; // bank name
> > > dma_buf[12] = 0x46454231; // bank name
> > > dma_buf[13] = 0x6; // bank type TID_DWORD
> > > dma_buf[14] = 0x3*4; // data size
> > > dma_buf[15] = 0xAFFEAFFE; // data
> > > dma_buf[16] = 0xAFFEAFFE; // data
> > > dma_buf[17] = 0xAFFEAFFE; // data
> > > 
> > > // event 2
> > > .....
> > > 
> > > dma_buf[fpga_pointer] = 0xXXXXXXXX;
> > > 
> > > 
> > > And we do something like:
> > > 
> > > while{true}
> > >    // obtain buffer space
> > >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> > >    fpga_pointer = fpga.read_last_data_add();
> > > 
> > >    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
> > >    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
> > >    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> > > 
> > >    last_fpga_pointer = fpga_pointer;
> > > 
> > > Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also 
increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> > > 
> > > The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. 
So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> > > 
> > > Remark: dma_buf is volatile so memcpy is not possible here.
> > > 
> > > Cheers,
> > > Marius

> rb_xxx functions are midas event agnostic. The receiving side in mfe.cxx (lines 1418 in receive_trigger_event) however pulls one event at a time. If you 
> have some inconsistency I would put some debugging code there.

I agree with Stefan, I do not think there is any bugs in the ring buffer code.

But. I do not think we ever did DMA the data directly into the ring buffer. Hmm...

I just checked, this is what we do (and this worked in the ALPHA Si-strip DAQ system for 10 years now):

- mfe.cxx multithread equipment
- mfe readout thread grabs pointer from ring buffer
- mfe creates event headers, etc
- calls our read_event() function
- creates data bank
- DMA data into the data bank (this is the DMA from VME block reads, using DMA controller inside the UniverseII and tsi148 VME-to-PCI bridges)
- close data bank
- return to mfe
- mfe readout thread increments the ring buffer
- mfe main thread grabs events from ring buffer, sends them to the mserver

So there could be trouble:
a) the ring buffer code does not have the required "volatile" (ahem, "atomic") annotations, so DMA may have a bad interaction with compiler optimizations (values stored in registers 
instead of in memory, etc)
b) the DMA driver must doctor the memory settings to (1) mark the DMA target memory uncachable or (1b) invalidate the cache after DMA completes, (2) mark the DMA target 
memory unswappable.

So I see possibilities for the ring buffer to malfunction.

But now I am curious, which DMA controller you use? The Altera or Xilinx PCIe block with the vendor supplied DMA driver? Or you do DMA on an ARM SoC FPGA? (no PCI/PCIe, 
different DMA controller, different DMA driver).

I am curious because we will be implementing pretty much what you do on ARM SoC FPGAs pretty soon, so good to know
if there is trouble to expect.

But I will probably use the tmfe.h c++ frontend and a "pure c++" ring buffer instead of mfe.cxx and the midas "rb" ring buffer.

(I did not look at your code at all, there could be a bug right there, this ring buffer stuff is tricky. With luck there is no bug
in your dma driver. The dma drivers for our vme bridges did do have bugs).

K.O.

We also agree and found the problem now. Since we build everything (MIDAS Event Header, Bank Header, Banks etc.) in the FPGA we had some struggle with the MIDAS data format (http://lmu.web.psi.ch/docu/manuals/bulk_manuals/software/midas195/html/AppendixA.html). We thought that only the MIDAS Event needs to be aligned to 64 bit but as it turned out also the bank data (Stefan updated the wiki page already) needs to be aligned. Since we are using the BANK32 it was a bit unclear for us since the bank header is not 64 bit aligned. But we managed this now by adding empty data and the system is running now.

Our setup looks like this:

Software:
- mfe.cxx multithread equipment
- mfe readout thread grabs pointer from dma ring buffer 
- since the dma buffer is volatile we do copy_n for transforming the data to MIDAS 
- the data is already in the MIDAS format so done from our side :)
- mfe readout thread increments the ring buffer
- mfe main thread grabs events from ring buffer, sends them to the mserver

Firmware:
- Arria 10 development board
- Altera PCIe block
- Own DMA engine since we are doing burst writing DMA with PCIe 3.0. 
- Own device driver
- no interrupts 

If you have more questions fell free to ask.

Actually the cause of all of the is a real bug in the midas functions. We want each bank 8-byte aligned, so there is code in bk_close like:

midas.cxx:14788:
     ((BANK_HEADER *) event)->data_size += sizeof(BANK32) + ALIGN8(pbk32->data_size);

While the old sizeof(BANK)=8, the extended sizeof(BANK32)=12, so not 8-byte aligned. This code should rather be:

     ((BANK_HEADER *) event)->data_size += ALIGN8(sizeof(BANK32) +pbk32->data_size);

But if we change that, it would break every midas data file on this planet!

The only chance I see is to use the "flags" in the BANK_HEADER to distinguish a current bank from a "correct" bank. 
So we could introduce a flag BANK_FORMAT_ALIGNED which distinguishes between the two pieces of code above. 
Then bk_iterate32 would look at that flag and do the right thing.

Any thoughts?

Best,
Stefan

Hi, Stefan - is this our famous 64-bit misalignement? Where we have each alternating bank aligned and misaligned at 64 bits? Without changing the data
format, one can always store data in 64-bit aligned banks by inserting a dummy banks between real banks:

event header
bank header
bank1 --- 64-bit aligned --- with data
bank2 --- misaligned, no data
bank3 --- 64-bit aligned --- with data
bank4 --- misaligned, no data
...

for sure, wastes space for bank2, bank4, etc, but at 12 bytes per bank, maybe this is negligible overhead compared to total event size.

BTW, aligned-to-64-bit is old news. The the PWB FPGA, I have 128-bit data paths to DDR RAM, the data has to be aligned to 128 bits, or else!

K.O.



> Actually the cause of all of the is a real bug in the midas functions. We want each bank 8-byte aligned, so there is code in bk_close like:
> 
> midas.cxx:14788:
>      ((BANK_HEADER *) event)->data_size += sizeof(BANK32) + ALIGN8(pbk32->data_size);
> 
> While the old sizeof(BANK)=8, the extended sizeof(BANK32)=12, so not 8-byte aligned. This code should rather be:
> 
>      ((BANK_HEADER *) event)->data_size += ALIGN8(sizeof(BANK32) +pbk32->data_size);
> 
> But if we change that, it would break every midas data file on this planet!
> 
> The only chance I see is to use the "flags" in the BANK_HEADER to distinguish a current bank from a "correct" bank. 
> So we could introduce a flag BANK_FORMAT_ALIGNED which distinguishes between the two pieces of code above. 
> Then bk_iterate32 would look at that flag and do the right thing.
> 
> Any thoughts?
> 
> Best,
> Stefan

> Hi, Stefan - is this our famous 64-bit misalignement? Where we have each alternating bank aligned and misaligned at 64 bits? Without changing the data
> format, one can always store data in 64-bit aligned banks by inserting a dummy banks between real banks:
> 
> event header
> bank header
> bank1 --- 64-bit aligned --- with data
> bank2 --- misaligned, no data
> bank3 --- 64-bit aligned --- with data
> bank4 --- misaligned, no data
> ...
> 
> for sure, wastes space for bank2, bank4, etc, but at 12 bytes per bank, maybe this is negligible overhead compared to total event size.
> 
> BTW, aligned-to-64-bit is old news. The the PWB FPGA, I have 128-bit data paths to DDR RAM, the data has to be aligned to 128 bits, or else!

Ok, so what about the following: When we do a bk_init32, we add a parameter "alignment", which might be 1,4,8,16 and "old". We store this alignment in the bank header, so the 
decoding works correctly. Now "old" means the current encoding, which is screwed up and produces the results you mention above, but we have to keep it (actually make it the 
default!) for backward compatibility. But then we can ask for 64-bit alignment or even 128-bit alignment if that helps the DAQ speed.

The only problem I see is if one writes data with the new library using 128-bit alignment for example, and wants to read it back with old code. Then it would explode. So if we 
make this modification, we have to announce it carefully and also adjust all ROOTANA & Co libraries to read back any midas data.

Stefan

> We also agree and found the problem now.

Good. what was wrong?

> - Own DMA engine since we are doing burst writing DMA with PCIe 3.0. 
> - Own device driver

Scary stuff.

> - no interrupts 

Right. Best I can tell, interrupts no longer useful in Linux - interrupt handler cannot do any real work, has to hand off to a kernel thread, resulting
in so much latency and overhead that one might as well poll for the data... And for DMA data transfers, the data rate is well known,
so easy to predict how long the DMA will run for and sleep for that amount of time instead of waiting for an interrupt.

K.O.

We also agree and found the problem now. Since we build everything (MIDAS Event Header, Bank Header, Banks etc.) in the FPGA we had some struggle with the MIDAS data format (http://lmu.web.psi.ch/docu/manuals/bulk_manuals/software/midas195/html/AppendixA.html). We thought that only the MIDAS Event needs to be aligned to 64 bit but as it turned out also the bank data (Stefan updated the wiki page already) needs to be aligned. Since we are using the BANK32 it was a bit unclear for us since the bank header is not 64 bit aligned. But we managed this now by adding empty data and the system is running now.

Our setup looks like this:

- mfe.cxx multithread equipment
- mfe readout thread grabs pointer from dma ring buffer 
- since the dma buffer is volatile we do copy_n for transforming the data to MIDAS 
- the data is already in the MIDAS format so done from our side :)
- mfe readout thread increments the ring buffer
- mfe main thread grabs events from ring buffer, sends them to the mserver

From the firmware side we have an Arria 10 development board and 

But now I am curious, which DMA controller you use? The Altera or Xilinx PCIe block with the vendor supplied DMA driver? Or you do DMA on an ARM SoC FPGA? (no PCI/PCIe, 
different DMA controller, different DMA driver).

I am curious because we will be implementing pretty much what you do on ARM SoC FPGAs pretty soon, so good to know
if there is trouble to expect.

But I will probably use the tmfe.h c++ frontend and a "pure c++" ring buffer instead of mfe.cxx and the midas "rb" ring buffer.

(I did not look at your code at all, there could be a bug right there, this ring buffer stuff is tricky. With luck there is no bug
in your dma driver. The dma drivers for our vme bridges did do have bugs).

K.O.

Dear all,

we are trying to build Midas events on FPGAs and send them directly to the midas
ring buffer via copy_n. According to the wiki
https://midas.triumf.ca/MidasWiki/index.php/Event_Structure Event Data Size is:
"The event data size contains the size of the event in bytes excluding the
header." and All Bank Size is: "Size in bytes of the following data plus the
size of the bank header". So are they actually the same or what header is the
header in the first sentence also the bank header?

Cheers,
Marius

 

Thanks for pointing out this error. The "All Bank Size" contains the size of all banks including their 
bank headers, but NOT the global bank header itself. I modified the documentation accordingly.

If you want to study the C code which tells you how to fill these headers, look at midas.cxx line 
14788.

Best,
Stefan

> Dear all,
> 
> we are trying to build Midas events on FPGAs and send them directly to the midas
> ring buffer via copy_n. According to the wiki
> https://midas.triumf.ca/MidasWiki/index.php/Event_Structure Event Data Size is:
> "The event data size contains the size of the event in bytes excluding the
> header." and All Bank Size is: "Size in bytes of the following data plus the
> size of the bank header". So are they actually the same or what header is the
> header in the first sentence also the bank header?
> 
> Cheers,
> Marius
> 
>  

> Thanks for pointing out this error. The "All Bank Size" contains the size of all banks including their 
> bank headers, but NOT the global bank header itself. I modified the documentation accordingly.
> 
> If you want to study the C code which tells you how to fill these headers, look at midas.cxx line 
> 14788.

Also take a look at the midas event parser in ROOTANA midasio.cxx, the code is pretty clean c++
https://bitbucket.org/tmidas/rootana/src/master/libMidasInterface/midasio.cxx

But Stefan's code in midas.cxx and in the documentation is the authoritative information.

K.O.

Hi,
for some reason we occasionally get JSON errors in the browser when accessing MIDAS. It is then not possible to open a new window or tab, see attachment. The unexpected token is \0x6, so the acknowledge symbol.
If this happens, then all "alive sessions" keep being usable despite error messages, but show similar error messages:
>RPC Error
>json parser exception: SyntaxError: JSON.parse: bad control character in string literal at line 80 column 30 of the JSON data, method: "db_get_valus", params: [object Object], id: 1582020074098.

Do you have any idea why db_get_values yields ACK or other control characters?

Thanks

You are the first one reporting this error, so it must be due to your values in the ODB. Can you track it down to specific ODB contents? If so, can you post it so that I can reproduce your error?

Stefan

> The unexpected token is \0x6
> RPC Error json parser exception: SyntaxError: JSON.parse: bad control character in string literal at line 80 column 30 of the JSON data, method: "db_get_valus", params: [object Object], id: 1582020074098.

Yes, there is a problem.

Traditionally, midas strings in ODB have no restriction on the content (I think even the '\0x0' char is permitted).

But web browser javascript strings are supposed to be valid unicode (UTF-16, if I read this right: https://tc39.es/ecma262/#sec-ecmascript-language-types-string-type).

The collision between the two happens when ODB values are json-encoded by midas, then json-decoded by the web browser.

The midas json encoder (mjson.h, mjson.cxx) encodes ODB strings according to JSON rules, but does not ensure that the result is valid UTF-8. (valid UTF-8 is not required, if I read the specs correctly http://www.ecma-
international.org/publications/files/ECMA-ST/ECMA-404.pdf and https://www.json.org/json-en.html)

The web browser json decoder requires valid UTF-8 and throws exceptions if it does not like something. Different browsers it slightly differently, so we have an error handler for this in the mjsonrpc results processor.

What does this mean in practice?

Now that MIDAS is very web oriented, MIDAS strings must be web browser friendly, too:

a) all ODB key names (subdirectory names, link names, etc) must be UTF-8 unicode, and this has been enforced by ODB for some time now.
b) all ODB string values must be valid UTF-8 unicode. This is not enforced right now.

Historically, it was okey to use ODB TID_STRING to store arbitrary binary data, but now, I think, we must deprecate this,
at least for any ODB entries that could be returned to a web browser (which means all of them, after we implement a fully
html+javascript odb editor). For storing binary data, arrays of TID_CHAR, TID_DWORD & co are probably a better match.

The MIDAS and ROOTANA json decoders (the same mjson.h, mjson.cxx) do not care about UTF-8, so ODB dumps
in JSON format are not affected by any of this. (But I am not sure about the JSON decoder in ROOT).

Bottom line:

I think db_validate() should check for invalid UTF-8 in ODB key names and in TID_STRING values
and at least warn the user. (I am not sure if invalid UTF-8 can be fixed automatically). db_create()
should reject key names that are not valid UTF-8 (it already does this, I think). db_set_value(TID_STRING) should
probably reject invalid UTF-8 strings, this needs to be discussed some more.

https://bitbucket.org/tmidas/midas/issues/215/everything-in-odb-must-be-valid-utf-8

K.O.

INT idle (EQUIPMENT * pequipment)

Hi, Giorgio - I think you encountered a fundamental problem with what to do at the begin of 
run. There are two ways of thinking about it.

Some experiments want to start the run as quickly as possible, so they do not want 
begin_of_run() to do too much stuff.

Other experiments want to record all the current settings and conditions before starting a 
run, their begin_of_run() will read all the slow controls, interrogate all the power supplies, 
read all the voltages, temperatures, pressures, etc. By necessity this will slow down the 
starting of the run quite significantly.

The best I understand the midas class driver structure, it is more geared for the first case - 
fast starting of runs.

The thinking behind this choice considers the nature of most slow control data in typical 
physics experiments:
- if the data does not change quickly (say, room temperature, atmospheric pressure, etc), 
and you read it say every 1 minute, then you do not need to read it again at begin run time - 
the 1 minute old measurement is still good enough - nothing changed much since then
- if the opposite is true, the data changes wildly (i.e. detector high voltage current goes up 
and down in response to the quickly changing beam current), measuring it at the start of 
the run does us no good - by the time the first event comes around, it has already changed 
completely.

Hopefully Stefan can help you with your specific problem, he has better understanding of 
the midas class drivers.


K.O.


[quote="Pintaudi Giorgio"]Hello!
As you know, the generic MIDAS frontend has a class driver, device driver, bus driver
structure. Assuming a slow device frontend, its class driver should have a routine of type   
[CODE]INT idle (EQUIPMENT * pequipment)[/CODE]
This routine is called with a rate controlled by the 
"[I]/Equipment/<frontend name>/Common/Event limit[/I]" parameter.
The idle routine usually reads one channel of the frontend and stores the results
in the "[I]/Equipment/<frontend name>/Variables[/I]" ODB folder.

[B]My question is: it is possible to force (from the code) the frontend to call the idle routine 
at a 
certain point. This is because I need to update the "[I]/Equipment/<frontend 
name>/Variables[/I]"
variables inside the "[I]begin_of_run[/I]" routine, at a very specific time.[/B]

One dirty solution would be to increase a lot the reading rate ... but I need this
increased reading rate only during the run start while I need a low reading rate
during the run. So the question: is it possible to increase and decrease the reading
rate (event limit) of a frontend without stopping and restarting it?

If you need more info, please let me know.
Thank you
Giorgio[/quote]

@@ -411,6 +411,17 @@ static INT register_equipment(void)
             ss_sleep(3000);
             return 0;
          }
+#ifdef WAGASCI_OPEN_ODB_HOTLINK
+         status = db_open_record(hDB, hKey, eq_info, sizeof(EQUIPMENT), MODE_READ,
+                                 nullptr, nullptr);
+         if (status != DB_SUCCESS) {
+            printf("ERROR: Cannot open hotlink with equipment record \"%s\", db_open_record() status %d\n",
+                   str, status);
+            cm_disconnect_experiment();
+            ss_sleep(3000);
+            return 0;
+         }
+#endif
       } else if (status == DB_STRUCT_MISMATCH) {
          cm_msg(MINFO, "register_equipment", "Correcting \"%s\", db_check_record() status %d", str, status);
          db_create_record(hDB, 0, str, EQUIPMENT_COMMON_STR);

• Every change to the equipment "/Equipment/<frontend name>/Common" is instantaneously applied (and might crash the frontend?)
  

INT idle (EQUIPMENT * pequipment)

> We had a case where a detector produced electrostatic discharges which only lasted for a second or so
> and we were happy to detect this in spikes in the HV current. With measurements of only one per minute
> we would not have realized that so quicky.

For the T2K/ND280 TPC we implemented something similar. The TPC uses MicroMegas detector which sparks during 
normal operation. We asked Wiener/ISEG to implement a "spark counting mode" for us (and they did). In this mode,
high voltage over-current (a micromegas spark) sets a special flag (does not trip the high voltage). Our midas frontend
reads this flag at rate about 1/min, if flag is set, clears it, increments the software spark counter, reads the flag again,
if the flag is still set (failed to clear), it means this was not a normal spark but a high voltage breakdown
and the offending channel is shut down. I believe this mode is still part of the ISEG normal firmware.

Because the Wiener/ISEG interface uses SNMP to "read all data in one operation", the MIDAS "device driver" structure
was not useful, the readout was a simple loop, the readout frequency was easy to control, and indeed,
we read the high voltage with increased frequency during ramping. This was easy to implement because we
did not have to fight the MIDAS "device driver" framework.

If you want a similar solution, talk to the device, interpret the data, record values to odb and history, generate
midas events - all without hand holding from (arm wrestling with the rest of) midas - I recommend
the new tmfe.h/tmfe.cxx c++ frontend - see the two examples in midas/progs/fetest_tmfe.cxx
and fetest_tmfe_thread.cxx (single-threaded and multi-threaded).

K.O.

Hi, Stefan & co - now that midas is c++11 and c++11 comes with a threads library, should we 
switch midas to use the c++11 threads instead of pthreads? (Of course on Linux c++11 
threads are a layer on top of pthreads, the best I know).

This should remove the dependency on pthreads.h and use a more native implementation of 
threads on MacOS and Windows. (again, the best I can tell).

Of course this depends on c++11 threads having all the functions we need. Specifically, "lock 
with timeout" is useful to deal with "gah! everything stopped! what do I do!", a problem 
bedeviling midas in the early days (and still happens today!). Current midas kills everything 
after 5 minutes of deadlock - then the user knows how to restart everything and the developer 
has core dumps to look at. (to see which program/thread holds the lock and would not give it 
up).

Any thoughts on this?

K.O.

I'm thinking of this already since some time, and it was part of my motivation switching to C++11.
I was delighted to see that what we do in system.c (encapsulate system functions such as threads
and shared memory) is now done natively in C++11, and it's done by experts and not amateurs like us.

I see that with std::timed_mutex and try_lock_for we have the desired timeout function. Pity that
C++11 does not contain inter-process communication like semaphores, so there we still have to use
our old functions. 

But for threads switching to std::thread, I'm all in.

Stefan

> Hi, Stefan & co - now that midas is c++11 and c++11 comes with a threads library, should we 
> switch midas to use the c++11 threads instead of pthreads? (Of course on Linux c++11 
> threads are a layer on top of pthreads, the best I know).
> 
> This should remove the dependency on pthreads.h and use a more native implementation of 
> threads on MacOS and Windows. (again, the best I can tell).
> 
> Of course this depends on c++11 threads having all the functions we need. Specifically, "lock 
> with timeout" is useful to deal with "gah! everything stopped! what do I do!", a problem 
> bedeviling midas in the early days (and still happens today!). Current midas kills everything 
> after 5 minutes of deadlock - then the user knows how to restart everything and the developer 
> has core dumps to look at. (to see which program/thread holds the lock and would not give it 
> up).
> 
> Any thoughts on this?
> 
> K.O.

> Hi, Stefan & co - now that midas is c++11 and c++11 comes with a threads library, should we 
> switch midas to use the c++11 threads instead of pthreads? (Of course on Linux c++11 
> threads are a layer on top of pthreads, the best I know).
> 
> This should remove the dependency on pthreads.h and use a more native implementation of 
> threads on MacOS and Windows. (again, the best I can tell).
> 
> Of course this depends on c++11 threads having all the functions we need. Specifically, "lock 
> with timeout" is useful to deal with "gah! everything stopped! what do I do!", a problem 
> bedeviling midas in the early days (and still happens today!). Current midas kills everything 
> after 5 minutes of deadlock - then the user knows how to restart everything and the developer 
> has core dumps to look at. (to see which program/thread holds the lock and would not give it 
> up).
> 
> Any thoughts on this?
> 
> K.O.

Hi, I just wanted to say that I have seen this post and maybe that is the solution to the pthreads compiler problem in OS 10.15, but 
of course I am a total amateur in here. Thanks for thinking about this and I will wait and hold to see what gets decided. Thanks

Berta  

Hi all, 

I have a Mac with OS 10.13.6 and Xcode 10.1. I am following the instructions in the wiki to install Midas. 
I have installed openssl via MacPorts as per the instructions. But then I get an error related to open ssl 
when I try to compile Midas. See the results of cmake .. and make install 

Darrens-Mac-mini:build betacage$ cmake ..
-- MIDAS: cmake version: 3.16.3
-- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
-- MIDAS: Found ROOT version 6.18/04
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: MySQL not found
-- MIDAS: ODBC not found
-- MIDAS: Found SQLITE /usr/include/sqlite3.h
-- MIDAS: nvidia-smi not found
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS example/experiment: Found ROOT version 6.18/04
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/betacage/packages/midas/build
Darrens-Mac-mini:build betacage$ make install 
[  1%] Built target mfeo
[  3%] Built target mfe
[  4%] Built target rmana
[  5%] Built target manao
[  6%] Built target mana
[  6%] Built target git_revision_h
[ 30%] Built target objlib
[ 31%] Built target midas
[ 32%] Built target rmanao
[ 32%] Built target objlib-c-compat
[ 32%] Built target midas-c-compat
[ 33%] Built target midas-shared
[ 35%] Built target rmlogger
[ 36%] Linking CXX executable mhttpd
Undefined symbols for architecture x86_64:
  "_OPENSSL_init_ssl", referenced from:
      _mg_mgr_init in mongoose6.cxx.o
  "_SSL_CTX_set_options", referenced from:
      _mg_set_ssl in mongoose6.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [progs/mhttpd] Error 1
make[1]: *** [progs/CMakeFiles/mhttpd.dir/all] Error 2
make: *** [all] Error 2


I hope that this is the right forum to submit this kind of reports.
Any idea what do I have to do to continue ? Thanks is advance !

Berta 

Hi, yes, this is the right place to report problems and to ask questions about midas.

As for your trouble, I have the same macos 10.13.6, so we should be able to figure out what goes wrong.

Unfortunately, the posted instructions (to run cmake directly) hide all the information needed to debug build problems.

Instead of "cd build; cmake ..", please run "make cmake" and post the output. Printed should be all the compiler
and linker commands executed by the build. You can examine the mhttpd linker command to see if somehow
the wrong openssl library is being used.

If you do not need built-in https support in mhttpd, you can run "make cmake NO_SSL=1".

K.O.

> Hi all, 
> 
> I have a Mac with OS 10.13.6 and Xcode 10.1. I am following the instructions in the wiki to install Midas. 
> I have installed openssl via MacPorts as per the instructions. But then I get an error related to open ssl 
> when I try to compile Midas. See the results of cmake .. and make install 
> 
> Darrens-Mac-mini:build betacage$ cmake ..
> -- MIDAS: cmake version: 3.16.3
> -- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
> -- MIDAS: Found ROOT version 6.18/04
> -- MIDAS: Found ZLIB version 1.2.11
> -- MIDAS: Found OpenSSL version 1.1.1d
> -- MIDAS: MySQL not found
> -- MIDAS: ODBC not found
> -- MIDAS: Found SQLITE /usr/include/sqlite3.h
> -- MIDAS: nvidia-smi not found
> -- MIDAS example/experiment: MIDAS in-tree-build
> -- MIDAS: Found ZLIB version 1.2.11
> -- MIDAS example/experiment: Found ROOT version 6.18/04
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /Users/betacage/packages/midas/build
> Darrens-Mac-mini:build betacage$ make install 
> [  1%] Built target mfeo
> [  3%] Built target mfe
> [  4%] Built target rmana
> [  5%] Built target manao
> [  6%] Built target mana
> [  6%] Built target git_revision_h
> [ 30%] Built target objlib
> [ 31%] Built target midas
> [ 32%] Built target rmanao
> [ 32%] Built target objlib-c-compat
> [ 32%] Built target midas-c-compat
> [ 33%] Built target midas-shared
> [ 35%] Built target rmlogger
> [ 36%] Linking CXX executable mhttpd
> Undefined symbols for architecture x86_64:
>   "_OPENSSL_init_ssl", referenced from:
>       _mg_mgr_init in mongoose6.cxx.o
>   "_SSL_CTX_set_options", referenced from:
>       _mg_set_ssl in mongoose6.cxx.o
> ld: symbol(s) not found for architecture x86_64
> clang: error: linker command failed with exit code 1 (use -v to see invocation)
> make[2]: *** [progs/mhttpd] Error 1
> make[1]: *** [progs/CMakeFiles/mhttpd.dir/all] Error 2
> make: *** [all] Error 2
> 
> 
> I hope that this is the right forum to submit this kind of reports.
> Any idea what do I have to do to continue ? Thanks is advance !
> 
> Berta 

I tried on my Mac (macOS 10.14.6, Xcode 11.3.1, current develop branch, openssl 1.1.1d) and it woks fine. Below is the transcript. I 
see that your cmake output is much shorter (no C compiler listed etc.). Did you remove some lines? For such comparisons, it's 
always good to start with an empty build directory.

OpenSSL libraries are the same (1.1.1d). Just for comparison, I run the build process with "make VERBOSE=1" and extract the line 
which fails for you when linking mhttp, so maybe you can compare.

Linking mhttpd
---------------

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -O2 -g -DNDEBUG -isysroot 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-
min=10.14 -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o 
CMakeFiles/mhttpd.dir/mongoose6.cxx.o CMakeFiles/mhttpd.dir/mgd.cxx.o CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o 
mhttpd  -L/opt/local/lib/mysql57/mysql -lmysqlclient -lz ../libmidas.a /opt/local/lib/libssl.dylib /opt/local/lib/libcrypto.dylib -lz -
L/opt/local/lib/mysql57/mysql -lmysqlclient -lz 


Here is the full transcript
-------------------------

/midas/build$ cmake ..
-- MIDAS: cmake version: 3.16.3
-- The C compiler identification is AppleClang 11.0.0.11000033
-- The CXX compiler identification is AppleClang 11.0.0.11000033
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- 
works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- MIDAS: CMAKE_INSTALL_PREFIX: /midas
-- MIDAS: Found ROOT version 6.16/00
-- Found ZLIB: /opt/local/lib/libz.dylib (found version "1.2.11") 
-- MIDAS: Found ZLIB version 1.2.11
-- Found OpenSSL: /opt/local/lib/libcrypto.dylib (found version "1.1.1d")  
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: Found MySQL version 5.7.26
-- MIDAS: ODBC not found
-- MIDAS: SQLITE not found
-- MIDAS: nvidia-smi not found
-- Setting default build type to "RelWithDebInfo"
-- Found Git: /usr/bin/git (found version "2.21.1 (Apple Git-122.3)") 
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS example/experiment: Found ROOT version 6.16/00
-- Configuring done
-- Generating done
-- Build files have been written to: /midas/build
/midas/build$ make
Scanning dependencies of target rmana
Scanning dependencies of target git_revision_h
Scanning dependencies of target mfe
Scanning dependencies of target manao
Scanning dependencies of target mfeo
Scanning dependencies of target objlib-c-compat
Scanning dependencies of target mana
Scanning dependencies of target rmanao
[  0%] Building CXX object CMakeFiles/objlib-c-compat.dir/src/midas_c_compat.cxx.o
[  2%] Building CXX object CMakeFiles/mfe.dir/src/mfe.cxx.o
[  2%] Building CXX object CMakeFiles/mfeo.dir/src/mfe.cxx.o
[  3%] Building CXX object CMakeFiles/mana.dir/src/mana.cxx.o
[  4%] Building CXX object CMakeFiles/manao.dir/src/mana.cxx.o
[  5%] Building CXX object CMakeFiles/rmanao.dir/src/mana.cxx.o
[  6%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
[  6%] Built target git_revision_h
Scanning dependencies of target objlib
[  7%] Building CXX object CMakeFiles/objlib.dir/src/midas.cxx.o
[  7%] Built target objlib-c-compat
[  8%] Building CXX object CMakeFiles/objlib.dir/src/midas_cxx.cxx.o
[  8%] Linking CXX static library libmana.a
[  8%] Built target manao
[  8%] Building CXX object CMakeFiles/objlib.dir/src/odb.cxx.o
[  8%] Built target mana
[  9%] Building CXX object CMakeFiles/objlib.dir/src/device_driver.cxx.o
[  9%] Built target mfeo
[ 10%] Linking CXX static library libmfe.a
[ 11%] Building CXX object CMakeFiles/objlib.dir/src/system.cxx.o
[ 11%] Built target mfe
[ 11%] Building CXX object CMakeFiles/objlib.dir/src/alarm.cxx.o
[ 12%] Building CXX object CMakeFiles/objlib.dir/src/elog.cxx.o
[ 13%] Building CXX object CMakeFiles/objlib.dir/src/mrpc.cxx.o
[ 13%] Building CXX object CMakeFiles/objlib.dir/src/mjson.cxx.o
[ 14%] Building CXX object CMakeFiles/objlib.dir/src/tmfe.cxx.o
[ 15%] Building CXX object CMakeFiles/objlib.dir/src/mvodb.cxx.o
[ 15%] Built target rmanao
[ 15%] Linking CXX static library librmana.a
[ 16%] Building CXX object CMakeFiles/objlib.dir/src/nullodb.cxx.o
[ 16%] Building CXX object CMakeFiles/objlib.dir/src/midasodb.cxx.o
[ 16%] Built target rmana
[ 17%] Building CXX object CMakeFiles/objlib.dir/src/mxmlodb.cxx.o
[ 18%] Building CXX object CMakeFiles/objlib.dir/src/mjsonodb.cxx.o
[ 18%] Building CXX object CMakeFiles/objlib.dir/src/json_paste.cxx.o
[ 19%] Building CXX object CMakeFiles/objlib.dir/src/mjsonrpc.cxx.o
[ 20%] Building CXX object CMakeFiles/objlib.dir/src/mjsonrpc_user.cxx.o
[ 21%] Building CXX object CMakeFiles/objlib.dir/src/history.cxx.o
[ 21%] Building CXX object CMakeFiles/objlib.dir/src/history_common.cxx.o
[ 22%] Building CXX object CMakeFiles/objlib.dir/src/history_odbc.cxx.o
[ 23%] Building CXX object CMakeFiles/objlib.dir/src/history_schema.cxx.o
[ 23%] Building CXX object CMakeFiles/objlib.dir/src/lz4.cxx.o
[ 24%] Building CXX object CMakeFiles/objlib.dir/src/lz4frame.cxx.o
[ 25%] Building CXX object CMakeFiles/objlib.dir/src/lz4hc.cxx.o
[ 26%] Building CXX object CMakeFiles/objlib.dir/src/xxhash.cxx.o
[ 26%] Building CXX object CMakeFiles/objlib.dir/src/crc32c.cxx.o
[ 27%] Building CXX object CMakeFiles/objlib.dir/src/sha256.cxx.o
[ 28%] Building CXX object CMakeFiles/objlib.dir/src/sha512.cxx.o
[ 28%] Building CXX object CMakeFiles/objlib.dir/src/ftplib.cxx.o
[ 29%] Building CXX object CMakeFiles/objlib.dir/src/mdsupport.cxx.o
[ 30%] Building CXX object CMakeFiles/objlib.dir/mxml/mxml.cxx.o
[ 31%] Building CXX object CMakeFiles/objlib.dir/mxml/strlcpy.cxx.o
[ 31%] Built target objlib
Scanning dependencies of target midas-shared
Scanning dependencies of target midas
[ 33%] Linking CXX shared library libmidas-shared.dylib
[ 33%] Linking CXX static library libmidas.a
[ 33%] Built target midas-shared
[ 33%] Built target midas
Scanning dependencies of target mhttpd
Scanning dependencies of target feudp
Scanning dependencies of target odb_lock_test
Scanning dependencies of target rmlogger
Scanning dependencies of target mchart
Scanning dependencies of target msysmon
Scanning dependencies of target midas-c-compat
Scanning dependencies of target mfe_link_test
[ 33%] Building CXX object progs/CMakeFiles/mfe_link_test.dir/mfe_link_test.cxx.o
[ 33%] Linking CXX shared library libmidas-c-compat.dylib
[ 34%] Building CXX object progs/CMakeFiles/feudp.dir/feudp.cxx.o
[ 35%] Building CXX object progs/CMakeFiles/msysmon.dir/msysmon.cxx.o
[ 36%] Building CXX object progs/CMakeFiles/mchart.dir/mchart.cxx.o
[ 37%] Building CXX object progs/CMakeFiles/odb_lock_test.dir/odb_lock_test.cxx.o
[ 38%] Building CXX object progs/CMakeFiles/rmlogger.dir/mlogger.cxx.o
[ 39%] Building CXX object progs/CMakeFiles/mhttpd.dir/mhttpd.cxx.o
[ 39%] Built target midas-c-compat
[ 40%] Building CXX object progs/CMakeFiles/mhttpd.dir/mongoose6.cxx.o
[ 41%] Linking CXX executable mfe_link_test
[ 42%] Linking CXX executable odb_lock_test
[ 42%] Built target mfe_link_test
Scanning dependencies of target mjson_test
[ 43%] Building CXX object progs/CMakeFiles/mjson_test.dir/mjson_test.cxx.o
[ 43%] Linking CXX executable mchart
[ 43%] Built target odb_lock_test
Scanning dependencies of target odbinit
[ 44%] Building CXX object progs/CMakeFiles/odbinit.dir/odbinit.cxx.o
[ 45%] Linking CXX executable feudp
[ 45%] Built target mchart
[ 46%] Building CXX object progs/CMakeFiles/mhttpd.dir/mgd.cxx.o
[ 46%] Built target feudp
[ 46%] Building CXX object progs/CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o
[ 47%] Linking CXX executable mjson_test
[ 47%] Built target mjson_test
Scanning dependencies of target fetest_tmfe_thread
[ 47%] Building CXX object progs/CMakeFiles/fetest_tmfe_thread.dir/fetest_tmfe_thread.cxx.o
[ 47%] Linking CXX executable msysmon
[ 47%] Built target msysmon
[ 48%] Linking CXX executable odbinit
Scanning dependencies of target fetest_tmfe
[ 49%] Building CXX object progs/CMakeFiles/fetest_tmfe.dir/fetest_tmfe.cxx.o
[ 49%] Built target odbinit
[ 50%] Linking CXX executable fetest_tmfe_thread
Scanning dependencies of target mh2sql
[ 51%] Building CXX object progs/CMakeFiles/mh2sql.dir/mh2sql.cxx.o
Scanning dependencies of target odbhist
[ 51%] Building CXX object progs/CMakeFiles/odbhist.dir/odbhist.cxx.o
[ 51%] Built target fetest_tmfe_thread
Scanning dependencies of target melog
[ 51%] Building CXX object progs/CMakeFiles/melog.dir/melog.cxx.o
[ 52%] Linking CXX executable odbhist
[ 53%] Built target odbhist
[ 53%] Linking CXX executable melog
[ 54%] Linking CXX executable fetest_tmfe
Scanning dependencies of target mfe_link_test_cxx
[ 55%] Building CXX object progs/CMakeFiles/mfe_link_test_cxx.dir/mfe_link_test_cxx.cxx.o
[ 55%] Built target melog
Scanning dependencies of target crc32c_sum
[ 55%] Built target fetest_tmfe
Scanning dependencies of target odbedit
[ 56%] Building CXX object progs/CMakeFiles/crc32c_sum.dir/crc32c_sum.cxx.o
[ 56%] Building CXX object progs/CMakeFiles/odbedit.dir/odbedit.cxx.o
[ 57%] Linking CXX executable crc32c_sum
[ 58%] Linking CXX executable mh2sql
[ 58%] Built target crc32c_sum
[ 59%] Building CXX object progs/CMakeFiles/odbedit.dir/cmdedit.cxx.o
[ 59%] Linking CXX executable mfe_link_test_cxx
Scanning dependencies of target mdump
[ 59%] Built target mh2sql
Scanning dependencies of target mhdump
[ 60%] Building CXX object progs/CMakeFiles/mdump.dir/mdump.cxx.o
[ 60%] Building CXX object progs/CMakeFiles/mhdump.dir/mhdump.cxx.o
[ 60%] Built target mfe_link_test_cxx
Scanning dependencies of target lazylogger
[ 61%] Building CXX object progs/CMakeFiles/lazylogger.dir/lazylogger.cxx.o
Scanning dependencies of target mtransition
[ 61%] Building CXX object progs/CMakeFiles/mtransition.dir/mtransition.cxx.o
[ 62%] Linking CXX executable mdump
[ 63%] Linking CXX executable rmlogger
[ 63%] Built target mdump
Scanning dependencies of target mserver
[ 64%] Building CXX object progs/CMakeFiles/mserver.dir/mserver.cxx.o
[ 65%] Linking CXX executable mtransition
[ 65%] Built target rmlogger
Scanning dependencies of target mhist
[ 66%] Building CXX object progs/CMakeFiles/mhist.dir/mhist.cxx.o
[ 66%] Built target mtransition
Scanning dependencies of target get_record_test
[ 67%] Building CXX object progs/CMakeFiles/get_record_test.dir/get_record_test.cxx.o
[ 68%] Linking CXX executable mhdump
Scanning dependencies of target msequencer
[ 68%] Building CXX object progs/CMakeFiles/msequencer.dir/msequencer.cxx.o
[ 68%] Built target mhdump
Scanning dependencies of target fetest
[ 69%] Linking CXX executable odbedit
[ 70%] Building CXX object progs/CMakeFiles/fetest.dir/fetest.cxx.o
[ 70%] Built target odbedit
[ 70%] Linking CXX executable mserver
Scanning dependencies of target mstat
[ 70%] Linking CXX executable get_record_test
[ 71%] Building CXX object progs/CMakeFiles/mstat.dir/mstat.cxx.o
[ 71%] Built target mserver
Scanning dependencies of target mlogger
[ 71%] Built target get_record_test
[ 71%] Building CXX object progs/CMakeFiles/mlogger.dir/mlogger.cxx.o
Scanning dependencies of target analyzer
[ 71%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/analyzer.cxx.o
[ 71%] Linking CXX executable mhist
[ 71%] Linking CXX executable fetest
[ 72%] Linking CXX executable lazylogger
[ 72%] Built target mhist
[ 72%] Built target fetest
[ 73%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/adccalib.cxx.o
[ 74%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/adcsum.cxx.o
[ 74%] Built target lazylogger
Scanning dependencies of target frontend
[ 74%] Building CXX object examples/experiment/CMakeFiles/frontend.dir/frontend.cxx.o
[ 75%] Linking CXX executable mstat
Scanning dependencies of target mscb_fe
[ 76%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/mscb_fe.cxx.o
[ 76%] Built target mstat
[ 76%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/scaler.cxx.o
[ 77%] Linking CXX executable frontend
Scanning dependencies of target scfe
[ 77%] Built target frontend
[ 77%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/class/hv.cxx.o
[ 78%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/class/multi.cxx.o
[ 79%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/scfe.cxx.o
Scanning dependencies of target mtfe
[ 80%] Building CXX object examples/mtfe/CMakeFiles/mtfe.dir/mtfe.cxx.o
[ 81%] Linking CXX executable analyzer
[ 81%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/class/hv.cxx.o
[ 82%] Linking CXX executable mtfe
[ 82%] Built target analyzer
[ 83%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/device/nulldev.cxx.o
Scanning dependencies of target rpc_srvr
[ 83%] Built target mtfe
[ 84%] Building CXX object examples/lowlevel/CMakeFiles/rpc_srvr.dir/rpc_srvr.cxx.o
[ 85%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/class/multi.cxx.o
Scanning dependencies of target rpc_clnt
[ 85%] Building CXX object examples/lowlevel/CMakeFiles/rpc_clnt.dir/rpc_clnt.cxx.o
[ 85%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/bus/null.cxx.o
[ 85%] Linking CXX executable rpc_srvr
[ 85%] Built target rpc_srvr
[ 86%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/device/nulldev.cxx.o
[ 87%] Linking CXX executable rpc_clnt
[ 88%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/device/mscbdev.cxx.o
Scanning dependencies of target rpc_test
[ 89%] Building CXX object examples/lowlevel/CMakeFiles/rpc_test.dir/rpc_test.cxx.o
[ 89%] Built target rpc_clnt
[ 90%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/mscb/src/mscb.cxx.o
[ 90%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/bus/null.cxx.o
[ 91%] Linking CXX executable mlogger
Scanning dependencies of target consume
[ 92%] Building CXX object examples/lowlevel/CMakeFiles/consume.dir/consume.cxx.o
[ 93%] Linking CXX executable rpc_test
[ 94%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/device/mscbdev.cxx.o
[ 94%] Built target mlogger
Scanning dependencies of target produce
[ 94%] Built target rpc_test
[ 94%] Building CXX object examples/lowlevel/CMakeFiles/produce.dir/produce.cxx.o
[ 95%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/mscb/src/mscb.cxx.o
[ 96%] Linking CXX executable msequencer
[ 96%] Built target msequencer
[ 96%] Linking CXX executable consume
[ 96%] Built target consume
[ 97%] Linking CXX executable produce
[ 97%] Built target produce
[ 98%] Linking CXX executable mscb_fe
[ 98%] Built target mscb_fe
[ 99%] Linking CXX executable scfe
[ 99%] Built target scfe
[100%] Linking CXX executable mhttpd
[100%] Built target mhttpd
/midas/build$ 

>
> .../c++  ...  /opt/local/lib/libssl.dylib
>

I get the same, libssl from /opt/local, so we are not using openssl shipped with mac os.

We still do not know were OP's libssl comes from and if there is a mismatch between
openssl header files and library (header files from one openssl, library from a different openssl).

OP's cmake output: (no VERBOSE=1)
> -- MIDAS: cmake version: 3.16.3
> -- MIDAS: Found OpenSSL version 1.1.1d

Stefan's and my cmake output: (VERBOSE=1)
> -- MIDAS: cmake version: 3.16.3
> -- Found OpenSSL: /opt/local/lib/libcrypto.dylib (found version "1.1.1d")  
> -- MIDAS: Found OpenSSL version 1.1.1d

Without VERBOSE=1 cmake does not tell is which OpenSSL it found (not useful).
With VERBOSE=1 cmake outputs a flood of gunk (not useful).

My solution is to filter the cmake output with grep in "make cmake" ("make cmake3").

So please use that: "make cmake" - the output is roughly the same as normal make - compiler command lines (CFLAGS, library paths, etc),
compiler error message and all of the useful cmake output.

K.O.

> >
> > .../c++  ...  /opt/local/lib/libssl.dylib
> >
> 
> I get the same, libssl from /opt/local, so we are not using openssl shipped with mac os.
> 

I note that latest mongoose 6.16 finally has a virtual ssl layer and appears to support mbedtls (polarssl) in addition to openssl.

I now think I should see if it works - as it gives us a way to support https without relying on the user having
pre-installed working openssl packages - we consistently run into problems with openssl on macos, and even
on linux there was trouble with preinstalled openssl packages and libraries.

With mbedtls, one will have to "git pull" and "make" it, but historically this causes much less trouble.

Also, with luck, mbedtls has better support for certificate expiration (I would really love to have openssl report an error
or a warning or at least some hint if I feed it an expired certificate) and (gasp!) certbot integration.

K.O.

> ... support for certbot

The certbot tool to use instead of certbot is this: https://github.com/ndilieto/uacme

K.O.

> > ... support for certbot
> 
> The certbot tool to use instead of certbot is this: https://github.com/ndilieto/uacme
> 
> K.O.


HI Stefan and Konstantin, 

Thanks a lot for your messages. Sorry for my late reply, I only work on this project from Tuesday to Thursdays. I have 
run " make cmake" instead of "cd build; cmake" and this is the output regarding mhttpd:

/Library/Developer/CommandLineTools/usr/bin/c++  -O2 -g -DNDEBUG -Wl,-search_paths_first -Wl,-
headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o CMakeFiles/mhttpd.dir/mongoose6.cxx.o 
CMakeFiles/mhttpd.dir/mgd.cxx.o CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o mhttpd  -lsqlite3 ../libmidas.a 
/usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz -lsqlite3 /usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz 
Undefined symbols for architecture x86_64:
  "_OPENSSL_init_ssl", referenced from:
      _mg_mgr_init in mongoose6.cxx.o
  "_SSL_CTX_set_options", referenced from:
      _mg_set_ssl in mongoose6.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)


I see that in your outputs the openssl libs are in /opt/local/lib/ while mine are in /usr/lib/, and that is the only difference. 
I have checked that the libraries libssl.dylib and libcrypto.dylib are in my /usr/lib/, and indeed they are, so I don't 
understand the reason for the error, I will continue investigating. 

Thanks 

Berta 

> I only work on this project from Tuesday to Thursdays.

No problem. No hurry.

> I have run " make cmake" instead of "cd build; cmake" and this is the output regarding mhttpd: ...

There should be also a line where mhttpd.cxx is compiled into mhttpd.o, we need to see what compiler
flags are used - I suspect the compiler uses header files from /usr/local/include while the linker
is using libraries from /usr/lib, a mismatch.

To save time, please attach the full output of "make cmake". There may be something else I want to see there.

If you do not use the mhttpd built-in https support (for best security I recommend using https from the apache httpd password protected https proxy),
then it is perfectly fine to build midas with NO_SSL=1.

K.O.

We're using the History Logger MIDAS feature and writing to mySQL tables, but 
in some cases have run into issues installing mySQL on centos7 systems.

Has anyone ever tried running this MIDAS feature with MariaDB rather than 
mySQL?

> We're using the History Logger MIDAS feature and writing to mySQL tables, but 
> in some cases have run into issues installing mySQL on centos7 systems.
> 
> Has anyone ever tried running this MIDAS feature with MariaDB rather than 
> mySQL?

Hi

I've used MIDAS Logger feature with MariaDB and had no issue thus far .

> We're using the History Logger MIDAS feature and writing to mySQL tables, but 
> in some cases have run into issues installing mySQL on centos7 systems.
> 
> Has anyone ever tried running this MIDAS feature with MariaDB rather than 
> mySQL?

The best I can tell, MariaDB is *the* mysql. "the other thing" is an abandonware fork.

I personally at this moment do not run any daq stations with mysql logging, and I accidentally
removed all the 25 versions and flavours of mysql from my home laptop, so if there
is any problems with mysql, please holler, and I will reinstall mysql (ahem, mariadb) and restore
my ability to test midas against it.

As a connected question, is there any need to have postgres support in midas as well?

K.O.

While upgrading to the latest MIDAS version

MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-2019-09-i-1-gd93944ce-dirty on branch develop

ODB version: 3

I encountered two issues using cmake

1. on machines with NVIDIA drivers:

(nvml.h doesn't seem to be part of the standard nvidia driver package.)

2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 and with ROOT 6.18/04 on Fedora 31:

A workaround (which works for me) is to compile with

cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1

(please post messages in "plain" mode, they are much easier to answer)

- nvidia problems - this code was contributed by Joseph (I think?), with luck he will look into 
this problem.

- ROOT problem - it looks like the error is thrown by the ROOT header files, has nothing to do 
with MIDAS?

So what ROOT are you using? I recommend installing ROOT by following instructions at 
root.cern.ch.

Perhaps you used the ROOT packages from the EPEL repository? I have seen trouble with 
those packages before (miscompiled; important optional features turned off; very old 
versions; etc).

P.S.

Historically, ROOT has caused so many reports of "cannot build midas" that I consistently 
vote to "remove ROOT support from MIDAS". But Stefan's code for writing MIDAS data into 
ROOT files is so neat, cannot throw it away. And some people do use it. So at the latest MIDAS 
bash this Summer we decided to keep it.

(Only build targets to use ROOT are the rmlogger executable and the rmana.o object file (and 
it's one-man-army library)).

But.

In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
get built and installed, except for the code that failed to build.

Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
into the "bin" directory. So one must discover and use "NO_ROOT=1" (which becomes sticky 
until the next "make cclean". Some people are not used to sticky "make" options, I just got 
burned by this very thing last week).

Perhaps there is a way to tell cmake to ignore compile errors for rmlogger and rmana.


K.O.



<p>&nbsp;</p>

<table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
	<tbody>
		<tr>
			<td style="background-color:#486090">Peter Kunz wrote:</td>
		</tr>
		<tr>
			<td style="background-color:#FFFFB0">
			<p>While upgrading to the latest MIDAS version</p>

			<p>MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-
2019-09-i-1-gd93944ce-dirty on branch develop</p>

			<p>ODB version: 3</p>

			<p>I encountered two issues using cmake</p>

			<p>1. on machines with NVIDIA drivers:</p>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 
10px;">nvml.h: no such file or directory</div>

			<p>(nvml.h doesn&#39;t seem to be part of the standard nvidia driver 
package.)</p>

			<p>2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 
and with ROOT 6.18/04 on Fedora 31:</p>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">[ 29%] 
Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o In file included from 
/usr/include/root/TString.h:28, from /usr/include/root/TCollection.h:29, from 
/usr/include/root/TSeqCollection.h:25, from /usr/include/root/TList.h:25, from 
/usr/include/root/TQObject.h:40, from /usr/include/root/TApplication.h:30, from 
/home/pkunz/packages/midas/src/mana.cxx:60:</div>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 
10px;">/usr/include/root/ROOT/RStringView.hxx:32:37: error: &lsquo;experimental&rsquo; in 
namespace &lsquo;std&rsquo; does not name a type 32 | using basic_string_view = 
::std::experimental::basic_string_view&lt;_CharT,_Traits&gt;;</div>

			<p>A workaround (which works for me) is to compile with</p>

			<p><tt>cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1</tt></p>
			</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

Re: ROOT problem

I looked into how my ROOT based MIDAS analyzer compiles. It is using the flag -std=c++14.
MIDAS cmake compiles with -std=gnu++11 with the result:

 29%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
/usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=gnu++11 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
In file included from /usr/include/root/TString.h:28,
                 from /usr/include/root/TCollection.h:29,
                 from /usr/include/root/TSeqCollection.h:25,
                 from /usr/include/root/TList.h:25,
                 from /usr/include/root/TQObject.h:40,
                 from /usr/include/root/TApplication.h:30,
                 from /home/pkunz/packages/midas/src/mana.cxx:60:
/usr/include/root/ROOT/RStringView.hxx:32:37: error: ‘experimental’ in namespace ‘std’ does not name a type
   32 |    using basic_string_view = ::std::experimental::basic_string_view<_CharT,_Traits>;


If I change this to

/usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=c++14 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx

it compiles without error. I hope this helps. 






> (please post messages in "plain" mode, they are much easier to answer)
> 
> - nvidia problems - this code was contributed by Joseph (I think?), with luck he will look into 
> this problem.
> 
> - ROOT problem - it looks like the error is thrown by the ROOT header files, has nothing to do 
> with MIDAS?
> 
> So what ROOT are you using? I recommend installing ROOT by following instructions at 
> root.cern.ch.
> 
> Perhaps you used the ROOT packages from the EPEL repository? I have seen trouble with 
> those packages before (miscompiled; important optional features turned off; very old 
> versions; etc).
> 
> P.S.
> 
> Historically, ROOT has caused so many reports of "cannot build midas" that I consistently 
> vote to "remove ROOT support from MIDAS". But Stefan's code for writing MIDAS data into 
> ROOT files is so neat, cannot throw it away. And some people do use it. So at the latest MIDAS 
> bash this Summer we decided to keep it.
> 
> (Only build targets to use ROOT are the rmlogger executable and the rmana.o object file (and 
> it's one-man-army library)).
> 
> But.
> 
> In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
> get built and installed, except for the code that failed to build.
> 
> Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
> into the "bin" directory. So one must discover and use "NO_ROOT=1" (which becomes sticky 
> until the next "make cclean". Some people are not used to sticky "make" options, I just got 
> burned by this very thing last week).
> 
> Perhaps there is a way to tell cmake to ignore compile errors for rmlogger and rmana.
> 
> 
> K.O.
> 
> 
> 
> <p>&nbsp;</p>
> 
> <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> 	<tbody>
> 		<tr>
> 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> 		</tr>
> 		<tr>
> 			<td style="background-color:#FFFFB0">
> 			<p>While upgrading to the latest MIDAS version</p>
> 
> 			<p>MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-
> 2019-09-i-1-gd93944ce-dirty on branch develop</p>
> 
> 			<p>ODB version: 3</p>
> 
> 			<p>I encountered two issues using cmake</p>
> 
> 			<p>1. on machines with NVIDIA drivers:</p>
> 
> 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> 10px;">nvml.h: no such file or directory</div>
> 
> 			<p>(nvml.h doesn&#39;t seem to be part of the standard nvidia driver 
> package.)</p>
> 
> 			<p>2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 
> and with ROOT 6.18/04 on Fedora 31:</p>
> 
> 			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">[ 29%] 
> Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o In file included from 
> /usr/include/root/TString.h:28, from /usr/include/root/TCollection.h:29, from 
> /usr/include/root/TSeqCollection.h:25, from /usr/include/root/TList.h:25, from 
> /usr/include/root/TQObject.h:40, from /usr/include/root/TApplication.h:30, from 
> /home/pkunz/packages/midas/src/mana.cxx:60:</div>
> 
> 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> 10px;">/usr/include/root/ROOT/RStringView.hxx:32:37: error: &lsquo;experimental&rsquo; in 
> namespace &lsquo;std&rsquo; does not name a type 32 | using basic_string_view = 
> ::std::experimental::basic_string_view&lt;_CharT,_Traits&gt;;</div>
> 
> 			<p>A workaround (which works for me) is to compile with</p>
> 
> 			<p><tt>cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1</tt></p>
> 			</td>
> 		</tr>
> 	</tbody>
> </table>
> 
> <p>&nbsp;</p>

Right. So the problem is mismatch in ROOT compile flags. The old Makefile build used CFLAGS from ROOT to build rmana and rmlogger, cmake uses some kind of generic CFLAGS, so here we have it. No idea how to fix it.

K.O.


> Re: ROOT problem
> 
> I looked into how my ROOT based MIDAS analyzer compiles. It is using the flag -std=c++14.
> MIDAS cmake compiles with -std=gnu++11 with the result:
> 
>  29%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
> /usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=gnu++11 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
> In file included from /usr/include/root/TString.h:28,
>                  from /usr/include/root/TCollection.h:29,
>                  from /usr/include/root/TSeqCollection.h:25,
>                  from /usr/include/root/TList.h:25,
>                  from /usr/include/root/TQObject.h:40,
>                  from /usr/include/root/TApplication.h:30,
>                  from /home/pkunz/packages/midas/src/mana.cxx:60:
> /usr/include/root/ROOT/RStringView.hxx:32:37: error: ‘experimental’ in namespace ‘std’ does not name a type
>    32 |    using basic_string_view = ::std::experimental::basic_string_view<_CharT,_Traits>;
> 
> 
> If I change this to
> 
> /usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=c++14 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
> 
> it compiles without error. I hope this helps. 
> 
> 
> 
> 
> 
> 
> > (please post messages in "plain" mode, they are much easier to answer)
> > 
> > - nvidia problems - this code was contributed by Joseph (I think?), with luck he will look into 
> > this problem.
> > 
> > - ROOT problem - it looks like the error is thrown by the ROOT header files, has nothing to do 
> > with MIDAS?
> > 
> > So what ROOT are you using? I recommend installing ROOT by following instructions at 
> > root.cern.ch.
> > 
> > Perhaps you used the ROOT packages from the EPEL repository? I have seen trouble with 
> > those packages before (miscompiled; important optional features turned off; very old 
> > versions; etc).
> > 
> > P.S.
> > 
> > Historically, ROOT has caused so many reports of "cannot build midas" that I consistently 
> > vote to "remove ROOT support from MIDAS". But Stefan's code for writing MIDAS data into 
> > ROOT files is so neat, cannot throw it away. And some people do use it. So at the latest MIDAS 
> > bash this Summer we decided to keep it.
> > 
> > (Only build targets to use ROOT are the rmlogger executable and the rmana.o object file (and 
> > it's one-man-army library)).
> > 
> > But.
> > 
> > In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
> > get built and installed, except for the code that failed to build.
> > 
> > Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
> > into the "bin" directory. So one must discover and use "NO_ROOT=1" (which becomes sticky 
> > until the next "make cclean". Some people are not used to sticky "make" options, I just got 
> > burned by this very thing last week).
> > 
> > Perhaps there is a way to tell cmake to ignore compile errors for rmlogger and rmana.
> > 
> > 
> > K.O.
> > 
> > 
> > 
> > <p>&nbsp;</p>
> > 
> > <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> > 	<tbody>
> > 		<tr>
> > 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> > 		</tr>
> > 		<tr>
> > 			<td style="background-color:#FFFFB0">
> > 			<p>While upgrading to the latest MIDAS version</p>
> > 
> > 			<p>MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-
> > 2019-09-i-1-gd93944ce-dirty on branch develop</p>
> > 
> > 			<p>ODB version: 3</p>
> > 
> > 			<p>I encountered two issues using cmake</p>
> > 
> > 			<p>1. on machines with NVIDIA drivers:</p>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> > 10px;">nvml.h: no such file or directory</div>
> > 
> > 			<p>(nvml.h doesn&#39;t seem to be part of the standard nvidia driver 
> > package.)</p>
> > 
> > 			<p>2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 
> > and with ROOT 6.18/04 on Fedora 31:</p>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">[ 29%] 
> > Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o In file included from 
> > /usr/include/root/TString.h:28, from /usr/include/root/TCollection.h:29, from 
> > /usr/include/root/TSeqCollection.h:25, from /usr/include/root/TList.h:25, from 
> > /usr/include/root/TQObject.h:40, from /usr/include/root/TApplication.h:30, from 
> > /home/pkunz/packages/midas/src/mana.cxx:60:</div>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> > 10px;">/usr/include/root/ROOT/RStringView.hxx:32:37: error: &lsquo;experimental&rsquo; in 
> > namespace &lsquo;std&rsquo; does not name a type 32 | using basic_string_view = 
> > ::std::experimental::basic_string_view&lt;_CharT,_Traits&gt;;</div>
> > 
> > 			<p>A workaround (which works for me) is to compile with</p>
> > 
> > 			<p><tt>cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1</tt></p>
> > 			</td>
> > 		</tr>
> > 	</tbody>
> > </table>
> > 
> > <p>&nbsp;</p>

Thanks for tracing the problem further down. Now I realized that the CMakeLists.txt for the ROOT analyzer did not contain the usual ROOT flags. I added that and committed the change, so please try again. Here is the diff:

examples/experiment/CMakeLists.txt:

       execute_process(COMMAND root-config --incdir OUTPUT_VARIABLE ROOT_INC)
       execute_process(COMMAND root-config --libs OUTPUT_VARIABLE ROOT_LIBS)
+      execute_process(COMMAND root-config --cflags OUTPUT_VARIABLE ROOT_CFLAGS)
       string(STRIP ${ROOT_LIBS} ROOT_LIBS)
-      string(REGEX REPLACE "\n$" "" ROOT_INC "${ROOT_INC}")
+      string(REGEX REPLACE "\n" "" ROOT_INC ${ROOT_INC})
+      separate_arguments(ROOT_CFLAGS UNIX_COMMAND "${ROOT_CFLAGS}")
       set(CMAKE_CXX_STANDARD 11)

       target_include_directories(analyzer PUBLIC ${ROOT_INC} ${INC_PATH})
+      target_compile_options(analyzer PUBLIC ${ROOT_CFLAGS})
       target_link_libraries(analyzer rmana midas ${ROOT_LIBS} ${LIBS})


Best,
Stefan

> Re: ROOT problem
> 
> I looked into how my ROOT based MIDAS analyzer compiles. It is using the flag -std=c++14.
> MIDAS cmake compiles with -std=gnu++11 with the result:
> 
>  29%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
> /usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=gnu++11 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
> In file included from /usr/include/root/TString.h:28,
>                  from /usr/include/root/TCollection.h:29,
>                  from /usr/include/root/TSeqCollection.h:25,
>                  from /usr/include/root/TList.h:25,
>                  from /usr/include/root/TQObject.h:40,
>                  from /usr/include/root/TApplication.h:30,
>                  from /home/pkunz/packages/midas/src/mana.cxx:60:
> /usr/include/root/ROOT/RStringView.hxx:32:37: error: ‘experimental’ in namespace ‘std’ does not name a type
>    32 |    using basic_string_view = ::std::experimental::basic_string_view<_CharT,_Traits>;
> 
> 
> If I change this to
> 
> /usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=c++14 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
> 
> it compiles without error. I hope this helps. 
> 
> 
> 
> 
> 
> 
> > (please post messages in "plain" mode, they are much easier to answer)
> > 
> > - nvidia problems - this code was contributed by Joseph (I think?), with luck he will look into 
> > this problem.
> > 
> > - ROOT problem - it looks like the error is thrown by the ROOT header files, has nothing to do 
> > with MIDAS?
> > 
> > So what ROOT are you using? I recommend installing ROOT by following instructions at 
> > root.cern.ch.
> > 
> > Perhaps you used the ROOT packages from the EPEL repository? I have seen trouble with 
> > those packages before (miscompiled; important optional features turned off; very old 
> > versions; etc).
> > 
> > P.S.
> > 
> > Historically, ROOT has caused so many reports of "cannot build midas" that I consistently 
> > vote to "remove ROOT support from MIDAS". But Stefan's code for writing MIDAS data into 
> > ROOT files is so neat, cannot throw it away. And some people do use it. So at the latest MIDAS 
> > bash this Summer we decided to keep it.
> > 
> > (Only build targets to use ROOT are the rmlogger executable and the rmana.o object file (and 
> > it's one-man-army library)).
> > 
> > But.
> > 
> > In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
> > get built and installed, except for the code that failed to build.
> > 
> > Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
> > into the "bin" directory. So one must discover and use "NO_ROOT=1" (which becomes sticky 
> > until the next "make cclean". Some people are not used to sticky "make" options, I just got 
> > burned by this very thing last week).
> > 
> > Perhaps there is a way to tell cmake to ignore compile errors for rmlogger and rmana.
> > 
> > 
> > K.O.
> > 
> > 
> > 
> > <p>&nbsp;</p>
> > 
> > <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> > 	<tbody>
> > 		<tr>
> > 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> > 		</tr>
> > 		<tr>
> > 			<td style="background-color:#FFFFB0">
> > 			<p>While upgrading to the latest MIDAS version</p>
> > 
> > 			<p>MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-
> > 2019-09-i-1-gd93944ce-dirty on branch develop</p>
> > 
> > 			<p>ODB version: 3</p>
> > 
> > 			<p>I encountered two issues using cmake</p>
> > 
> > 			<p>1. on machines with NVIDIA drivers:</p>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> > 10px;">nvml.h: no such file or directory</div>
> > 
> > 			<p>(nvml.h doesn&#39;t seem to be part of the standard nvidia driver 
> > package.)</p>
> > 
> > 			<p>2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 
> > and with ROOT 6.18/04 on Fedora 31:</p>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">[ 29%] 
> > Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o In file included from 
> > /usr/include/root/TString.h:28, from /usr/include/root/TCollection.h:29, from 
> > /usr/include/root/TSeqCollection.h:25, from /usr/include/root/TList.h:25, from 
> > /usr/include/root/TQObject.h:40, from /usr/include/root/TApplication.h:30, from 
> > /home/pkunz/packages/midas/src/mana.cxx:60:</div>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> > 10px;">/usr/include/root/ROOT/RStringView.hxx:32:37: error: &lsquo;experimental&rsquo; in 
> > namespace &lsquo;std&rsquo; does not name a type 32 | using basic_string_view = 
> > ::std::experimental::basic_string_view&lt;_CharT,_Traits&gt;;</div>
> > 
> > 			<p>A workaround (which works for me) is to compile with</p>
> > 
> > 			<p><tt>cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1</tt></p>
> > 			</td>
> > 		</tr>
> > 	</tbody>
> > </table>
> > 
> > <p>&nbsp;</p>

> In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
> get built and installed, except for the code that failed to build.
> 
> Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
> into the "bin" directory.

That's not correct. Indeed a "make -k" stops after an error, but "make -i" still works. Try it! Event "make install -i" 
installs all executables into the bin directory which successfully compiled even if there is an error.

Stefan

> Complile including ROOT throws an error with ROOT 6.18/04 on Fedora 31 ...
> /usr/include/root/ROOT/RStringView.hxx:32:37: error: ‘experimental’ in namespace ‘std’ ...

I was puzzled by this error about "experimental" in ROOT header files.

All ROOT kits that I have are configured and built with c++11, and I do not see this error. I now guess that you
only saw this on Fedora 31 with gcc 9 and ROOT that was configured for c++14 or c++17.

We now realize that MIDAS cmake does not use the ROOT CFLAGS when building code that uses ROOT,
and this is probably a mistake. Stefan added a fix to use ROOT CFLAGS to build the analyzer,
but I am not sure if this is sufficient, rmana and rmlogger probably also should use the ROOT CFLAGS.

In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).

K.O.

> In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).

Are there good reasons to not compile MIDAS with set(CMAKE_CXX_STANDARD 14)? So far this was an easier "fix" for me than to recompile ROOT with c++11.

> > In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).
> 
> Are there good reasons to not compile MIDAS with set(CMAKE_CXX_STANDARD 14)? So far this was an easier "fix" for me than to recompile ROOT with c++11.

Yes. We agreed to base MIDAS on C++11, not C++14, since C++11 is available on all relevant systems. By putting that requirement explicitly into the CMake file, we know that it 
correctly compiles with C++11. Of course feel free to overwrite this in your local installation.

Stefan

> > In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).
> 
> Are there good reasons to not compile MIDAS with set(CMAKE_CXX_STANDARD 14)? So far this was an easier "fix" for me than to recompile ROOT with c++11.

We already have trouble enough even with requiring C++11:

We still run a number of SL6 DAQ stations at TRIUMF, and it is well known that SL6 gcc does not even do c++11.

The suggested solution to "simply just use the devtoolset-8" compilers turned out to be missing a critical 32-bit
library and cannot build midas for 32-bit VME processors. (straight or cross-compiled).

(A solution for this can be kludged using unofficial packages: https://midas.triumf.ca/elog/Midas/1748)

No, we cannot "just upgrade" SL6 and no, we cannot "just replace" the 32-bit-only VME processors.

If we move the midas requirements from c++11 to c++14, again, we have to see what compilers
are available on what platforms, and I am sure again we will miss some important case.

And then, where does this train stop?

I see c++17 has some really nice to use improvements. And I see c++20 added even some more nice improvements.

I say we follow ROOT: https://root.cern.ch/supported-platforms

> Supported Platforms
> ROOT 6
> The compiler needs to support -std=c++11 to be able to build version 6.

K.O.

> > In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).
> 
> Are there good reasons to not compile MIDAS with set(CMAKE_CXX_STANDARD 14)? So far this was an easier "fix" for me than to recompile ROOT with c++11.

Per  https://root.cern.ch/supported-platforms, ROOTv6 is C++11.

Binary kits distributed from root.cern.ch seem to be built with c++11.

Where does ROOT built with c++14 come from?

K.O.

I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:

/home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
            , &(info->caid[i].chan_id))

/home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
            , &(info->caid[i].evt_id))

This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 

> I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
> However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:
> 
> /home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
>             , &(info->caid[i].chan_id))
> 
> /home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
>             , &(info->caid[i].evt_id))
> 
> This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
> I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 

Hi, Peter - it looks like epics_ca.cxx needs to be updated to proper C++ (char* warnings, etc).

As for the "function not declared" errors, it's the C++ way to say "you are calling a function with wrong arguments",
again, something I should be able to fix without too much trouble.

Thank you for reporting this, the midas version of epics_ca.cxx definitely needs fixing.

K.O.

I fixed the compiler errors in epics_ca.cxx, can you try again? (see https://bitbucket.org/tmidas/midas/commits/)

But, I do not see errors with ca_create_channel() and ca_create_subscription().

Maybe I am using the wrong epics. If you still see these errors, let me know what epics you have, I will try the same one.

This is what I do (and I get epics 7.0).

mkdir -p $HOME/git
cd $HOME/git
git clone https://git.launchpad.net/epics-base
cd epics-base
make
ls -l include/cadef.h
ls -l lib/darwin-x86/libca.a # also linux-x86/libca.a

K.O.

> > I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
> > However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:
> > 
> > /home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
> >             , &(info->caid[i].chan_id))
> > 
> > /home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
> >             , &(info->caid[i].evt_id))
> > 
> > This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
> > I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 
> 
> Hi, Peter - it looks like epics_ca.cxx needs to be updated to proper C++ (char* warnings, etc).
> 
> As for the "function not declared" errors, it's the C++ way to say "you are calling a function with wrong arguments",
> again, something I should be able to fix without too much trouble.
> 
> Thank you for reporting this, the midas version of epics_ca.cxx definitely needs fixing.
> 
> K.O.

Hi Konstantin,
I have EPICS Base Release 3.14.8.2 and got your example running with it, though I had to make one change.
For some reason it wouldn't find libca.so, but when I linked the static library libca.a instead, it worked.
Also my midas frontend works now with the updated epics_ca.xx. There was a problem with an old local version of cadef.h
(probably a leftover from some previous testing). After removing it everything compiled well.

Thanks,
Peter


> I fixed the compiler errors in epics_ca.cxx, can you try again? (see https://bitbucket.org/tmidas/midas/commits/)
> 
> But, I do not see errors with ca_create_channel() and ca_create_subscription().
> 
> Maybe I am using the wrong epics. If you still see these errors, let me know what epics you have, I will try the same one.
> 
> This is what I do (and I get epics 7.0).
> 
> mkdir -p $HOME/git
> cd $HOME/git
> git clone https://git.launchpad.net/epics-base
> cd epics-base
> make
> ls -l include/cadef.h
> ls -l lib/darwin-x86/libca.a # also linux-x86/libca.a
> 
> K.O.
> 
> > > I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
> > > However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:
> > > 
> > > /home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
> > >             , &(info->caid[i].chan_id))
> > > 
> > > /home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
> > >             , &(info->caid[i].evt_id))
> > > 
> > > This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
> > > I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 
> > 
> > Hi, Peter - it looks like epics_ca.cxx needs to be updated to proper C++ (char* warnings, etc).
> > 
> > As for the "function not declared" errors, it's the C++ way to say "you are calling a function with wrong arguments",
> > again, something I should be able to fix without too much trouble.
> > 
> > Thank you for reporting this, the midas version of epics_ca.cxx definitely needs fixing.
> > 
> > K.O.

> I have EPICS Base Release 3.14.8.2 and got your example running with it...

Ok, good.

> I had to make one change.
> For some reason it wouldn't find libca.so, but when I linked the static library libca.a instead, it worked.

Yes, libca.a is what I see in my epics, too. I recommend static linking (libxxx.a instead of libxxx.so) as it eliminates
all possible problems with "cannot find the shared library" and "found the wrong shared library" and "found
a shared library from a different version of epics".

> Also my midas frontend works now with the updated epics_ca.xx. There was a problem with an old local version of cadef.h
> (probably a leftover from some previous testing). After removing it everything compiled well.

Ok. good. I also see some old epics header files in the midas repository. I guess I should remove them.

K.O.


> 
> Thanks,
> Peter
> 
> 
> > I fixed the compiler errors in epics_ca.cxx, can you try again? (see https://bitbucket.org/tmidas/midas/commits/)
> > 
> > But, I do not see errors with ca_create_channel() and ca_create_subscription().
> > 
> > Maybe I am using the wrong epics. If you still see these errors, let me know what epics you have, I will try the same one.
> > 
> > This is what I do (and I get epics 7.0).
> > 
> > mkdir -p $HOME/git
> > cd $HOME/git
> > git clone https://git.launchpad.net/epics-base
> > cd epics-base
> > make
> > ls -l include/cadef.h
> > ls -l lib/darwin-x86/libca.a # also linux-x86/libca.a
> > 
> > K.O.
> > 
> > > > I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
> > > > However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:
> > > > 
> > > > /home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
> > > >             , &(info->caid[i].chan_id))
> > > > 
> > > > /home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
> > > >             , &(info->caid[i].evt_id))
> > > > 
> > > > This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
> > > > I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 
> > > 
> > > Hi, Peter - it looks like epics_ca.cxx needs to be updated to proper C++ (char* warnings, etc).
> > > 
> > > As for the "function not declared" errors, it's the C++ way to say "you are calling a function with wrong arguments",
> > > again, something I should be able to fix without too much trouble.
> > > 
> > > Thank you for reporting this, the midas version of epics_ca.cxx definitely needs fixing.
> > > 
> > > K.O.

>
> > I have EPICS Base Release 3.14.8.2 ...
>

For the record, according to https://epics.anl.gov/

the 3.14 branch is "Closed, please upgrade!"
the 3.15 branch seems to be the current "stable" branch, and
the 7.0 branch is the "current release"

I forget which version of EPICS is currently recommended by the TRIUMF controls group. I will have to go and ask them.

K.O.

After upgrading to the lastes MIDAS version I got the DAQ frontend of my application running by changing all compiler directives from cc to g++ and using

#include "mfe.h"

extern HNDLE hDB

 extern "C" { 
 #include <CAENComm.h>  
 }

With these changes everything seems to work fine.

However, I'm having trouble with a slow control frontend using a tcpip driver. It compiled well with the older MIDAS version. Even though all the functions in question are defined in the frontend code, the following error comes up:

 I guess the the aforementioned DAQ frontend compiles because its equipment definitions don't call on the function `initialize_equipment()', but I can't figure out why it doesn't work. Help is appreciated. P.K.

(please use the "plain" text, much easier to answer).

Hi, Peter, I think you misread the error message. There is no error about initialize_equipment(), the error 
is about interrupt_configure(). initialize_equipment() is just one of the functions that calls it.

The cause of the error most likely is a mismatch between the declaration of interrupt_configure() in 
mfe.h and the definition of this function in your program (in feMotor.c, I guess).

Sometimes this mismatch is hard to identify just by looking at the code.

One fool-proof method to debug this is to extract the actual function prototypes from your object files, 
both the declaration ("U") in mfe.o and the definition ("T") in your program should be identical:

nm feMotor.o |  grep -i interrupt | c++filt
0000000000000f90 T interrupt_configure(int, int, long) <--- should be this

nm ~/packages/midas/lib/libmfe.a | grep -i interrupt | c++filt
U interrupt_configure(int, int, long)

If they are different, you adjust your program until they match. One way to ensure the match is to copy 
the declaration from mfe.h into your program.

K.O.




<p>&nbsp;</p>

<table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
	<tbody>
		<tr>
			<td style="background-color:#486090">Peter Kunz wrote:</td>
		</tr>
		<tr>
			<td style="background-color:#FFFFB0">
			<p>After upgrading to the lastes MIDAS version I got the DAQ frontend of my application 
running by changing all compiler directives from cc to g++ and using</p>

			<p>#include &quot;mfe.h&quot;</p>

			<p>extern HNDLE hDB</p>

			<p>&nbsp;extern &quot;C&quot; {&nbsp;<br />
			&nbsp;#include &lt;CAENComm.h&gt; &nbsp;<br />
			&nbsp;}</p>

			<p>With these changes everything seems to work fine.</p>

			<p>However, I&#39;m having trouble with a slow control frontend using a tcpip driver. It 
compiled well with the older MIDAS version. Even though all the functions in question are defined in the 
frontend code, the following error comes up:</p>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">g++ -o feMotor 
-DOS_LINUX -Dextname -g -O2 -fPIC -Wall -Wuninitialized &nbsp;-fpermissive -
I/home/pkunz/packages/midas/include -I. -I/home/pkunz/packages/midas/drivers/bus &nbsp; 
/home/pkunz/packages/midas/lib/libmidas.a feMotor.o 
/home/pkunz/packages/midas/drivers/bus/tcpip.o cd_Galil.o 
/home/pkunz/packages/midas/lib/libmidas.a /home/pkunz/packages/midas/lib/mfe.o -lm -lz -lutil -lnsl -
lpthread -lrt<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
`initialize_equipment()&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:687: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
`readout_enable(unsigned int)&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:1236: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:1238: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function `main&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:2791: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:2792: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			collect2: error: ld returned 1 exit status<br />
			make: *** [Makefile:36: feMotor] Error 1</div>

			<p>&nbsp;I guess the the aforementioned DAQ frontend compiles because its equipment 
definitions don&#39;t call on the function `initialize_equipment()&#39;, but I can&#39;t figure out why 
it doesn&#39;t work. Help is appreciated. P.K.</p>
			</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

Thanks for explaining this, Konstantin. 
After updating the function to

INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
  return CM_SUCCESS;
}

it compiled without errors. In the original code the "INT source" variable was missing.



> (please use the "plain" text, much easier to answer).
> 
> Hi, Peter, I think you misread the error message. There is no error about initialize_equipment(), the error 
> is about interrupt_configure(). initialize_equipment() is just one of the functions that calls it.
> 
> The cause of the error most likely is a mismatch between the declaration of interrupt_configure() in 
> mfe.h and the definition of this function in your program (in feMotor.c, I guess).
> 
> Sometimes this mismatch is hard to identify just by looking at the code.
> 
> One fool-proof method to debug this is to extract the actual function prototypes from your object files, 
> both the declaration ("U") in mfe.o and the definition ("T") in your program should be identical:
> 
> nm feMotor.o |  grep -i interrupt | c++filt
> 0000000000000f90 T interrupt_configure(int, int, long) <--- should be this
> 
> nm ~/packages/midas/lib/libmfe.a | grep -i interrupt | c++filt
> U interrupt_configure(int, int, long)
> 
> If they are different, you adjust your program until they match. One way to ensure the match is to copy 
> the declaration from mfe.h into your program.
> 
> K.O.
> 
> 
> 
> 
> <p>&nbsp;</p>
> 
> <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> 	<tbody>
> 		<tr>
> 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> 		</tr>
> 		<tr>
> 			<td style="background-color:#FFFFB0">
> 			<p>After upgrading to the lastes MIDAS version I got the DAQ frontend of my application 
> running by changing all compiler directives from cc to g++ and using</p>
> 
> 			<p>#include &quot;mfe.h&quot;</p>
> 
> 			<p>extern HNDLE hDB</p>
> 
> 			<p>&nbsp;extern &quot;C&quot; {&nbsp;<br />
> 			&nbsp;#include &lt;CAENComm.h&gt; &nbsp;<br />
> 			&nbsp;}</p>
> 
> 			<p>With these changes everything seems to work fine.</p>
> 
> 			<p>However, I&#39;m having trouble with a slow control frontend using a tcpip driver. It 
> compiled well with the older MIDAS version. Even though all the functions in question are defined in the 
> frontend code, the following error comes up:</p>
> 
> 			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">g++ -o feMotor 
> -DOS_LINUX -Dextname -g -O2 -fPIC -Wall -Wuninitialized &nbsp;-fpermissive -
> I/home/pkunz/packages/midas/include -I. -I/home/pkunz/packages/midas/drivers/bus &nbsp; 
> /home/pkunz/packages/midas/lib/libmidas.a feMotor.o 
> /home/pkunz/packages/midas/drivers/bus/tcpip.o cd_Galil.o 
> /home/pkunz/packages/midas/lib/libmidas.a /home/pkunz/packages/midas/lib/mfe.o -lm -lz -lutil -lnsl -
> lpthread -lrt<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
> `initialize_equipment()&#39;:<br />
> 			/home/pkunz/packages/midas/src/mfe.cxx:687: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
> `readout_enable(unsigned int)&#39;:<br />
> 			/home/pkunz/packages/midas/src/mfe.cxx:1236: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:1238: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function `main&#39;:<br />
> 			/home/pkunz/packages/midas/src/mfe.cxx:2791: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:2792: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			collect2: error: ld returned 1 exit status<br />
> 			make: *** [Makefile:36: feMotor] Error 1</div>
> 
> 			<p>&nbsp;I guess the the aforementioned DAQ frontend compiles because its equipment 
> definitions don&#39;t call on the function `initialize_equipment()&#39;, but I can&#39;t figure out why 
> it doesn&#39;t work. Help is appreciated. P.K.</p>
> 			</td>
> 		</tr>
> 	</tbody>
> </table>
> 
> <p>&nbsp;</p>

We updated midas/examples/experiment/frontend.cxx to correctly contain

/*-- Interrupt configuration ---------------------------------------*/

INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
   switch (cmd) {
   case CMD_INTERRUPT_ENABLE:
      break;
   case CMD_INTERRUPT_DISABLE:
      break;
   case CMD_INTERRUPT_ATTACH:
      break;
   case CMD_INTERRUPT_DETACH:
      break;
   }
   return SUCCESS;
}

but if you upgrade from C to C++ from your own old frontend code you might be hit by that issue. 

Maybe Konstantin can update elog:1526 to contain a hint about "INT source".

Stefan

Actually now I see that 

a4) poll_event() and interrupt_configure() have "source" as "int[]" instead of "int" (why did this work before?)

mention this already, but maybe it's not completely clear that one has to change "int[] source" to "int source"

Stefan


> We updated midas/examples/experiment/frontend.cxx to correctly contain
> 
> /*-- Interrupt configuration ---------------------------------------*/
> 
> INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
> {
>    switch (cmd) {
>    case CMD_INTERRUPT_ENABLE:
>       break;
>    case CMD_INTERRUPT_DISABLE:
>       break;
>    case CMD_INTERRUPT_ATTACH:
>       break;
>    case CMD_INTERRUPT_DETACH:
>       break;
>    }
>    return SUCCESS;
> }
> 
> but if you upgrade from C to C++ from your own old frontend code you might be hit by that issue. 
> 
> Maybe Konstantin can update elog:1526 to contain a hint about "INT source".
> 
> Stefan

(please use the "plain" text, much easier to answer).

Hi, Peter, I think you misread the error message. There is no error about 
initialize_equipment(), the error is about interrupt_configure(). initialize_equipment() is just 
one of the functions that calls it.

The cause of the error most likely is a mismatch between the declaration of 
interrupt_configure() in mfe.h and the definition of this function in your program (in 
feMotor.c, I guess).

Sometimes this mismatch is hard to identify just by looking at the code.

One fool-proof method to debug this is to extract the actual function prototypes from your 
object files, both the declaration ("U") in mfe.o and the definition ("T") in your program 
should be identical:

nm feMotor.o |  grep -i interrupt | c++filt
0000000000000f90 T interrupt_configure(int, int, long) <--- should be this

nm ~/packages/midas/lib/libmfe.a | grep -i interrupt | c++filt
U interrupt_configure(int, int, long)

If they are different, you adjust your program until they match. One way to ensure the 
match is to copy the declaration from mfe.h into your program.

K.O.




<p>&nbsp;</p>

<table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
	<tbody>
		<tr>
			<td style="background-color:#486090">Peter Kunz wrote:</td>
		</tr>
		<tr>
			<td style="background-color:#FFFFB0">
			<p>After upgrading to the lastes MIDAS version I got the DAQ frontend of my 
application running by changing all compiler directives from cc to g++ and using</p>

			<p>#include &quot;mfe.h&quot;</p>

			<p>extern HNDLE hDB</p>

			<p>&nbsp;extern &quot;C&quot; {&nbsp;<br />
			&nbsp;#include &lt;CAENComm.h&gt; &nbsp;<br />
			&nbsp;}</p>

			<p>With these changes everything seems to work fine.</p>

			<p>However, I&#39;m having trouble with a slow control frontend using a 
tcpip driver. It compiled well with the older MIDAS version. Even though all the functions in 
question are defined in the frontend code, the following error comes up:</p>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">g++ 
-o feMotor -DOS_LINUX -Dextname -g -O2 -fPIC -Wall -Wuninitialized &nbsp;-fpermissive 
-I/home/pkunz/packages/midas/include -I. -I/home/pkunz/packages/midas/drivers/bus 
&nbsp; /home/pkunz/packages/midas/lib/libmidas.a feMotor.o 
/home/pkunz/packages/midas/drivers/bus/tcpip.o cd_Galil.o 
/home/pkunz/packages/midas/lib/libmidas.a /home/pkunz/packages/midas/lib/mfe.o -lm -lz 
-lutil -lnsl -lpthread -lrt<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
`initialize_equipment()&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:687: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
`readout_enable(unsigned int)&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:1236: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:1238: undefined 
reference to `interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function `main&#39;:
<br />
			/home/pkunz/packages/midas/src/mfe.cxx:2791: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:2792: undefined 
reference to `interrupt_configure(int, int, long)&#39;<br />
			collect2: error: ld returned 1 exit status<br />
			make: *** [Makefile:36: feMotor] Error 1</div>

			<p>&nbsp;I guess the the aforementioned DAQ frontend compiles because 
its equipment definitions don&#39;t call on the function `initialize_equipment()&#39;, but I 
can&#39;t figure out why it doesn&#39;t work. Help is appreciated. P.K.</p>
			</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

MIDAS version:      2.1
GIT revision:       Tue Dec 31 17:40:14 2019 +0100 - midas-2019-09-i-1-gd93944ce-dirty on branch develop

/Logger/Channels/0/Settings

With the settings above the file last.json generated for a new run is empty and the events 0x8000 and 0x8001 are missing in the .mid file.

When setting "ODB dump format" to "xml", events 0x8000 and 0x8001 are included in the .mid file, however, the file last.xml is not created.

(Please post messages in "plain" mode, they are much easier to answer)

Thank you for reporting this problem. I will try to reproduce it.

In addition, I will say a few words about your version of midas:

> GIT revision: midas-2019-09-i-1-gd93944ce-dirty on branch develop

I recommend that for production systems one used the tagged release versions of midas. 
(i.e. see https://midas.triumf.ca/elog/Midas/1750).

(Your midas is "1 commit after the latest tag" - the "-1" in the git revision).

I apply bug fixes to both the release branch and the develop branch, but for you to get 
these fixes, on the develop branch you will also "get" all the unrelated changes that may 
come with new bugs. On the release branch, you will only get the bug fixes.

In your midas version it says "-dirty" which means that you have local modifications to the 
midas sources. With luck those changes are not related to the bug that you see. (but I 
cannot tell). You can do "git status" and "git diff" to see what the local changes are.

It is much better if bugs are reported against "clean" builds of MIDAS (no "-dirty").


K.O.


<p>&nbsp;</p>

<table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
	<tbody>
		<tr>
			<td style="background-color:#486090">Peter Kunz wrote:</td>
		</tr>
		<tr>
			<td style="background-color:#FFFFB0">
			<p>MIDAS version: &nbsp; &nbsp; &nbsp;2.1<br />
			GIT revision: &nbsp; &nbsp; &nbsp; Tue Dec 31 17:40:14 2019 +0100 - 
midas-2019-09-i-1-gd93944ce-dirty on branch develop</p>

			<p>/Logger/Channels/0/Settings</p>

			<table border="3" cellpadding="1" class="dialogTable">
				<tbody>
					<tr>
						<td>ODB dump</td>
						<td>y</td>
					</tr>
					<tr>
						<td>ODB dump format</td>
						<td>json</td>
					</tr>
				</tbody>
			</table>

			<p>With the settings above the file last.json generated for a new run is 
empty and the events 0x8000 and 0x8001 are missing in the .mid file.</p>

			<p>When setting &quot;ODB dump format&quot; to &quot;xml&quot;, events 
0x8000 and 0x8001 are included in the .mid file, however, the file last.xml is not created.
</p>

			<p>&nbsp;</p>

			<p>&nbsp;</p>

			<p>&nbsp;</p>
			</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

Re: MIDAS versions

Thanks for pointing that out. I wasn't actually aware that there is a release branch and a development branch.
I was just following the installation instructions using

git clone https://bitbucket.org/tmidas/midas --recursive

Apparently that gave me the development branch. How can I get the release version?

(I think my version showed up a "dirty" because I played around with one of the examples. I didn't touch the actual source code.)

> (Please post messages in "plain" mode, they are much easier to answer)
> 
> Thank you for reporting this problem. I will try to reproduce it.
> 
> In addition, I will say a few words about your version of midas:
> 
> > GIT revision: midas-2019-09-i-1-gd93944ce-dirty on branch develop
> 
> I recommend that for production systems one used the tagged release versions of midas. 
> (i.e. see https://midas.triumf.ca/elog/Midas/1750).
> 
> (Your midas is "1 commit after the latest tag" - the "-1" in the git revision).
> 
> I apply bug fixes to both the release branch and the develop branch, but for you to get 
> these fixes, on the develop branch you will also "get" all the unrelated changes that may 
> come with new bugs. On the release branch, you will only get the bug fixes.
> 
> In your midas version it says "-dirty" which means that you have local modifications to the 
> midas sources. With luck those changes are not related to the bug that you see. (but I 
> cannot tell). You can do "git status" and "git diff" to see what the local changes are.
> 
> It is much better if bugs are reported against "clean" builds of MIDAS (no "-dirty").
> 
> 
> K.O.
> 
> 
> <p>&nbsp;</p>
> 
> <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> 	<tbody>
> 		<tr>
> 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> 		</tr>
> 		<tr>
> 			<td style="background-color:#FFFFB0">
> 			<p>MIDAS version: &nbsp; &nbsp; &nbsp;2.1<br />
> 			GIT revision: &nbsp; &nbsp; &nbsp; Tue Dec 31 17:40:14 2019 +0100 - 
> midas-2019-09-i-1-gd93944ce-dirty on branch develop</p>
> 
> 			<p>/Logger/Channels/0/Settings</p>
> 
> 			<table border="3" cellpadding="1" class="dialogTable">
> 				<tbody>
> 					<tr>
> 						<td>ODB dump</td>
> 						<td>y</td>
> 					</tr>
> 					<tr>
> 						<td>ODB dump format</td>
> 						<td>json</td>
> 					</tr>
> 				</tbody>
> 			</table>
> 
> 			<p>With the settings above the file last.json generated for a new run is 
> empty and the events 0x8000 and 0x8001 are missing in the .mid file.</p>
> 
> 			<p>When setting &quot;ODB dump format&quot; to &quot;xml&quot;, events 
> 0x8000 and 0x8001 are included in the .mid file, however, the file last.xml is not created.
> </p>
> 
> 			<p>&nbsp;</p>
> 
> 			<p>&nbsp;</p>
> 
> 			<p>&nbsp;</p>
> 			</td>
> 		</tr>
> 	</tbody>
> </table>
> 
> <p>&nbsp;</p>

For using the release branch read the messages in this thread. Most of the time, the develop branch is fine, except when we are developing something 
new, and the only way to tell is to watch the git activity on bitbucket or see the release branch announcements I post on the midas forum.

https://midas.triumf.ca/elog/Midas/1706

K.O.


> Re: MIDAS versions
> 
> Thanks for pointing that out. I wasn't actually aware that there is a release branch and a development branch.
> I was just following the installation instructions using
> 
> git clone https://bitbucket.org/tmidas/midas --recursive
> 
> Apparently that gave me the development branch. How can I get the release version?
> 
> (I think my version showed up a "dirty" because I played around with one of the examples. I didn't touch the actual source code.)
> 
> > (Please post messages in "plain" mode, they are much easier to answer)
> > 
> > Thank you for reporting this problem. I will try to reproduce it.
> > 
> > In addition, I will say a few words about your version of midas:
> > 
> > > GIT revision: midas-2019-09-i-1-gd93944ce-dirty on branch develop
> > 
> > I recommend that for production systems one used the tagged release versions of midas. 
> > (i.e. see https://midas.triumf.ca/elog/Midas/1750).
> > 
> > (Your midas is "1 commit after the latest tag" - the "-1" in the git revision).
> > 
> > I apply bug fixes to both the release branch and the develop branch, but for you to get 
> > these fixes, on the develop branch you will also "get" all the unrelated changes that may 
> > come with new bugs. On the release branch, you will only get the bug fixes.
> > 
> > In your midas version it says "-dirty" which means that you have local modifications to the 
> > midas sources. With luck those changes are not related to the bug that you see. (but I 
> > cannot tell). You can do "git status" and "git diff" to see what the local changes are.
> > 
> > It is much better if bugs are reported against "clean" builds of MIDAS (no "-dirty").
> > 
> > 
> > K.O.
> > 
> > 
> > <p>&nbsp;</p>
> > 
> > <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> > 	<tbody>
> > 		<tr>
> > 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> > 		</tr>
> > 		<tr>
> > 			<td style="background-color:#FFFFB0">
> > 			<p>MIDAS version: &nbsp; &nbsp; &nbsp;2.1<br />
> > 			GIT revision: &nbsp; &nbsp; &nbsp; Tue Dec 31 17:40:14 2019 +0100 - 
> > midas-2019-09-i-1-gd93944ce-dirty on branch develop</p>
> > 
> > 			<p>/Logger/Channels/0/Settings</p>
> > 
> > 			<table border="3" cellpadding="1" class="dialogTable">
> > 				<tbody>
> > 					<tr>
> > 						<td>ODB dump</td>
> > 						<td>y</td>
> > 					</tr>
> > 					<tr>
> > 						<td>ODB dump format</td>
> > 						<td>json</td>
> > 					</tr>
> > 				</tbody>
> > 			</table>
> > 
> > 			<p>With the settings above the file last.json generated for a new run is 
> > empty and the events 0x8000 and 0x8001 are missing in the .mid file.</p>
> > 
> > 			<p>When setting &quot;ODB dump format&quot; to &quot;xml&quot;, events 
> > 0x8000 and 0x8001 are included in the .mid file, however, the file last.xml is not created.
> > </p>
> > 
> > 			<p>&nbsp;</p>
> > 
> > 			<p>&nbsp;</p>
> > 
> > 			<p>&nbsp;</p>
> > 			</td>
> > 		</tr>
> > 	</tbody>
> > </table>
> > 
> > <p>&nbsp;</p>