Hi all, 

I have updated our daq computer to the latest OS 10.15 with the idea that then I will get all our daq 
programs running and will not need to update again in years to come. 
But now I am running unto even more problems trying to compile Midas. 
OS 10.15.3, Xcode 11.3 with commad line tools 

Darrens-Mac-mini:~ betacage$ cd packages/midas/build/
Darrens-Mac-mini:build betacage$ cmake .. 
-- MIDAS: cmake version: 3.16.3
-- The C compiler identification is AppleClang 11.0.0.11000033
-- The CXX compiler identification is AppleClang 11.0.0.11000033
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
-- MIDAS: Found ROOT version 6.18/04
-- Found ZLIB: /usr/lib/libz.dylib  
-- MIDAS: Found ZLIB version 
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: MySQL not found
-- MIDAS: ODBC not found
-- MIDAS: SQLITE not found
-- MIDAS: nvidia-smi not found
-- Found Git: /usr/bin/git (found version "2.21.0 (Apple Git-122.2)") 
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 
-- MIDAS example/experiment: Found ROOT version 6.18/04
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/betacage/packages/midas/build
Darrens-Mac-mini:build betacage$ make install
[  1%] Building CXX object CMakeFiles/mfeo.dir/src/mfe.cxx.o
In file included from /Users/betacage/packages/midas/src/mfe.cxx:15:
In file included from /Users/betacage/packages/midas/include/mfe.h:13:
/Users/betacage/packages/midas/include/midas.h:159:10: fatal error: 'pthread.h' file not found
#include <pthread.h>
         ^~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/mfeo.dir/src/mfe.cxx.o] Error 1
make[1]: *** [CMakeFiles/mfeo.dir/all] Error 2


I guess that I am maybe the first one trying to install MIDAS in this OS, so I am willing to help as much as I 
can with getting this going. I have found a solution to this by downgrading Xcode to version 10 
https://stackoverflow.com/questions/58524715/failing-to-compile-a-c-application-under-macos-catalina-10-15
If you don't have other solutions I will try to do that tomorrow.

Thanks 

Berta 

> I have updated our daq computer to the latest OS 10.15 ...

FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?

> In file included from /Users/betacage/packages/midas/include/mfe.h:13:
> /Users/betacage/packages/midas/include/midas.h:159:10: fatal error: 'pthread.h' file not found
> #include <pthread.h>
>          ^~~~~~~~~~~

Hmm... pthread.h should be there, I do not see any notice of it's removal in macos.

So I suspect a mis-installation of c++ compilers. In the past we had problems with this,
in addition to installing Xcode from the app store, one most install some missing stuff
manually ("XCode command line tools"), if I have it right, the command is "xcode-select --install".

If you figure out how to build it, I think midas will work just fine on the latest macos.

K.O.

> > I have updated our daq computer to the latest OS 10.15 ...
> 
> FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?

No, I'm stuck with 10.14 due to the current lack of 64-bit support of some programs.

I would try KOs proposal to do a "xcode-select --install".

Stefan

> > > I have updated our daq computer to the latest OS 10.15 ...
> > 
> > FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?
> 
> No, I'm stuck with 10.14 due to the current lack of 64-bit support of some programs.
> 

Ok, in this case, I will update my office mac mini to 10.15.

K.O.

> 
> Ok, in this case, I will update my office mac mini to 10.15.
> 
> K.O.


Hi Konstantin, 

Any luck with Midas in OS 10.15? 
I have downgraded to Xcode 10.2.1 as suggested in the post linked in my first post, without any luck, I keep getting the same pthead.h error. 
MY gcc uses the 10.15 sdk despite having v 10.2.1 for Xcode and command line tools 

Darrens-Mac-mini:~ betacage$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-
dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin19.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin


and that is where I suspect the problem is. Will keep investigating next week. Thanks for your help.

Berta 

> Any luck with Midas in OS 10.15? 

Best I can tell, the problem is not in midas: pthread.h should be there, somewhere.

K.O.

> > Any luck with Midas in OS 10.15? 
> 
> Best I can tell, the problem is not in midas: pthread.h should be there, somewhere.
> 
> K.O.


HI Konstain, 

Thanks for your reply. I have been investigating this issue of pthread in OS 10.15.  Found this forum post 
https://github.com/wjakob/nori/issues/16

that seem to suggest that pthread.h is in a different directory from OS 10.14 and on. And the end of the post the 
developer suggests that he has found a way to fix the Cmake to work so I have checked his code updates in here 
https://github.com/wjakob/nori/commit/be46cccc01a75b21dad1a3f61baa108fe644fc4b

I have added his lines 


if(APPLE)
   # Try to auto-detect a suitable SDK
   execute_process(COMMAND bash -c "xcodebuild -version -sdk | grep MacOSX | grep Path | head -n 1 | cut -f 2 -
d ' '" OUTPUT_VARIABLE CMAKE_OSX_SYSROOT)
   string(REGEX REPLACE "(\r?\n)+$" "" CMAKE_OSX_SYSROOT "${CMAKE_OSX_SYSROOT}")
   string(REGEX REPLACE "^.*X([0-9.]*).sdk$" "\\1" CMAKE_OSX_DEPLOYMENT_TARGET 
"${CMAKE_OSX_SYSROOT}")
 endif()

to my local pakages/midas/CMakeLists.txt and run cmake .., make install again but now I get a different error 

[  1%] Building CXX object CMakeFiles/mfeo.dir/src/mfe.cxx.o
clang: error: invalid version number in '-mmacosx-version-
min=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/DriverKit19.0.sdk'
clang: warning: using sysroot for 'DriverKit' but targeting 'MacOSX' [-Wincompatible-sysroot]


I will continue investigating this issue tomorrow, but please let me know if you have any suggestions.

Also my version of cmake is 3.16.3

Thanks 

Berta 

For your reference, here on my MacOSX 10.14.6 with XCode 11.3.1 the pthread.h file is present in locations listed below.

Did you execute "xcode-select --install" ?


~$ locate pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h

> For your reference, here on my MacOSX 10.14.6 with XCode 11.3.1 the pthread.h file is present in locations listed below.
> 
> Did you execute "xcode-select --install" ?
> 
> 
> ~$ locate pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread/pthread.h
> /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread.h
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread.h

Hi Stefan,

Thanks for looking into this. Yes, I have done  "xcode-select --install"

This is my output when I try to locate pthread.h

Darrens-Mac-mini:~ betacage$ locate pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread/pthread.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/pthread/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/pthread/pthread.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/pthread.h

and this is what I see in my SDKs folder 

Darrens-Mac-mini:SDKs betacage$ pwd
/Library/Developer/CommandLineTools/SDKs
Darrens-Mac-mini:SDKs betacage$ ls
MacOSX.sdk	MacOSX10.14.sdk	MacOSX10.15.sdk

So it looks like in this computer one needs to read the version of the SDK as pthread.h is not in MacOSX.sdk but in MacOSX10.15.sdk

Thanks 

Berta 

Another thought: Can you delete the midas build directory and run cmake again? Like

$ cd midas/build
$ rm -rf *
$ cmake ..
$ make VERBOSE=1

If you have the old build cache and upgraded your OS in meantime, the cache needs to be rebuild. The VERBOSE 
flag tells you the compiler options, and you see where the compiler looks for the SDK directory.

Cheers,
Stefan

> Another thought: Can you delete the midas build directory and run cmake again? Like
> 
> $ cd midas/build
> $ rm -rf *
> $ cmake ..
> $ make VERBOSE=1
> 
> If you have the old build cache and upgraded your OS in meantime, the cache needs to be rebuild. The VERBOSE 
> flag tells you the compiler options, and you see where the compiler looks for the SDK directory.
> 
> Cheers,
> Stefan

Hi Stefan, 

Thanks again for your reply. Yes! the suggestion of rebuilding the cache was the right one, this is the output of cmake ..

Darrens-Mac-mini:~ betacage$ cd packages/midas/build/
Darrens-Mac-mini:build betacage$ rm -rf *
Darrens-Mac-mini:build betacage$ cmake ..
-- MIDAS: cmake version: 3.16.3
-- The C compiler identification is AppleClang 11.0.0.11000033
-- The CXX compiler identification is AppleClang 11.0.0.11000033
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
-- MIDAS: Found ROOT version 6.18/04
-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.11") 
-- MIDAS: Found ZLIB version 1.2.11
-- Found OpenSSL: /usr/lib/libcrypto.dylib (found version "1.1.1d")  
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: MySQL not found
-- MIDAS: ODBC not found
-- MIDAS: SQLITE not found
-- MIDAS: nvidia-smi not found
-- Setting default build type to "RelWithDebInfo"
-- Found Git: /usr/bin/git (found version "2.21.1 (Apple Git-122.3)") 
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS example/experiment: Found ROOT version 6.18/04
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/betacage/packages/midas/build


Unfortunately running make VERBOSE=1 the compilation crashes, with the following error 

[ 39%] Linking CXX executable mhttpd
cd /Users/betacage/packages/midas/build/progs && /Applications/CMake.app/Contents/bin/cmake -E cmake_link_script CMakeFiles/mhttpd.dir/link.txt 
--verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -O2 -g -DNDEBUG -isysroot 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -Wl,-search_paths_first -Wl,-
headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o CMakeFiles/mhttpd.dir/mongoose6.cxx.o CMakeFiles/mhttpd.dir/mgd.cxx.o 
CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o mhttpd  ../libmidas.a /usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz 
ld: cannot link directly with dylib/framework, your binary is not an allowed client of /usr/lib/libcrypto.dylib for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [progs/mhttpd] Error 1
make[1]: *** [progs/CMakeFiles/mhttpd.dir/all] Error 2
make: *** [all] Error 2


Libcrypto seems to be part of the OpenSSL, so I am back to the original problem again. I did install OpenSSL via MacPorts. 

I have found this tread regarding this problem. But have to go and pick up my kids from school, so I don't have time to investigate today.
https://stackoverflow.com/questions/58446253/xcode-11-ld-error-your-binary-is-not-an-allowed-client-of-usr-lib-libcrypto-dy

Thanks again for staying with me and for responding so promptly. 

Berta 

Now you are stuck with openssl, which is optional for mhttpd. If you only use mhttpd locally, you maybe don't need SSL support. In that case you can jus do

[midas/build] $ cmake -D NO_SSL=1 ..

To disable that. If you do need SSL, maybe you can try to install openssl11 via mac ports.

Stefan

> Now you are stuck with openssl, which is optional for mhttpd. If you only use mhttpd locally, you maybe don't need SSL support. In that case you can jus do
> 
> [midas/build] $ cmake -D NO_SSL=1 ..
> 
> To disable that. If you do need SSL, maybe you can try to install openssl11 via mac ports.
> 
> Stefan

Thanks Stefan. 
If I run the compilation with the flag NO_SSL it works just fine. Still I think that Mhttpd is going to be important for us in the future as we may want to control the experiment remotely, so I will keep trying. But at least I 
can get started.

Thanks 

Berta 

> > [midas/build] $ cmake -D NO_SSL=1 ..
> If I run the compilation with the flag NO_SSL it works just fine. ...

FYI, the mongoose616 branch now has support for using the mbedtls https library,
this library seems to build easily from sources and removes our dependency
on where/how/which openssl library is installed. I hope to have this included
in the next release of midas.

K.O.

Thanks Konstantin,

I will keep an eye for the next release so that I can update my Midas to include ssl libraries.

Thanks 



> > > [midas/build] $ cmake -D NO_SSL=1 ..
> > If I run the compilation with the flag NO_SSL it works just fine. ...
> 
> FYI, the mongoose616 branch now has support for using the mbedtls https library,
> this library seems to build easily from sources and removes our dependency
> on where/how/which openssl library is installed. I hope to have this included
> in the next release of midas.
> 
> K.O.

Dear all,

we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:

// event 1
dma_buf[0] = 0x00000001; // Trigger and Event ID
dma_buf[1] = 0x00000001; // Serial number
dma_buf[2] = TIME; // time
dma_buf[3] = 18*4-4*4; // event size
dma_buf[4] = 18*4-6*4; // all bank size
dma_buf[5] = 0x11; // flags
// bank 0
dma_buf[6] = 0x46454230; // bank name
dma_buf[7] = 0x6; // bank type TID_DWORD
dma_buf[8] = 0x3*4; // data size
dma_buf[9] = 0xAFFEAFFE; // data
dma_buf[10] = 0xAFFEAFFE; // data
dma_buf[11] = 0xAFFEAFFE; // data
// bank 1
dma_buf[12] = 0x1; // bank name
dma_buf[12] = 0x46454231; // bank name
dma_buf[13] = 0x6; // bank type TID_DWORD
dma_buf[14] = 0x3*4; // data size
dma_buf[15] = 0xAFFEAFFE; // data
dma_buf[16] = 0xAFFEAFFE; // data
dma_buf[17] = 0xAFFEAFFE; // data

// event 2
.....

dma_buf[fpga_pointer] = 0xXXXXXXXX;


And we do something like:

while{true}
   // obtain buffer space
   status = rb_get_wp(rbh, (void **)&pdata, 10);
   fpga_pointer = fpga.read_last_data_add();

   wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
   copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
   rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte

   last_fpga_pointer = fpga_pointer;

Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 

The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.

Remark: dma_buf is volatile so memcpy is not possible here.

Cheers,
Marius

The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:

1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. It's your responsibility to not write more bytes into pdata then 
what you specified as max event size in rb_create()

2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I don't see this behaviour in your code. You would need something 
like

do {
   status = rb_get_wp(rbh, (void **)&pdata, 10);
   if (status == DB_TIMEOUT)
      ss_sleep(10);
   } while (status == DB_TIMEOUT);

Best,
Stefan


> Dear all,
> 
> we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:
> 
> // event 1
> dma_buf[0] = 0x00000001; // Trigger and Event ID
> dma_buf[1] = 0x00000001; // Serial number
> dma_buf[2] = TIME; // time
> dma_buf[3] = 18*4-4*4; // event size
> dma_buf[4] = 18*4-6*4; // all bank size
> dma_buf[5] = 0x11; // flags
> // bank 0
> dma_buf[6] = 0x46454230; // bank name
> dma_buf[7] = 0x6; // bank type TID_DWORD
> dma_buf[8] = 0x3*4; // data size
> dma_buf[9] = 0xAFFEAFFE; // data
> dma_buf[10] = 0xAFFEAFFE; // data
> dma_buf[11] = 0xAFFEAFFE; // data
> // bank 1
> dma_buf[12] = 0x1; // bank name
> dma_buf[12] = 0x46454231; // bank name
> dma_buf[13] = 0x6; // bank type TID_DWORD
> dma_buf[14] = 0x3*4; // data size
> dma_buf[15] = 0xAFFEAFFE; // data
> dma_buf[16] = 0xAFFEAFFE; // data
> dma_buf[17] = 0xAFFEAFFE; // data
> 
> // event 2
> .....
> 
> dma_buf[fpga_pointer] = 0xXXXXXXXX;
> 
> 
> And we do something like:
> 
> while{true}
>    // obtain buffer space
>    status = rb_get_wp(rbh, (void **)&pdata, 10);
>    fpga_pointer = fpga.read_last_data_add();
> 
>    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
>    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
>    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> 
>    last_fpga_pointer = fpga_pointer;
> 
> Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> 
> The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> 
> Remark: dma_buf is volatile so memcpy is not possible here.
> 
> Cheers,
> Marius

Hello Stefan,
is there a difference for the later data processing (after writing the ring buffer blocks)
if we write single events or multiple in one rb_get_wp - memcopy - rb_increment_wp cycle?
Both Marius and me have seen some inconsistencies in the number of events produced that is reported in the status page when writing multiple events in one go,
so I was wondering if this is due to us treating the buffer badly or the way midas handles the events after that.

Given that we produce the full event in our (FPGA) domain, an option would be to always copy one event from the dma to the midas-system buffer in a loop.
The question is if there is a difference (for midas) between
[pseudo code, much simplified]

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
    dma_read_index+=event_size;
    continue;   
  }
  copy_n(dma_buffer, pdata, event_size);
  rb_increment_wp(event_size);
  dma_read_index+=event_size;
} 

and

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
     ...
  };
  total_size=max_n_events_that_fit_in_rb_block();
  copy_n(dma_buffer, pdata, total_size);
  rb_increment_wp(total_size);
  dma_read_index+=total_size;
}

Cheers,
Konrad

> The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:
> 
> 1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. It's your responsibility to not write more bytes into pdata then 
> what you specified as max event size in rb_create()
> 
> 2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
> function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I don't see this behaviour in your code. You would need something 
> like
> 
> do {
>    status = rb_get_wp(rbh, (void **)&pdata, 10);
>    if (status == DB_TIMEOUT)
>       ss_sleep(10);
>    } while (status == DB_TIMEOUT);
> 
> Best,
> Stefan
> 
> 
> > Dear all,
> > 
> > we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:
> > 
> > // event 1
> > dma_buf[0] = 0x00000001; // Trigger and Event ID
> > dma_buf[1] = 0x00000001; // Serial number
> > dma_buf[2] = TIME; // time
> > dma_buf[3] = 18*4-4*4; // event size
> > dma_buf[4] = 18*4-6*4; // all bank size
> > dma_buf[5] = 0x11; // flags
> > // bank 0
> > dma_buf[6] = 0x46454230; // bank name
> > dma_buf[7] = 0x6; // bank type TID_DWORD
> > dma_buf[8] = 0x3*4; // data size
> > dma_buf[9] = 0xAFFEAFFE; // data
> > dma_buf[10] = 0xAFFEAFFE; // data
> > dma_buf[11] = 0xAFFEAFFE; // data
> > // bank 1
> > dma_buf[12] = 0x1; // bank name
> > dma_buf[12] = 0x46454231; // bank name
> > dma_buf[13] = 0x6; // bank type TID_DWORD
> > dma_buf[14] = 0x3*4; // data size
> > dma_buf[15] = 0xAFFEAFFE; // data
> > dma_buf[16] = 0xAFFEAFFE; // data
> > dma_buf[17] = 0xAFFEAFFE; // data
> > 
> > // event 2
> > .....
> > 
> > dma_buf[fpga_pointer] = 0xXXXXXXXX;
> > 
> > 
> > And we do something like:
> > 
> > while{true}
> >    // obtain buffer space
> >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> >    fpga_pointer = fpga.read_last_data_add();
> > 
> >    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
> >    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
> >    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> > 
> >    last_fpga_pointer = fpga_pointer;
> > 
> > Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> > 
> > The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> > 
> > Remark: dma_buf is volatile so memcpy is not possible here.
> > 
> > Cheers,
> > Marius

rb_xxx functions are midas event agnostic. The receiving side in mfe.cxx (lines 1418 in receive_trigger_event) however pulls one event at a time. If you 
have some inconsistency I would put some debugging code there.

Stefan

> Hello Stefan,
> is there a difference for the later data processing (after writing the ring buffer blocks)
> if we write single events or multiple in one rb_get_wp - memcopy - rb_increment_wp cycle?
> Both Marius and me have seen some inconsistencies in the number of events produced that is reported in the status page when writing multiple 
events in one go,
> so I was wondering if this is due to us treating the buffer badly or the way midas handles the events after that.
> 
> Given that we produce the full event in our (FPGA) domain, an option would be to always copy one event from the dma to the midas-system buffer 
in a loop.
> The question is if there is a difference (for midas) between
> [pseudo code, much simplified]
> 
> while(dma_read_index < last_dma_write_index){
>   if(rb_get_wp(pdata)!=SUCCESS){
>     dma_read_index+=event_size;
>     continue;   
>   }
>   copy_n(dma_buffer, pdata, event_size);
>   rb_increment_wp(event_size);
>   dma_read_index+=event_size;
> } 
> 
> and
> 
> while(dma_read_index < last_dma_write_index){
>   if(rb_get_wp(pdata)!=SUCCESS){
>      ...
>   };
>   total_size=max_n_events_that_fit_in_rb_block();
>   copy_n(dma_buffer, pdata, total_size);
>   rb_increment_wp(total_size);
>   dma_read_index+=total_size;
> }
> 
> Cheers,
> Konrad
> 
> > The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:
> > 
> > 1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection 
any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. 
It's your responsibility to not write more bytes into pdata then 
> > what you specified as max event size in rb_create()
> > 
> > 2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the 
producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. 
If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
> > function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. 
Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I 
don't see this behaviour in your code. You would need something 
> > like
> > 
> > do {
> >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> >    if (status == DB_TIMEOUT)
> >       ss_sleep(10);
> >    } while (status == DB_TIMEOUT);
> > 
> > Best,
> > Stefan
> > 
> > 
> > > Dear all,
> > > 
> > > we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA 
sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The 
buffer looks something like:
> > > 
> > > // event 1
> > > dma_buf[0] = 0x00000001; // Trigger and Event ID
> > > dma_buf[1] = 0x00000001; // Serial number
> > > dma_buf[2] = TIME; // time
> > > dma_buf[3] = 18*4-4*4; // event size
> > > dma_buf[4] = 18*4-6*4; // all bank size
> > > dma_buf[5] = 0x11; // flags
> > > // bank 0
> > > dma_buf[6] = 0x46454230; // bank name
> > > dma_buf[7] = 0x6; // bank type TID_DWORD
> > > dma_buf[8] = 0x3*4; // data size
> > > dma_buf[9] = 0xAFFEAFFE; // data
> > > dma_buf[10] = 0xAFFEAFFE; // data
> > > dma_buf[11] = 0xAFFEAFFE; // data
> > > // bank 1
> > > dma_buf[12] = 0x1; // bank name
> > > dma_buf[12] = 0x46454231; // bank name
> > > dma_buf[13] = 0x6; // bank type TID_DWORD
> > > dma_buf[14] = 0x3*4; // data size
> > > dma_buf[15] = 0xAFFEAFFE; // data
> > > dma_buf[16] = 0xAFFEAFFE; // data
> > > dma_buf[17] = 0xAFFEAFFE; // data
> > > 
> > > // event 2
> > > .....
> > > 
> > > dma_buf[fpga_pointer] = 0xXXXXXXXX;
> > > 
> > > 
> > > And we do something like:
> > > 
> > > while{true}
> > >    // obtain buffer space
> > >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> > >    fpga_pointer = fpga.read_last_data_add();
> > > 
> > >    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
> > >    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
> > >    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> > > 
> > >    last_fpga_pointer = fpga_pointer;
> > > 
> > > Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also 
increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> > > 
> > > The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. 
So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> > > 
> > > Remark: dma_buf is volatile so memcpy is not possible here.
> > > 
> > > Cheers,
> > > Marius

> rb_xxx functions are midas event agnostic. The receiving side in mfe.cxx (lines 1418 in receive_trigger_event) however pulls one event at a time. If you 
> have some inconsistency I would put some debugging code there.

I agree with Stefan, I do not think there is any bugs in the ring buffer code.

But. I do not think we ever did DMA the data directly into the ring buffer. Hmm...

I just checked, this is what we do (and this worked in the ALPHA Si-strip DAQ system for 10 years now):

- mfe.cxx multithread equipment
- mfe readout thread grabs pointer from ring buffer
- mfe creates event headers, etc
- calls our read_event() function
- creates data bank
- DMA data into the data bank (this is the DMA from VME block reads, using DMA controller inside the UniverseII and tsi148 VME-to-PCI bridges)
- close data bank
- return to mfe
- mfe readout thread increments the ring buffer
- mfe main thread grabs events from ring buffer, sends them to the mserver

So there could be trouble:
a) the ring buffer code does not have the required "volatile" (ahem, "atomic") annotations, so DMA may have a bad interaction with compiler optimizations (values stored in registers 
instead of in memory, etc)
b) the DMA driver must doctor the memory settings to (1) mark the DMA target memory uncachable or (1b) invalidate the cache after DMA completes, (2) mark the DMA target 
memory unswappable.

So I see possibilities for the ring buffer to malfunction.

But now I am curious, which DMA controller you use? The Altera or Xilinx PCIe block with the vendor supplied DMA driver? Or you do DMA on an ARM SoC FPGA? (no PCI/PCIe, 
different DMA controller, different DMA driver).

I am curious because we will be implementing pretty much what you do on ARM SoC FPGAs pretty soon, so good to know
if there is trouble to expect.

But I will probably use the tmfe.h c++ frontend and a "pure c++" ring buffer instead of mfe.cxx and the midas "rb" ring buffer.

(I did not look at your code at all, there could be a bug right there, this ring buffer stuff is tricky. With luck there is no bug
in your dma driver. The dma drivers for our vme bridges did do have bugs).

K.O.

We also agree and found the problem now. Since we build everything (MIDAS Event Header, Bank Header, Banks etc.) in the FPGA we had some struggle with the MIDAS data format (http://lmu.web.psi.ch/docu/manuals/bulk_manuals/software/midas195/html/AppendixA.html). We thought that only the MIDAS Event needs to be aligned to 64 bit but as it turned out also the bank data (Stefan updated the wiki page already) needs to be aligned. Since we are using the BANK32 it was a bit unclear for us since the bank header is not 64 bit aligned. But we managed this now by adding empty data and the system is running now.

Our setup looks like this:

Software:
- mfe.cxx multithread equipment
- mfe readout thread grabs pointer from dma ring buffer 
- since the dma buffer is volatile we do copy_n for transforming the data to MIDAS 
- the data is already in the MIDAS format so done from our side :)
- mfe readout thread increments the ring buffer
- mfe main thread grabs events from ring buffer, sends them to the mserver

Firmware:
- Arria 10 development board
- Altera PCIe block
- Own DMA engine since we are doing burst writing DMA with PCIe 3.0. 
- Own device driver
- no interrupts 

If you have more questions fell free to ask.

Actually the cause of all of the is a real bug in the midas functions. We want each bank 8-byte aligned, so there is code in bk_close like:

midas.cxx:14788:
     ((BANK_HEADER *) event)->data_size += sizeof(BANK32) + ALIGN8(pbk32->data_size);

While the old sizeof(BANK)=8, the extended sizeof(BANK32)=12, so not 8-byte aligned. This code should rather be:

     ((BANK_HEADER *) event)->data_size += ALIGN8(sizeof(BANK32) +pbk32->data_size);

But if we change that, it would break every midas data file on this planet!

The only chance I see is to use the "flags" in the BANK_HEADER to distinguish a current bank from a "correct" bank. 
So we could introduce a flag BANK_FORMAT_ALIGNED which distinguishes between the two pieces of code above. 
Then bk_iterate32 would look at that flag and do the right thing.

Any thoughts?

Best,
Stefan

Hi, Stefan - is this our famous 64-bit misalignement? Where we have each alternating bank aligned and misaligned at 64 bits? Without changing the data
format, one can always store data in 64-bit aligned banks by inserting a dummy banks between real banks:

event header
bank header
bank1 --- 64-bit aligned --- with data
bank2 --- misaligned, no data
bank3 --- 64-bit aligned --- with data
bank4 --- misaligned, no data
...

for sure, wastes space for bank2, bank4, etc, but at 12 bytes per bank, maybe this is negligible overhead compared to total event size.

BTW, aligned-to-64-bit is old news. The the PWB FPGA, I have 128-bit data paths to DDR RAM, the data has to be aligned to 128 bits, or else!

K.O.



> Actually the cause of all of the is a real bug in the midas functions. We want each bank 8-byte aligned, so there is code in bk_close like:
> 
> midas.cxx:14788:
>      ((BANK_HEADER *) event)->data_size += sizeof(BANK32) + ALIGN8(pbk32->data_size);
> 
> While the old sizeof(BANK)=8, the extended sizeof(BANK32)=12, so not 8-byte aligned. This code should rather be:
> 
>      ((BANK_HEADER *) event)->data_size += ALIGN8(sizeof(BANK32) +pbk32->data_size);
> 
> But if we change that, it would break every midas data file on this planet!
> 
> The only chance I see is to use the "flags" in the BANK_HEADER to distinguish a current bank from a "correct" bank. 
> So we could introduce a flag BANK_FORMAT_ALIGNED which distinguishes between the two pieces of code above. 
> Then bk_iterate32 would look at that flag and do the right thing.
> 
> Any thoughts?
> 
> Best,
> Stefan

> Hi, Stefan - is this our famous 64-bit misalignement? Where we have each alternating bank aligned and misaligned at 64 bits? Without changing the data
> format, one can always store data in 64-bit aligned banks by inserting a dummy banks between real banks:
> 
> event header
> bank header
> bank1 --- 64-bit aligned --- with data
> bank2 --- misaligned, no data
> bank3 --- 64-bit aligned --- with data
> bank4 --- misaligned, no data
> ...
> 
> for sure, wastes space for bank2, bank4, etc, but at 12 bytes per bank, maybe this is negligible overhead compared to total event size.
> 
> BTW, aligned-to-64-bit is old news. The the PWB FPGA, I have 128-bit data paths to DDR RAM, the data has to be aligned to 128 bits, or else!

Ok, so what about the following: When we do a bk_init32, we add a parameter "alignment", which might be 1,4,8,16 and "old". We store this alignment in the bank header, so the 
decoding works correctly. Now "old" means the current encoding, which is screwed up and produces the results you mention above, but we have to keep it (actually make it the 
default!) for backward compatibility. But then we can ask for 64-bit alignment or even 128-bit alignment if that helps the DAQ speed.

The only problem I see is if one writes data with the new library using 128-bit alignment for example, and wants to read it back with old code. Then it would explode. So if we 
make this modification, we have to announce it carefully and also adjust all ROOTANA & Co libraries to read back any midas data.

Stefan

> We also agree and found the problem now.

Good. what was wrong?

> - Own DMA engine since we are doing burst writing DMA with PCIe 3.0. 
> - Own device driver

Scary stuff.

> - no interrupts 

Right. Best I can tell, interrupts no longer useful in Linux - interrupt handler cannot do any real work, has to hand off to a kernel thread, resulting
in so much latency and overhead that one might as well poll for the data... And for DMA data transfers, the data rate is well known,
so easy to predict how long the DMA will run for and sleep for that amount of time instead of waiting for an interrupt.

K.O.

We also agree and found the problem now. Since we build everything (MIDAS Event Header, Bank Header, Banks etc.) in the FPGA we had some struggle with the MIDAS data format (http://lmu.web.psi.ch/docu/manuals/bulk_manuals/software/midas195/html/AppendixA.html). We thought that only the MIDAS Event needs to be aligned to 64 bit but as it turned out also the bank data (Stefan updated the wiki page already) needs to be aligned. Since we are using the BANK32 it was a bit unclear for us since the bank header is not 64 bit aligned. But we managed this now by adding empty data and the system is running now.

Our setup looks like this:

- mfe.cxx multithread equipment
- mfe readout thread grabs pointer from dma ring buffer 
- since the dma buffer is volatile we do copy_n for transforming the data to MIDAS 
- the data is already in the MIDAS format so done from our side :)
- mfe readout thread increments the ring buffer
- mfe main thread grabs events from ring buffer, sends them to the mserver

From the firmware side we have an Arria 10 development board and 

But now I am curious, which DMA controller you use? The Altera or Xilinx PCIe block with the vendor supplied DMA driver? Or you do DMA on an ARM SoC FPGA? (no PCI/PCIe, 
different DMA controller, different DMA driver).

I am curious because we will be implementing pretty much what you do on ARM SoC FPGAs pretty soon, so good to know
if there is trouble to expect.

But I will probably use the tmfe.h c++ frontend and a "pure c++" ring buffer instead of mfe.cxx and the midas "rb" ring buffer.

(I did not look at your code at all, there could be a bug right there, this ring buffer stuff is tricky. With luck there is no bug
in your dma driver. The dma drivers for our vme bridges did do have bugs).

K.O.

Dear all,

we are trying to build Midas events on FPGAs and send them directly to the midas
ring buffer via copy_n. According to the wiki
https://midas.triumf.ca/MidasWiki/index.php/Event_Structure Event Data Size is:
"The event data size contains the size of the event in bytes excluding the
header." and All Bank Size is: "Size in bytes of the following data plus the
size of the bank header". So are they actually the same or what header is the
header in the first sentence also the bank header?

Cheers,
Marius

 

Thanks for pointing out this error. The "All Bank Size" contains the size of all banks including their 
bank headers, but NOT the global bank header itself. I modified the documentation accordingly.

If you want to study the C code which tells you how to fill these headers, look at midas.cxx line 
14788.

Best,
Stefan

> Dear all,
> 
> we are trying to build Midas events on FPGAs and send them directly to the midas
> ring buffer via copy_n. According to the wiki
> https://midas.triumf.ca/MidasWiki/index.php/Event_Structure Event Data Size is:
> "The event data size contains the size of the event in bytes excluding the
> header." and All Bank Size is: "Size in bytes of the following data plus the
> size of the bank header". So are they actually the same or what header is the
> header in the first sentence also the bank header?
> 
> Cheers,
> Marius
> 
>  

> Thanks for pointing out this error. The "All Bank Size" contains the size of all banks including their 
> bank headers, but NOT the global bank header itself. I modified the documentation accordingly.
> 
> If you want to study the C code which tells you how to fill these headers, look at midas.cxx line 
> 14788.

Also take a look at the midas event parser in ROOTANA midasio.cxx, the code is pretty clean c++
https://bitbucket.org/tmidas/rootana/src/master/libMidasInterface/midasio.cxx

But Stefan's code in midas.cxx and in the documentation is the authoritative information.

K.O.

Hi,
for some reason we occasionally get JSON errors in the browser when accessing MIDAS. It is then not possible to open a new window or tab, see attachment. The unexpected token is \0x6, so the acknowledge symbol.
If this happens, then all "alive sessions" keep being usable despite error messages, but show similar error messages:
>RPC Error
>json parser exception: SyntaxError: JSON.parse: bad control character in string literal at line 80 column 30 of the JSON data, method: "db_get_valus", params: [object Object], id: 1582020074098.

Do you have any idea why db_get_values yields ACK or other control characters?

Thanks

You are the first one reporting this error, so it must be due to your values in the ODB. Can you track it down to specific ODB contents? If so, can you post it so that I can reproduce your error?

Stefan

> The unexpected token is \0x6
> RPC Error json parser exception: SyntaxError: JSON.parse: bad control character in string literal at line 80 column 30 of the JSON data, method: "db_get_valus", params: [object Object], id: 1582020074098.

Yes, there is a problem.

Traditionally, midas strings in ODB have no restriction on the content (I think even the '\0x0' char is permitted).

But web browser javascript strings are supposed to be valid unicode (UTF-16, if I read this right: https://tc39.es/ecma262/#sec-ecmascript-language-types-string-type).

The collision between the two happens when ODB values are json-encoded by midas, then json-decoded by the web browser.

The midas json encoder (mjson.h, mjson.cxx) encodes ODB strings according to JSON rules, but does not ensure that the result is valid UTF-8. (valid UTF-8 is not required, if I read the specs correctly http://www.ecma-
international.org/publications/files/ECMA-ST/ECMA-404.pdf and https://www.json.org/json-en.html)

The web browser json decoder requires valid UTF-8 and throws exceptions if it does not like something. Different browsers it slightly differently, so we have an error handler for this in the mjsonrpc results processor.

What does this mean in practice?

Now that MIDAS is very web oriented, MIDAS strings must be web browser friendly, too:

a) all ODB key names (subdirectory names, link names, etc) must be UTF-8 unicode, and this has been enforced by ODB for some time now.
b) all ODB string values must be valid UTF-8 unicode. This is not enforced right now.

Historically, it was okey to use ODB TID_STRING to store arbitrary binary data, but now, I think, we must deprecate this,
at least for any ODB entries that could be returned to a web browser (which means all of them, after we implement a fully
html+javascript odb editor). For storing binary data, arrays of TID_CHAR, TID_DWORD & co are probably a better match.

The MIDAS and ROOTANA json decoders (the same mjson.h, mjson.cxx) do not care about UTF-8, so ODB dumps
in JSON format are not affected by any of this. (But I am not sure about the JSON decoder in ROOT).

Bottom line:

I think db_validate() should check for invalid UTF-8 in ODB key names and in TID_STRING values
and at least warn the user. (I am not sure if invalid UTF-8 can be fixed automatically). db_create()
should reject key names that are not valid UTF-8 (it already does this, I think). db_set_value(TID_STRING) should
probably reject invalid UTF-8 strings, this needs to be discussed some more.

https://bitbucket.org/tmidas/midas/issues/215/everything-in-odb-must-be-valid-utf-8

K.O.

INT idle (EQUIPMENT * pequipment)

Hi, Giorgio - I think you encountered a fundamental problem with what to do at the begin of 
run. There are two ways of thinking about it.

Some experiments want to start the run as quickly as possible, so they do not want 
begin_of_run() to do too much stuff.

Other experiments want to record all the current settings and conditions before starting a 
run, their begin_of_run() will read all the slow controls, interrogate all the power supplies, 
read all the voltages, temperatures, pressures, etc. By necessity this will slow down the 
starting of the run quite significantly.

The best I understand the midas class driver structure, it is more geared for the first case - 
fast starting of runs.

The thinking behind this choice considers the nature of most slow control data in typical 
physics experiments:
- if the data does not change quickly (say, room temperature, atmospheric pressure, etc), 
and you read it say every 1 minute, then you do not need to read it again at begin run time - 
the 1 minute old measurement is still good enough - nothing changed much since then
- if the opposite is true, the data changes wildly (i.e. detector high voltage current goes up 
and down in response to the quickly changing beam current), measuring it at the start of 
the run does us no good - by the time the first event comes around, it has already changed 
completely.

Hopefully Stefan can help you with your specific problem, he has better understanding of 
the midas class drivers.


K.O.


[quote="Pintaudi Giorgio"]Hello!
As you know, the generic MIDAS frontend has a class driver, device driver, bus driver
structure. Assuming a slow device frontend, its class driver should have a routine of type   
[CODE]INT idle (EQUIPMENT * pequipment)[/CODE]
This routine is called with a rate controlled by the 
"[I]/Equipment/<frontend name>/Common/Event limit[/I]" parameter.
The idle routine usually reads one channel of the frontend and stores the results
in the "[I]/Equipment/<frontend name>/Variables[/I]" ODB folder.

[B]My question is: it is possible to force (from the code) the frontend to call the idle routine 
at a 
certain point. This is because I need to update the "[I]/Equipment/<frontend 
name>/Variables[/I]"
variables inside the "[I]begin_of_run[/I]" routine, at a very specific time.[/B]

One dirty solution would be to increase a lot the reading rate ... but I need this
increased reading rate only during the run start while I need a low reading rate
during the run. So the question: is it possible to increase and decrease the reading
rate (event limit) of a frontend without stopping and restarting it?

If you need more info, please let me know.
Thank you
Giorgio[/quote]

@@ -411,6 +411,17 @@ static INT register_equipment(void)
             ss_sleep(3000);
             return 0;
          }
+#ifdef WAGASCI_OPEN_ODB_HOTLINK
+         status = db_open_record(hDB, hKey, eq_info, sizeof(EQUIPMENT), MODE_READ,
+                                 nullptr, nullptr);
+         if (status != DB_SUCCESS) {
+            printf("ERROR: Cannot open hotlink with equipment record \"%s\", db_open_record() status %d\n",
+                   str, status);
+            cm_disconnect_experiment();
+            ss_sleep(3000);
+            return 0;
+         }
+#endif
       } else if (status == DB_STRUCT_MISMATCH) {
          cm_msg(MINFO, "register_equipment", "Correcting \"%s\", db_check_record() status %d", str, status);
          db_create_record(hDB, 0, str, EQUIPMENT_COMMON_STR);

• Every change to the equipment "/Equipment/<frontend name>/Common" is instantaneously applied (and might crash the frontend?)
  

INT idle (EQUIPMENT * pequipment)

> We had a case where a detector produced electrostatic discharges which only lasted for a second or so
> and we were happy to detect this in spikes in the HV current. With measurements of only one per minute
> we would not have realized that so quicky.

For the T2K/ND280 TPC we implemented something similar. The TPC uses MicroMegas detector which sparks during 
normal operation. We asked Wiener/ISEG to implement a "spark counting mode" for us (and they did). In this mode,
high voltage over-current (a micromegas spark) sets a special flag (does not trip the high voltage). Our midas frontend
reads this flag at rate about 1/min, if flag is set, clears it, increments the software spark counter, reads the flag again,
if the flag is still set (failed to clear), it means this was not a normal spark but a high voltage breakdown
and the offending channel is shut down. I believe this mode is still part of the ISEG normal firmware.

Because the Wiener/ISEG interface uses SNMP to "read all data in one operation", the MIDAS "device driver" structure
was not useful, the readout was a simple loop, the readout frequency was easy to control, and indeed,
we read the high voltage with increased frequency during ramping. This was easy to implement because we
did not have to fight the MIDAS "device driver" framework.

If you want a similar solution, talk to the device, interpret the data, record values to odb and history, generate
midas events - all without hand holding from (arm wrestling with the rest of) midas - I recommend
the new tmfe.h/tmfe.cxx c++ frontend - see the two examples in midas/progs/fetest_tmfe.cxx
and fetest_tmfe_thread.cxx (single-threaded and multi-threaded).

K.O.

Hi, Stefan & co - now that midas is c++11 and c++11 comes with a threads library, should we 
switch midas to use the c++11 threads instead of pthreads? (Of course on Linux c++11 
threads are a layer on top of pthreads, the best I know).

This should remove the dependency on pthreads.h and use a more native implementation of 
threads on MacOS and Windows. (again, the best I can tell).

Of course this depends on c++11 threads having all the functions we need. Specifically, "lock 
with timeout" is useful to deal with "gah! everything stopped! what do I do!", a problem 
bedeviling midas in the early days (and still happens today!). Current midas kills everything 
after 5 minutes of deadlock - then the user knows how to restart everything and the developer 
has core dumps to look at. (to see which program/thread holds the lock and would not give it 
up).

Any thoughts on this?

K.O.

I'm thinking of this already since some time, and it was part of my motivation switching to C++11.
I was delighted to see that what we do in system.c (encapsulate system functions such as threads
and shared memory) is now done natively in C++11, and it's done by experts and not amateurs like us.

I see that with std::timed_mutex and try_lock_for we have the desired timeout function. Pity that
C++11 does not contain inter-process communication like semaphores, so there we still have to use
our old functions. 

But for threads switching to std::thread, I'm all in.

Stefan

> Hi, Stefan & co - now that midas is c++11 and c++11 comes with a threads library, should we 
> switch midas to use the c++11 threads instead of pthreads? (Of course on Linux c++11 
> threads are a layer on top of pthreads, the best I know).
> 
> This should remove the dependency on pthreads.h and use a more native implementation of 
> threads on MacOS and Windows. (again, the best I can tell).
> 
> Of course this depends on c++11 threads having all the functions we need. Specifically, "lock 
> with timeout" is useful to deal with "gah! everything stopped! what do I do!", a problem 
> bedeviling midas in the early days (and still happens today!). Current midas kills everything 
> after 5 minutes of deadlock - then the user knows how to restart everything and the developer 
> has core dumps to look at. (to see which program/thread holds the lock and would not give it 
> up).
> 
> Any thoughts on this?
> 
> K.O.

> Hi, Stefan & co - now that midas is c++11 and c++11 comes with a threads library, should we 
> switch midas to use the c++11 threads instead of pthreads? (Of course on Linux c++11 
> threads are a layer on top of pthreads, the best I know).
> 
> This should remove the dependency on pthreads.h and use a more native implementation of 
> threads on MacOS and Windows. (again, the best I can tell).
> 
> Of course this depends on c++11 threads having all the functions we need. Specifically, "lock 
> with timeout" is useful to deal with "gah! everything stopped! what do I do!", a problem 
> bedeviling midas in the early days (and still happens today!). Current midas kills everything 
> after 5 minutes of deadlock - then the user knows how to restart everything and the developer 
> has core dumps to look at. (to see which program/thread holds the lock and would not give it 
> up).
> 
> Any thoughts on this?
> 
> K.O.

Hi, I just wanted to say that I have seen this post and maybe that is the solution to the pthreads compiler problem in OS 10.15, but 
of course I am a total amateur in here. Thanks for thinking about this and I will wait and hold to see what gets decided. Thanks

Berta  

Hi all, 

I have a Mac with OS 10.13.6 and Xcode 10.1. I am following the instructions in the wiki to install Midas. 
I have installed openssl via MacPorts as per the instructions. But then I get an error related to open ssl 
when I try to compile Midas. See the results of cmake .. and make install 

Darrens-Mac-mini:build betacage$ cmake ..
-- MIDAS: cmake version: 3.16.3
-- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
-- MIDAS: Found ROOT version 6.18/04
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: MySQL not found
-- MIDAS: ODBC not found
-- MIDAS: Found SQLITE /usr/include/sqlite3.h
-- MIDAS: nvidia-smi not found
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS example/experiment: Found ROOT version 6.18/04
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/betacage/packages/midas/build
Darrens-Mac-mini:build betacage$ make install 
[  1%] Built target mfeo
[  3%] Built target mfe
[  4%] Built target rmana
[  5%] Built target manao
[  6%] Built target mana
[  6%] Built target git_revision_h
[ 30%] Built target objlib
[ 31%] Built target midas
[ 32%] Built target rmanao
[ 32%] Built target objlib-c-compat
[ 32%] Built target midas-c-compat
[ 33%] Built target midas-shared
[ 35%] Built target rmlogger
[ 36%] Linking CXX executable mhttpd
Undefined symbols for architecture x86_64:
  "_OPENSSL_init_ssl", referenced from:
      _mg_mgr_init in mongoose6.cxx.o
  "_SSL_CTX_set_options", referenced from:
      _mg_set_ssl in mongoose6.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [progs/mhttpd] Error 1
make[1]: *** [progs/CMakeFiles/mhttpd.dir/all] Error 2
make: *** [all] Error 2


I hope that this is the right forum to submit this kind of reports.
Any idea what do I have to do to continue ? Thanks is advance !

Berta 

Hi, yes, this is the right place to report problems and to ask questions about midas.

As for your trouble, I have the same macos 10.13.6, so we should be able to figure out what goes wrong.

Unfortunately, the posted instructions (to run cmake directly) hide all the information needed to debug build problems.

Instead of "cd build; cmake ..", please run "make cmake" and post the output. Printed should be all the compiler
and linker commands executed by the build. You can examine the mhttpd linker command to see if somehow
the wrong openssl library is being used.

If you do not need built-in https support in mhttpd, you can run "make cmake NO_SSL=1".

K.O.

> Hi all, 
> 
> I have a Mac with OS 10.13.6 and Xcode 10.1. I am following the instructions in the wiki to install Midas. 
> I have installed openssl via MacPorts as per the instructions. But then I get an error related to open ssl 
> when I try to compile Midas. See the results of cmake .. and make install 
> 
> Darrens-Mac-mini:build betacage$ cmake ..
> -- MIDAS: cmake version: 3.16.3
> -- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
> -- MIDAS: Found ROOT version 6.18/04
> -- MIDAS: Found ZLIB version 1.2.11
> -- MIDAS: Found OpenSSL version 1.1.1d
> -- MIDAS: MySQL not found
> -- MIDAS: ODBC not found
> -- MIDAS: Found SQLITE /usr/include/sqlite3.h
> -- MIDAS: nvidia-smi not found
> -- MIDAS example/experiment: MIDAS in-tree-build
> -- MIDAS: Found ZLIB version 1.2.11
> -- MIDAS example/experiment: Found ROOT version 6.18/04
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /Users/betacage/packages/midas/build
> Darrens-Mac-mini:build betacage$ make install 
> [  1%] Built target mfeo
> [  3%] Built target mfe
> [  4%] Built target rmana
> [  5%] Built target manao
> [  6%] Built target mana
> [  6%] Built target git_revision_h
> [ 30%] Built target objlib
> [ 31%] Built target midas
> [ 32%] Built target rmanao
> [ 32%] Built target objlib-c-compat
> [ 32%] Built target midas-c-compat
> [ 33%] Built target midas-shared
> [ 35%] Built target rmlogger
> [ 36%] Linking CXX executable mhttpd
> Undefined symbols for architecture x86_64:
>   "_OPENSSL_init_ssl", referenced from:
>       _mg_mgr_init in mongoose6.cxx.o
>   "_SSL_CTX_set_options", referenced from:
>       _mg_set_ssl in mongoose6.cxx.o
> ld: symbol(s) not found for architecture x86_64
> clang: error: linker command failed with exit code 1 (use -v to see invocation)
> make[2]: *** [progs/mhttpd] Error 1
> make[1]: *** [progs/CMakeFiles/mhttpd.dir/all] Error 2
> make: *** [all] Error 2
> 
> 
> I hope that this is the right forum to submit this kind of reports.
> Any idea what do I have to do to continue ? Thanks is advance !
> 
> Berta 

I tried on my Mac (macOS 10.14.6, Xcode 11.3.1, current develop branch, openssl 1.1.1d) and it woks fine. Below is the transcript. I 
see that your cmake output is much shorter (no C compiler listed etc.). Did you remove some lines? For such comparisons, it's 
always good to start with an empty build directory.

OpenSSL libraries are the same (1.1.1d). Just for comparison, I run the build process with "make VERBOSE=1" and extract the line 
which fails for you when linking mhttp, so maybe you can compare.

Linking mhttpd
---------------

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -O2 -g -DNDEBUG -isysroot 
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-
min=10.14 -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o 
CMakeFiles/mhttpd.dir/mongoose6.cxx.o CMakeFiles/mhttpd.dir/mgd.cxx.o CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o 
mhttpd  -L/opt/local/lib/mysql57/mysql -lmysqlclient -lz ../libmidas.a /opt/local/lib/libssl.dylib /opt/local/lib/libcrypto.dylib -lz -
L/opt/local/lib/mysql57/mysql -lmysqlclient -lz 


Here is the full transcript
-------------------------

/midas/build$ cmake ..
-- MIDAS: cmake version: 3.16.3
-- The C compiler identification is AppleClang 11.0.0.11000033
-- The CXX compiler identification is AppleClang 11.0.0.11000033
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- 
works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- MIDAS: CMAKE_INSTALL_PREFIX: /midas
-- MIDAS: Found ROOT version 6.16/00
-- Found ZLIB: /opt/local/lib/libz.dylib (found version "1.2.11") 
-- MIDAS: Found ZLIB version 1.2.11
-- Found OpenSSL: /opt/local/lib/libcrypto.dylib (found version "1.1.1d")  
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: Found MySQL version 5.7.26
-- MIDAS: ODBC not found
-- MIDAS: SQLITE not found
-- MIDAS: nvidia-smi not found
-- Setting default build type to "RelWithDebInfo"
-- Found Git: /usr/bin/git (found version "2.21.1 (Apple Git-122.3)") 
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS example/experiment: Found ROOT version 6.16/00
-- Configuring done
-- Generating done
-- Build files have been written to: /midas/build
/midas/build$ make
Scanning dependencies of target rmana
Scanning dependencies of target git_revision_h
Scanning dependencies of target mfe
Scanning dependencies of target manao
Scanning dependencies of target mfeo
Scanning dependencies of target objlib-c-compat
Scanning dependencies of target mana
Scanning dependencies of target rmanao
[  0%] Building CXX object CMakeFiles/objlib-c-compat.dir/src/midas_c_compat.cxx.o
[  2%] Building CXX object CMakeFiles/mfe.dir/src/mfe.cxx.o
[  2%] Building CXX object CMakeFiles/mfeo.dir/src/mfe.cxx.o
[  3%] Building CXX object CMakeFiles/mana.dir/src/mana.cxx.o
[  4%] Building CXX object CMakeFiles/manao.dir/src/mana.cxx.o
[  5%] Building CXX object CMakeFiles/rmanao.dir/src/mana.cxx.o
[  6%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
[  6%] Built target git_revision_h
Scanning dependencies of target objlib
[  7%] Building CXX object CMakeFiles/objlib.dir/src/midas.cxx.o
[  7%] Built target objlib-c-compat
[  8%] Building CXX object CMakeFiles/objlib.dir/src/midas_cxx.cxx.o
[  8%] Linking CXX static library libmana.a
[  8%] Built target manao
[  8%] Building CXX object CMakeFiles/objlib.dir/src/odb.cxx.o
[  8%] Built target mana
[  9%] Building CXX object CMakeFiles/objlib.dir/src/device_driver.cxx.o
[  9%] Built target mfeo
[ 10%] Linking CXX static library libmfe.a
[ 11%] Building CXX object CMakeFiles/objlib.dir/src/system.cxx.o
[ 11%] Built target mfe
[ 11%] Building CXX object CMakeFiles/objlib.dir/src/alarm.cxx.o
[ 12%] Building CXX object CMakeFiles/objlib.dir/src/elog.cxx.o
[ 13%] Building CXX object CMakeFiles/objlib.dir/src/mrpc.cxx.o
[ 13%] Building CXX object CMakeFiles/objlib.dir/src/mjson.cxx.o
[ 14%] Building CXX object CMakeFiles/objlib.dir/src/tmfe.cxx.o
[ 15%] Building CXX object CMakeFiles/objlib.dir/src/mvodb.cxx.o
[ 15%] Built target rmanao
[ 15%] Linking CXX static library librmana.a
[ 16%] Building CXX object CMakeFiles/objlib.dir/src/nullodb.cxx.o
[ 16%] Building CXX object CMakeFiles/objlib.dir/src/midasodb.cxx.o
[ 16%] Built target rmana
[ 17%] Building CXX object CMakeFiles/objlib.dir/src/mxmlodb.cxx.o
[ 18%] Building CXX object CMakeFiles/objlib.dir/src/mjsonodb.cxx.o
[ 18%] Building CXX object CMakeFiles/objlib.dir/src/json_paste.cxx.o
[ 19%] Building CXX object CMakeFiles/objlib.dir/src/mjsonrpc.cxx.o
[ 20%] Building CXX object CMakeFiles/objlib.dir/src/mjsonrpc_user.cxx.o
[ 21%] Building CXX object CMakeFiles/objlib.dir/src/history.cxx.o
[ 21%] Building CXX object CMakeFiles/objlib.dir/src/history_common.cxx.o
[ 22%] Building CXX object CMakeFiles/objlib.dir/src/history_odbc.cxx.o
[ 23%] Building CXX object CMakeFiles/objlib.dir/src/history_schema.cxx.o
[ 23%] Building CXX object CMakeFiles/objlib.dir/src/lz4.cxx.o
[ 24%] Building CXX object CMakeFiles/objlib.dir/src/lz4frame.cxx.o
[ 25%] Building CXX object CMakeFiles/objlib.dir/src/lz4hc.cxx.o
[ 26%] Building CXX object CMakeFiles/objlib.dir/src/xxhash.cxx.o
[ 26%] Building CXX object CMakeFiles/objlib.dir/src/crc32c.cxx.o
[ 27%] Building CXX object CMakeFiles/objlib.dir/src/sha256.cxx.o
[ 28%] Building CXX object CMakeFiles/objlib.dir/src/sha512.cxx.o
[ 28%] Building CXX object CMakeFiles/objlib.dir/src/ftplib.cxx.o
[ 29%] Building CXX object CMakeFiles/objlib.dir/src/mdsupport.cxx.o
[ 30%] Building CXX object CMakeFiles/objlib.dir/mxml/mxml.cxx.o
[ 31%] Building CXX object CMakeFiles/objlib.dir/mxml/strlcpy.cxx.o
[ 31%] Built target objlib
Scanning dependencies of target midas-shared
Scanning dependencies of target midas
[ 33%] Linking CXX shared library libmidas-shared.dylib
[ 33%] Linking CXX static library libmidas.a
[ 33%] Built target midas-shared
[ 33%] Built target midas
Scanning dependencies of target mhttpd
Scanning dependencies of target feudp
Scanning dependencies of target odb_lock_test
Scanning dependencies of target rmlogger
Scanning dependencies of target mchart
Scanning dependencies of target msysmon
Scanning dependencies of target midas-c-compat
Scanning dependencies of target mfe_link_test
[ 33%] Building CXX object progs/CMakeFiles/mfe_link_test.dir/mfe_link_test.cxx.o
[ 33%] Linking CXX shared library libmidas-c-compat.dylib
[ 34%] Building CXX object progs/CMakeFiles/feudp.dir/feudp.cxx.o
[ 35%] Building CXX object progs/CMakeFiles/msysmon.dir/msysmon.cxx.o
[ 36%] Building CXX object progs/CMakeFiles/mchart.dir/mchart.cxx.o
[ 37%] Building CXX object progs/CMakeFiles/odb_lock_test.dir/odb_lock_test.cxx.o
[ 38%] Building CXX object progs/CMakeFiles/rmlogger.dir/mlogger.cxx.o
[ 39%] Building CXX object progs/CMakeFiles/mhttpd.dir/mhttpd.cxx.o
[ 39%] Built target midas-c-compat
[ 40%] Building CXX object progs/CMakeFiles/mhttpd.dir/mongoose6.cxx.o
[ 41%] Linking CXX executable mfe_link_test
[ 42%] Linking CXX executable odb_lock_test
[ 42%] Built target mfe_link_test
Scanning dependencies of target mjson_test
[ 43%] Building CXX object progs/CMakeFiles/mjson_test.dir/mjson_test.cxx.o
[ 43%] Linking CXX executable mchart
[ 43%] Built target odb_lock_test
Scanning dependencies of target odbinit
[ 44%] Building CXX object progs/CMakeFiles/odbinit.dir/odbinit.cxx.o
[ 45%] Linking CXX executable feudp
[ 45%] Built target mchart
[ 46%] Building CXX object progs/CMakeFiles/mhttpd.dir/mgd.cxx.o
[ 46%] Built target feudp
[ 46%] Building CXX object progs/CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o
[ 47%] Linking CXX executable mjson_test
[ 47%] Built target mjson_test
Scanning dependencies of target fetest_tmfe_thread
[ 47%] Building CXX object progs/CMakeFiles/fetest_tmfe_thread.dir/fetest_tmfe_thread.cxx.o
[ 47%] Linking CXX executable msysmon
[ 47%] Built target msysmon
[ 48%] Linking CXX executable odbinit
Scanning dependencies of target fetest_tmfe
[ 49%] Building CXX object progs/CMakeFiles/fetest_tmfe.dir/fetest_tmfe.cxx.o
[ 49%] Built target odbinit
[ 50%] Linking CXX executable fetest_tmfe_thread
Scanning dependencies of target mh2sql
[ 51%] Building CXX object progs/CMakeFiles/mh2sql.dir/mh2sql.cxx.o
Scanning dependencies of target odbhist
[ 51%] Building CXX object progs/CMakeFiles/odbhist.dir/odbhist.cxx.o
[ 51%] Built target fetest_tmfe_thread
Scanning dependencies of target melog
[ 51%] Building CXX object progs/CMakeFiles/melog.dir/melog.cxx.o
[ 52%] Linking CXX executable odbhist
[ 53%] Built target odbhist
[ 53%] Linking CXX executable melog
[ 54%] Linking CXX executable fetest_tmfe
Scanning dependencies of target mfe_link_test_cxx
[ 55%] Building CXX object progs/CMakeFiles/mfe_link_test_cxx.dir/mfe_link_test_cxx.cxx.o
[ 55%] Built target melog
Scanning dependencies of target crc32c_sum
[ 55%] Built target fetest_tmfe
Scanning dependencies of target odbedit
[ 56%] Building CXX object progs/CMakeFiles/crc32c_sum.dir/crc32c_sum.cxx.o
[ 56%] Building CXX object progs/CMakeFiles/odbedit.dir/odbedit.cxx.o
[ 57%] Linking CXX executable crc32c_sum
[ 58%] Linking CXX executable mh2sql
[ 58%] Built target crc32c_sum
[ 59%] Building CXX object progs/CMakeFiles/odbedit.dir/cmdedit.cxx.o
[ 59%] Linking CXX executable mfe_link_test_cxx
Scanning dependencies of target mdump
[ 59%] Built target mh2sql
Scanning dependencies of target mhdump
[ 60%] Building CXX object progs/CMakeFiles/mdump.dir/mdump.cxx.o
[ 60%] Building CXX object progs/CMakeFiles/mhdump.dir/mhdump.cxx.o
[ 60%] Built target mfe_link_test_cxx
Scanning dependencies of target lazylogger
[ 61%] Building CXX object progs/CMakeFiles/lazylogger.dir/lazylogger.cxx.o
Scanning dependencies of target mtransition
[ 61%] Building CXX object progs/CMakeFiles/mtransition.dir/mtransition.cxx.o
[ 62%] Linking CXX executable mdump
[ 63%] Linking CXX executable rmlogger
[ 63%] Built target mdump
Scanning dependencies of target mserver
[ 64%] Building CXX object progs/CMakeFiles/mserver.dir/mserver.cxx.o
[ 65%] Linking CXX executable mtransition
[ 65%] Built target rmlogger
Scanning dependencies of target mhist
[ 66%] Building CXX object progs/CMakeFiles/mhist.dir/mhist.cxx.o
[ 66%] Built target mtransition
Scanning dependencies of target get_record_test
[ 67%] Building CXX object progs/CMakeFiles/get_record_test.dir/get_record_test.cxx.o
[ 68%] Linking CXX executable mhdump
Scanning dependencies of target msequencer
[ 68%] Building CXX object progs/CMakeFiles/msequencer.dir/msequencer.cxx.o
[ 68%] Built target mhdump
Scanning dependencies of target fetest
[ 69%] Linking CXX executable odbedit
[ 70%] Building CXX object progs/CMakeFiles/fetest.dir/fetest.cxx.o
[ 70%] Built target odbedit
[ 70%] Linking CXX executable mserver
Scanning dependencies of target mstat
[ 70%] Linking CXX executable get_record_test
[ 71%] Building CXX object progs/CMakeFiles/mstat.dir/mstat.cxx.o
[ 71%] Built target mserver
Scanning dependencies of target mlogger
[ 71%] Built target get_record_test
[ 71%] Building CXX object progs/CMakeFiles/mlogger.dir/mlogger.cxx.o
Scanning dependencies of target analyzer
[ 71%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/analyzer.cxx.o
[ 71%] Linking CXX executable mhist
[ 71%] Linking CXX executable fetest
[ 72%] Linking CXX executable lazylogger
[ 72%] Built target mhist
[ 72%] Built target fetest
[ 73%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/adccalib.cxx.o
[ 74%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/adcsum.cxx.o
[ 74%] Built target lazylogger
Scanning dependencies of target frontend
[ 74%] Building CXX object examples/experiment/CMakeFiles/frontend.dir/frontend.cxx.o
[ 75%] Linking CXX executable mstat
Scanning dependencies of target mscb_fe
[ 76%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/mscb_fe.cxx.o
[ 76%] Built target mstat
[ 76%] Building CXX object examples/experiment/CMakeFiles/analyzer.dir/scaler.cxx.o
[ 77%] Linking CXX executable frontend
Scanning dependencies of target scfe
[ 77%] Built target frontend
[ 77%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/class/hv.cxx.o
[ 78%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/class/multi.cxx.o
[ 79%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/scfe.cxx.o
Scanning dependencies of target mtfe
[ 80%] Building CXX object examples/mtfe/CMakeFiles/mtfe.dir/mtfe.cxx.o
[ 81%] Linking CXX executable analyzer
[ 81%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/class/hv.cxx.o
[ 82%] Linking CXX executable mtfe
[ 82%] Built target analyzer
[ 83%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/device/nulldev.cxx.o
Scanning dependencies of target rpc_srvr
[ 83%] Built target mtfe
[ 84%] Building CXX object examples/lowlevel/CMakeFiles/rpc_srvr.dir/rpc_srvr.cxx.o
[ 85%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/class/multi.cxx.o
Scanning dependencies of target rpc_clnt
[ 85%] Building CXX object examples/lowlevel/CMakeFiles/rpc_clnt.dir/rpc_clnt.cxx.o
[ 85%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/bus/null.cxx.o
[ 85%] Linking CXX executable rpc_srvr
[ 85%] Built target rpc_srvr
[ 86%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/device/nulldev.cxx.o
[ 87%] Linking CXX executable rpc_clnt
[ 88%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/drivers/device/mscbdev.cxx.o
Scanning dependencies of target rpc_test
[ 89%] Building CXX object examples/lowlevel/CMakeFiles/rpc_test.dir/rpc_test.cxx.o
[ 89%] Built target rpc_clnt
[ 90%] Building CXX object examples/slowcont/CMakeFiles/mscb_fe.dir/__/__/mscb/src/mscb.cxx.o
[ 90%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/bus/null.cxx.o
[ 91%] Linking CXX executable mlogger
Scanning dependencies of target consume
[ 92%] Building CXX object examples/lowlevel/CMakeFiles/consume.dir/consume.cxx.o
[ 93%] Linking CXX executable rpc_test
[ 94%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/drivers/device/mscbdev.cxx.o
[ 94%] Built target mlogger
Scanning dependencies of target produce
[ 94%] Built target rpc_test
[ 94%] Building CXX object examples/lowlevel/CMakeFiles/produce.dir/produce.cxx.o
[ 95%] Building CXX object examples/slowcont/CMakeFiles/scfe.dir/__/__/mscb/src/mscb.cxx.o
[ 96%] Linking CXX executable msequencer
[ 96%] Built target msequencer
[ 96%] Linking CXX executable consume
[ 96%] Built target consume
[ 97%] Linking CXX executable produce
[ 97%] Built target produce
[ 98%] Linking CXX executable mscb_fe
[ 98%] Built target mscb_fe
[ 99%] Linking CXX executable scfe
[ 99%] Built target scfe
[100%] Linking CXX executable mhttpd
[100%] Built target mhttpd
/midas/build$ 

>
> .../c++  ...  /opt/local/lib/libssl.dylib
>

I get the same, libssl from /opt/local, so we are not using openssl shipped with mac os.

We still do not know were OP's libssl comes from and if there is a mismatch between
openssl header files and library (header files from one openssl, library from a different openssl).

OP's cmake output: (no VERBOSE=1)
> -- MIDAS: cmake version: 3.16.3
> -- MIDAS: Found OpenSSL version 1.1.1d

Stefan's and my cmake output: (VERBOSE=1)
> -- MIDAS: cmake version: 3.16.3
> -- Found OpenSSL: /opt/local/lib/libcrypto.dylib (found version "1.1.1d")  
> -- MIDAS: Found OpenSSL version 1.1.1d

Without VERBOSE=1 cmake does not tell is which OpenSSL it found (not useful).
With VERBOSE=1 cmake outputs a flood of gunk (not useful).

My solution is to filter the cmake output with grep in "make cmake" ("make cmake3").

So please use that: "make cmake" - the output is roughly the same as normal make - compiler command lines (CFLAGS, library paths, etc),
compiler error message and all of the useful cmake output.

K.O.

> >
> > .../c++  ...  /opt/local/lib/libssl.dylib
> >
> 
> I get the same, libssl from /opt/local, so we are not using openssl shipped with mac os.
> 

I note that latest mongoose 6.16 finally has a virtual ssl layer and appears to support mbedtls (polarssl) in addition to openssl.

I now think I should see if it works - as it gives us a way to support https without relying on the user having
pre-installed working openssl packages - we consistently run into problems with openssl on macos, and even
on linux there was trouble with preinstalled openssl packages and libraries.

With mbedtls, one will have to "git pull" and "make" it, but historically this causes much less trouble.

Also, with luck, mbedtls has better support for certificate expiration (I would really love to have openssl report an error
or a warning or at least some hint if I feed it an expired certificate) and (gasp!) certbot integration.

K.O.

> ... support for certbot

The certbot tool to use instead of certbot is this: https://github.com/ndilieto/uacme

K.O.

> > ... support for certbot
> 
> The certbot tool to use instead of certbot is this: https://github.com/ndilieto/uacme
> 
> K.O.


HI Stefan and Konstantin, 

Thanks a lot for your messages. Sorry for my late reply, I only work on this project from Tuesday to Thursdays. I have 
run " make cmake" instead of "cd build; cmake" and this is the output regarding mhttpd:

/Library/Developer/CommandLineTools/usr/bin/c++  -O2 -g -DNDEBUG -Wl,-search_paths_first -Wl,-
headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o CMakeFiles/mhttpd.dir/mongoose6.cxx.o 
CMakeFiles/mhttpd.dir/mgd.cxx.o CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o mhttpd  -lsqlite3 ../libmidas.a 
/usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz -lsqlite3 /usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz 
Undefined symbols for architecture x86_64:
  "_OPENSSL_init_ssl", referenced from:
      _mg_mgr_init in mongoose6.cxx.o
  "_SSL_CTX_set_options", referenced from:
      _mg_set_ssl in mongoose6.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)


I see that in your outputs the openssl libs are in /opt/local/lib/ while mine are in /usr/lib/, and that is the only difference. 
I have checked that the libraries libssl.dylib and libcrypto.dylib are in my /usr/lib/, and indeed they are, so I don't 
understand the reason for the error, I will continue investigating. 

Thanks 

Berta 

> I only work on this project from Tuesday to Thursdays.

No problem. No hurry.

> I have run " make cmake" instead of "cd build; cmake" and this is the output regarding mhttpd: ...

There should be also a line where mhttpd.cxx is compiled into mhttpd.o, we need to see what compiler
flags are used - I suspect the compiler uses header files from /usr/local/include while the linker
is using libraries from /usr/lib, a mismatch.

To save time, please attach the full output of "make cmake". There may be something else I want to see there.

If you do not use the mhttpd built-in https support (for best security I recommend using https from the apache httpd password protected https proxy),
then it is perfectly fine to build midas with NO_SSL=1.

K.O.

We're using the History Logger MIDAS feature and writing to mySQL tables, but 
in some cases have run into issues installing mySQL on centos7 systems.

Has anyone ever tried running this MIDAS feature with MariaDB rather than 
mySQL?

> We're using the History Logger MIDAS feature and writing to mySQL tables, but 
> in some cases have run into issues installing mySQL on centos7 systems.
> 
> Has anyone ever tried running this MIDAS feature with MariaDB rather than 
> mySQL?

Hi

I've used MIDAS Logger feature with MariaDB and had no issue thus far .

> We're using the History Logger MIDAS feature and writing to mySQL tables, but 
> in some cases have run into issues installing mySQL on centos7 systems.
> 
> Has anyone ever tried running this MIDAS feature with MariaDB rather than 
> mySQL?

The best I can tell, MariaDB is *the* mysql. "the other thing" is an abandonware fork.

I personally at this moment do not run any daq stations with mysql logging, and I accidentally
removed all the 25 versions and flavours of mysql from my home laptop, so if there
is any problems with mysql, please holler, and I will reinstall mysql (ahem, mariadb) and restore
my ability to test midas against it.

As a connected question, is there any need to have postgres support in midas as well?

K.O.

While upgrading to the latest MIDAS version

MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-2019-09-i-1-gd93944ce-dirty on branch develop

ODB version: 3

I encountered two issues using cmake

1. on machines with NVIDIA drivers:

(nvml.h doesn't seem to be part of the standard nvidia driver package.)

2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 and with ROOT 6.18/04 on Fedora 31:

A workaround (which works for me) is to compile with

cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1

(please post messages in "plain" mode, they are much easier to answer)

- nvidia problems - this code was contributed by Joseph (I think?), with luck he will look into 
this problem.

- ROOT problem - it looks like the error is thrown by the ROOT header files, has nothing to do 
with MIDAS?

So what ROOT are you using? I recommend installing ROOT by following instructions at 
root.cern.ch.

Perhaps you used the ROOT packages from the EPEL repository? I have seen trouble with 
those packages before (miscompiled; important optional features turned off; very old 
versions; etc).

P.S.

Historically, ROOT has caused so many reports of "cannot build midas" that I consistently 
vote to "remove ROOT support from MIDAS". But Stefan's code for writing MIDAS data into 
ROOT files is so neat, cannot throw it away. And some people do use it. So at the latest MIDAS 
bash this Summer we decided to keep it.

(Only build targets to use ROOT are the rmlogger executable and the rmana.o object file (and 
it's one-man-army library)).

But.

In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
get built and installed, except for the code that failed to build.

Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
into the "bin" directory. So one must discover and use "NO_ROOT=1" (which becomes sticky 
until the next "make cclean". Some people are not used to sticky "make" options, I just got 
burned by this very thing last week).

Perhaps there is a way to tell cmake to ignore compile errors for rmlogger and rmana.


K.O.



<p>&nbsp;</p>

<table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
	<tbody>
		<tr>
			<td style="background-color:#486090">Peter Kunz wrote:</td>
		</tr>
		<tr>
			<td style="background-color:#FFFFB0">
			<p>While upgrading to the latest MIDAS version</p>

			<p>MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-
2019-09-i-1-gd93944ce-dirty on branch develop</p>

			<p>ODB version: 3</p>

			<p>I encountered two issues using cmake</p>

			<p>1. on machines with NVIDIA drivers:</p>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 
10px;">nvml.h: no such file or directory</div>

			<p>(nvml.h doesn&#39;t seem to be part of the standard nvidia driver 
package.)</p>

			<p>2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 
and with ROOT 6.18/04 on Fedora 31:</p>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">[ 29%] 
Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o In file included from 
/usr/include/root/TString.h:28, from /usr/include/root/TCollection.h:29, from 
/usr/include/root/TSeqCollection.h:25, from /usr/include/root/TList.h:25, from 
/usr/include/root/TQObject.h:40, from /usr/include/root/TApplication.h:30, from 
/home/pkunz/packages/midas/src/mana.cxx:60:</div>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 
10px;">/usr/include/root/ROOT/RStringView.hxx:32:37: error: &lsquo;experimental&rsquo; in 
namespace &lsquo;std&rsquo; does not name a type 32 | using basic_string_view = 
::std::experimental::basic_string_view&lt;_CharT,_Traits&gt;;</div>

			<p>A workaround (which works for me) is to compile with</p>

			<p><tt>cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1</tt></p>
			</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

Re: ROOT problem

I looked into how my ROOT based MIDAS analyzer compiles. It is using the flag -std=c++14.
MIDAS cmake compiles with -std=gnu++11 with the result:

 29%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
/usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=gnu++11 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
In file included from /usr/include/root/TString.h:28,
                 from /usr/include/root/TCollection.h:29,
                 from /usr/include/root/TSeqCollection.h:25,
                 from /usr/include/root/TList.h:25,
                 from /usr/include/root/TQObject.h:40,
                 from /usr/include/root/TApplication.h:30,
                 from /home/pkunz/packages/midas/src/mana.cxx:60:
/usr/include/root/ROOT/RStringView.hxx:32:37: error: ‘experimental’ in namespace ‘std’ does not name a type
   32 |    using basic_string_view = ::std::experimental::basic_string_view<_CharT,_Traits>;


If I change this to

/usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=c++14 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx

it compiles without error. I hope this helps. 






> (please post messages in "plain" mode, they are much easier to answer)
> 
> - nvidia problems - this code was contributed by Joseph (I think?), with luck he will look into 
> this problem.
> 
> - ROOT problem - it looks like the error is thrown by the ROOT header files, has nothing to do 
> with MIDAS?
> 
> So what ROOT are you using? I recommend installing ROOT by following instructions at 
> root.cern.ch.
> 
> Perhaps you used the ROOT packages from the EPEL repository? I have seen trouble with 
> those packages before (miscompiled; important optional features turned off; very old 
> versions; etc).
> 
> P.S.
> 
> Historically, ROOT has caused so many reports of "cannot build midas" that I consistently 
> vote to "remove ROOT support from MIDAS". But Stefan's code for writing MIDAS data into 
> ROOT files is so neat, cannot throw it away. And some people do use it. So at the latest MIDAS 
> bash this Summer we decided to keep it.
> 
> (Only build targets to use ROOT are the rmlogger executable and the rmana.o object file (and 
> it's one-man-army library)).
> 
> But.
> 
> In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
> get built and installed, except for the code that failed to build.
> 
> Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
> into the "bin" directory. So one must discover and use "NO_ROOT=1" (which becomes sticky 
> until the next "make cclean". Some people are not used to sticky "make" options, I just got 
> burned by this very thing last week).
> 
> Perhaps there is a way to tell cmake to ignore compile errors for rmlogger and rmana.
> 
> 
> K.O.
> 
> 
> 
> <p>&nbsp;</p>
> 
> <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> 	<tbody>
> 		<tr>
> 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> 		</tr>
> 		<tr>
> 			<td style="background-color:#FFFFB0">
> 			<p>While upgrading to the latest MIDAS version</p>
> 
> 			<p>MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-
> 2019-09-i-1-gd93944ce-dirty on branch develop</p>
> 
> 			<p>ODB version: 3</p>
> 
> 			<p>I encountered two issues using cmake</p>
> 
> 			<p>1. on machines with NVIDIA drivers:</p>
> 
> 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> 10px;">nvml.h: no such file or directory</div>
> 
> 			<p>(nvml.h doesn&#39;t seem to be part of the standard nvidia driver 
> package.)</p>
> 
> 			<p>2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 
> and with ROOT 6.18/04 on Fedora 31:</p>
> 
> 			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">[ 29%] 
> Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o In file included from 
> /usr/include/root/TString.h:28, from /usr/include/root/TCollection.h:29, from 
> /usr/include/root/TSeqCollection.h:25, from /usr/include/root/TList.h:25, from 
> /usr/include/root/TQObject.h:40, from /usr/include/root/TApplication.h:30, from 
> /home/pkunz/packages/midas/src/mana.cxx:60:</div>
> 
> 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> 10px;">/usr/include/root/ROOT/RStringView.hxx:32:37: error: &lsquo;experimental&rsquo; in 
> namespace &lsquo;std&rsquo; does not name a type 32 | using basic_string_view = 
> ::std::experimental::basic_string_view&lt;_CharT,_Traits&gt;;</div>
> 
> 			<p>A workaround (which works for me) is to compile with</p>
> 
> 			<p><tt>cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1</tt></p>
> 			</td>
> 		</tr>
> 	</tbody>
> </table>
> 
> <p>&nbsp;</p>

Right. So the problem is mismatch in ROOT compile flags. The old Makefile build used CFLAGS from ROOT to build rmana and rmlogger, cmake uses some kind of generic CFLAGS, so here we have it. No idea how to fix it.

K.O.


> Re: ROOT problem
> 
> I looked into how my ROOT based MIDAS analyzer compiles. It is using the flag -std=c++14.
> MIDAS cmake compiles with -std=gnu++11 with the result:
> 
>  29%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
> /usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=gnu++11 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
> In file included from /usr/include/root/TString.h:28,
>                  from /usr/include/root/TCollection.h:29,
>                  from /usr/include/root/TSeqCollection.h:25,
>                  from /usr/include/root/TList.h:25,
>                  from /usr/include/root/TQObject.h:40,
>                  from /usr/include/root/TApplication.h:30,
>                  from /home/pkunz/packages/midas/src/mana.cxx:60:
> /usr/include/root/ROOT/RStringView.hxx:32:37: error: ‘experimental’ in namespace ‘std’ does not name a type
>    32 |    using basic_string_view = ::std::experimental::basic_string_view<_CharT,_Traits>;
> 
> 
> If I change this to
> 
> /usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=c++14 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
> 
> it compiles without error. I hope this helps. 
> 
> 
> 
> 
> 
> 
> > (please post messages in "plain" mode, they are much easier to answer)
> > 
> > - nvidia problems - this code was contributed by Joseph (I think?), with luck he will look into 
> > this problem.
> > 
> > - ROOT problem - it looks like the error is thrown by the ROOT header files, has nothing to do 
> > with MIDAS?
> > 
> > So what ROOT are you using? I recommend installing ROOT by following instructions at 
> > root.cern.ch.
> > 
> > Perhaps you used the ROOT packages from the EPEL repository? I have seen trouble with 
> > those packages before (miscompiled; important optional features turned off; very old 
> > versions; etc).
> > 
> > P.S.
> > 
> > Historically, ROOT has caused so many reports of "cannot build midas" that I consistently 
> > vote to "remove ROOT support from MIDAS". But Stefan's code for writing MIDAS data into 
> > ROOT files is so neat, cannot throw it away. And some people do use it. So at the latest MIDAS 
> > bash this Summer we decided to keep it.
> > 
> > (Only build targets to use ROOT are the rmlogger executable and the rmana.o object file (and 
> > it's one-man-army library)).
> > 
> > But.
> > 
> > In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
> > get built and installed, except for the code that failed to build.
> > 
> > Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
> > into the "bin" directory. So one must discover and use "NO_ROOT=1" (which becomes sticky 
> > until the next "make cclean". Some people are not used to sticky "make" options, I just got 
> > burned by this very thing last week).
> > 
> > Perhaps there is a way to tell cmake to ignore compile errors for rmlogger and rmana.
> > 
> > 
> > K.O.
> > 
> > 
> > 
> > <p>&nbsp;</p>
> > 
> > <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> > 	<tbody>
> > 		<tr>
> > 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> > 		</tr>
> > 		<tr>
> > 			<td style="background-color:#FFFFB0">
> > 			<p>While upgrading to the latest MIDAS version</p>
> > 
> > 			<p>MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-
> > 2019-09-i-1-gd93944ce-dirty on branch develop</p>
> > 
> > 			<p>ODB version: 3</p>
> > 
> > 			<p>I encountered two issues using cmake</p>
> > 
> > 			<p>1. on machines with NVIDIA drivers:</p>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> > 10px;">nvml.h: no such file or directory</div>
> > 
> > 			<p>(nvml.h doesn&#39;t seem to be part of the standard nvidia driver 
> > package.)</p>
> > 
> > 			<p>2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 
> > and with ROOT 6.18/04 on Fedora 31:</p>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">[ 29%] 
> > Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o In file included from 
> > /usr/include/root/TString.h:28, from /usr/include/root/TCollection.h:29, from 
> > /usr/include/root/TSeqCollection.h:25, from /usr/include/root/TList.h:25, from 
> > /usr/include/root/TQObject.h:40, from /usr/include/root/TApplication.h:30, from 
> > /home/pkunz/packages/midas/src/mana.cxx:60:</div>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> > 10px;">/usr/include/root/ROOT/RStringView.hxx:32:37: error: &lsquo;experimental&rsquo; in 
> > namespace &lsquo;std&rsquo; does not name a type 32 | using basic_string_view = 
> > ::std::experimental::basic_string_view&lt;_CharT,_Traits&gt;;</div>
> > 
> > 			<p>A workaround (which works for me) is to compile with</p>
> > 
> > 			<p><tt>cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1</tt></p>
> > 			</td>
> > 		</tr>
> > 	</tbody>
> > </table>
> > 
> > <p>&nbsp;</p>

Thanks for tracing the problem further down. Now I realized that the CMakeLists.txt for the ROOT analyzer did not contain the usual ROOT flags. I added that and committed the change, so please try again. Here is the diff:

examples/experiment/CMakeLists.txt:

       execute_process(COMMAND root-config --incdir OUTPUT_VARIABLE ROOT_INC)
       execute_process(COMMAND root-config --libs OUTPUT_VARIABLE ROOT_LIBS)
+      execute_process(COMMAND root-config --cflags OUTPUT_VARIABLE ROOT_CFLAGS)
       string(STRIP ${ROOT_LIBS} ROOT_LIBS)
-      string(REGEX REPLACE "\n$" "" ROOT_INC "${ROOT_INC}")
+      string(REGEX REPLACE "\n" "" ROOT_INC ${ROOT_INC})
+      separate_arguments(ROOT_CFLAGS UNIX_COMMAND "${ROOT_CFLAGS}")
       set(CMAKE_CXX_STANDARD 11)

       target_include_directories(analyzer PUBLIC ${ROOT_INC} ${INC_PATH})
+      target_compile_options(analyzer PUBLIC ${ROOT_CFLAGS})
       target_link_libraries(analyzer rmana midas ${ROOT_LIBS} ${LIBS})


Best,
Stefan

> Re: ROOT problem
> 
> I looked into how my ROOT based MIDAS analyzer compiles. It is using the flag -std=c++14.
> MIDAS cmake compiles with -std=gnu++11 with the result:
> 
>  29%] Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o
> /usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=gnu++11 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
> In file included from /usr/include/root/TString.h:28,
>                  from /usr/include/root/TCollection.h:29,
>                  from /usr/include/root/TSeqCollection.h:25,
>                  from /usr/include/root/TList.h:25,
>                  from /usr/include/root/TQObject.h:40,
>                  from /usr/include/root/TApplication.h:30,
>                  from /home/pkunz/packages/midas/src/mana.cxx:60:
> /usr/include/root/ROOT/RStringView.hxx:32:37: error: ‘experimental’ in namespace ‘std’ does not name a type
>    32 |    using basic_string_view = ::std::experimental::basic_string_view<_CharT,_Traits>;
> 
> 
> If I change this to
> 
> /usr/bin/c++  -D_LARGEFILE64_SOURCE -I/home/pkunz/packages/midas/include -I/home/pkunz/packages/midas/mxml -I/usr/include/root  -O2 -g -DNDEBUG   -DHAVE_ZLIB -DHAVE_FTPLIB -Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function -DHAVE_ROOT -std=c++14 -o CMakeFiles/rmana.dir/src/mana.cxx.o -c /home/pkunz/packages/midas/src/mana.cxx
> 
> it compiles without error. I hope this helps. 
> 
> 
> 
> 
> 
> 
> > (please post messages in "plain" mode, they are much easier to answer)
> > 
> > - nvidia problems - this code was contributed by Joseph (I think?), with luck he will look into 
> > this problem.
> > 
> > - ROOT problem - it looks like the error is thrown by the ROOT header files, has nothing to do 
> > with MIDAS?
> > 
> > So what ROOT are you using? I recommend installing ROOT by following instructions at 
> > root.cern.ch.
> > 
> > Perhaps you used the ROOT packages from the EPEL repository? I have seen trouble with 
> > those packages before (miscompiled; important optional features turned off; very old 
> > versions; etc).
> > 
> > P.S.
> > 
> > Historically, ROOT has caused so many reports of "cannot build midas" that I consistently 
> > vote to "remove ROOT support from MIDAS". But Stefan's code for writing MIDAS data into 
> > ROOT files is so neat, cannot throw it away. And some people do use it. So at the latest MIDAS 
> > bash this Summer we decided to keep it.
> > 
> > (Only build targets to use ROOT are the rmlogger executable and the rmana.o object file (and 
> > it's one-man-army library)).
> > 
> > But.
> > 
> > In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
> > get built and installed, except for the code that failed to build.
> > 
> > Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
> > into the "bin" directory. So one must discover and use "NO_ROOT=1" (which becomes sticky 
> > until the next "make cclean". Some people are not used to sticky "make" options, I just got 
> > burned by this very thing last week).
> > 
> > Perhaps there is a way to tell cmake to ignore compile errors for rmlogger and rmana.
> > 
> > 
> > K.O.
> > 
> > 
> > 
> > <p>&nbsp;</p>
> > 
> > <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> > 	<tbody>
> > 		<tr>
> > 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> > 		</tr>
> > 		<tr>
> > 			<td style="background-color:#FFFFB0">
> > 			<p>While upgrading to the latest MIDAS version</p>
> > 
> > 			<p>MIDAS version: 2.1 GIT revision: Tue Dec 31 17:40:14 2019 +0100 - midas-
> > 2019-09-i-1-gd93944ce-dirty on branch develop</p>
> > 
> > 			<p>ODB version: 3</p>
> > 
> > 			<p>I encountered two issues using cmake</p>
> > 
> > 			<p>1. on machines with NVIDIA drivers:</p>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> > 10px;">nvml.h: no such file or directory</div>
> > 
> > 			<p>(nvml.h doesn&#39;t seem to be part of the standard nvidia driver 
> > package.)</p>
> > 
> > 			<p>2. Complile including ROOT throws an error with ROOT 6.12/06 on Centos7 
> > and with ROOT 6.18/04 on Fedora 31:</p>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">[ 29%] 
> > Building CXX object CMakeFiles/rmana.dir/src/mana.cxx.o In file included from 
> > /usr/include/root/TString.h:28, from /usr/include/root/TCollection.h:29, from 
> > /usr/include/root/TSeqCollection.h:25, from /usr/include/root/TList.h:25, from 
> > /usr/include/root/TQObject.h:40, from /usr/include/root/TApplication.h:30, from 
> > /home/pkunz/packages/midas/src/mana.cxx:60:</div>
> > 
> > 			<div style="background:#eee;border:1px solid #ccc;padding:5px 
> > 10px;">/usr/include/root/ROOT/RStringView.hxx:32:37: error: &lsquo;experimental&rsquo; in 
> > namespace &lsquo;std&rsquo; does not name a type 32 | using basic_string_view = 
> > ::std::experimental::basic_string_view&lt;_CharT,_Traits&gt;;</div>
> > 
> > 			<p>A workaround (which works for me) is to compile with</p>
> > 
> > 			<p><tt>cmake .. -DNO_ROOT=1 -DNO_NVIDIA=1</tt></p>
> > 			</td>
> > 		</tr>
> > 	</tbody>
> > </table>
> > 
> > <p>&nbsp;</p>

> In the past, one could use "make -k" to get past the errors caused by ROOT, everything will 
> get built and installed, except for the code that failed to build.
> 
> Now with cmake, it is "all or nothing", if there is any compilation error, nothing gets installed 
> into the "bin" directory.

That's not correct. Indeed a "make -k" stops after an error, but "make -i" still works. Try it! Event "make install -i" 
installs all executables into the bin directory which successfully compiled even if there is an error.

Stefan

> Complile including ROOT throws an error with ROOT 6.18/04 on Fedora 31 ...
> /usr/include/root/ROOT/RStringView.hxx:32:37: error: ‘experimental’ in namespace ‘std’ ...

I was puzzled by this error about "experimental" in ROOT header files.

All ROOT kits that I have are configured and built with c++11, and I do not see this error. I now guess that you
only saw this on Fedora 31 with gcc 9 and ROOT that was configured for c++14 or c++17.

We now realize that MIDAS cmake does not use the ROOT CFLAGS when building code that uses ROOT,
and this is probably a mistake. Stefan added a fix to use ROOT CFLAGS to build the analyzer,
but I am not sure if this is sufficient, rmana and rmlogger probably also should use the ROOT CFLAGS.

In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).

K.O.

> In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).

Are there good reasons to not compile MIDAS with set(CMAKE_CXX_STANDARD 14)? So far this was an easier "fix" for me than to recompile ROOT with c++11.

> > In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).
> 
> Are there good reasons to not compile MIDAS with set(CMAKE_CXX_STANDARD 14)? So far this was an easier "fix" for me than to recompile ROOT with c++11.

Yes. We agreed to base MIDAS on C++11, not C++14, since C++11 is available on all relevant systems. By putting that requirement explicitly into the CMake file, we know that it 
correctly compiles with C++11. Of course feel free to overwrite this in your local installation.

Stefan

> > In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).
> 
> Are there good reasons to not compile MIDAS with set(CMAKE_CXX_STANDARD 14)? So far this was an easier "fix" for me than to recompile ROOT with c++11.

We already have trouble enough even with requiring C++11:

We still run a number of SL6 DAQ stations at TRIUMF, and it is well known that SL6 gcc does not even do c++11.

The suggested solution to "simply just use the devtoolset-8" compilers turned out to be missing a critical 32-bit
library and cannot build midas for 32-bit VME processors. (straight or cross-compiled).

(A solution for this can be kludged using unofficial packages: https://midas.triumf.ca/elog/Midas/1748)

No, we cannot "just upgrade" SL6 and no, we cannot "just replace" the 32-bit-only VME processors.

If we move the midas requirements from c++11 to c++14, again, we have to see what compilers
are available on what platforms, and I am sure again we will miss some important case.

And then, where does this train stop?

I see c++17 has some really nice to use improvements. And I see c++20 added even some more nice improvements.

I say we follow ROOT: https://root.cern.ch/supported-platforms

> Supported Platforms
> ROOT 6
> The compiler needs to support -std=c++11 to be able to build version 6.

K.O.

> > In any case, we are setup for an eventual collision between MIDAS CFLAGS (-std=gnu++11) and ROOT CFLAGS (-std=something else).
> 
> Are there good reasons to not compile MIDAS with set(CMAKE_CXX_STANDARD 14)? So far this was an easier "fix" for me than to recompile ROOT with c++11.

Per  https://root.cern.ch/supported-platforms, ROOTv6 is C++11.

Binary kits distributed from root.cern.ch seem to be built with c++11.

Where does ROOT built with c++14 come from?

K.O.

I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:

/home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
            , &(info->caid[i].chan_id))

/home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
            , &(info->caid[i].evt_id))

This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 

> I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
> However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:
> 
> /home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
>             , &(info->caid[i].chan_id))
> 
> /home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
>             , &(info->caid[i].evt_id))
> 
> This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
> I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 

Hi, Peter - it looks like epics_ca.cxx needs to be updated to proper C++ (char* warnings, etc).

As for the "function not declared" errors, it's the C++ way to say "you are calling a function with wrong arguments",
again, something I should be able to fix without too much trouble.

Thank you for reporting this, the midas version of epics_ca.cxx definitely needs fixing.

K.O.

I fixed the compiler errors in epics_ca.cxx, can you try again? (see https://bitbucket.org/tmidas/midas/commits/)

But, I do not see errors with ca_create_channel() and ca_create_subscription().

Maybe I am using the wrong epics. If you still see these errors, let me know what epics you have, I will try the same one.

This is what I do (and I get epics 7.0).

mkdir -p $HOME/git
cd $HOME/git
git clone https://git.launchpad.net/epics-base
cd epics-base
make
ls -l include/cadef.h
ls -l lib/darwin-x86/libca.a # also linux-x86/libca.a

K.O.

> > I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
> > However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:
> > 
> > /home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
> >             , &(info->caid[i].chan_id))
> > 
> > /home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
> >             , &(info->caid[i].evt_id))
> > 
> > This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
> > I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 
> 
> Hi, Peter - it looks like epics_ca.cxx needs to be updated to proper C++ (char* warnings, etc).
> 
> As for the "function not declared" errors, it's the C++ way to say "you are calling a function with wrong arguments",
> again, something I should be able to fix without too much trouble.
> 
> Thank you for reporting this, the midas version of epics_ca.cxx definitely needs fixing.
> 
> K.O.

Hi Konstantin,
I have EPICS Base Release 3.14.8.2 and got your example running with it, though I had to make one change.
For some reason it wouldn't find libca.so, but when I linked the static library libca.a instead, it worked.
Also my midas frontend works now with the updated epics_ca.xx. There was a problem with an old local version of cadef.h
(probably a leftover from some previous testing). After removing it everything compiled well.

Thanks,
Peter


> I fixed the compiler errors in epics_ca.cxx, can you try again? (see https://bitbucket.org/tmidas/midas/commits/)
> 
> But, I do not see errors with ca_create_channel() and ca_create_subscription().
> 
> Maybe I am using the wrong epics. If you still see these errors, let me know what epics you have, I will try the same one.
> 
> This is what I do (and I get epics 7.0).
> 
> mkdir -p $HOME/git
> cd $HOME/git
> git clone https://git.launchpad.net/epics-base
> cd epics-base
> make
> ls -l include/cadef.h
> ls -l lib/darwin-x86/libca.a # also linux-x86/libca.a
> 
> K.O.
> 
> > > I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
> > > However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:
> > > 
> > > /home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
> > >             , &(info->caid[i].chan_id))
> > > 
> > > /home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
> > >             , &(info->caid[i].evt_id))
> > > 
> > > This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
> > > I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 
> > 
> > Hi, Peter - it looks like epics_ca.cxx needs to be updated to proper C++ (char* warnings, etc).
> > 
> > As for the "function not declared" errors, it's the C++ way to say "you are calling a function with wrong arguments",
> > again, something I should be able to fix without too much trouble.
> > 
> > Thank you for reporting this, the midas version of epics_ca.cxx definitely needs fixing.
> > 
> > K.O.

> I have EPICS Base Release 3.14.8.2 and got your example running with it...

Ok, good.

> I had to make one change.
> For some reason it wouldn't find libca.so, but when I linked the static library libca.a instead, it worked.

Yes, libca.a is what I see in my epics, too. I recommend static linking (libxxx.a instead of libxxx.so) as it eliminates
all possible problems with "cannot find the shared library" and "found the wrong shared library" and "found
a shared library from a different version of epics".

> Also my midas frontend works now with the updated epics_ca.xx. There was a problem with an old local version of cadef.h
> (probably a leftover from some previous testing). After removing it everything compiled well.

Ok. good. I also see some old epics header files in the midas repository. I guess I should remove them.

K.O.


> 
> Thanks,
> Peter
> 
> 
> > I fixed the compiler errors in epics_ca.cxx, can you try again? (see https://bitbucket.org/tmidas/midas/commits/)
> > 
> > But, I do not see errors with ca_create_channel() and ca_create_subscription().
> > 
> > Maybe I am using the wrong epics. If you still see these errors, let me know what epics you have, I will try the same one.
> > 
> > This is what I do (and I get epics 7.0).
> > 
> > mkdir -p $HOME/git
> > cd $HOME/git
> > git clone https://git.launchpad.net/epics-base
> > cd epics-base
> > make
> > ls -l include/cadef.h
> > ls -l lib/darwin-x86/libca.a # also linux-x86/libca.a
> > 
> > K.O.
> > 
> > > > I'm still trying to upgrade my MIDAS system to midas-2019-09-i. Most frontends work fine with the modifications already discussed.
> > > > However, I ran into some trouble with the epics frontend. Even with the modifications it throws a lot of warnings and errors (see attached log file). I can reduce the errors with -fpermissive, but the following two errors are persistent:
> > > > 
> > > > /home/ays/packages/midas/drivers/device/epics_ca.cxx:167:38: error: ‘ca_create_channel’ was not declared in this scope
> > > >             , &(info->caid[i].chan_id))
> > > > 
> > > > /home/ays/packages/midas/drivers/device/epics_ca.cxx:178:37: error: ‘ca_create_subscription’ was not declared in this scope
> > > >             , &(info->caid[i].evt_id))
> > > > 
> > > > This is strange because the functions seem to be declared in base/include/cadef.h along with similar functions that don't throw an error.
> > > > I don't know what's going on. The frontend which is almost identical to the example in the midas distribution compiles without warnings or errors under the Midas2017 version. 
> > > 
> > > Hi, Peter - it looks like epics_ca.cxx needs to be updated to proper C++ (char* warnings, etc).
> > > 
> > > As for the "function not declared" errors, it's the C++ way to say "you are calling a function with wrong arguments",
> > > again, something I should be able to fix without too much trouble.
> > > 
> > > Thank you for reporting this, the midas version of epics_ca.cxx definitely needs fixing.
> > > 
> > > K.O.

>
> > I have EPICS Base Release 3.14.8.2 ...
>

For the record, according to https://epics.anl.gov/

the 3.14 branch is "Closed, please upgrade!"
the 3.15 branch seems to be the current "stable" branch, and
the 7.0 branch is the "current release"

I forget which version of EPICS is currently recommended by the TRIUMF controls group. I will have to go and ask them.

K.O.

After upgrading to the lastes MIDAS version I got the DAQ frontend of my application running by changing all compiler directives from cc to g++ and using

#include "mfe.h"

extern HNDLE hDB

 extern "C" { 
 #include <CAENComm.h>  
 }

With these changes everything seems to work fine.

However, I'm having trouble with a slow control frontend using a tcpip driver. It compiled well with the older MIDAS version. Even though all the functions in question are defined in the frontend code, the following error comes up:

 I guess the the aforementioned DAQ frontend compiles because its equipment definitions don't call on the function `initialize_equipment()', but I can't figure out why it doesn't work. Help is appreciated. P.K.

(please use the "plain" text, much easier to answer).

Hi, Peter, I think you misread the error message. There is no error about initialize_equipment(), the error 
is about interrupt_configure(). initialize_equipment() is just one of the functions that calls it.

The cause of the error most likely is a mismatch between the declaration of interrupt_configure() in 
mfe.h and the definition of this function in your program (in feMotor.c, I guess).

Sometimes this mismatch is hard to identify just by looking at the code.

One fool-proof method to debug this is to extract the actual function prototypes from your object files, 
both the declaration ("U") in mfe.o and the definition ("T") in your program should be identical:

nm feMotor.o |  grep -i interrupt | c++filt
0000000000000f90 T interrupt_configure(int, int, long) <--- should be this

nm ~/packages/midas/lib/libmfe.a | grep -i interrupt | c++filt
U interrupt_configure(int, int, long)

If they are different, you adjust your program until they match. One way to ensure the match is to copy 
the declaration from mfe.h into your program.

K.O.




<p>&nbsp;</p>

<table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
	<tbody>
		<tr>
			<td style="background-color:#486090">Peter Kunz wrote:</td>
		</tr>
		<tr>
			<td style="background-color:#FFFFB0">
			<p>After upgrading to the lastes MIDAS version I got the DAQ frontend of my application 
running by changing all compiler directives from cc to g++ and using</p>

			<p>#include &quot;mfe.h&quot;</p>

			<p>extern HNDLE hDB</p>

			<p>&nbsp;extern &quot;C&quot; {&nbsp;<br />
			&nbsp;#include &lt;CAENComm.h&gt; &nbsp;<br />
			&nbsp;}</p>

			<p>With these changes everything seems to work fine.</p>

			<p>However, I&#39;m having trouble with a slow control frontend using a tcpip driver. It 
compiled well with the older MIDAS version. Even though all the functions in question are defined in the 
frontend code, the following error comes up:</p>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">g++ -o feMotor 
-DOS_LINUX -Dextname -g -O2 -fPIC -Wall -Wuninitialized &nbsp;-fpermissive -
I/home/pkunz/packages/midas/include -I. -I/home/pkunz/packages/midas/drivers/bus &nbsp; 
/home/pkunz/packages/midas/lib/libmidas.a feMotor.o 
/home/pkunz/packages/midas/drivers/bus/tcpip.o cd_Galil.o 
/home/pkunz/packages/midas/lib/libmidas.a /home/pkunz/packages/midas/lib/mfe.o -lm -lz -lutil -lnsl -
lpthread -lrt<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
`initialize_equipment()&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:687: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
`readout_enable(unsigned int)&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:1236: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:1238: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function `main&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:2791: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:2792: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			collect2: error: ld returned 1 exit status<br />
			make: *** [Makefile:36: feMotor] Error 1</div>

			<p>&nbsp;I guess the the aforementioned DAQ frontend compiles because its equipment 
definitions don&#39;t call on the function `initialize_equipment()&#39;, but I can&#39;t figure out why 
it doesn&#39;t work. Help is appreciated. P.K.</p>
			</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

Thanks for explaining this, Konstantin. 
After updating the function to

INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
  return CM_SUCCESS;
}

it compiled without errors. In the original code the "INT source" variable was missing.



> (please use the "plain" text, much easier to answer).
> 
> Hi, Peter, I think you misread the error message. There is no error about initialize_equipment(), the error 
> is about interrupt_configure(). initialize_equipment() is just one of the functions that calls it.
> 
> The cause of the error most likely is a mismatch between the declaration of interrupt_configure() in 
> mfe.h and the definition of this function in your program (in feMotor.c, I guess).
> 
> Sometimes this mismatch is hard to identify just by looking at the code.
> 
> One fool-proof method to debug this is to extract the actual function prototypes from your object files, 
> both the declaration ("U") in mfe.o and the definition ("T") in your program should be identical:
> 
> nm feMotor.o |  grep -i interrupt | c++filt
> 0000000000000f90 T interrupt_configure(int, int, long) <--- should be this
> 
> nm ~/packages/midas/lib/libmfe.a | grep -i interrupt | c++filt
> U interrupt_configure(int, int, long)
> 
> If they are different, you adjust your program until they match. One way to ensure the match is to copy 
> the declaration from mfe.h into your program.
> 
> K.O.
> 
> 
> 
> 
> <p>&nbsp;</p>
> 
> <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> 	<tbody>
> 		<tr>
> 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> 		</tr>
> 		<tr>
> 			<td style="background-color:#FFFFB0">
> 			<p>After upgrading to the lastes MIDAS version I got the DAQ frontend of my application 
> running by changing all compiler directives from cc to g++ and using</p>
> 
> 			<p>#include &quot;mfe.h&quot;</p>
> 
> 			<p>extern HNDLE hDB</p>
> 
> 			<p>&nbsp;extern &quot;C&quot; {&nbsp;<br />
> 			&nbsp;#include &lt;CAENComm.h&gt; &nbsp;<br />
> 			&nbsp;}</p>
> 
> 			<p>With these changes everything seems to work fine.</p>
> 
> 			<p>However, I&#39;m having trouble with a slow control frontend using a tcpip driver. It 
> compiled well with the older MIDAS version. Even though all the functions in question are defined in the 
> frontend code, the following error comes up:</p>
> 
> 			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">g++ -o feMotor 
> -DOS_LINUX -Dextname -g -O2 -fPIC -Wall -Wuninitialized &nbsp;-fpermissive -
> I/home/pkunz/packages/midas/include -I. -I/home/pkunz/packages/midas/drivers/bus &nbsp; 
> /home/pkunz/packages/midas/lib/libmidas.a feMotor.o 
> /home/pkunz/packages/midas/drivers/bus/tcpip.o cd_Galil.o 
> /home/pkunz/packages/midas/lib/libmidas.a /home/pkunz/packages/midas/lib/mfe.o -lm -lz -lutil -lnsl -
> lpthread -lrt<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
> `initialize_equipment()&#39;:<br />
> 			/home/pkunz/packages/midas/src/mfe.cxx:687: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
> `readout_enable(unsigned int)&#39;:<br />
> 			/home/pkunz/packages/midas/src/mfe.cxx:1236: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:1238: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function `main&#39;:<br />
> 			/home/pkunz/packages/midas/src/mfe.cxx:2791: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:2792: undefined reference to 
> `interrupt_configure(int, int, long)&#39;<br />
> 			collect2: error: ld returned 1 exit status<br />
> 			make: *** [Makefile:36: feMotor] Error 1</div>
> 
> 			<p>&nbsp;I guess the the aforementioned DAQ frontend compiles because its equipment 
> definitions don&#39;t call on the function `initialize_equipment()&#39;, but I can&#39;t figure out why 
> it doesn&#39;t work. Help is appreciated. P.K.</p>
> 			</td>
> 		</tr>
> 	</tbody>
> </table>
> 
> <p>&nbsp;</p>

We updated midas/examples/experiment/frontend.cxx to correctly contain

/*-- Interrupt configuration ---------------------------------------*/

INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
   switch (cmd) {
   case CMD_INTERRUPT_ENABLE:
      break;
   case CMD_INTERRUPT_DISABLE:
      break;
   case CMD_INTERRUPT_ATTACH:
      break;
   case CMD_INTERRUPT_DETACH:
      break;
   }
   return SUCCESS;
}

but if you upgrade from C to C++ from your own old frontend code you might be hit by that issue. 

Maybe Konstantin can update elog:1526 to contain a hint about "INT source".

Stefan

Actually now I see that 

a4) poll_event() and interrupt_configure() have "source" as "int[]" instead of "int" (why did this work before?)

mention this already, but maybe it's not completely clear that one has to change "int[] source" to "int source"

Stefan


> We updated midas/examples/experiment/frontend.cxx to correctly contain
> 
> /*-- Interrupt configuration ---------------------------------------*/
> 
> INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
> {
>    switch (cmd) {
>    case CMD_INTERRUPT_ENABLE:
>       break;
>    case CMD_INTERRUPT_DISABLE:
>       break;
>    case CMD_INTERRUPT_ATTACH:
>       break;
>    case CMD_INTERRUPT_DETACH:
>       break;
>    }
>    return SUCCESS;
> }
> 
> but if you upgrade from C to C++ from your own old frontend code you might be hit by that issue. 
> 
> Maybe Konstantin can update elog:1526 to contain a hint about "INT source".
> 
> Stefan

(please use the "plain" text, much easier to answer).

Hi, Peter, I think you misread the error message. There is no error about 
initialize_equipment(), the error is about interrupt_configure(). initialize_equipment() is just 
one of the functions that calls it.

The cause of the error most likely is a mismatch between the declaration of 
interrupt_configure() in mfe.h and the definition of this function in your program (in 
feMotor.c, I guess).

Sometimes this mismatch is hard to identify just by looking at the code.

One fool-proof method to debug this is to extract the actual function prototypes from your 
object files, both the declaration ("U") in mfe.o and the definition ("T") in your program 
should be identical:

nm feMotor.o |  grep -i interrupt | c++filt
0000000000000f90 T interrupt_configure(int, int, long) <--- should be this

nm ~/packages/midas/lib/libmfe.a | grep -i interrupt | c++filt
U interrupt_configure(int, int, long)

If they are different, you adjust your program until they match. One way to ensure the 
match is to copy the declaration from mfe.h into your program.

K.O.




<p>&nbsp;</p>

<table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
	<tbody>
		<tr>
			<td style="background-color:#486090">Peter Kunz wrote:</td>
		</tr>
		<tr>
			<td style="background-color:#FFFFB0">
			<p>After upgrading to the lastes MIDAS version I got the DAQ frontend of my 
application running by changing all compiler directives from cc to g++ and using</p>

			<p>#include &quot;mfe.h&quot;</p>

			<p>extern HNDLE hDB</p>

			<p>&nbsp;extern &quot;C&quot; {&nbsp;<br />
			&nbsp;#include &lt;CAENComm.h&gt; &nbsp;<br />
			&nbsp;}</p>

			<p>With these changes everything seems to work fine.</p>

			<p>However, I&#39;m having trouble with a slow control frontend using a 
tcpip driver. It compiled well with the older MIDAS version. Even though all the functions in 
question are defined in the frontend code, the following error comes up:</p>

			<div style="background:#eee;border:1px solid #ccc;padding:5px 10px;">g++ 
-o feMotor -DOS_LINUX -Dextname -g -O2 -fPIC -Wall -Wuninitialized &nbsp;-fpermissive 
-I/home/pkunz/packages/midas/include -I. -I/home/pkunz/packages/midas/drivers/bus 
&nbsp; /home/pkunz/packages/midas/lib/libmidas.a feMotor.o 
/home/pkunz/packages/midas/drivers/bus/tcpip.o cd_Galil.o 
/home/pkunz/packages/midas/lib/libmidas.a /home/pkunz/packages/midas/lib/mfe.o -lm -lz 
-lutil -lnsl -lpthread -lrt<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
`initialize_equipment()&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:687: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function 
`readout_enable(unsigned int)&#39;:<br />
			/home/pkunz/packages/midas/src/mfe.cxx:1236: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:1238: undefined 
reference to `interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/lib/mfe.o: in function `main&#39;:
<br />
			/home/pkunz/packages/midas/src/mfe.cxx:2791: undefined reference to 
`interrupt_configure(int, int, long)&#39;<br />
			/usr/bin/ld: /home/pkunz/packages/midas/src/mfe.cxx:2792: undefined 
reference to `interrupt_configure(int, int, long)&#39;<br />
			collect2: error: ld returned 1 exit status<br />
			make: *** [Makefile:36: feMotor] Error 1</div>

			<p>&nbsp;I guess the the aforementioned DAQ frontend compiles because 
its equipment definitions don&#39;t call on the function `initialize_equipment()&#39;, but I 
can&#39;t figure out why it doesn&#39;t work. Help is appreciated. P.K.</p>
			</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

MIDAS version:      2.1
GIT revision:       Tue Dec 31 17:40:14 2019 +0100 - midas-2019-09-i-1-gd93944ce-dirty on branch develop

/Logger/Channels/0/Settings

With the settings above the file last.json generated for a new run is empty and the events 0x8000 and 0x8001 are missing in the .mid file.

When setting "ODB dump format" to "xml", events 0x8000 and 0x8001 are included in the .mid file, however, the file last.xml is not created.

(Please post messages in "plain" mode, they are much easier to answer)

Thank you for reporting this problem. I will try to reproduce it.

In addition, I will say a few words about your version of midas:

> GIT revision: midas-2019-09-i-1-gd93944ce-dirty on branch develop

I recommend that for production systems one used the tagged release versions of midas. 
(i.e. see https://midas.triumf.ca/elog/Midas/1750).

(Your midas is "1 commit after the latest tag" - the "-1" in the git revision).

I apply bug fixes to both the release branch and the develop branch, but for you to get 
these fixes, on the develop branch you will also "get" all the unrelated changes that may 
come with new bugs. On the release branch, you will only get the bug fixes.

In your midas version it says "-dirty" which means that you have local modifications to the 
midas sources. With luck those changes are not related to the bug that you see. (but I 
cannot tell). You can do "git status" and "git diff" to see what the local changes are.

It is much better if bugs are reported against "clean" builds of MIDAS (no "-dirty").


K.O.


<p>&nbsp;</p>

<table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
	<tbody>
		<tr>
			<td style="background-color:#486090">Peter Kunz wrote:</td>
		</tr>
		<tr>
			<td style="background-color:#FFFFB0">
			<p>MIDAS version: &nbsp; &nbsp; &nbsp;2.1<br />
			GIT revision: &nbsp; &nbsp; &nbsp; Tue Dec 31 17:40:14 2019 +0100 - 
midas-2019-09-i-1-gd93944ce-dirty on branch develop</p>

			<p>/Logger/Channels/0/Settings</p>

			<table border="3" cellpadding="1" class="dialogTable">
				<tbody>
					<tr>
						<td>ODB dump</td>
						<td>y</td>
					</tr>
					<tr>
						<td>ODB dump format</td>
						<td>json</td>
					</tr>
				</tbody>
			</table>

			<p>With the settings above the file last.json generated for a new run is 
empty and the events 0x8000 and 0x8001 are missing in the .mid file.</p>

			<p>When setting &quot;ODB dump format&quot; to &quot;xml&quot;, events 
0x8000 and 0x8001 are included in the .mid file, however, the file last.xml is not created.
</p>

			<p>&nbsp;</p>

			<p>&nbsp;</p>

			<p>&nbsp;</p>
			</td>
		</tr>
	</tbody>
</table>

<p>&nbsp;</p>

Re: MIDAS versions

Thanks for pointing that out. I wasn't actually aware that there is a release branch and a development branch.
I was just following the installation instructions using

git clone https://bitbucket.org/tmidas/midas --recursive

Apparently that gave me the development branch. How can I get the release version?

(I think my version showed up a "dirty" because I played around with one of the examples. I didn't touch the actual source code.)

> (Please post messages in "plain" mode, they are much easier to answer)
> 
> Thank you for reporting this problem. I will try to reproduce it.
> 
> In addition, I will say a few words about your version of midas:
> 
> > GIT revision: midas-2019-09-i-1-gd93944ce-dirty on branch develop
> 
> I recommend that for production systems one used the tagged release versions of midas. 
> (i.e. see https://midas.triumf.ca/elog/Midas/1750).
> 
> (Your midas is "1 commit after the latest tag" - the "-1" in the git revision).
> 
> I apply bug fixes to both the release branch and the develop branch, but for you to get 
> these fixes, on the develop branch you will also "get" all the unrelated changes that may 
> come with new bugs. On the release branch, you will only get the bug fixes.
> 
> In your midas version it says "-dirty" which means that you have local modifications to the 
> midas sources. With luck those changes are not related to the bug that you see. (but I 
> cannot tell). You can do "git status" and "git diff" to see what the local changes are.
> 
> It is much better if bugs are reported against "clean" builds of MIDAS (no "-dirty").
> 
> 
> K.O.
> 
> 
> <p>&nbsp;</p>
> 
> <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> 	<tbody>
> 		<tr>
> 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> 		</tr>
> 		<tr>
> 			<td style="background-color:#FFFFB0">
> 			<p>MIDAS version: &nbsp; &nbsp; &nbsp;2.1<br />
> 			GIT revision: &nbsp; &nbsp; &nbsp; Tue Dec 31 17:40:14 2019 +0100 - 
> midas-2019-09-i-1-gd93944ce-dirty on branch develop</p>
> 
> 			<p>/Logger/Channels/0/Settings</p>
> 
> 			<table border="3" cellpadding="1" class="dialogTable">
> 				<tbody>
> 					<tr>
> 						<td>ODB dump</td>
> 						<td>y</td>
> 					</tr>
> 					<tr>
> 						<td>ODB dump format</td>
> 						<td>json</td>
> 					</tr>
> 				</tbody>
> 			</table>
> 
> 			<p>With the settings above the file last.json generated for a new run is 
> empty and the events 0x8000 and 0x8001 are missing in the .mid file.</p>
> 
> 			<p>When setting &quot;ODB dump format&quot; to &quot;xml&quot;, events 
> 0x8000 and 0x8001 are included in the .mid file, however, the file last.xml is not created.
> </p>
> 
> 			<p>&nbsp;</p>
> 
> 			<p>&nbsp;</p>
> 
> 			<p>&nbsp;</p>
> 			</td>
> 		</tr>
> 	</tbody>
> </table>
> 
> <p>&nbsp;</p>

For using the release branch read the messages in this thread. Most of the time, the develop branch is fine, except when we are developing something 
new, and the only way to tell is to watch the git activity on bitbucket or see the release branch announcements I post on the midas forum.

https://midas.triumf.ca/elog/Midas/1706

K.O.


> Re: MIDAS versions
> 
> Thanks for pointing that out. I wasn't actually aware that there is a release branch and a development branch.
> I was just following the installation instructions using
> 
> git clone https://bitbucket.org/tmidas/midas --recursive
> 
> Apparently that gave me the development branch. How can I get the release version?
> 
> (I think my version showed up a "dirty" because I played around with one of the examples. I didn't touch the actual source code.)
> 
> > (Please post messages in "plain" mode, they are much easier to answer)
> > 
> > Thank you for reporting this problem. I will try to reproduce it.
> > 
> > In addition, I will say a few words about your version of midas:
> > 
> > > GIT revision: midas-2019-09-i-1-gd93944ce-dirty on branch develop
> > 
> > I recommend that for production systems one used the tagged release versions of midas. 
> > (i.e. see https://midas.triumf.ca/elog/Midas/1750).
> > 
> > (Your midas is "1 commit after the latest tag" - the "-1" in the git revision).
> > 
> > I apply bug fixes to both the release branch and the develop branch, but for you to get 
> > these fixes, on the develop branch you will also "get" all the unrelated changes that may 
> > come with new bugs. On the release branch, you will only get the bug fixes.
> > 
> > In your midas version it says "-dirty" which means that you have local modifications to the 
> > midas sources. With luck those changes are not related to the bug that you see. (but I 
> > cannot tell). You can do "git status" and "git diff" to see what the local changes are.
> > 
> > It is much better if bugs are reported against "clean" builds of MIDAS (no "-dirty").
> > 
> > 
> > K.O.
> > 
> > 
> > <p>&nbsp;</p>
> > 
> > <table align="center" cellspacing="1" style="border:1px solid #486090; width:98%">
> > 	<tbody>
> > 		<tr>
> > 			<td style="background-color:#486090">Peter Kunz wrote:</td>
> > 		</tr>
> > 		<tr>
> > 			<td style="background-color:#FFFFB0">
> > 			<p>MIDAS version: &nbsp; &nbsp; &nbsp;2.1<br />
> > 			GIT revision: &nbsp; &nbsp; &nbsp; Tue Dec 31 17:40:14 2019 +0100 - 
> > midas-2019-09-i-1-gd93944ce-dirty on branch develop</p>
> > 
> > 			<p>/Logger/Channels/0/Settings</p>
> > 
> > 			<table border="3" cellpadding="1" class="dialogTable">
> > 				<tbody>
> > 					<tr>
> > 						<td>ODB dump</td>
> > 						<td>y</td>
> > 					</tr>
> > 					<tr>
> > 						<td>ODB dump format</td>
> > 						<td>json</td>
> > 					</tr>
> > 				</tbody>
> > 			</table>
> > 
> > 			<p>With the settings above the file last.json generated for a new run is 
> > empty and the events 0x8000 and 0x8001 are missing in the .mid file.</p>
> > 
> > 			<p>When setting &quot;ODB dump format&quot; to &quot;xml&quot;, events 
> > 0x8000 and 0x8001 are included in the .mid file, however, the file last.xml is not created.
> > </p>
> > 
> > 			<p>&nbsp;</p>
> > 
> > 			<p>&nbsp;</p>
> > 
> > 			<p>&nbsp;</p>
> > 			</td>
> > 		</tr>
> > 	</tbody>
> > </table>
> > 
> > <p>&nbsp;</p>

I just happened to check the current situation with self-signed https certificates as implemented in mhttpd.

(To remember, the powers-that-be are pushing for universal use of https for all web access. The https
implementation in mhttpd at the moment can only generate self-signed certificates, so...)

plain unencrypted http:
- both google chrome and firefox say "connection not secure", but connect without any fuss.
- apple safari does not say anything

https with self-signed certificate:
- google chrome goes through an "are you sure?" page, "red not secure" status in toolbar
- firefox does the same thing, requires adding a security exception, but still shows "not secure" status in toolbar
- apple safari goes through a sequence of "are you sure?" pages, asks for the user password to add the self-signed certificate to 
the macos key store, then marks the connection as "secure" (good)

So clearly powers-that-be do not want us to use self-signed certificates for https. (And frown on use of unencrypted
http even for localhost connections). Properly signed certificates can be obtained from letsencrypt almost
automatically, but of course mhttpd needs to know how to use them and how to do handle their automatic renewals.

I plan to update the mongoose web server library inside mhttpd and with luck I will straighten some of this certificate business at 
the same time.

In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache 
httpd, etc).

K.O.

> In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache 
> httpd, etc).

I guess this is what moste people do anyhow these days. Do I understand correctly that this then rules out the usage of letsencrype certificates, since the 
host needs to be accessed from outside, which is not possible if running behind a password protected firewall.

Stefan

> > In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache 
> > httpd, etc).
> 
> I guess this is what moste people do anyhow these days. Do I understand correctly that this then rules out the usage of letsencrype certificates, since the 
> host needs to be accessed from outside, which is not possible if running behind a password protected firewall.
> 
> Stefan

Careful, firewall != proxy, very different things.

A firewall prevents network communications, period. (Like fences and locked doors, there are good reasons to have them).

An https proxy is a way to have encrypted (protected) web communications with a machine behind a firewall.

Basically, we have 4 main cases, all with trouble.

1) mhttpd running on localhost, "just for testing", is in trouble. there is no simple way to get a "blessed" certificate, and self-signed certificates are now "almost forbidden". http is "okey 
for now", but the writing is on the wall. There is no special exception for "local-only" connections.

2a) mhttpd running on an internet-connected machine, with apache httpd, our best case. To get this working one has to configure both apache httpd and the "blessed certificate" 
certbot tool. With luck, both tools work smoothly on current OSes (they do NOT).

2b) same, but without apache httpd. One still has to run certbot, and the "glue" between mhttpd and certbot is currently missing: need a way to point mhttpd to the certbot certificate 
files and a way to reload mhttpd when the certificate is auto-renewed.

3) mhttpd running on a machine behind a corporate firewall. worst case. if firewall Gods make an opening for ports 80 and 443, it becomes case (2a/b), otherwise, one must use some 
kind of https proxy. (Plus there is no trivial way to setup an encrypted secure communication channel between mhttpd and this proxy, a double bad).

K.O.

P.S. I guess one can use nginx as the https proxy instead of apache httpd. I did not try yet. My impression is that everybody uses nginx, except for people who started with apache httpd 
and are too lazy to try nginx.

K.O.

> > > In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache 
> > > httpd, etc).

There we go. google-chrome 74 refuses to connect to mhttpd configured with a self-signed certificate generated per instructions printed by mhttpd.

Here is the full error text (there is no button to "let me connect to it anyway"):

Your connection is not private
Attackers might be trying to steal your information from musr03.triumf.ca (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_AUTHORITY_INVALID
 
Help improve Safe Browsing by sending some system information and page content to Google. Privacy policy
musr03.triumf.ca normally uses encryption to protect your information. When Google Chrome tried to connect to musr03.triumf.ca this time, the website sent back unusual and incorrect credentials. This may happen when an 
attacker is trying to pretend to be musr03.triumf.ca, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Google Chrome stopped the connection before any data was exchanged.

You cannot visit musr03.triumf.ca right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

Now firefox returns the same error. version 72.0.1.

> daqlabpc.triumf.ca has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site.
> Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT

I think the problem is with HSTS. I enabled HSTS (in mhttpd and in apache httpd) because
SSLlabs encourage it and because my reading of it's description at
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security
makes it sound like a good idea without any downsides.

However, the actual HSTS RFC says something completely different:
https://tools.ietf.org/html/rfc6797

"The aim is to prevent click-through insecurity and address other potential threats".

To me this explains what I see. In contrast to the description at developer.mozilla.org,
firefox (and google chrome) disable "click-through" exceptions for "I do not like this https certificate",
and there is no way to connect to self-signed https.

Bottom line, either use certbot to get blessed https certificate or no https for you.

K.O.


> > > > In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache 
> > > > httpd, etc).
> 
> There we go. google-chrome 74 refuses to connect to mhttpd configured with a self-signed certificate generated per instructions printed by mhttpd.
> 
> Here is the full error text (there is no button to "let me connect to it anyway"):
> 
> Your connection is not private
> Attackers might be trying to steal your information from musr03.triumf.ca (for example, passwords, messages, or credit cards). Learn more
> NET::ERR_CERT_AUTHORITY_INVALID
>  
> Help improve Safe Browsing by sending some system information and page content to Google. Privacy policy
> musr03.triumf.ca normally uses encryption to protect your information. When Google Chrome tried to connect to musr03.triumf.ca this time, the website sent back unusual and incorrect credentials. This may happen when an 
> attacker is trying to pretend to be musr03.triumf.ca, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Google Chrome stopped the connection before any data was exchanged.
> 
> You cannot visit musr03.triumf.ca right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

To convert a MIDAS frontend to C++ follow this checklist:

a) add #include "mfe.h" after include of midas.h and fix all compilation errors.

NOTE: there should be no "extern C"  brackets around MIDAS include files.

NOTE: Expect to see following problems:

a1) duplicate or mismatched declarations of functions defined in mfe.h
a2) frontend_name and frontend_file_name should be "const char*" instead of "char*"
a3) duplicate "HNDLE hDB" collision with hDB from mfe.c - not sure why it worked before, either use HNDLE hDB from mfe.h or use "extern HNDLE hDB".
a4) poll_event() and interrupt_configure() have "source" as "int[]" instead of "int" (why did this work before?)
a5) use of "extern int frontend_index" instead of get_frontend_index() from mfe.h
a6) bk_create() last argument needs to be cast to (void**)
a7) "bool debug" collides with "debug" from mfe.h (why did this work before?)

b) remove no longer needed "extern C" brackets around mfe related code. Ideally there should be no "extern C" brackets anywhere.

c) in the Makefile, change CC=gcc to CC=g++ for compiling and linking everything as C++

c1) fix all compilation problems. most valid C code will compile as valid C++, but there is some known trouble:
- return value of malloc() & co needs to be cast to the correct data type: "char* s = (char*)malloc(...)"
- some C++ compilers complain about mismatch between signed and unsigned values

If you need help with converting your frontend from C to C++, I will be most happy
to assist you - post your compiler error messages to this forum or email them to me privately.

Good luck,
K.O.

> To convert a MIDAS frontend to C++ follow this checklist:

Pierre A.-A. reminded me that include files for CAEN libraries have  to
use "extern C" brackets:

some 3rd party libraries (CAEN, etc) are written in C (or require C linkage),
if their include files are not C++ compatible (do not have "extern C" brackets
for all exported symbols), the experiment frontend code must say something like this:

extern "C" {
#include "3rd-party-c-library.h"
}

Note: "#ifdef cplusplus" is not needed because we already know we are C++, not C.

K.O.

Big thanks to Peter Kunz - specifically when using the CAEN libraries:

>
> After upgrading to the lastes MIDAS version I got the DAQ frontend of my application running by
> changing all compiler directives from cc to g++ and using
>
> #include "mfe.h"
>
> extern HNDLE hDB
> 
>  extern "C" { 
>  #include <CAENComm.h>  
>  }
>
> With these changes everything seems to work fine.
>


K.O.



> > To convert a MIDAS frontend to C++ follow this checklist:
> 
> Pierre A.-A. reminded me that include files for CAEN libraries have  to
> use "extern C" brackets:
> 
> some 3rd party libraries (CAEN, etc) are written in C (or require C linkage),
> if their include files are not C++ compatible (do not have "extern C" brackets
> for all exported symbols), the experiment frontend code must say something like this:
> 
> extern "C" {
> #include "3rd-party-c-library.h"
> }
> 
> Note: "#ifdef cplusplus" is not needed because we already know we are C++, not C.
> 
> K.O.

I am moving our fe code to c++ midas with cmake. I did encounter your a) problems.

How do I solve mismatched declarations in the mfe (or other places in the midas code)? It is having issues with the midas defined BOOL/... types. This 
is what I get for a minimal scfe:

[ 12%] Building CXX object CMakeFiles/sc_fe_mini.dir/sc_fe_mini.cpp.o
[ 25%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/class/hv.cxx.o
[ 37%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/class/multi.cxx.o
[ 50%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/device/nulldev.cxx.o
[ 62%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/bus/null.cxx.o
[ 75%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/device/mscbdev.cxx.o
[ 87%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/mscb/src/mscb.cxx.o
[100%] Linking CXX executable sc_fe_mini
/home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `_readout_thread':
/home/frederik/packages/midas/src/mfe.cxx:1271: undefined reference to `poll_event(int, int, unsigned int)'
/home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `check_polled_events':
/home/frederik/packages/midas/src/mfe.cxx:1601: undefined reference to `poll_event(int, int, unsigned int)'
/home/frederik/packages/midas/src/mfe.cxx:1643: undefined reference to `poll_event(int, int, unsigned int)'
/home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `readout_enable(unsigned int)':
/home/frederik/packages/midas/src/mfe.cxx:1158: undefined reference to `interrupt_configure(int, int, long)'
/home/frederik/packages/midas/src/mfe.cxx:1156: undefined reference to `interrupt_configure(int, int, long)'
/home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `initialize_equipment':
/home/frederik/packages/midas/src/mfe.cxx:614: undefined reference to `interrupt_configure(int, int, long)'
/home/frederik/packages/midas/src/mfe.cxx:649: undefined reference to `poll_event(int, int, unsigned int)'
/home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `scheduler':
/home/frederik/packages/midas/src/mfe.cxx:1890: undefined reference to `poll_event(int, int, unsigned int)'
/home/frederik/packages/midas/src/mfe.cxx:1932: undefined reference to `poll_event(int, int, unsigned int)'
/home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `main':
/home/frederik/packages/midas/src/mfe.cxx:2701: undefined reference to `interrupt_configure(int, int, long)'
/home/frederik/packages/midas/src/mfe.cxx:2702: undefined reference to `interrupt_configure(int, int, long)'
collect2: error: ld returned 1 exit status
make[2]: *** [sc_fe_mini] Error 1
make[1]: *** [CMakeFiles/sc_fe_mini.dir/all] Error 2
make: *** [all] Error 2


This is my cmakelists for my user code:

#
# cmake for the muX software
#
cmake_minimum_required(VERSION 3.3)

project(muX)

#
# find installations
#
set(MIDAS_DIR $ENV{MIDASSYS})
message("MIDAS dir: " ${MIDAS_DIR})

#
# set directories
#
set(MIDASBUILD_DIR ${MIDAS_DIR}/build)
set(MIDASINCLUDE_DIR ${MIDAS_DIR}/include)
set(MXML_DIR ${MIDAS_DIR}/mxml)
set(MSCB_DIR ${MIDAS_DIR}/mscb)
set(DRV_DIR ${MIDAS_DIR}/drivers)


#
# drivers, libs
#
set(DRIVERS
    ${MIDAS_DIR}/drivers/class/hv
    ${MIDAS_DIR}/drivers/class/multi
    ${MIDAS_DIR}/drivers/device/nulldev
    ${MIDAS_DIR}/drivers/bus/null
)
set(MIDASLIB ${MIDASBUILD_DIR}/libmidas.a)
set(FELIB ${MIDASBUILD_DIR}/libmfe.a)

#
# sc_fe
#
add_executable(sc_fe_mini
        sc_fe_mini.cpp
        ${DRIVERS}
        ${MIDAS_DIR}/drivers/device/mscbdev
        ${MIDAS_DIR}/mscb/src/mscb)

target_include_directories(sc_fe_mini PRIVATE ${DRV_DIR} ${MIDAS_DIR}/mscb/include ${MIDAS_DIR}/include)
target_link_libraries(sc_fe_mini ${LIBS} ${MIDASLIB} ${FELIB} rt pthread util)



I seem to be able to compile the current midas distributions, including the scfe frontend



> To convert a MIDAS frontend to C++ follow this checklist:
> 
> a) add #include "mfe.h" after include of midas.h and fix all compilation errors.
> 
> NOTE: there should be no "extern C"  brackets around MIDAS include files.
> 
> NOTE: Expect to see following problems:
> 
> a1) duplicate or mismatched declarations of functions defined in mfe.h
> a2) frontend_name and frontend_file_name should be "const char*" instead of "char*"
> a3) duplicate "HNDLE hDB" collision with hDB from mfe.c - not sure why it worked before, either use HNDLE hDB from mfe.h or use "extern HNDLE hDB".
> a4) poll_event() and interrupt_configure() have "source" as "int[]" instead of "int" (why did this work before?)
> a5) use of "extern int frontend_index" instead of get_frontend_index() from mfe.h
> a6) bk_create() last argument needs to be cast to (void**)
> a7) "bool debug" collides with "debug" from mfe.h (why did this work before?)
> 
> b) remove no longer needed "extern C" brackets around mfe related code. Ideally there should be no "extern C" brackets anywhere.
> 
> c) in the Makefile, change CC=gcc to CC=g++ for compiling and linking everything as C++
> 
> c1) fix all compilation problems. most valid C code will compile as valid C++, but there is some known trouble:
> - return value of malloc() & co needs to be cast to the correct data type: "char* s = (char*)malloc(...)"
> - some C++ compilers complain about mismatch between signed and unsigned values
> 
> If you need help with converting your frontend from C to C++, I will be most happy
> to assist you - post your compiler error messages to this forum or email them to me privately.
> 
> Good luck,
> K.O.

Did you include mfe.h as written in elog:1526 ?

Stefan

> Did you include mfe.h as written in elog:1526 ?
> 
> Stefan


Yes I did

this is my include 

#include <stdio.h>
#include <string.h>
#include <assert.h>
#include <math.h>
#include <pthread.h>


#include "midas.h"
#include "mscb.h"
#include "multi.h"
#include "mscbdev.h"
#include "mfe.h"

(I attach my dummy fe)

What confuses me is that I can compile examples/experiment/ if I copy that to a 
fresh dir.

I also copied the CMakeLists from the example:

#
# cmake for the muX software
#
cmake_minimum_required(VERSION 3.3)

project(muX)

#
# find midas installation, from CMakeLists in examples/experiment
#
set(MIDAS_DIR $ENV{MIDASSYS})
message("MIDAS dir: " ${MIDAS_DIR})
if (NOT EXISTS $ENV{MIDASSYS})
  message(FATAL_ERROR "Environment variable $MIDASSYS not defined, aborting.")
endif()

set(INC_PATH ${MIDAS_DIR}/include ${MIDAS_DIR}/mxml ${MIDAS_DIR}/mscb/include 
${MIDAS_DIR}/drivers/class ${MIDAS_DIR}/drivers/device)
link_directories($ENV{MIDASSYS}/lib)

# enable certain compile warnings
add_compile_options(-Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-
aliasing -Wuninitialized -Wno-unused-function)

set(LIBS -lpthread -lutil -lrt) 


add_executable(sc_fe_mini sc_fe_mini.cpp)
target_include_directories(sc_fe_mini PRIVATE ${INC_PATH})
target_link_libraries(sc_fe_mini mfe midas ${LIBS})

Have you left any "extern C" in your frontend program or in any of the used header file. Seems 
like the linker cannot find the poll_event in your frontend code. If it's there, but it's compiled 
with C calling (instead of C++), the name mangling causes it to be invisible to the linker. That 
usually happens if there is somewhere

extern C {
  INT poll_event();
  ... 
}

while it is NOT defined as "extern C" in mfe.h which is used by mfe.cxx.

Stefan

> > Did you include mfe.h as written in elog:1526 ?
> > 
> > Stefan
> 
> 
> Yes I did
> 
> this is my include 
> 
> #include <stdio.h>
> #include <string.h>
> #include <assert.h>
> #include <math.h>
> #include <pthread.h>
> 
> 
> #include "midas.h"
> #include "mscb.h"
> #include "multi.h"
> #include "mscbdev.h"
> #include "mfe.h"
> 
> (I attach my dummy fe)
> 
> What confuses me is that I can compile examples/experiment/ if I copy that to a 
> fresh dir.
> 
> I also copied the CMakeLists from the example:
> 
> #
> # cmake for the muX software
> #
> cmake_minimum_required(VERSION 3.3)
> 
> project(muX)
> 
> #
> # find midas installation, from CMakeLists in examples/experiment
> #
> set(MIDAS_DIR $ENV{MIDASSYS})
> message("MIDAS dir: " ${MIDAS_DIR})
> if (NOT EXISTS $ENV{MIDASSYS})
>   message(FATAL_ERROR "Environment variable $MIDASSYS not defined, aborting.")
> endif()
> 
> set(INC_PATH ${MIDAS_DIR}/include ${MIDAS_DIR}/mxml ${MIDAS_DIR}/mscb/include 
> ${MIDAS_DIR}/drivers/class ${MIDAS_DIR}/drivers/device)
> link_directories($ENV{MIDASSYS}/lib)
> 
> # enable certain compile warnings
> add_compile_options(-Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-
> aliasing -Wuninitialized -Wno-unused-function)
> 
> set(LIBS -lpthread -lutil -lrt) 
> 
> 
> add_executable(sc_fe_mini sc_fe_mini.cpp)
> target_include_directories(sc_fe_mini PRIVATE ${INC_PATH})
> target_link_libraries(sc_fe_mini mfe midas ${LIBS})

Can you post the exact command that cmake executes to link sc_fe_mini (with make VERBOSE=1)?

I have noticed similar linking problems that depended on the order when linking. In my case, it 
compiled when "-lpthread -lutil -lrt" were at the end of the command, but not before mfe.o and 
libmidas.a. Unfortunately I haven't found a way to tell cmake the "correct" order of the link 
libraries.

Maybe this can be fixed by adding midas as a subdirectory in your cmake project and just linking 
against the "mfe" target instead of libmfe.a.

Ok, so the detail that I missed was that the dummy functions

INT poll_event(INT source[], INT count, BOOL test)
{
   return 1;
};

as shown in some of the older examples, work when you set

INT poll_event(INT source, INT count, BOOL test)
{
   return 1;
};


as a side comment, not all drivers are c++ compatible yet (e.g. mscbvr), so changes needed are small 

> as a side comment, not all drivers are c++ compatible yet (e.g. mscbvr), so changes needed are small 

Right. We recently switched the whole midas to c++, but we could not cover all drivers. Most of them just need some type 
casting to compile under c++. I got already patches from several people which I'm happy to merge in. If you got mscbhvr 
or any other driver compile under c++, please send me the diff.

Stefan

> 
> Ok, so the detail that I missed was that the dummy functions
> 
> INT poll_event(INT source[], INT count, BOOL test)
> {
>    return 1;
> };
> 
> as shown in some of the older examples, work when you set
> 
> INT poll_event(INT source, INT count, BOOL test)
> {
>    return 1;
> };

If you would have read elog:1526 more carefully (point a4) you would have saved yourself a lot of time.

Stefan

> How do I solve mismatched declarations in the mfe (or other places in the midas code)?

I run into such problems all the time. My solution? I grep for the function name in my code and in the header file,
then look very carefully at the definition to confirm that all the argument declarations are the same in both
places. Sometimes my eyes do not see the difference and I ask for a "second pair of eyes".

In your case, you have a mismatch between functions in mfe.h and in your frontend. The difference
is "int source" in mfe.h and "int source[]" in your code.

Because C++ permits functions with identical namesm but different arguments, the compiler thinks
you did this on purpose and does not complain. Later, of course, the linker bombs,
but all it can report at this stage, is what you see "function not found"... Then you grep your code
for the missing function, check arguments, rinse, repeat.

Before C++, the C compiler would probably had complained about the mismatch, except that MIDAS
did not have an mfe.h header file definitions for all this stuff until just now, so again, the mismatch would
have gone unnoticed, unfixed.

K.O.



> It is having issues with the midas defined BOOL/... types. This 
> is what I get for a minimal scfe:
> 
> [ 12%] Building CXX object CMakeFiles/sc_fe_mini.dir/sc_fe_mini.cpp.o
> [ 25%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/class/hv.cxx.o
> [ 37%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/class/multi.cxx.o
> [ 50%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/device/nulldev.cxx.o
> [ 62%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/bus/null.cxx.o
> [ 75%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/drivers/device/mscbdev.cxx.o
> [ 87%] Building CXX object CMakeFiles/sc_fe_mini.dir/home/frederik/packages/midas/mscb/src/mscb.cxx.o
> [100%] Linking CXX executable sc_fe_mini
> /home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `_readout_thread':
> /home/frederik/packages/midas/src/mfe.cxx:1271: undefined reference to `poll_event(int, int, unsigned int)'
> /home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `check_polled_events':
> /home/frederik/packages/midas/src/mfe.cxx:1601: undefined reference to `poll_event(int, int, unsigned int)'
> /home/frederik/packages/midas/src/mfe.cxx:1643: undefined reference to `poll_event(int, int, unsigned int)'
> /home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `readout_enable(unsigned int)':
> /home/frederik/packages/midas/src/mfe.cxx:1158: undefined reference to `interrupt_configure(int, int, long)'
> /home/frederik/packages/midas/src/mfe.cxx:1156: undefined reference to `interrupt_configure(int, int, long)'
> /home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `initialize_equipment':
> /home/frederik/packages/midas/src/mfe.cxx:614: undefined reference to `interrupt_configure(int, int, long)'
> /home/frederik/packages/midas/src/mfe.cxx:649: undefined reference to `poll_event(int, int, unsigned int)'
> /home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `scheduler':
> /home/frederik/packages/midas/src/mfe.cxx:1890: undefined reference to `poll_event(int, int, unsigned int)'
> /home/frederik/packages/midas/src/mfe.cxx:1932: undefined reference to `poll_event(int, int, unsigned int)'
> /home/frederik/packages/midas/build/libmfe.a(mfe.cxx.o): In function `main':
> /home/frederik/packages/midas/src/mfe.cxx:2701: undefined reference to `interrupt_configure(int, int, long)'
> /home/frederik/packages/midas/src/mfe.cxx:2702: undefined reference to `interrupt_configure(int, int, long)'
> collect2: error: ld returned 1 exit status
> make[2]: *** [sc_fe_mini] Error 1
> make[1]: *** [CMakeFiles/sc_fe_mini.dir/all] Error 2
> make: *** [all] Error 2
> 
> 
> This is my cmakelists for my user code:
> 
> #
> # cmake for the muX software
> #
> cmake_minimum_required(VERSION 3.3)
> 
> project(muX)
> 
> #
> # find installations
> #
> set(MIDAS_DIR $ENV{MIDASSYS})
> message("MIDAS dir: " ${MIDAS_DIR})
> 
> #
> # set directories
> #
> set(MIDASBUILD_DIR ${MIDAS_DIR}/build)
> set(MIDASINCLUDE_DIR ${MIDAS_DIR}/include)
> set(MXML_DIR ${MIDAS_DIR}/mxml)
> set(MSCB_DIR ${MIDAS_DIR}/mscb)
> set(DRV_DIR ${MIDAS_DIR}/drivers)
> 
> 
> #
> # drivers, libs
> #
> set(DRIVERS
>     ${MIDAS_DIR}/drivers/class/hv
>     ${MIDAS_DIR}/drivers/class/multi
>     ${MIDAS_DIR}/drivers/device/nulldev
>     ${MIDAS_DIR}/drivers/bus/null
> )
> set(MIDASLIB ${MIDASBUILD_DIR}/libmidas.a)
> set(FELIB ${MIDASBUILD_DIR}/libmfe.a)
> 
> #
> # sc_fe
> #
> add_executable(sc_fe_mini
>         sc_fe_mini.cpp
>         ${DRIVERS}
>         ${MIDAS_DIR}/drivers/device/mscbdev
>         ${MIDAS_DIR}/mscb/src/mscb)
> 
> target_include_directories(sc_fe_mini PRIVATE ${DRV_DIR} ${MIDAS_DIR}/mscb/include ${MIDAS_DIR}/include)
> target_link_libraries(sc_fe_mini ${LIBS} ${MIDASLIB} ${FELIB} rt pthread util)
> 
> 
> 
> I seem to be able to compile the current midas distributions, including the scfe frontend
> 
> 
> 
> > To convert a MIDAS frontend to C++ follow this checklist:
> > 
> > a) add #include "mfe.h" after include of midas.h and fix all compilation errors.
> > 
> > NOTE: there should be no "extern C"  brackets around MIDAS include files.
> > 
> > NOTE: Expect to see following problems:
> > 
> > a1) duplicate or mismatched declarations of functions defined in mfe.h
> > a2) frontend_name and frontend_file_name should be "const char*" instead of "char*"
> > a3) duplicate "HNDLE hDB" collision with hDB from mfe.c - not sure why it worked before, either use HNDLE hDB from mfe.h or use "extern HNDLE hDB".
> > a4) poll_event() and interrupt_configure() have "source" as "int[]" instead of "int" (why did this work before?)
> > a5) use of "extern int frontend_index" instead of get_frontend_index() from mfe.h
> > a6) bk_create() last argument needs to be cast to (void**)
> > a7) "bool debug" collides with "debug" from mfe.h (why did this work before?)
> > 
> > b) remove no longer needed "extern C" brackets around mfe related code. Ideally there should be no "extern C" brackets anywhere.
> > 
> > c) in the Makefile, change CC=gcc to CC=g++ for compiling and linking everything as C++
> > 
> > c1) fix all compilation problems. most valid C code will compile as valid C++, but there is some known trouble:
> > - return value of malloc() & co needs to be cast to the correct data type: "char* s = (char*)malloc(...)"
> > - some C++ compilers complain about mismatch between signed and unsigned values
> > 
> > If you need help with converting your frontend from C to C++, I will be most happy
> > to assist you - post your compiler error messages to this forum or email them to me privately.
> > 
> > Good luck,
> > K.O.

I now have a centos-8 computer and I tried midas on it:

- the develop and midas-2019-09 branches build, mhttpd runs
- there are compiler warnings about use of strncpy() that need to be looked into, but see https://stackoverflow.com/questions/50198319/gcc-8-wstringop-truncation-what-is-
the-good-practice
- mhttpd built-in https support does not seem to work (see the other forum thread)
- apache httpd proxy for https can be made to work, but there are problems with certbot.

K.O.

Hello,

I am quite new in both Linux and MIDAS.
I have install MIDAS on my desktop by going through this link: 
https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux 

in the last step when I send "mhttpd" command and try to open the link 
https://localhost:8443 (of course, changing the localhost with my host name), it 
failed to connect and shows this error: SSL_ERROR_NO_CYPHER_OVERLAP (please see 
attached file includes a screenshot of the error).

I have tried many ways to solve this problem: In Firefox: going to option/privacy 
and security/ security and uncheck the option "Block dangerous and deceptive 
content". but it does not help.

Looking forward your help
Thanks
Mehran

> I am quite new in both Linux and MIDAS.
> I have install MIDAS on my desktop by going through this link: 
> https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux 
> 
> in the last step when I send "mhttpd" command and try to open the link 
> https://localhost:8443 (of course, changing the localhost with my host name), it 
> failed to connect and shows this error: SSL_ERROR_NO_CYPHER_OVERLAP (please see 
> attached file includes a screenshot of the error).

What Linux? (on most linuxes, run "lsb_release -a")
What version of midas? (run odbedit "ver" command)
What version of firefox? (from the "about firefox" menu)

> I have tried many ways to solve this problem: In Firefox: going to option/privacy 
> and security/ security and uncheck the option "Block dangerous and deceptive 
> content". but it does not help.

No you cannot fix it from inside firefox. The issue is that the overlap of encryption methods
supported by your firefox and by your openssl library (used by mhttpd) is an empty set.
No common language, so to say, communication is impossible.

So either you have a very old openssl but very new firefox, or a very new openssl but very old 
firefox. Both very old or both very new can talk to each other, difficulties start with greater  
difference in age, as new (better) encryption methods are added and old (no-longer-secure) 
methods are banished.

BTW, for good security we recommend using apache httpd as the https proxy (instead of built-in 
https support in mhttpd). (I am not sure what it says in the current documentation). (But apache 
httpd will use the same openssl library, so this may not solve your problem. Let's see what 
versions of software you are using, per questions above, first).

K.O.

Hi Konstantin,
Thanks for your reply, 

> What Linux? (on most linuxes, run "lsb_release -a")
> What version of midas? (run odbedit "ver" command)
I am using CentOS 8

> What version of firefox? (from the "about firefox" menu)
Firefox 71.0

Thanks 
Mehran

> No you cannot fix it from inside firefox. The issue is that the overlap of encryption methods
> supported by your firefox and by your openssl library (used by mhttpd) is an empty set.
> No common language, so to say, communication is impossible.
> 
> So either you have a very old openssl but very new firefox, or a very new openssl but very old 
> firefox. Both very old or both very new can talk to each other, difficulties start with greater  
> difference in age, as new (better) encryption methods are added and old (no-longer-secure) 
> methods are banished.
> 
> BTW, for good security we recommend using apache httpd as the https proxy (instead of built-in 
> https support in mhttpd). (I am not sure what it says in the current documentation). (But apache 
> httpd will use the same openssl library, so this may not solve your problem. Let's see what 
> versions of software you are using, per questions above, first).
> 
> K.O.

Hi, I have not run midas on Centos-8 yet. Maybe there is a problem with the openssl library there. The Centos-7 
instructions for setting up apache httpd proxy are here, with luck they work on centos-8:
https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29

K.O.


> Hi Konstantin,
> Thanks for your reply, 
> 
> > What Linux? (on most linuxes, run "lsb_release -a")
> > What version of midas? (run odbedit "ver" command)
> I am using CentOS 8
> 
> > What version of firefox? (from the "about firefox" menu)
> Firefox 71.0
> 
> Thanks 
> Mehran
> 
> > No you cannot fix it from inside firefox. The issue is that the overlap of encryption methods
> > supported by your firefox and by your openssl library (used by mhttpd) is an empty set.
> > No common language, so to say, communication is impossible.
> > 
> > So either you have a very old openssl but very new firefox, or a very new openssl but very old 
> > firefox. Both very old or both very new can talk to each other, difficulties start with greater  
> > difference in age, as new (better) encryption methods are added and old (no-longer-secure) 
> > methods are banished.
> > 
> > BTW, for good security we recommend using apache httpd as the https proxy (instead of built-in 
> > https support in mhttpd). (I am not sure what it says in the current documentation). (But apache 
> > httpd will use the same openssl library, so this may not solve your problem. Let's see what 
> > versions of software you are using, per questions above, first).
> > 
> > K.O.

Hi,
As, the link suggests, I perform "yum install -y mod_ssl certwatch crypto-utils" but it complains as:
No match for argument: certwatch
No match for argument: crypto-utils

You may have a look on this link: https://blog.cloudware.bg/en/whats-new-in-centos-linux-8/
What’s gone?
In with the new, out with the old. CentOS 8 also says goodbye to some features. The OS removes several security functionalities. Among them is the Clevis HTTP pin, Coolkey and crypto-utils.
Cent OS 8 comes with securetty disabled by default. The configuration file is no longer included. You can add it back, but you will have to do it yourself. Another change is that shadow-utils no longer allow all-numeric user and group names.

Thanks
Mehran

> Hi, I have not run midas on Centos-8 yet. Maybe there is a problem with the openssl library there. The Centos-7 
> instructions for setting up apache httpd proxy are here, with luck they work on centos-8:
> https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
> 
> K.O.
> 

> > The Centos-7 instructions for setting up apache httpd proxy are here, with luck they work on centos-8:
> > https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29

I now have a centos-8 computer, I followed my instructions and they generally worked.

There is a number of problems with the certbot package that prevent me from writing coherent production quality instructions for centos-8.

But at the end I was successful, httpd runs, gets "A+" rating from SSLlabs, forwards the requests to mhttpd, I can access the midas status page etc.

With luck the certbot packages for centos-8 will be sorted out soon (the apache plugin seems to be missing, this causes the automatic
certificate renewal to not work) and I will update my instructions to include centos-8.

Until then, I recommend that people continue to use centos-7 or the current Ubuntu LTS release.

K.O.

> I am using CentOS 8 [and]
> Firefox 71.0

I now have a centos-8 machine, I successfully built midas and I confirm that there is a problem.

But I get different errors from you:

- google chrome - does not connect at all (without any useful error message: "This site can’t be reached. The 
connection was reset.")
- firefox complains about the self-signed certificate, but connects ok, I see the midas status page and it works. "page 
info" reports connection is TLS 1.3, encryption TLS_AES_128_GCM_SHA256. However, the function "view certificate" 
does not work (without any useful error message).

I tried to run the SSLlabs tool to get some more information from mhttpd, but it does not want to run against mhttpd on 
port 8443... I do have a port redirect program somewhere... need to find it...

K.O.

I created the release branch for midas-2019-09 and tag midas-2019-09-a.

Since the previous release midas-2019-06, some news:

- new history graphics (Stefan)
- c++ frontend framework mvodb.h and tmfe.h merged from ALPHA-g (K.O.)
- we think we have all the fallout from switching to cmake and to c++11 sorted out

There is a number of known problems with the current code, see the bitbucket bug tracker:
https://bitbucket.org/tmidas/midas/issues?status=new&status=open

Hopefully we can use this release as a baseline for more testing and with luck we will
fix all the pending bugs and add all the pending missing code (the new sequencer web pages,
the "m" analyzer, etc) quickly and our next release midas-2019-10 will be the best midas ever.

To obtain this release, either checkout the top of branch feature/midas-2019-09 (recommended)
or checkout the tag midas-2019-09-a.

If you are using the last pre-cmake/c++ release midas-2019-03, I recommend that you stay with it
until our next release midas-2019-10.

K.O.

> I created the release branch for midas-2019-09 and tag midas-2019-09-a.
> Since the previous release midas-2019-06, some news:
> 
> - new history graphics (Stefan)
> - c++ frontend framework mvodb.h and tmfe.h merged from ALPHA-g (K.O.)
> - we think we have all the fallout from switching to cmake and to c++11 sorted out
> 

midas-2019-09-e is here.

- the new history plots now work both for Stefan *and* for me, please try them out!
- no new problems with cmake and c++11.
- fixes for some reported bugs
- some bugs remain to be fixed, so with luck, there will by a midas-2019-09-f.

> add all the pending missing code (the new sequencer web pages, the "m" analyzer, etc

pending for midas-2019-12:

- new sequencer web pages
- the "m" analyzer merge (from rootana)
- python-client branch merge (thanks to Ben!)
- simplified odb settings for mlogger and mhttpd configuration
- mhttpd update to mongoose 6.16

To obtain this release, either checkout the top of branch feature/midas-2019-09 (recommended)
or checkout the tag midas-2019-09-e.

K.O.

midas-2019-09-g is here.

- the last bug in the new history plots is fixed, please try them out, plus
- "<<" and "<<<" buttons now work for going back to the old data
- "+" and "-" buttons are added for zooming in and out.

- the new sequencer web pages have been activated, the old sequencer page moved to "OldSequencer"

To obtain this release, either checkout the top of branch feature/midas-2019-09 (recommended)
or checkout the tag midas-2019-09-g.

K.O.


> > I created the release branch for midas-2019-09 and tag midas-2019-09-a.
> > Since the previous release midas-2019-06, some news:
> > 
> > - new history graphics (Stefan)
> > - c++ frontend framework mvodb.h and tmfe.h merged from ALPHA-g (K.O.)
> > - we think we have all the fallout from switching to cmake and to c++11 sorted out
> > 
> 
> midas-2019-09-e is here.
> 
> - the new history plots now work both for Stefan *and* for me, please try them out!
> - no new problems with cmake and c++11.
> - fixes for some reported bugs
> - some bugs remain to be fixed, so with luck, there will by a midas-2019-09-f.
> 
> > add all the pending missing code (the new sequencer web pages, the "m" analyzer, etc
> 
> pending for midas-2019-12:
> 
> - new sequencer web pages
> - the "m" analyzer merge (from rootana)
> - python-client branch merge (thanks to Ben!)
> - simplified odb settings for mlogger and mhttpd configuration
> - mhttpd update to mongoose 6.16
> 
> To obtain this release, either checkout the top of branch feature/midas-2019-09 (recommended)
> or checkout the tag midas-2019-09-e.
> 
> K.O.

midas-2019-09-i is here.

- the new sequencer web pages written in html+javascript (NewSequencer), the old c-generated sequencer pages still work (Sequencer)
- python-client from Ben Smith merged in, see documentation at https://bitbucket.org/tmidas/midas/src/develop/python/

To obtain this release, either checkout the top of branch feature/midas-2019-09 (recommended)
or checkout the tag midas-2019-09-i.

K.O.

The default el6 (RHEL/SL/CentOS-6) compiler is gcc-4.4.7, it does not support c++11, not even a little bit.

Do this to install newer c++ compilers and build MIDAS with c++11:

ssh root@sl6machine
# yum install centos-release-scl-rh
# yum install devtoolset-8
# yum install cmake3
# scl -l
devtoolset-8
...

$ ssh user@sl6machine
$ scl enable devtoolset-8 bash
$ gcc -v
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) 
$ cd git/midas
$ make cclean
$ make cmake3
$ ls -l bin/odbedit

K.O.

> The default el6 (RHEL/SL/CentOS-6) compiler is gcc-4.4.7, it does not support c++11, not even a little bit.

The previously posted instructions are incomplete - one cannot cross-compile 32-bit executables (i.e. for running on 32-bit VME 
processors) because 64-bit packages are missing 4 files for the 32-bit C++ standard library (libstdc++_nonshared.a).

After a bit of searching I found the missing files, i.e. here:
https://copr-be.cloud.fedoraproject.org/results/mayeut/devtoolset-8/epel-6-i386/01045166-devtoolset-8-gcc/

There are 2 options:

a) install the 32-bit development package:
rpm -vh --install https://ladd00.triumf.ca/~olchansk/devtoolset-8/devtoolset-8-libstdc++-devel-8.3.1-3.1.el6.i686.rpm

b) install just the 4 missing files from here:
https://ladd00.triumf.ca/~olchansk/devtoolset-8/i686-redhat-linux/8/
into
/opt/rh/devtoolset-8/root/usr/lib/gcc/i686-redhat-linux/8/

After doing this, "make linux32" builds. (requires latest midas-2019-09 for minor Makefile fixes)

K.O.


> 
> Do this to install newer c++ compilers and build MIDAS with c++11:
> 
> ssh root@sl6machine
> # yum install centos-release-scl-rh
> # yum install devtoolset-8
> # yum install cmake3
> # scl -l
> devtoolset-8
> ...
> 
> $ ssh user@sl6machine
> $ scl enable devtoolset-8 bash
> $ gcc -v
> COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
> gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) 
> $ cd git/midas
> $ make cclean
> $ make cmake3
> $ ls -l bin/odbedit
> 
> K.O.

I have written a system monitor tool for MIDAS, that has been merged in the develop branch today: sysmon

https://bitbucket.org/tmidas/midas/pull-requests/8/system-monitoring-a-new-frontend-to-log/diff

To use it, simply run the new program
sysmon
on any host that you want to monitor, no configuring required.




The program is a frontend for MIDAS, there is no need for configuration, as upon initialisation it builds a history display for you. Simply run one instance per machine you want to monitor. By default, it only logs once per 10 seconds.

The equipment name is derived from the hostname, so multiple instances can be run across multiple machines without conflict. A new history display will be created for each host.

sysmon uses the /proc pseudo-filesystem, so unfortunately only linux is supported. It does however work with multiple architectures, so x86 and ARM processors are supported.

If the build machine has NVIDIA drivers installed, there is an additional version of sysmon that gets built: sysmon-nvidia. This will log the GPU temperature and usage, as well as CPU, memory and swap. A host should only run either sysmon or sysmon-nvidia

elog:1727/1 shows the History Display generated by sysmon-nvidia. sysmon would only generate the first two displays (sysmon/localhost and sysmon/localhost-CPU)

A little advertised feature of the modifications needed support the msysmon program is 
that MIDAS equipment names can support the injecting of the hostname of the system 
running the frontend at runtime (register_equipment(void)).

https://midas.triumf.ca/MidasWiki/index.php/Equipment_List_Parameters#Equipment_Name

A special string ${HOSTNAME} can be put in any position in the equipment name. It will 
be replaced with the hostname of the computer running the frontend at run-time. Note, 
the frontend_name string will be trimmed down to 32 characters.
Example usage: msysmon


EQUIPMENT equipment[] = {

  { "${HOSTNAME}_msysmon",   /* equipment name */    {
      EVID_MONITOR, 0,      /* event ID, trigger mask */
      "SYSTEM",             /* event buffer */
      EQ_PERIODIC,          /* equipment type */
      0,                    /* event source */
      "MIDAS",              /* format */
      TRUE,                 /* enabled */
      RO_ALWAYS,            /* Read when running */
      10000,                /* poll every so milliseconds */
      0,                    /* stop run after this event limit */
      0,                    /* number of sub events */
      1,                    /* history period */
      "", "", ""
    },
    read_system_load,/* readout routine */
  },
  { "" }
};