> /code/midas/manalyzer/manalyzer.cxx:799:27: error: ‘pow’ was not declared in this scope
>   799 |       bins[i] = TimeRange*pow(1.1,i)/pow(1.1,Nbins);

math.h added, pushed. nice catch.

implicit include of math.h came through TFile.h (ROOT v6.34.02), perhaps you have a newer ROOT
and they jiggled the include files somehow.

TFile.h -> TDirectoryFile.h -> TDirectory.h -> TNamed.h -> TString.h -> TMathBase.h -> cmath -> math.h

K.O.

Thanks. Are you happy for me to update the submodule commit in Midas to use this fix? I should have sufficient permission if you agree.

> > /code/midas/manalyzer/manalyzer.cxx:799:27: error: ‘pow’ was not declared in this scope
> >   799 |       bins[i] = TimeRange*pow(1.1,i)/pow(1.1,Nbins);
> 
> math.h added, pushed. nice catch.
> 
> implicit include of math.h came through TFile.h (ROOT v6.34.02), perhaps you have a newer ROOT
> and they jiggled the include files somehow.
> 
> TFile.h -> TDirectoryFile.h -> TDirectory.h -> TNamed.h -> TString.h -> TMathBase.h -> cmath -> math.h
> 
> K.O.

> TFile.h -> TDirectoryFile.h -> TDirectory.h -> TNamed.h -> TString.h -> TMathBase.h -> cmath -> math.h

reading ROOT release notes, 6.38 removed TMathBase.h from TString.h, with a warning "This change may cause errors during compilation of 
ROOT-based code". Upright citizens, nice guys!

> Thanks. Are you happy for me to update the submodule commit in Midas to use this fix? I should have sufficient permission if you agree.

I am doing some last minute tests, will pull it into midas and rootana later today.

K.O.

When processing MIDAS files offline, JSROOT did not work, -Rxxx worked, http 
connection would open, but would not serve any histograms. This should now be 
fixed.

In addition, normally, after processing all input MIDAS files, manalyzer would 
exit, JSROOT would abruptly stop. To look at final results one had to open the 
ROOT files using some other method (roody, TBrowser, mjsroot, etc).

I now added a command line switch "--jsroot", if supplied, after processing all 
input MIDAS files, manalyzer will keep running in the JSROOT server mode (same as 
mjsroot).

"manalyzer -R8082 --jsroot run*.mid.lz4" now does something useful: open 
http://localhost:8082 (or ssh tunnel or mhttpd proxy per my mjsroot message) and 
watch histograms fill in real time, after analysis finishes, keep looking at the 
final results until bored. stop manalyzer using Ctrl-C. (we should add a "Stop 
JSROOT" botton to the JSROOT main page).

MIDAS commit 1d0d6448c3ec4ffd225b8d2030fe13e379fcd007

K.O.

We are considering to remove the analyzer framework mana.cxx from MIDAS. It 
currently has some compiler warnings and we wonder if we should fix them which 
would take some time or just remove the file. We have now to much more modern 
analyzer frameworks "manalyzer" and "ROOTANA" which should be used instead.

Is anybody still using the mana.cxx framework?

/Stefan

Ok, no relevant complains so far, so I removed mana and rmana from the CMake build 
process, but left the file mana.cxx still in the repository for educational 
purposes ;-)

Stefan

> Ok, no relevant complains so far, so I removed mana and rmana from the CMake build 
> process, but left the file mana.cxx still in the repository for educational 
> purposes ;-)

+1

K.O.

Stephan, why did you prohibit building mana.c without ROOT and HBOOK
support? I think such a configuration is valid and should be allowed.

Also, this prohibition broke the Midas Makefile, it now bombs building
mana.c. The Makefile is setup for building hmana.c with HBOOK support,
rmana.c with ROOT support (if ROOTSYS is set) and mana.c without HBOOK and
ROOT support (currently bombs on #error in mana.c).

K.O.

> Stephan, why did you prohibit building mana.c without ROOT and HBOOK
> support? I think such a configuration is valid and should be allowed.

Oops, sorry, my fault. I forgto that people use mana.c without ROOT and 
HBOOK. The reason I made the change was that people forgot the -DHVAE_HBOOK 
in their makefile. In that case, no HBOOK init is done in mana.c and the 
first histogram booking in the user code crashes HBOOK.

So please take the #error statement out of mana.c (I'm away in two hours for 
one week), but think about preventing the above mentionend problem. I don't 
know any way for the makefile or mana.c to figure out if there is any HF1 
call in the user code. Actually HF1 should return a "proper" error message 
than just crashing.

One possibility is that we put an additional layer on top of the histogram 
boooking/filling. These macros are converted to their HBOOK or ROOT 
equivalents depending on the HAVE_HBOOK/HAVE_ROOT. If none of both is 
present, the histogram booking macro can produce a runtime error. This has 
the additional advantage that users can switch from HBOOK to ROOT without 
change of their user code.

> > Stephan, why did you prohibit building mana.c without ROOT and HBOOK
> > support? I think such a configuration is valid and should be allowed.
> 
> Oops, sorry, my fault. I forgto that people use mana.c without ROOT and 
> HBOOK. The reason I made the change was that people forgot the -DHVAE_HBOOK 
> in their makefile. In that case, no HBOOK init is done in mana.c and the 
> first histogram booking in the user code crashes HBOOK.

Ahem. There is only so much rope we can give out to prevent people from shooting
themselves in the foot...

> So please take the #error statement out of mana.c

Done.

> One possibility is that we put an additional layer on top of the histogram 
> boooking/filling. These macros are converted to their HBOOK or ROOT 
> equivalents depending on the HAVE_HBOOK/HAVE_ROOT. If none of both is 
> present, the histogram booking macro can produce a runtime error. This has 
> the additional advantage that users can switch from HBOOK to ROOT without 
> change of their user code.

I can't think of anything other than wrapping every HBOOK call with "if
(!hbook_is_initialized) initialize_hbook();". But then, where is PAWC
coming from anyway?!?

We could also print a warning message "This mana.c has no HBOOK support. If you
see HBOOK crashes, please relink with hmana,c". Ugly, but informative, plus it
points anybody who knows how to read towards a solution.

K.O.

Reproduced on el7 (CentOS7). Same thing works on el6 (SL6).

The error is in the SSE4.2-assembly-accelerated library for computing crc32c checksums. I do 
not understand this assembly stuff enough to tell what goes wrong.

In any case, "make linux32" is intended for VME processors that cannot run 64-bit code. These 
processors also happen to not have the SSE4.2 instructions needed for this code to actually 
work, so one solution would be to always disable crc32c SSE4.2-assembly-acceleration for the 
linux32 target.

Note that the original reported was running "make linux32" with the idea of generating code for 
the 32-bit ARM processor.

Here the situation is like this: the required CRC32C instructions are present on 64-bit capable 
ARM processors (RPi3, etc) and probably work in 32-bit mode, and I found the assembly-
language crc32c library that uses them. It needs to be added to MIDAS and tested.

For the older 32-bit-only ARM processors (Cyclone5 FPGA, Rpi2 and older) I do not see any 
hardware accelerated crc32c implementations, so it uses software computed CRC32 always. 
(P.S.: I see the current linux kernels have a hardware accelerated library for CRC32C, not sure if 
it can be imported into MIDAS easily).

K.O.


> $ make linux32 ...
> g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
> Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
> DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -
DMG_ENABLE_SSL 
> -DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
> src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
> src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
>                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
>                                                                   ^
> src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
>                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
>                                                                   ^
> src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
> src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
>                  : "%ebx", "%edx"); \
>                                   ^
> src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
>      SSE42(sse42);
>      ^
> make[1]: *** [lib/crc32c.o] Error 1
> make[1]: Leaving directory `/home/hh19285/packages/midas'
> make: *** [linux32] Error 2
> 
> Could you please help with getting past this? otherwise we may need to change 
> our whole experimental setup.
> 
> Thank you in advance

> Reproduced on el7 (CentOS7). Same thing works on el6 (SL6).

Fixed in commit dd937e6. Only enable SSE4.2 crc32c for 64-bit compilation. Still not sure why it worked for 32-bit 
compilation on el6 (SL6).

K.O.


> 
> The error is in the SSE4.2-assembly-accelerated library for computing crc32c checksums. I do 
> not understand this assembly stuff enough to tell what goes wrong.
> 
> In any case, "make linux32" is intended for VME processors that cannot run 64-bit code. These 
> processors also happen to not have the SSE4.2 instructions needed for this code to actually 
> work, so one solution would be to always disable crc32c SSE4.2-assembly-acceleration for the 
> linux32 target.
> 
> Note that the original reported was running "make linux32" with the idea of generating code for 
> the 32-bit ARM processor.
> 
> Here the situation is like this: the required CRC32C instructions are present on 64-bit capable 
> ARM processors (RPi3, etc) and probably work in 32-bit mode, and I found the assembly-
> language crc32c library that uses them. It needs to be added to MIDAS and tested.
> 
> For the older 32-bit-only ARM processors (Cyclone5 FPGA, Rpi2 and older) I do not see any 
> hardware accelerated crc32c implementations, so it uses software computed CRC32 always. 
> (P.S.: I see the current linux kernels have a hardware accelerated library for CRC32C, not sure if 
> it can be imported into MIDAS easily).
> 
> K.O.
> 
> 
> > $ make linux32 ...
> > g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
> > Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
> > DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -
> DMG_ENABLE_SSL 
> > -DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
> > src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
> > src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
> >                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
> >                                                                   ^
> > src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
> >                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
> >                                                                   ^
> > src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
> > src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
> >                  : "%ebx", "%edx"); \
> >                                   ^
> > src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
> >      SSE42(sse42);
> >      ^
> > make[1]: *** [lib/crc32c.o] Error 1
> > make[1]: Leaving directory `/home/hh19285/packages/midas'
> > make: *** [linux32] Error 2
> > 
> > Could you please help with getting past this? otherwise we may need to change 
> > our whole experimental setup.
> > 
> > Thank you in advance

The Makefile targets for crosscompiling MIDAS are now documented in the MIDAS
Doxygen documentation:

make linux32 & make clean32
make linux64 & make clean64
make crosscompile
make dox

This has to do with which flavour of MIDAS is built by default: 32-bit or 64-bit.

This is how this works now.

Default flavour is determined by ROOT. If ROOTSYS points to 32-bit ROOT, then
32-bit MIDAS is built, if 64-bit ROOT, then 64-bit MIDAS. This works well after
the ROOT team added the correct "-m32" and "-m64" flags to "rootconfig --cflags".

If for some reason, we also need a non-default flavour of MIDAS, for example
when the main daq computer runs 64-bit MIDAS, but one frontend has to run on a
"32-bit only" VME processor, you say "make linux32". This creates the
"linux-m32/{lib,bin}" tree that you then reference in the Makefile of your
special frontend (i.e. instead of "-L$MIDASSYS/linux/lib" say
"-L$MIDASSYS/linux-m32/lib"). "make linux64" works the same way.

These non-default flavours of MIDAS are compiled with most special features
disabled: no ROOT, no MYSQL, etc.

When building "make linux32", you may also see errors caused by missing 32-bit
libraries - many 64-bit Linux distributions do not install the full 32-bit
development environment by default - so some header files and libraries may be
reported as missing. These not-installed-by-default 32-bit packages are usually
easy to install using commands like "yum install libxxx-devel.i386".

K.O.

The capability to generate doxygen documentation of MIDAS was restored.

Use "make dox" and "make cleandox",
find generated documentation in ./html,
look at it via "firefox html/index.html".

The documentation is not generated by default - it takes quite a long time to build all the call graphs.

And the call graphs is what makes this documentation useful - without some visual graphical 
representation it is quite difficult to understand some parts of MIDAS. Both caller and callee graphs are 
generated.

Note that doxygen documentation for the javascript functions in mhttpd.js is also generated, making a 
handy reference in addition to the full documentation on the MIDAS Wiki.

K.O.

> The capability to generate doxygen documentation of MIDAS was restored.
> 
> Use "make dox" and "make cleandox",
> find generated documentation in ./html,
> look at it via "firefox html/index.html".
> 
> The documentation is not generated by default - it takes quite a long time to build all the call graphs.
> 
> And the call graphs is what makes this documentation useful - without some visual graphical 
> representation it is quite difficult to understand some parts of MIDAS. Both caller and callee graphs are 
> generated.
> 
> Note that doxygen documentation for the javascript functions in mhttpd.js is also generated, making a 
> handy reference in addition to the full documentation on the MIDAS Wiki.
> 
> K.O.

To generate the files, you need doxygen installed which not everybody has. Is there a web site where one can see the generated graphs?

/Stefan

> > The capability to generate doxygen documentation of MIDAS was restored.
> > 
> > Use "make dox" and "make cleandox",
> > find generated documentation in ./html,
> > look at it via "firefox html/index.html".
> > 
> 
> To generate the files, you need doxygen installed which not everybody has.

On most Linux systems, doxygen is easy to install. Red Hat instructions are here: 
http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Install_packages_needed_for_QUARTUS.2C_ROOT.2C_EPICS_and_MIDAS_DAQ

On MacOS, doxygen is easy to install via macports: sudo port install doxygen

> Is there a web site where one can see the generated graphs?

http://ladd00.triumf.ca/~olchansk/midas/index.html

there is no cron job to update this, but I may update it infrequently.

K.O.

> On most Linux systems, doxygen is easy to install. Red Hat instructions are here: 
> http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Install_packages_needed_for_QUARTUS.2C_ROOT.2C_EPICS_and_MIDAS_DAQ
> 
> On MacOS, doxygen is easy to install via macports: sudo port install doxygen
> 
> > Is there a web site where one can see the generated graphs?
> 
> http://ladd00.triumf.ca/~olchansk/midas/index.html
> 
> there is no cron job to update this, but I may update it infrequently.
> 
> K.O.

Great, thanks a lot!

-Stefan

Some time ago Ben Smith added test coverage reports using GCC -fprofile-arcs -
ftest-coverage and lcov.

It reports code coverage after running "make test". Reported code coverage is 
surprisingly high for the very little code executed by "make test" - create ODB, 
start a frontend,  start run, stop run, check that mlogger created data files 
and history files.

(Of course coverage can never reach 100%, some obscure branches are unreachable 
without using an automated fuzzer, and some error paths are unreachable without 
also using a system call fault injector).

Simplest way to generate a coverage report:

make clean
make cmake YES_COVERAGE=1
make coverage
open cov_html/index.html

K.O.

I added Makefile rules for running MIDAS through the clang-tidy static code 
analyzer. (rules for running cppcheck have gone missing, I hope I find them).

Reports from clang-tidy are somewhat repetitive (complain about the same class of 
non-problem problems repeatedly), but several warnings and errors identified real 
bugs.

Specifically, detection of memory leaks in error paths and detection of NULL 
pointer dereference and use of uninitialized variables is pretty solid.

Fixes for the worst problems reported by cppcheck and clang-tidy are already 
committed to midas git.

K.O.

As of svn rev 4794, midas builds, runs and should be fully usable on MacOS 10.6.4. Previous revisions did 
not compile due to assorted Linuxisms and did not run because of a sizeof() problem in ss_gettid(). Also 
one of the system header files (mtio.h?) present in MacOS 10.5 vanished from 10.6.

Please continue reporting all problems with midas on macos to this forum.
K.O.