Back Midas Rome Roody Rootana
  Root Analyzer Framework, Page 2 of 3  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
IDup Date Author Topic Subject
  21   22 May 2020 Konstantin OlchanskiReleaserootana-2020-03
I cut a release branch and tag for rootana-2020-03. This release is meant to be used with 
midas release 2020-03.

More recent versions of midas have several incompatibilities with ROOTANA, and I will
be working on resolving them:

- new incompatible data bank format in midas .mid files. (to fix the 64-bit misalignment bug)
- renaming of MIDAS TID_xxx types
- renaming of MIDAS data types in XML ODB dump files.

When we tag the next midas release, we will tag the compatible ROOTANA release and going 
forward hope to have MIDAS and ROOTANA releases done in lock-step.

K.O.
  22   22 May 2020 Konstantin OlchanskiInforootana updates
I am updating the rootana code to include all the latest developments in midas and 
elsewhere:

- mxml, mjson (and soon mvodb, midasio and manalyzer) are moving into git submodules 
(shared with midas and usable as standalone code)
- VirtualOdb is gone, use MVOdb instead
- TMidasFile is gone, use TMReader and TMWriter in midasio.h
- TMidasEvent remains, I will update it with Stefan's to the midas bank format.
- TMEvent in midasio.h will also be updated for this
- TMEvent has incomplete implementation for creating new events and adding data banks, I 
hope to fix this. This should make midasio classes useful for writing MIDAS frontends.
- I am still thinking about what to do with roody and the XmlServer, MidasServer and 
NetDirectory code. Most likely remove it all and concentrate in supporting JSROOT and the 
associated ROOT-built-in web server. Maybe update roody to use the JSROOT interface to 
get data from the analyzer.

If you do not want to see all these changes, please use the rootana-2020-03 release 
branch.

After all changes are implemented and tested, I will cut a release branch and post an 
announcement.

K.O.
  23   12 Jul 2020 Thomas LindnerBug ReportCompiling rootana on Macos 10.15.4
I had some trouble building rootana on my new laptop, running MacOS 10.15.4 (Xcode 11.5).

Most of rootana compiled fine, but the steps with rootcint fail.  For instance:

rootcint -f obj/TMainDisplayWindowDict.cxx -DHAVE_ROOT   -DOS_LINUX -DOS_DARWIN -I./include 
include/TMainDisplayWindow.hxx include/TMainDisplayWindow_LinkDef.h
In file included from input_line_12:12:
In file included from ./include/TMainDisplayWindow.hxx:4:
In file included from /Applications/root_v6.18.99/include/TGClient.h:27:
In file included from /Applications/root_v6.18.99/include/TString.h:26:
In file included from /Applications/root_v6.18.99/include/TMathBase.h:32:
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/cmath:317:9: error: no member named 
'signbit' in the global namespace
using ::signbit;
      ~~^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/cmath:318:9: error: no member named 
'fpclassify' in the global namespace
using ::fpclassify;
      ~~^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/cmath:319:9: error: no member named 
'isfinite' in the global namespace
using ::isfinite;
      ~~^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/cmath:320:9: error: no member named 'isinf' 
in the global namespace
using ::isinf;
      ~~^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/cmath:321:9: error: no member named 'isnan' 
in the global namespace

This is probably a problem with my Xcode installation.  But I found a work-around, which was explicitly 
defining CPATH before the rootcint command in the Makefile:
obj/TMainDisplayWindowDict.cxx obj/TRootanaDisplayDict.cxx obj/TFancyHistogramCanvasDict.cxx: 
obj/%Dict.cxx:
    CPATH=/usr/local/include rootcint -f $@ $(CXXFLAGS_ROOTCINT) -I./include include/$*.hxx 
include/$*_LinkDef.h

obj/TNetDirectoryDict.cxx: obj/%Dict.cxx:
    CPATH=/usr/local/include rootcint -f $@ $(CXXFLAGS_ROOTCINT) -I./include include/$*.h 
include/$*_LinkDef.h

As I say, this is probably a problem with my installation; but I include the work-around in case it is helpful.

In the medium term we will try to remove the dependence on the old, ugly rootcint, since it often has 
problems.
  24   06 Nov 2020 Alexandr KozlinskiySuggestioncmake build
hi,

In mu3e we use cmake to build our software,
and as rootana does not use cmake build system it is
very difficult to integrate it in our software.

I have posted pull request
'https://bitbucket.org/tmidas/rootana/pull-requests/25'
that implements cmake build for rootana.
It is tested standalone on several systems
and also as part of mutrigana-base repo
(which is also used in online repo of mu3e).

Tests and comments are welcome.
  25   07 Dec 2020 Isaac Labrie BoulayForumThe FIXED format.
Hi folks,

I am thinking of using the FIXED format event to avoid the overhead of the bank 
structure. The Wiki says under "FIXED Format Event" that the standard MIDAS 
analyzer cannot work with this format. What exactly is meant by this? Is it saying 
that the specific ROOTANA analyzer template will not work with the event format? 

Thanks for helping me out!

Isaac
  26   10 Dec 2020 Isaac Labrie BoulayForumROOTANA MidasWiki
Hi folks,

I'm currently looking at the ROOTANA wiki. I will be building an online MIDAS 
analyzer and I'm wondering if the wiki still contains useful information if I want 
to use the more recent 'manalyzer' template.

Thanks!

Isaac
  27   16 Dec 2020 Konstantin OlchanskiSuggestioncmake build
> hi,
> 
> In mu3e we use cmake to build our software,
> and as rootana does not use cmake build system it is
> very difficult to integrate it in our software.
> 
> I have posted pull request
> 'https://bitbucket.org/tmidas/rootana/pull-requests/25'
> that implements cmake build for rootana.
> It is tested standalone on several systems
> and also as part of mutrigana-base repo
> (which is also used in online repo of mu3e).
> 
> Tests and comments are welcome.

Thank you for contributing this. I will be working on MIDAS and ROOTANA soon and I 
hope to add cmake support. The main issue it to finish the split or mvodb, midasio 
and manalyzer into separate git submodules.

K.O.
  28   16 Dec 2020 Konstantin OlchanskiForumThe FIXED format.
> I am thinking of using the FIXED format event to avoid the overhead of the bank 
> structure. The Wiki says under "FIXED Format Event" that the standard MIDAS 
> analyzer cannot work with this format. What exactly is meant by this? Is it saying 
> that the specific ROOTANA analyzer template will not work with the event format? 

I am not sure if current code can deal with anything other than MIDAS data that uses 
the bank format. (even bk_init32a() support is incomplete in ROOTANA!).

I am curious why you have a problem with the bank format.

For bank overhead, there are several solutions:

a) if you have 1000 banks, 1 data word in each bank, you can create 1 MIDAS bank, store 
your data inside using your own optimized data format. (MIDAS overhead reduces from 
4000 words to 4 words, overhead from your custom data format goes from 0 to ???, so you 
may not gain much).

b) if you have 1000 data events, 1 data word in each event. Overhead is crazy: many 
words of event header, many words of bank header, etc. I would solve this storing
all your events inside one single midas event (say, 1000 data events per 1 midas 
event). Each data event can be stored in a midas bank (it is okey to have 1000 banks 
named "AAAA"), or you use solution (a). The m-analyzer is especially suitable for 
working with such "super events". In the unpacking module, AnalyzeEvent() would unpack 
each data event into a FlowEvent, in all the other modules, your FlowEvent-encapsulated 
data events will show up in the AnalyzeFlowEvent() method.

Next time I work on the midasio library and the TMEvent class I should check that it 
works with and that is usable with non-midas-bank data.

K.O.
  29   16 Dec 2020 Konstantin OlchanskiForumROOTANA MidasWiki
> I'm currently looking at the ROOTANA wiki. I will be building an online MIDAS 
> analyzer and I'm wondering if the wiki still contains useful information if I want 
> to use the more recent 'manalyzer' template.

All the documentation for manalyzer is in it's README.md file. I am not sure if the ROOTANA wiki makes 
correct reference to this.

The big idea is to make manalyzer part of midas, as replacement to the old mana.c analyzer.

For now, I plan to make manalyzer into a git submodule in midas and rootana (same as mjson, mvodb and 
midasio).

K.O.
  30   16 Dec 2020 Isaac Labrie BoulayForumThe FIXED format.
> > I am thinking of using the FIXED format event to avoid the overhead of the bank 
> > structure. The Wiki says under "FIXED Format Event" that the standard MIDAS 
> > analyzer cannot work with this format. What exactly is meant by this? Is it saying 
> > that the specific ROOTANA analyzer template will not work with the event format? 
> 
> I am not sure if current code can deal with anything other than MIDAS data that uses 
> the bank format. (even bk_init32a() support is incomplete in ROOTANA!).
> 
> I am curious why you have a problem with the bank format.
> 
> For bank overhead, there are several solutions:
> 
> a) if you have 1000 banks, 1 data word in each bank, you can create 1 MIDAS bank, store 
> your data inside using your own optimized data format. (MIDAS overhead reduces from 
> 4000 words to 4 words, overhead from your custom data format goes from 0 to ???, so you 
> may not gain much).
> 
> b) if you have 1000 data events, 1 data word in each event. Overhead is crazy: many 
> words of event header, many words of bank header, etc. I would solve this storing
> all your events inside one single midas event (say, 1000 data events per 1 midas 
> event). Each data event can be stored in a midas bank (it is okey to have 1000 banks 
> named "AAAA"), or you use solution (a). The m-analyzer is especially suitable for 
> working with such "super events". In the unpacking module, AnalyzeEvent() would unpack 
> each data event into a FlowEvent, in all the other modules, your FlowEvent-encapsulated 
> data events will show up in the AnalyzeFlowEvent() method.
> 
> Next time I work on the midasio library and the TMEvent class I should check that it 
> works with and that is usable with non-midas-bank data.
> 
> K.O.

Hey,

Yeah I had not yet really dove into the Event Structure documentation at the time that I 
wrote this email. The BANK format and I get along now. No problem there anymore.

Thanks for getting back to me sir.

Isaac
  31   18 Jan 2021 Isaac Labrie BoulayForumWarning: the time for this bank is more than 10 sec older than current time.
Hi all,

I get this warning when I run anaDisplay.exe (analyzer display mode) of the old 
Rootana analyzer. It seems to be because the analyzer cannot keep up with the data 
gathered by my frontend. The same issue seems to happen when I run the newer 
"manalyzer_example_root_graphic.exe" in graphical mode (-g). Is my computer cpu 
just not powerful enough to run both the data acquisition and the graphical 
analyzer simultaneously? Both the frontend and analyzer run on the same PC.

As always, thanks for the help!

Isaac the intern
  32   21 Jan 2021 Thomas LindnerForumWarning: the time for this bank is more than 10 sec older than current time.
> Hi all,
> 
> I get this warning when I run anaDisplay.exe (analyzer display mode) of the old 
> Rootana analyzer. It seems to be because the analyzer cannot keep up with the data 
> gathered by my frontend. The same issue seems to happen when I run the newer 
> "manalyzer_example_root_graphic.exe" in graphical mode (-g). Is my computer cpu 
> just not powerful enough to run both the data acquisition and the graphical 
> analyzer simultaneously? Both the frontend and analyzer run on the same PC.
> 
> As always, thanks for the help!
> 
> Isaac the intern

Hi Isaac,

It is certainly possible that your analyzer cannot keep up with the data taking.  It will depend (at least partly) on what you are trying to do in your analyzer.  Suppose you are taking 1000Hz of events from some waveform digitizer.  If you are trying to do something simple (like count how many bytes are in each data packet) then your analyzer will probably be able to keep up with the data taking.  But if you are trying to do some very complicated maximum likelihood fit to each waveform, then it is possible that your computer (or any computer) would not be able to keep up with the data taking.  In that case maybe your analyzer is only able to process 100Hz of data or 10% of the events (for example).

But it is not always a problem if your analyzer can't keep up.  The goal of the online analyzer is to allow you to check data in real time.  If you wanted to calculate the pulse height distribution of hits in your waveforms, then it is probably fine that you are only able to process 10% of the events online.  It might take you a little longer to accumulate your statistics for your online plots, but your online analyzer data is still useful.

On the other hand, there is other types of online analysis where it is important to process every event.  For instance, suppose you are trying to calculate online the rate of a certain type of event that involves making cuts in your online analysig. In that case you will want to be analyzing every event; missing 10% or 90% of events will bias your online analysis.

So that message is a warning; if you are doing an online analysis where it is critical that you process every event, then you have a problem if you see that message.  In that case you need to figure out how to speed up your online analysis.  You will need to use a profiler or some other tool to optimize your code.

It should be noted that the display itself can often be a limiting part of the online analyzer.  You might be able to process 1000Hz of waveform data, but you certainly can't update your plots at 1000Hz; so you need to take care with the updating and display of your plots if you really care about speed.
  33   21 Jan 2021 Isaac Labrie BoulayForumWarning: the time for this bank is more than 10 sec older than current time.
> > Hi all,
> > 
> > I get this warning when I run anaDisplay.exe (analyzer display mode) of the old 
> > Rootana analyzer. It seems to be because the analyzer cannot keep up with the data 
> > gathered by my frontend. The same issue seems to happen when I run the newer 
> > "manalyzer_example_root_graphic.exe" in graphical mode (-g). Is my computer cpu 
> > just not powerful enough to run both the data acquisition and the graphical 
> > analyzer simultaneously? Both the frontend and analyzer run on the same PC.
> > 
> > As always, thanks for the help!
> > 
> > Isaac the intern
> 
> Hi Isaac,
> 
> It is certainly possible that your analyzer cannot keep up with the data taking.  It will depend (at least partly) on what you are trying to do in your analyzer.  Suppose you are taking 1000Hz of events from some waveform digitizer.  If you are trying to do something simple (like count how many bytes are in each data packet) then your analyzer will probably be able to keep up with the data taking.  But if you are trying to do some very complicated maximum likelihood fit to each waveform, then it is possible that your computer (or any computer) would not be able to keep up with the data taking.  In that case maybe your analyzer is only able to process 100Hz of data or 10% of the events (for example).
> 
> But it is not always a problem if your analyzer can't keep up.  The goal of the online analyzer is to allow you to check data in real time.  If you wanted to calculate the pulse height distribution of hits in your waveforms, then it is probably fine that you are only able to process 10% of the events online.  It might take you a little longer to accumulate your statistics for your online plots, but your online analyzer data is still useful.
> 
> On the other hand, there is other types of online analysis where it is important to process every event.  For instance, suppose you are trying to calculate online the rate of a certain type of event that involves making cuts in your online analysig. In that case you will want to be analyzing every event; missing 10% or 90% of events will bias your online analysis.
> 
> So that message is a warning; if you are doing an online analysis where it is critical that you process every event, then you have a problem if you see that message.  In that case you need to figure out how to speed up your online analysis.  You will need to use a profiler or some other tool to optimize your code.
> 
> It should be noted that the display itself can often be a limiting part of the online analyzer.  You might be able to process 1000Hz of waveform data, but you certainly can't update your plots at 1000Hz; so you need to take care with the updating and display of your plots if you really care about speed.

Hi Thomas,

Thank you for the thorough answer. I can now see how this feature is important. Also, I was getting turtle speeds for my analysis rate. 
I found out that it was because I was 'drawing/updating' my histograms for every event. I'm new to ROOT and I didn't realize how badly these 
functions slow down your system.

Cheers.

Isaac
  34   05 Feb 2021 Frederik WautersForumana or manalyzer?
reworking our old (MuCap inherited) mana.cpp

To have a good start, what is advised:

a) following the Rootana Analyzer Framework as advertised in https://midas.triumf.ca/MidasWiki/index.php/Rootana_Analyzer_Framework with the TRootanaEventLoop

b) The manalyzer advertised in https://bitbucket.org/tmidas/rootana/src/master/manalyzer/

Option b) with it's module support?
  35   05 Feb 2021 Konstantin OlchanskiForumana or manalyzer?
> reworking our old (MuCap inherited) mana.cpp
> To have a good start, what is advised:
> a) following the Rootana Analyzer Framework as advertised in https://midas.triumf.ca/MidasWiki/index.php/Rootana_Analyzer_Framework with the TRootanaEventLoop
> b) The manalyzer advertised in https://bitbucket.org/tmidas/rootana/src/master/manalyzer/
> Option b) with it's module support?

It depends on your experiment and your data. If your data is just a few numbers,
either analyzer would do (even the old midas mana.c analyzer would do).

But if you have complicated data, especially streaming data, where midas events
and physics events do not have a 1-to-1 connection, the old analyzers will
not cut the mustard. I have done this kind of processing in the old analyzers,
came up with a generalized structure, the "m" analyzer is the result.

Historical note: I developed rootana maybe 15 years ago to process data
from the ALPHA experiment at CERN (anti-hydrogen trap), more recently, I developed
manalyzer to process more complicated data from the ALPHA-g experiment at CERN (vertical
anti-hydrogen trap, to study gravity effects). Joseph McKenna from CERN contributed
the automatic by-module multithreading and benchmarking.

To compare:

- both rootana and manalyzer use the same TMidasOnline class for connecting to live MIDAS data
- both use the same midasio classes to read and write MIDAS files
- rootana uses the older TMidasEvent class to access MIDAS banks
- manalyzer uses the newer TMEvent class. the main improvement is that TMEvent is
a well-behaved C++ class with a usable assignment and copy-constructor operators.
- graphics and web interface uses the same ROOT graphics, ROOT web server (a derivative
of mongoose web server library) and the same JS-ROOT.
- the manalyzer has better defined control flow and C++ object lifetime rules
wrt run start and run end (this corrects problems with life time of C++ objects
in the old rootana).
- manalyzer has automatic per-module multithreading "for free" (thanks, Joseph!),
I typically see the alphag analyzer use 8-10 CPU threads without having to use
any threading or mutex gunk in the analysis code.

The current plan is to include manalyzer into MIDAS as a git submodule, with luck
it will happen for the next midas release, the hold up is with the midasio class that needs
to be updated, git-submoduled and added to midas. Also the old TMidasOnline class needs
updating.

To conclude, we use manalyzer for ALPHA-2 and ALPHA-g, people who use it seem to be
generally happy (especially with the automatic per-module multithreading), we use
it for a couple of simpler experiments at TRIUMF.

I hope this answers your questions?

K.O.
  36   05 Feb 2021 Frederik WautersForumana or manalyzer?
yes, thanks! 

" manalyzer has automatic per-module multithreading "for free"  "

does that mean that if one needs modules to run sequentially (think first clustering, then making higher level physics objects), the user needs to enforce that?

> > reworking our old (MuCap inherited) mana.cpp
> > To have a good start, what is advised:
> > a) following the Rootana Analyzer Framework as advertised in https://midas.triumf.ca/MidasWiki/index.php/Rootana_Analyzer_Framework with the TRootanaEventLoop
> > b) The manalyzer advertised in https://bitbucket.org/tmidas/rootana/src/master/manalyzer/
> > Option b) with it's module support?
> 
> It depends on your experiment and your data. If your data is just a few numbers,
> either analyzer would do (even the old midas mana.c analyzer would do).
> 
> But if you have complicated data, especially streaming data, where midas events
> and physics events do not have a 1-to-1 connection, the old analyzers will
> not cut the mustard. I have done this kind of processing in the old analyzers,
> came up with a generalized structure, the "m" analyzer is the result.
> 
> Historical note: I developed rootana maybe 15 years ago to process data
> from the ALPHA experiment at CERN (anti-hydrogen trap), more recently, I developed
> manalyzer to process more complicated data from the ALPHA-g experiment at CERN (vertical
> anti-hydrogen trap, to study gravity effects). Joseph McKenna from CERN contributed
> the automatic by-module multithreading and benchmarking.
> 
> To compare:
> 
> - both rootana and manalyzer use the same TMidasOnline class for connecting to live MIDAS data
> - both use the same midasio classes to read and write MIDAS files
> - rootana uses the older TMidasEvent class to access MIDAS banks
> - manalyzer uses the newer TMEvent class. the main improvement is that TMEvent is
> a well-behaved C++ class with a usable assignment and copy-constructor operators.
> - graphics and web interface uses the same ROOT graphics, ROOT web server (a derivative
> of mongoose web server library) and the same JS-ROOT.
> - the manalyzer has better defined control flow and C++ object lifetime rules
> wrt run start and run end (this corrects problems with life time of C++ objects
> in the old rootana).
> - manalyzer has automatic per-module multithreading "for free" (thanks, Joseph!),
> I typically see the alphag analyzer use 8-10 CPU threads without having to use
> any threading or mutex gunk in the analysis code.
> 
> The current plan is to include manalyzer into MIDAS as a git submodule, with luck
> it will happen for the next midas release, the hold up is with the midasio class that needs
> to be updated, git-submoduled and added to midas. Also the old TMidasOnline class needs
> updating.
> 
> To conclude, we use manalyzer for ALPHA-2 and ALPHA-g, people who use it seem to be
> generally happy (especially with the automatic per-module multithreading), we use
> it for a couple of simpler experiments at TRIUMF.
> 
> I hope this answers your questions?
> 
> K.O.
  37   06 Feb 2021 Frederik WautersForumana or manalyzer?
ok, nevermind, it's in the help

" -mt: Enable multithreaded mode. Extra multithread config settings: "
  38   22 Mar 2021 Isaac Labrie BoulayForumManalyzer - Preventing BeginRun() from getting called when I stop the run.
Hi all,

I am running an analyzer using an manalyzer templates and the analysis seems to 
lag behind the events getting stored in the buffer by my frontend. This is because 
the live histogram (-g) is very slow to update and draw. The slight delay causes a 
run object to be created after I 'stop' the acquisition. I'm guessing that this is 
because there are still MIDAS events in the ring buffer. This causes the 
BeginRun() methods of all my modules to get called again.

What is the best way to prevent this? From what I understand, PreEndRun() methods 
should only be used to clear buffered flow events. Is there maybe a way to speed 
up the histogram drawing so that I don't get this lag in the analysis?

Any help will be greatly appreciated.

Thanks so much for your time.

Isaac
  39   04 Apr 2021 Konstantin OlchanskiInfodrop support for ROOT v5?
There is problem report about failing builds against ROOT v5. https://bitbucket.org/tmidas/rootana/issues/19/make-error-rtypesimph-should-only-be

At TRIUMF, do we even still have anybody who uses ROOT v5 with ROOTANA?

My newest copy of ROOT v5 is from 2016 and I am not sure it would run on any computer we still have.

So is ROOT v5 still relevant, should we officially say ROOT v5 is not supported?

K.O.
  40   12 Apr 2021 Gennaro TortoneForumrootana and TAnaManager
Hi,

I'm facing with a strange problem with rootana:
I fetched and built rootana (master branch) as usual (make / make install);

If I build 'examples' and launch:

./root_server.exe -v

I got a lot of this messages:

Info in <TCling::LoadPCM>: In-memory ROOT PCM candidate /opt/root/lib/.....

and at the end:


 *** Break *** segmentation violation
 Generating stack trace...
 0x00007f9a876a7df1 in ROOT::AddClassAlternate(char const*, char const*) + 0x31 
from /opt/root/lib/root/libCore.so
 0x0000557debfea956 in <unknown> from ./root_server.exe
 0x0000557debfead72 in <unknown> from ./root_server.exe
 0x0000557debfead9b in <unknown> from ./root_server.exe
 0x0000557dec034605 in __libc_csu_init + 0x45 from ./root_server.exe
 0x00007f9a856e802a in __libc_start_main + 0x7a from /lib/x86_64-linux-
gnu/libc.so.6
 0x0000557debf759fa in _start + 0x2a from ./root_server.exe


I'm using ROOT 6.22/06 compiled from source files on Debian 10 on a virtual
machine...

The problem seems to be less critical if I connect with SSH with -XC option
or if I set DISPLAY variable. In these case I got only TCling messages with
this warning:

Error in <TGClient::TGClient>: can't open display "0:0", switching to batch 
mode...
 In case you run from a remote ssh session, reconnect with ssh -Y

If I build manalyzer everything goes fine and it seems that problem is
related to TAnaManager in some ways...

Regards,
Gennaro
ELOG V3.1.4-2e1708b5