Back Midas Rome Roody Rootana
  Root Analyzer Framework, Page 2 of 4  Not logged in ELOG logo
ID Date Authorup Topic Subject
  17   14 Aug 2019 Konstantin OlchanskiBug ReportROOTANA installation fails (ss_suspend_set_dispatch_ipc(NULL); not declared)
(please send elog messages in "plain text" mode, otherwise I cannot quote your text)

The problem with missing ss_suspend_set_dispatch_ipc() is now fixed, rootana commit 
https://bitbucket.org/tmidas/rootana/commits/52fdedb5745fa9b6739a16a6faab8b9d40108045

If you run rootana online in graphical mode (or call TMidasOnline::sleep() yourself), you may re-encounter
the problem of recursive calls to the event handler, depending on which version of MIDAS you use. The
probability of this problem happening is probably reduced to zero in all 2019 MIDAS releases and it is
definitely fixed in the most recent commits to MIDAS (this will be the midas-2019-09 release, most likely).

Read more here:
https://midas.triumf.ca/elog/Midas/1663

K.O.
  18   14 Aug 2019 Konstantin OlchanskiBug Reportcannot find dictionary module TMainDisplayWindowDict_rdict.pcm
> cannot find dictionary module TMainDisplayWindowDict_rdict.pcm

this problem is not unique to ROOTANA and ROODY, take a search for this problem (and solutions) on the ROOT forum...

K.O.
  20   10 Mar 2020 Konstantin OlchanskiInfoweb display of MIDAS data using rootana+THttpServer
> > 
> > 2) The 'same-origin policy' means ...
> > https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
> >
> 
> The MIDAS web server mhttpd already implements CORS, so you can access mhttpd AJAX and JSON-RPC from external web pages, no need to proxy.
> Does the ROOT web server NOT implement CORS? We should file a bug report with ROOT if it does not.

For the record, the web server in ROOT 6.20 does implement CORS via the "cors=" option, see
https://github.com/root-project/jsroot/blob/master/docs/HttpServer.md

K.O.
  21   22 May 2020 Konstantin OlchanskiReleaserootana-2020-03
I cut a release branch and tag for rootana-2020-03. This release is meant to be used with 
midas release 2020-03.

More recent versions of midas have several incompatibilities with ROOTANA, and I will
be working on resolving them:

- new incompatible data bank format in midas .mid files. (to fix the 64-bit misalignment bug)
- renaming of MIDAS TID_xxx types
- renaming of MIDAS data types in XML ODB dump files.

When we tag the next midas release, we will tag the compatible ROOTANA release and going 
forward hope to have MIDAS and ROOTANA releases done in lock-step.

K.O.
  22   22 May 2020 Konstantin OlchanskiInforootana updates
I am updating the rootana code to include all the latest developments in midas and 
elsewhere:

- mxml, mjson (and soon mvodb, midasio and manalyzer) are moving into git submodules 
(shared with midas and usable as standalone code)
- VirtualOdb is gone, use MVOdb instead
- TMidasFile is gone, use TMReader and TMWriter in midasio.h
- TMidasEvent remains, I will update it with Stefan's to the midas bank format.
- TMEvent in midasio.h will also be updated for this
- TMEvent has incomplete implementation for creating new events and adding data banks, I 
hope to fix this. This should make midasio classes useful for writing MIDAS frontends.
- I am still thinking about what to do with roody and the XmlServer, MidasServer and 
NetDirectory code. Most likely remove it all and concentrate in supporting JSROOT and the 
associated ROOT-built-in web server. Maybe update roody to use the JSROOT interface to 
get data from the analyzer.

If you do not want to see all these changes, please use the rootana-2020-03 release 
branch.

After all changes are implemented and tested, I will cut a release branch and post an 
announcement.

K.O.
  27   16 Dec 2020 Konstantin OlchanskiSuggestioncmake build
> hi,
> 
> In mu3e we use cmake to build our software,
> and as rootana does not use cmake build system it is
> very difficult to integrate it in our software.
> 
> I have posted pull request
> 'https://bitbucket.org/tmidas/rootana/pull-requests/25'
> that implements cmake build for rootana.
> It is tested standalone on several systems
> and also as part of mutrigana-base repo
> (which is also used in online repo of mu3e).
> 
> Tests and comments are welcome.

Thank you for contributing this. I will be working on MIDAS and ROOTANA soon and I 
hope to add cmake support. The main issue it to finish the split or mvodb, midasio 
and manalyzer into separate git submodules.

K.O.
  28   16 Dec 2020 Konstantin OlchanskiForumThe FIXED format.
> I am thinking of using the FIXED format event to avoid the overhead of the bank 
> structure. The Wiki says under "FIXED Format Event" that the standard MIDAS 
> analyzer cannot work with this format. What exactly is meant by this? Is it saying 
> that the specific ROOTANA analyzer template will not work with the event format? 

I am not sure if current code can deal with anything other than MIDAS data that uses 
the bank format. (even bk_init32a() support is incomplete in ROOTANA!).

I am curious why you have a problem with the bank format.

For bank overhead, there are several solutions:

a) if you have 1000 banks, 1 data word in each bank, you can create 1 MIDAS bank, store 
your data inside using your own optimized data format. (MIDAS overhead reduces from 
4000 words to 4 words, overhead from your custom data format goes from 0 to ???, so you 
may not gain much).

b) if you have 1000 data events, 1 data word in each event. Overhead is crazy: many 
words of event header, many words of bank header, etc. I would solve this storing
all your events inside one single midas event (say, 1000 data events per 1 midas 
event). Each data event can be stored in a midas bank (it is okey to have 1000 banks 
named "AAAA"), or you use solution (a). The m-analyzer is especially suitable for 
working with such "super events". In the unpacking module, AnalyzeEvent() would unpack 
each data event into a FlowEvent, in all the other modules, your FlowEvent-encapsulated 
data events will show up in the AnalyzeFlowEvent() method.

Next time I work on the midasio library and the TMEvent class I should check that it 
works with and that is usable with non-midas-bank data.

K.O.
  29   16 Dec 2020 Konstantin OlchanskiForumROOTANA MidasWiki
> I'm currently looking at the ROOTANA wiki. I will be building an online MIDAS 
> analyzer and I'm wondering if the wiki still contains useful information if I want 
> to use the more recent 'manalyzer' template.

All the documentation for manalyzer is in it's README.md file. I am not sure if the ROOTANA wiki makes 
correct reference to this.

The big idea is to make manalyzer part of midas, as replacement to the old mana.c analyzer.

For now, I plan to make manalyzer into a git submodule in midas and rootana (same as mjson, mvodb and 
midasio).

K.O.
  35   05 Feb 2021 Konstantin OlchanskiForumana or manalyzer?
> reworking our old (MuCap inherited) mana.cpp
> To have a good start, what is advised:
> a) following the Rootana Analyzer Framework as advertised in https://midas.triumf.ca/MidasWiki/index.php/Rootana_Analyzer_Framework with the TRootanaEventLoop
> b) The manalyzer advertised in https://bitbucket.org/tmidas/rootana/src/master/manalyzer/
> Option b) with it's module support?

It depends on your experiment and your data. If your data is just a few numbers,
either analyzer would do (even the old midas mana.c analyzer would do).

But if you have complicated data, especially streaming data, where midas events
and physics events do not have a 1-to-1 connection, the old analyzers will
not cut the mustard. I have done this kind of processing in the old analyzers,
came up with a generalized structure, the "m" analyzer is the result.

Historical note: I developed rootana maybe 15 years ago to process data
from the ALPHA experiment at CERN (anti-hydrogen trap), more recently, I developed
manalyzer to process more complicated data from the ALPHA-g experiment at CERN (vertical
anti-hydrogen trap, to study gravity effects). Joseph McKenna from CERN contributed
the automatic by-module multithreading and benchmarking.

To compare:

- both rootana and manalyzer use the same TMidasOnline class for connecting to live MIDAS data
- both use the same midasio classes to read and write MIDAS files
- rootana uses the older TMidasEvent class to access MIDAS banks
- manalyzer uses the newer TMEvent class. the main improvement is that TMEvent is
a well-behaved C++ class with a usable assignment and copy-constructor operators.
- graphics and web interface uses the same ROOT graphics, ROOT web server (a derivative
of mongoose web server library) and the same JS-ROOT.
- the manalyzer has better defined control flow and C++ object lifetime rules
wrt run start and run end (this corrects problems with life time of C++ objects
in the old rootana).
- manalyzer has automatic per-module multithreading "for free" (thanks, Joseph!),
I typically see the alphag analyzer use 8-10 CPU threads without having to use
any threading or mutex gunk in the analysis code.

The current plan is to include manalyzer into MIDAS as a git submodule, with luck
it will happen for the next midas release, the hold up is with the midasio class that needs
to be updated, git-submoduled and added to midas. Also the old TMidasOnline class needs
updating.

To conclude, we use manalyzer for ALPHA-2 and ALPHA-g, people who use it seem to be
generally happy (especially with the automatic per-module multithreading), we use
it for a couple of simpler experiments at TRIUMF.

I hope this answers your questions?

K.O.
  39   04 Apr 2021 Konstantin OlchanskiInfodrop support for ROOT v5?
There is problem report about failing builds against ROOT v5. https://bitbucket.org/tmidas/rootana/issues/19/make-error-rtypesimph-should-only-be

At TRIUMF, do we even still have anybody who uses ROOT v5 with ROOTANA?

My newest copy of ROOT v5 is from 2016 and I am not sure it would run on any computer we still have.

So is ROOT v5 still relevant, should we officially say ROOT v5 is not supported?

K.O.
  44   29 Apr 2021 Konstantin OlchanskiInfomanalyzer split into a git submodule
In preparation to including the manalyzer into MIDAS, I split it into a separate 
repository and included it into ROOTANA as a git submodule.

https://bitbucket.org/tmidas/manalyzer/src/master/

This will cause a minor difficulty and require a manual adjustment next time you 
do a "git pull":
- check that (a) manalyzer directory is empty or (b) it only has .o and .exe 
files, in both cases delete it
- run "git submodule init; git submodule update", you should see it clone the 
manalyzer repository

K.O.
  46   27 Apr 2022 Konstantin OlchanskiSuggestionSupport for THttpServer Options
> I would like to have the possibility to add different options when I create a THttpServer.

I would like to know your "use case". What different options do you want to use in addition to what's already there?

I must ask because some options are insecure (i.e. exposing the webserver to external connections) and
while I have no problem with others shooting themselves in the foot, I think they should be warned
before they do it and I do not want to read it in the news that they did it using a gun I built.

TLDR follows:

> At the moment the manalyzer.cxx only adds the port:
> 2840: sprintf(str, "http:127.0.0.1:%d?cors", httpPort);

this is by design. it is only safe to bind thttpserver to localhost, exposing to external connections is not safe. (until somebody
can review the security situation with the version of civetweb, an antique clone of mongoose, inside ROOT).

same situation with mhttpd, it is not safe to expose it to external connections, it should always live
behind a password protected https proxy.

> But there are more options which one can use https://github.com/root-project/jsroot/blob/master/docs/HttpServer.md.

yes, I see there is more options now that there used to be before.

which ones do you want to use? and which ones you think are generally useful in what we do?

- threads - I am not sure what the latest thread-safe situation is on ROOT. is it useful to increase number of threads beyond 10?
- top=name - yes, useful, but I think jsroot overrides that.
- auth_file, auth_domain - digest authentication .htdigest file stores passwords effectively as plain text, if you steal the .htdigest file, you 
can login into the web server (as opposed to stealing /etc/passwd, you cannot login anywhere with it, not until you crack the hashes).
- loopback - this is what I want to enforce
- debug - could be useful, but is it?
- websocket - does this do anything useful for us?
- cors=domain - same as in midas mhttpd, I think we respond with CORS "*", is some other reponse useful?
- readonly - (is the default?)
- readwrite - I think should be *our* default
- global - (is the default, I agree)
- noglobal - any use case where we may want this?
- anything else?

For general use, I want maximum security configuration: bind to localhost, cors enabled, user can specify tcp port number, plus anything else from 
the list above?

For maximum flexibility we should probably have "--insecure-thttpd" to specify
the complete THttpServer() constructor argument. But it would be silly to force everybody
to use that just to set the "debug" option.

K.O.
  15   08 Aug 2019 Lauren MantonBug ReportROOTANA installation fails (ss_suspend_set_dispatch_ipc(NULL); not declared)

Hi,

Thank you, commenting out the line worked and we can now compile the code. However, when we try to run ana.exe or anaDisplay.exe, we get the following errors:

Error in <TCling::RegisterModule>: cannot find dictionary module TMainDisplayWindowDict_rdict.pcm
Error in <TCling::RegisterModule>: cannot find dictionary module TRootanaDisplayDict_rdict.pcm
Error in <TCling::RegisterModule>: cannot find dictionary module TFancyHistogramCanvasDict_rdict.pcm

 

We see that the files are in /rootana/obj but we cannot find a way to point the compiler to them.

Could you please advise how to proceed,

Many thanks

Paolo Baesso wrote:

Good afternoon, I am following the instructions provided here to install ROOTANA (and MIDAS, more in general). When I try to make I get the following error

libMidasInterface/TMidasOnline.cxx: In member function ‘bool TMidasOnline::sleep(int)’: libMidasInterface/TMidasOnline.cxx:194:3: error: ‘ss_suspend_set_dispatch_ipc’ was not declared in this scope
ss_suspend_set_dispatch_ipc(NULL);
^~~~~~~~~~~~~~~~~~~~~~~~~~~
libMidasInterface/TMidasOnline.cxx:194:3: note: suggested alternative: ‘ss_suspend_set_rpc_thread’
ss_suspend_set_dispatch_ipc(NULL);
^~~~~~~~~~~~~~~~~~~~~~~~~~~
ss_suspend_set_rpc_thread
Makefile:339: recipe for target 'obj/TMidasOnline.o' failed make: *** [obj/TMidasOnline.o] Error 1

 

And the compilation stops. Is this a known issue?
My system is Ubuntu 18 and I am running ROOT 6.18.00

 

  Draft   22 Oct 2019 Lukas GerritzenBug Report 
  45   25 Apr 2022 Marius KoeppelSuggestionSupport for THttpServer Options
Dear all,

I would like to have the possibility to add different options when I create a THttpServer. At the moment the manalyzer.cxx only adds the port:

2840: sprintf(str, "http:127.0.0.1:%d?cors", httpPort);

But there are more options which one can use https://github.com/root-project/jsroot/blob/master/docs/HttpServer.md.

Best regards,
Marius
  47   04 Apr 2023 Marius KoeppelSuggestionSupport for THttpServer Options
Hi Konstantin,

sorry I did not set the mail notification on on this elog. So I never got the info that you replayed to this topic.
Maybe its also time to have a elog for manalyzer since rootana is not really used anymore.

Following our e-mail conversion I would start with this topic first:

Mail from me: >  Supporting more function of the root browser
Mail from you: > yes, I would like to add/have more code to interact with jsroot. basically jsroot specifies/implements a simple RPC and we should have support for it in manalyzer. (unless your "the root browser" is something else).

> I would like to know your "use case". What different options do you want to use in addition to what's already there?
So the main problem I have in short is that the readwrite flag is not the default in manalyzer. If its set to rw one can simply get histos on custom page via:

let gH = await httpRequest(serverIP + '/PathToHisto/' + '/root.json', 'object');
redraw('NAME', gH, 'hist');

> I must ask because some options are insecure (i.e. exposing the webserver to external connections) and
> while I have no problem with others shooting themselves in the foot, I think they should be warned
> before they do it and I do not want to read it in the news that they did it using a gun I built.
I agree here one needs to at least warn the user what he is going to do now.

 
> > At the moment the manalyzer.cxx only adds the port:
> > 2840: sprintf(str, "http:127.0.0.1:%d?cors", httpPort);
> 
> this is by design. it is only safe to bind thttpserver to localhost, exposing to external connections is not safe. (until somebody
> can review the security situation with the version of civetweb, an antique clone of mongoose, inside ROOT).
I fully agree with having only localhost allowed. My comment was more about the different options like rw, ro etc.

> - threads - I am not sure what the latest thread-safe situation is on ROOT. is it useful to increase number of threads beyond 10?
We never had problems with the performance - but I can not tell for others so better have an option for this so the user can choose?

> - top=name - yes, useful, but I think jsroot overrides that.
Would like to have the option especially when you have multiple analyzers running in the network can get confusing. 

> - auth_file, auth_domain - digest authentication .htdigest file stores passwords effectively as plain text, if you steal the .htdigest file, you 
> can login into the web server (as opposed to stealing /etc/passwd, you cannot login anywhere with it, not until you crack the hashes).
Better not.

> - loopback - this is what I want to enforce
Yes, I agree.

> - debug - could be useful, but is it?
I had it on a couple of times while writing custom pages displaying histograms.

> - websocket - does this do anything useful for us?
I don't think so, but I also never look into it in detail.

> - cors=domain - same as in midas mhttpd, I think we respond with CORS "*", is some other reponse useful?
At the moment we have with sprintf(str, "http:127.0.0.1:%d?cors", httpPort); CORS "*" I think other responses are not super useful.

> - readonly - (is the default?)
This it default at the moment I would also like to have readwrite as default

> - readwrite - I think should be *our* default
Yes.

> - global - (is the default, I agree)
Yes should be default.

> - noglobal - any use case where we may want this?
Maybe when users run into performance issues?

That's all of my thoughts about what setting should be possible.

Best,
Marius
  48   04 Apr 2023 Marius KoeppelInfoDocumentation Event Flow
Hi all,

following the e-mail conversation from Konstantin. The next topic we discussed was the event flow.

What I want to have (especially for commissioning phases for an experiment) is the possibility to have some kind of online calibration setup. 
So one has some kind of detector X which sends data and another detector Y which also sends data and I want to correlate the two detectors after applying some kinds of cleaning steps to the data. 

Mail me: Having the event flow maybe „orthogonal“ - not sure how to say this correctly but at the moment there is only one flow in a linear direction.
So something like was my intuition:
eventX -> decodeX -flow-> cleanX -> buffer -> correlation of X/Y
                                      ^
                                      |
eventY -> decodeY -flow-> cleanY ->   |


Mail Konstantin: yes, you will have to explain what you mean. right now, the flow is linear, as is typical for most physics analysis.
                 however, at any point one is permitted to create new flow events and inject them to the top of the modules pipeline, this adds "loops".

So how would injection into the top help? So I clean x/y and than I inject them back to the top and create a third flow for the correlation?
If this is true maybe a simple example or some more words on the documentation would help. I would be also willing to do that. After I fully understand how this would be possible.

Best,
Marius
  49   04 Apr 2023 Marius KoeppelSuggestionPython Plotting
Hi all,

again an idea from the mail conversation:

Mail me: Having an „easy way“ to explore the data - maybe some simple Python bindings. Especially when the data protocol is not fully settled in the experiment it’s maybe nice to have a simple way of plotting histograms. But also in general more and more people go to Python.

Mail Konstantin: I am not sure how python can come into this. manalyzer is a C++ analysis framework for experiments with needs beyound "toy" analyzers written in scripting languages (in performance and in complexity). if we want a python analysis framework, we can create one, but I am not sure how much of manalyzer we can reuse, python != C++. if you are thinking of a mixed framework, lets see your examples and use cases.

So a bit of what we did in the past. At some point MIDAS did not have this nice REST API to get events directly by request. At that time we had a python flask server running which used the Python MIDAS binding and gets some events from the buffer and created a REST API so our event display could display the events. However, my idea for the analyzer would be to further extend this idea of having an REST API which provides the user analyzed data. So my "python binding" can be anything at the end (also still a c++ application). So the focus is more from an experiment side: A few people develop the an anlyzer for an experiment but when it's going to be complex users have a hard time to even plot a "simple" histogram if they are non-experts. But if each module could export its flow events via a REST API debugging and easy development for new histograms / checks etc. could be quite easy.

What do you thing?

Best,
Marius
  50   04 Apr 2023 Marius KoeppelSuggestionHistogram Class Templates
Hi all,

Having different histogram class templates to make processing easier.

My idea here was to create a parser which can take json input and creates automated default histograms from this. So here are a toy example how this could look like:

{
    "default1D": {
        "title": "bla",
        "x": "bla",
        "y": "#",
        "minX": "0",
        "maxX": 128,
        "nBins": 128,
    },
    "default2D": {
        "title": "bla",
        "x": "bla",
        "y": "#",
        "minX": "0",
        "maxX": 128,
        "nBinsX": 128,
        "minY": "0",
        "maxY": 128,
        "nBinsY": 128,
    },
    "detectorX": {
        "1DHistos":
            [
                {
                    "name": "FPGAID",
                    "type": "count",
                    "y": "#",
                    "minX": 0,
                    "maxX": 12,
                    "nBins": 12,
                    "value": "fpgaID",
                    "startChipID": 0,
                    "endChipID": 128,
                    "condition": "ROOT style: tree->Draw("X","X>10","");"
                },
                {
                    "name": "timeDiff",
                    "type": "diff",
                    "y": "#",
                    "xaxis": "col",
                    "minX": -256,
                    "maxX": 256,
                    "nBins": 512,
                    "value": "chipID",
                    "startChipID": 0,
                    "endChipID": 128,
                    "condition": "ROOT style: tree->Draw("X","X>10","");"
                },
           ]
....
}

Best,
Marius
  52   04 Apr 2023 Marius KoeppelSuggestionPython Plotting
Hi,

> Since a while, there is an API to request individual events. Have a look at the new "Event Dump" page (which is a pure custom page using mjsonrpc_call("bm_receive_event") ).
> In the mjsonrpc call you can specify the "midas even buffer", which is "SYSTEM" by default where all front-end dump their data. Nothing prevents you to create a new
> buffer "ANALYZER" and send analyzed events there from your analyzer. The logger only listens to SYSTEM, so does not store these analyzed events. But you can use
> the bm_receive_event call to request those analyzed events in a custom page and make a nice single event display from it. This technique would not require any 
> modification in midas, just add code to your analyzer to send events to a new buffer.

Question would be if this functionality should be implemented into the manalyzer framework so that everyone can use it if needed.

Best,
Marius
ELOG V3.1.4-2e1708b5