Hi Konstantin,

sorry I did not set the mail notification on on this elog. So I never got the info that you replayed to this topic.
Maybe its also time to have a elog for manalyzer since rootana is not really used anymore.

Following our e-mail conversion I would start with this topic first:

Mail from me: >  Supporting more function of the root browser
Mail from you: > yes, I would like to add/have more code to interact with jsroot. basically jsroot specifies/implements a simple RPC and we should have support for it in manalyzer. (unless your "the root browser" is something else).

> I would like to know your "use case". What different options do you want to use in addition to what's already there?
So the main problem I have in short is that the readwrite flag is not the default in manalyzer. If its set to rw one can simply get histos on custom page via:

let gH = await httpRequest(serverIP + '/PathToHisto/' + '/root.json', 'object');
redraw('NAME', gH, 'hist');

> I must ask because some options are insecure (i.e. exposing the webserver to external connections) and
> while I have no problem with others shooting themselves in the foot, I think they should be warned
> before they do it and I do not want to read it in the news that they did it using a gun I built.
I agree here one needs to at least warn the user what he is going to do now.

 
> > At the moment the manalyzer.cxx only adds the port:
> > 2840: sprintf(str, "http:127.0.0.1:%d?cors", httpPort);
> 
> this is by design. it is only safe to bind thttpserver to localhost, exposing to external connections is not safe. (until somebody
> can review the security situation with the version of civetweb, an antique clone of mongoose, inside ROOT).
I fully agree with having only localhost allowed. My comment was more about the different options like rw, ro etc.

> - threads - I am not sure what the latest thread-safe situation is on ROOT. is it useful to increase number of threads beyond 10?
We never had problems with the performance - but I can not tell for others so better have an option for this so the user can choose?

> - top=name - yes, useful, but I think jsroot overrides that.
Would like to have the option especially when you have multiple analyzers running in the network can get confusing. 

> - auth_file, auth_domain - digest authentication .htdigest file stores passwords effectively as plain text, if you steal the .htdigest file, you 
> can login into the web server (as opposed to stealing /etc/passwd, you cannot login anywhere with it, not until you crack the hashes).
Better not.

> - loopback - this is what I want to enforce
Yes, I agree.

> - debug - could be useful, but is it?
I had it on a couple of times while writing custom pages displaying histograms.

> - websocket - does this do anything useful for us?
I don't think so, but I also never look into it in detail.

> - cors=domain - same as in midas mhttpd, I think we respond with CORS "*", is some other reponse useful?
At the moment we have with sprintf(str, "http:127.0.0.1:%d?cors", httpPort); CORS "*" I think other responses are not super useful.

> - readonly - (is the default?)
This it default at the moment I would also like to have readwrite as default

> - readwrite - I think should be *our* default
Yes.

> - global - (is the default, I agree)
Yes should be default.

> - noglobal - any use case where we may want this?
Maybe when users run into performance issues?

That's all of my thoughts about what setting should be possible.

Best,
Marius

Hi all,

following the e-mail conversation from Konstantin. The next topic we discussed was the event flow.

What I want to have (especially for commissioning phases for an experiment) is the possibility to have some kind of online calibration setup. 
So one has some kind of detector X which sends data and another detector Y which also sends data and I want to correlate the two detectors after applying some kinds of cleaning steps to the data. 

Mail me: Having the event flow maybe „orthogonal“ - not sure how to say this correctly but at the moment there is only one flow in a linear direction.
So something like was my intuition:
eventX -> decodeX -flow-> cleanX -> buffer -> correlation of X/Y
                                      ^
                                      |
eventY -> decodeY -flow-> cleanY ->   |


Mail Konstantin: yes, you will have to explain what you mean. right now, the flow is linear, as is typical for most physics analysis.
                 however, at any point one is permitted to create new flow events and inject them to the top of the modules pipeline, this adds "loops".

So how would injection into the top help? So I clean x/y and than I inject them back to the top and create a third flow for the correlation?
If this is true maybe a simple example or some more words on the documentation would help. I would be also willing to do that. After I fully understand how this would be possible.

Best,
Marius

Hi all,

again an idea from the mail conversation:

Mail me: Having an „easy way“ to explore the data - maybe some simple Python bindings. Especially when the data protocol is not fully settled in the experiment it’s maybe nice to have a simple way of plotting histograms. But also in general more and more people go to Python.

Mail Konstantin: I am not sure how python can come into this. manalyzer is a C++ analysis framework for experiments with needs beyound "toy" analyzers written in scripting languages (in performance and in complexity). if we want a python analysis framework, we can create one, but I am not sure how much of manalyzer we can reuse, python != C++. if you are thinking of a mixed framework, lets see your examples and use cases.

So a bit of what we did in the past. At some point MIDAS did not have this nice REST API to get events directly by request. At that time we had a python flask server running which used the Python MIDAS binding and gets some events from the buffer and created a REST API so our event display could display the events. However, my idea for the analyzer would be to further extend this idea of having an REST API which provides the user analyzed data. So my "python binding" can be anything at the end (also still a c++ application). So the focus is more from an experiment side: A few people develop the an anlyzer for an experiment but when it's going to be complex users have a hard time to even plot a "simple" histogram if they are non-experts. But if each module could export its flow events via a REST API debugging and easy development for new histograms / checks etc. could be quite easy.

What do you thing?

Best,
Marius

Hi all,

Having different histogram class templates to make processing easier.

My idea here was to create a parser which can take json input and creates automated default histograms from this. So here are a toy example how this could look like:

{
    "default1D": {
        "title": "bla",
        "x": "bla",
        "y": "#",
        "minX": "0",
        "maxX": 128,
        "nBins": 128,
    },
    "default2D": {
        "title": "bla",
        "x": "bla",
        "y": "#",
        "minX": "0",
        "maxX": 128,
        "nBinsX": 128,
        "minY": "0",
        "maxY": 128,
        "nBinsY": 128,
    },
    "detectorX": {
        "1DHistos":
            [
                {
                    "name": "FPGAID",
                    "type": "count",
                    "y": "#",
                    "minX": 0,
                    "maxX": 12,
                    "nBins": 12,
                    "value": "fpgaID",
                    "startChipID": 0,
                    "endChipID": 128,
                    "condition": "ROOT style: tree->Draw("X","X>10","");"
                },
                {
                    "name": "timeDiff",
                    "type": "diff",
                    "y": "#",
                    "xaxis": "col",
                    "minX": -256,
                    "maxX": 256,
                    "nBins": 512,
                    "value": "chipID",
                    "startChipID": 0,
                    "endChipID": 128,
                    "condition": "ROOT style: tree->Draw("X","X>10","");"
                },
           ]
....
}

Best,
Marius

Hi,

> Since a while, there is an API to request individual events. Have a look at the new "Event Dump" page (which is a pure custom page using mjsonrpc_call("bm_receive_event") ).
> In the mjsonrpc call you can specify the "midas even buffer", which is "SYSTEM" by default where all front-end dump their data. Nothing prevents you to create a new
> buffer "ANALYZER" and send analyzed events there from your analyzer. The logger only listens to SYSTEM, so does not store these analyzed events. But you can use
> the bm_receive_event call to request those analyzed events in a custom page and make a nice single event display from it. This technique would not require any 
> modification in midas, just add code to your analyzer to send events to a new buffer.

Question would be if this functionality should be implemented into the manalyzer framework so that everyone can use it if needed.

Best,
Marius

Hi,

> The contents of the event is anyhow up to the user.
Correct me but the bm_send_event wants a event in the MIDAS format. I was more thinking about a json format.

BTW: at my current MIDAS setup I get: Error: Invalid URL "" or query "cmd=Event%20Dump" or command "Event Dump" and Error: Invalid URL "" or query "cmd=OldODB" or command "OldODB" when I try to open the Event Dump or OldODB pages - rest of the pages work.

Best,
Marius

Hi,

I have now a mini working setup to get the events from the new buffer:

TEST(Analyzer, GetBuffer)
{
  INT status, request_id;
  HNDLE hBufEvent;
  INT EVENT_BUFFER_SIZE = 10000 * (1 << 25);

  status = cm_connect_experiment("", "Mu3e", "Simple Analyzer", NULL);
  if (status != CM_SUCCESS)
    ASSERT_TRUE(false);

  bm_open_buffer("ANALYZER", EVENT_BUFFER_SIZE, &hBufEvent);
  bm_request_event(hBufEvent, 1, TRIGGER_ALL, GET_ALL, &request_id, process_event);

  cm_disconnect_experiment();

  ASSERT_TRUE(true);
}

For sending the events I am not sure how to do this in the correct way. Since all the methods like bk_init etc. require some kind of frontend logic. Is there an easy way to create a MIDAS event without having a frontend?

Also when I created the new buffer I saw that there is no option to create an uint ODB entry (see file).

Best,
Marius


 

> You have to update to the develop branch of midas and refresh your browser cache.
I already tried it and my local installation is 2 days old. I pulled again the dev branch now and I also got another error (actually a while ago but I forgot to report it):

Commit d5263e gives me the following error:

/home/makoeppe/mu3e/midas/progs/mhdump.cxx:393:42: error: ‘ctime’ was not declared in this scope
  393 |                            printf(" %s", ctime(&t));
      |                                          ^~~~~
/home/makoeppe/mu3e/midas/progs/mhdump.cxx:25:1: note: ‘ctime’ is defined in header ‘<ctime>’; did you forget to ‘#include <ctime>’?
   24 | #include <vector>
  +++ |+#include <ctime>
   25 | 

gcc (GCC) 12.2.0
Linux office 5.15.86-1-lts #1 SMP Sun, 01 Jan 2023 16:03:00 +0000 x86_64 GNU/Linux

After adding #include <ctime> it compiles fine.

However, after cleaning the ODB and setting up everything fresh I don't even have the pages on the left bar anymore.

Best,
Marius

> > > You have to update to the develop branch of midas and refresh your browser cache.
> > I already tried it and my local installation is 2 days old. I pulled again the dev branch now and I also got another error (actually a while ago but I forgot to report it):
> > 
> > Commit d5263e gives me the following error:
> > 
> > /home/makoeppe/mu3e/midas/progs/mhdump.cxx:393:42: error: ‘ctime’ was not declared in this scope
> >   393 |                            printf(" %s", ctime(&t));
> >       |                                          ^~~~~
> > /home/makoeppe/mu3e/midas/progs/mhdump.cxx:25:1: note: ‘ctime’ is defined in header ‘<ctime>’; did you forget to ‘#include <ctime>’?
> >    24 | #include <vector>
> >   +++ |+#include <ctime>
> >    25 | 
> > 
> > gcc (GCC) 12.2.0
> > Linux office 5.15.86-1-lts #1 SMP Sun, 01 Jan 2023 16:03:00 +0000 x86_64 GNU/Linux
> > 
> > After adding #include <ctime> it compiles fine.
> 
> 
> mhdump.cxx is written by Konstantin, so he should give his ok to add the include. 
Any news on this? We had this issue now on couple of more PCs and operating systems (Arch, OpenSuse and Fedora)

Best,
Marius

Hi,

Thank you, commenting out the line worked and we can now compile the code. However, when we try to run ana.exe or anaDisplay.exe, we get the following errors:

Error in <TCling::RegisterModule>: cannot find dictionary module TMainDisplayWindowDict_rdict.pcm
Error in <TCling::RegisterModule>: cannot find dictionary module TRootanaDisplayDict_rdict.pcm
Error in <TCling::RegisterModule>: cannot find dictionary module TFancyHistogramCanvasDict_rdict.pcm
 

We see that the files are in /rootana/obj but we cannot find a way to point the compiler to them.

Could you please advise how to proceed,

Many thanks

ROOT v6 has been around for a while now, finally got around to test it with ROOTANA.

a) using root-v6-06-08
b) everything compiles and links
c) the example applications - i.e. display_example.exe - seem to run correctly
d) user applications linked against librootana.a show a problem loading the ROOTANA rdict.pcm files.

Failure to load the rdict.pcm files seems to prevent some functions, at least in the GUIs.

I do not fully understand how ROOT is supposed to find these files and I am asking for help on the ROOT forum.

In the mean time, there is a work around:
- symlink all rdict.pcm files into same place as the executable: ln -sv $ROOTANASYS/*/*.pcm .
- symlink all (many) .h files into the same place (this is a bit extreme): ln -sv $ROOTANASYS/include/* .

K.O.

> ROOT v6 has been around for a while now, finally got around to test it with ROOTANA.

The problems with loading rdict.pcm files and c++ .h header files is resolved, ROOTANA should be fully usable with ROOTv6:

Put this in your login file:

export ROOTANASYS=$HOME/packages/rootana
export ROOT_INCLUDE_PATH=$ROOTANASYS/include
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROOTANASYS/obj

Explanation:
- LD_LIBRARY_PATH is used to search for rdict.pcm files.
- ROOT_INCLUDE_PATH is used to find the c++ .h header files.

K.O.

> 
> 2) The 'same-origin policy' means that you need to setup some sort of proxying so that the custom web page can access the 
> ROOT webpage.  So, let's suppose in your case that you had the following ports for web servers:
> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
>

The MIDAS web server mhttpd already implements CORS, so you can access mhttpd AJAX and JSON-RPC from external web pages, no need to proxy.

Does the ROOT web server NOT implement CORS? We should file a bug report with ROOT if it does not.

K.O.

> [lm17773@it038146 examples]$ ./ana.exe 
> Error in <TCling::RegisterModule>: cannot find dictionary module
> TFancyHistogramCanvasDict_rdict.pcm

It is a problem inside ROOT. The pcm files replaced the dict.h files during the switch from CINT in 
ROOTv5 to LLVM in ROOTv6. But they have a mistake and use the wrong path for finding these files. 
I.e. search for "cannot find dictionary module" on the ROOT forum https://root-forum.cern.ch

K.O.

(please send elog messages in "plain text" mode, otherwise I cannot quote your text)

The problem with missing ss_suspend_set_dispatch_ipc() is now fixed, rootana commit 
https://bitbucket.org/tmidas/rootana/commits/52fdedb5745fa9b6739a16a6faab8b9d40108045

If you run rootana online in graphical mode (or call TMidasOnline::sleep() yourself), you may re-encounter
the problem of recursive calls to the event handler, depending on which version of MIDAS you use. The
probability of this problem happening is probably reduced to zero in all 2019 MIDAS releases and it is
definitely fixed in the most recent commits to MIDAS (this will be the midas-2019-09 release, most likely).

Read more here:
https://midas.triumf.ca/elog/Midas/1663

K.O.

> cannot find dictionary module TMainDisplayWindowDict_rdict.pcm

this problem is not unique to ROOTANA and ROODY, take a search for this problem (and solutions) on the ROOT forum...

K.O.

> > 
> > 2) The 'same-origin policy' means ...
> > https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
> >
> 
> The MIDAS web server mhttpd already implements CORS, so you can access mhttpd AJAX and JSON-RPC from external web pages, no need to proxy.
> Does the ROOT web server NOT implement CORS? We should file a bug report with ROOT if it does not.

For the record, the web server in ROOT 6.20 does implement CORS via the "cors=" option, see
https://github.com/root-project/jsroot/blob/master/docs/HttpServer.md

K.O.

I cut a release branch and tag for rootana-2020-03. This release is meant to be used with 
midas release 2020-03.

More recent versions of midas have several incompatibilities with ROOTANA, and I will
be working on resolving them:

- new incompatible data bank format in midas .mid files. (to fix the 64-bit misalignment bug)
- renaming of MIDAS TID_xxx types
- renaming of MIDAS data types in XML ODB dump files.

When we tag the next midas release, we will tag the compatible ROOTANA release and going 
forward hope to have MIDAS and ROOTANA releases done in lock-step.

K.O.

I am updating the rootana code to include all the latest developments in midas and 
elsewhere:

- mxml, mjson (and soon mvodb, midasio and manalyzer) are moving into git submodules 
(shared with midas and usable as standalone code)
- VirtualOdb is gone, use MVOdb instead
- TMidasFile is gone, use TMReader and TMWriter in midasio.h
- TMidasEvent remains, I will update it with Stefan's to the midas bank format.
- TMEvent in midasio.h will also be updated for this
- TMEvent has incomplete implementation for creating new events and adding data banks, I 
hope to fix this. This should make midasio classes useful for writing MIDAS frontends.
- I am still thinking about what to do with roody and the XmlServer, MidasServer and 
NetDirectory code. Most likely remove it all and concentrate in supporting JSROOT and the 
associated ROOT-built-in web server. Maybe update roody to use the JSROOT interface to 
get data from the analyzer.

If you do not want to see all these changes, please use the rootana-2020-03 release 
branch.

After all changes are implemented and tested, I will cut a release branch and post an 
announcement.

K.O.