In running alphaAnalysis.exe, a instance of manalyser, I would like to use the ---usetimerange option. When I include it on the commandline I get back the messages including this option which suggests that it is input incorrectly, but I also don't see it anymore in the manalyzer.cxx code.
Has it been deliberately removed, and, if so, should not the help be amended to reflect this. This option would be highly useful to me.

> > mhdump.cxx is written by Konstantin, so he should give his ok to add the include. 
> Any news on this? We had this issue now on couple of more PCs and operating systems (Arch, OpenSuse and Fedora)

Since I didn't get any complaint in the last two months, I added the include.

Stefan

> > > You have to update to the develop branch of midas and refresh your browser cache.
> > I already tried it and my local installation is 2 days old. I pulled again the dev branch now and I also got another error (actually a while ago but I forgot to report it):
> > 
> > Commit d5263e gives me the following error:
> > 
> > /home/makoeppe/mu3e/midas/progs/mhdump.cxx:393:42: error: ‘ctime’ was not declared in this scope
> >   393 |                            printf(" %s", ctime(&t));
> >       |                                          ^~~~~
> > /home/makoeppe/mu3e/midas/progs/mhdump.cxx:25:1: note: ‘ctime’ is defined in header ‘<ctime>’; did you forget to ‘#include <ctime>’?
> >    24 | #include <vector>
> >   +++ |+#include <ctime>
> >    25 | 
> > 
> > gcc (GCC) 12.2.0
> > Linux office 5.15.86-1-lts #1 SMP Sun, 01 Jan 2023 16:03:00 +0000 x86_64 GNU/Linux
> > 
> > After adding #include <ctime> it compiles fine.
> 
> 
> mhdump.cxx is written by Konstantin, so he should give his ok to add the include. 
Any news on this? We had this issue now on couple of more PCs and operating systems (Arch, OpenSuse and Fedora)

Best,
Marius

> > You have to update to the develop branch of midas and refresh your browser cache.
> I already tried it and my local installation is 2 days old. I pulled again the dev branch now and I also got another error (actually a while ago but I forgot to report it):
> 
> Commit d5263e gives me the following error:
> 
> /home/makoeppe/mu3e/midas/progs/mhdump.cxx:393:42: error: ‘ctime’ was not declared in this scope
>   393 |                            printf(" %s", ctime(&t));
>       |                                          ^~~~~
> /home/makoeppe/mu3e/midas/progs/mhdump.cxx:25:1: note: ‘ctime’ is defined in header ‘<ctime>’; did you forget to ‘#include <ctime>’?
>    24 | #include <vector>
>   +++ |+#include <ctime>
>    25 | 
> 
> gcc (GCC) 12.2.0
> Linux office 5.15.86-1-lts #1 SMP Sun, 01 Jan 2023 16:03:00 +0000 x86_64 GNU/Linux
> 
> After adding #include <ctime> it compiles fine.


mhdump.cxx is written by Konstantin, so he should give his ok to add the include. 


> However, after cleaning the ODB and setting up everything fresh I don't even have the pages on the left bar anymore.

The menu on the left side is controlled via the ODB flags /Experiment/Menu. Probably the "Event dump" is of by default.

Stefan

> You have to update to the develop branch of midas and refresh your browser cache.
I already tried it and my local installation is 2 days old. I pulled again the dev branch now and I also got another error (actually a while ago but I forgot to report it):

Commit d5263e gives me the following error:

/home/makoeppe/mu3e/midas/progs/mhdump.cxx:393:42: error: ‘ctime’ was not declared in this scope
  393 |                            printf(" %s", ctime(&t));
      |                                          ^~~~~
/home/makoeppe/mu3e/midas/progs/mhdump.cxx:25:1: note: ‘ctime’ is defined in header ‘<ctime>’; did you forget to ‘#include <ctime>’?
   24 | #include <vector>
  +++ |+#include <ctime>
   25 | 

gcc (GCC) 12.2.0
Linux office 5.15.86-1-lts #1 SMP Sun, 01 Jan 2023 16:03:00 +0000 x86_64 GNU/Linux

After adding #include <ctime> it compiles fine.

However, after cleaning the ODB and setting up everything fresh I don't even have the pages on the left bar anymore.

Best,
Marius

> BTW: at my current MIDAS setup I get: Error: Invalid URL "" or query "cmd=Event%20Dump" or command "Event Dump" and Error: Invalid URL "" or query "cmd=OldODB" or command "OldODB" when I try to open the Event Dump or OldODB pages - rest of the pages work.

You have to update to the develop branch of midas and refresh your browser cache.

Stefan

> Also when I created the new buffer I saw that there is no option to create an uint ODB entry (see file).

> For sending the events I am not sure how to do this in the correct way. 

bm_compose_event()
bm_send_event()

You can use bk_init (better bk_init32) to create a bank structure inside an event, but that's not mandatory.
Nothing prevents you from putting a JSON string into your event, as long as the midas even header (created
via bm_compose_event() is correct).

Best,
Stefan

Hi,

I have now a mini working setup to get the events from the new buffer:

TEST(Analyzer, GetBuffer)
{
  INT status, request_id;
  HNDLE hBufEvent;
  INT EVENT_BUFFER_SIZE = 10000 * (1 << 25);

  status = cm_connect_experiment("", "Mu3e", "Simple Analyzer", NULL);
  if (status != CM_SUCCESS)
    ASSERT_TRUE(false);

  bm_open_buffer("ANALYZER", EVENT_BUFFER_SIZE, &hBufEvent);
  bm_request_event(hBufEvent, 1, TRIGGER_ALL, GET_ALL, &request_id, process_event);

  cm_disconnect_experiment();

  ASSERT_TRUE(true);
}

For sending the events I am not sure how to do this in the correct way. Since all the methods like bk_init etc. require some kind of frontend logic. Is there an easy way to create a MIDAS event without having a frontend?

Also when I created the new buffer I saw that there is no option to create an uint ODB entry (see file).

Best,
Marius


 

Hi,

> The contents of the event is anyhow up to the user.
Correct me but the bm_send_event wants a event in the MIDAS format. I was more thinking about a json format.

BTW: at my current MIDAS setup I get: Error: Invalid URL "" or query "cmd=Event%20Dump" or command "Event Dump" and Error: Invalid URL "" or query "cmd=OldODB" or command "OldODB" when I try to open the Event Dump or OldODB pages - rest of the pages work.

Best,
Marius

> Question would be if this functionality should be implemented into the manalyzer framework so that everyone can use it if needed.

You need to function calls for that bm_open_buffer and bm_send_event (both part of the MIDAS C API). The contents of the event
is anyhow up to the user. There also might be cases where one want not one but several different buffers, like for different
event types. So I would recommend start outside the manalyzer framework with that (since it's just two lines of code ... plus maybe
some error checking), and once we all know what is good we can promote this to the manalyzer framework.

Stefan

Hi,

> Since a while, there is an API to request individual events. Have a look at the new "Event Dump" page (which is a pure custom page using mjsonrpc_call("bm_receive_event") ).
> In the mjsonrpc call you can specify the "midas even buffer", which is "SYSTEM" by default where all front-end dump their data. Nothing prevents you to create a new
> buffer "ANALYZER" and send analyzed events there from your analyzer. The logger only listens to SYSTEM, so does not store these analyzed events. But you can use
> the bm_receive_event call to request those analyzed events in a custom page and make a nice single event display from it. This technique would not require any 
> modification in midas, just add code to your analyzer to send events to a new buffer.

Question would be if this functionality should be implemented into the manalyzer framework so that everyone can use it if needed.

Best,
Marius

I can add here my five cents:

Since a while, there is an API to request individual events. Have a look at the new "Event Dump" page (which is a pure custom page using mjsonrpc_call("bm_receive_event") ).
In the mjsonrpc call you can specify the "midas even buffer", which is "SYSTEM" by default where all front-end dump their data. Nothing prevents you to create a new
buffer "ANALYZER" and send analyzed events there from your analyzer. The logger only listens to SYSTEM, so does not store these analyzed events. But you can use
the bm_receive_event call to request those analyzed events in a custom page and make a nice single event display from it. This technique would not require any 
modification in midas, just add code to your analyzer to send events to a new buffer.

Best,
Stefan 

Hi all,

Having different histogram class templates to make processing easier.

My idea here was to create a parser which can take json input and creates automated default histograms from this. So here are a toy example how this could look like:

{
    "default1D": {
        "title": "bla",
        "x": "bla",
        "y": "#",
        "minX": "0",
        "maxX": 128,
        "nBins": 128,
    },
    "default2D": {
        "title": "bla",
        "x": "bla",
        "y": "#",
        "minX": "0",
        "maxX": 128,
        "nBinsX": 128,
        "minY": "0",
        "maxY": 128,
        "nBinsY": 128,
    },
    "detectorX": {
        "1DHistos":
            [
                {
                    "name": "FPGAID",
                    "type": "count",
                    "y": "#",
                    "minX": 0,
                    "maxX": 12,
                    "nBins": 12,
                    "value": "fpgaID",
                    "startChipID": 0,
                    "endChipID": 128,
                    "condition": "ROOT style: tree->Draw("X","X>10","");"
                },
                {
                    "name": "timeDiff",
                    "type": "diff",
                    "y": "#",
                    "xaxis": "col",
                    "minX": -256,
                    "maxX": 256,
                    "nBins": 512,
                    "value": "chipID",
                    "startChipID": 0,
                    "endChipID": 128,
                    "condition": "ROOT style: tree->Draw("X","X>10","");"
                },
           ]
....
}

Best,
Marius

Hi all,

again an idea from the mail conversation:

Mail me: Having an „easy way“ to explore the data - maybe some simple Python bindings. Especially when the data protocol is not fully settled in the experiment it’s maybe nice to have a simple way of plotting histograms. But also in general more and more people go to Python.

Mail Konstantin: I am not sure how python can come into this. manalyzer is a C++ analysis framework for experiments with needs beyound "toy" analyzers written in scripting languages (in performance and in complexity). if we want a python analysis framework, we can create one, but I am not sure how much of manalyzer we can reuse, python != C++. if you are thinking of a mixed framework, lets see your examples and use cases.

So a bit of what we did in the past. At some point MIDAS did not have this nice REST API to get events directly by request. At that time we had a python flask server running which used the Python MIDAS binding and gets some events from the buffer and created a REST API so our event display could display the events. However, my idea for the analyzer would be to further extend this idea of having an REST API which provides the user analyzed data. So my "python binding" can be anything at the end (also still a c++ application). So the focus is more from an experiment side: A few people develop the an anlyzer for an experiment but when it's going to be complex users have a hard time to even plot a "simple" histogram if they are non-experts. But if each module could export its flow events via a REST API debugging and easy development for new histograms / checks etc. could be quite easy.

What do you thing?

Best,
Marius

Hi all,

following the e-mail conversation from Konstantin. The next topic we discussed was the event flow.

What I want to have (especially for commissioning phases for an experiment) is the possibility to have some kind of online calibration setup. 
So one has some kind of detector X which sends data and another detector Y which also sends data and I want to correlate the two detectors after applying some kinds of cleaning steps to the data. 

Mail me: Having the event flow maybe „orthogonal“ - not sure how to say this correctly but at the moment there is only one flow in a linear direction.
So something like was my intuition:
eventX -> decodeX -flow-> cleanX -> buffer -> correlation of X/Y
                                      ^
                                      |
eventY -> decodeY -flow-> cleanY ->   |


Mail Konstantin: yes, you will have to explain what you mean. right now, the flow is linear, as is typical for most physics analysis.
                 however, at any point one is permitted to create new flow events and inject them to the top of the modules pipeline, this adds "loops".

So how would injection into the top help? So I clean x/y and than I inject them back to the top and create a third flow for the correlation?
If this is true maybe a simple example or some more words on the documentation would help. I would be also willing to do that. After I fully understand how this would be possible.

Best,
Marius

Hi Konstantin,

sorry I did not set the mail notification on on this elog. So I never got the info that you replayed to this topic.
Maybe its also time to have a elog for manalyzer since rootana is not really used anymore.

Following our e-mail conversion I would start with this topic first:

Mail from me: >  Supporting more function of the root browser
Mail from you: > yes, I would like to add/have more code to interact with jsroot. basically jsroot specifies/implements a simple RPC and we should have support for it in manalyzer. (unless your "the root browser" is something else).

> I would like to know your "use case". What different options do you want to use in addition to what's already there?
So the main problem I have in short is that the readwrite flag is not the default in manalyzer. If its set to rw one can simply get histos on custom page via:

let gH = await httpRequest(serverIP + '/PathToHisto/' + '/root.json', 'object');
redraw('NAME', gH, 'hist');

> I must ask because some options are insecure (i.e. exposing the webserver to external connections) and
> while I have no problem with others shooting themselves in the foot, I think they should be warned
> before they do it and I do not want to read it in the news that they did it using a gun I built.
I agree here one needs to at least warn the user what he is going to do now.

 
> > At the moment the manalyzer.cxx only adds the port:
> > 2840: sprintf(str, "http:127.0.0.1:%d?cors", httpPort);
> 
> this is by design. it is only safe to bind thttpserver to localhost, exposing to external connections is not safe. (until somebody
> can review the security situation with the version of civetweb, an antique clone of mongoose, inside ROOT).
I fully agree with having only localhost allowed. My comment was more about the different options like rw, ro etc.

> - threads - I am not sure what the latest thread-safe situation is on ROOT. is it useful to increase number of threads beyond 10?
We never had problems with the performance - but I can not tell for others so better have an option for this so the user can choose?

> - top=name - yes, useful, but I think jsroot overrides that.
Would like to have the option especially when you have multiple analyzers running in the network can get confusing. 

> - auth_file, auth_domain - digest authentication .htdigest file stores passwords effectively as plain text, if you steal the .htdigest file, you 
> can login into the web server (as opposed to stealing /etc/passwd, you cannot login anywhere with it, not until you crack the hashes).
Better not.

> - loopback - this is what I want to enforce
Yes, I agree.

> - debug - could be useful, but is it?
I had it on a couple of times while writing custom pages displaying histograms.

> - websocket - does this do anything useful for us?
I don't think so, but I also never look into it in detail.

> - cors=domain - same as in midas mhttpd, I think we respond with CORS "*", is some other reponse useful?
At the moment we have with sprintf(str, "http:127.0.0.1:%d?cors", httpPort); CORS "*" I think other responses are not super useful.

> - readonly - (is the default?)
This it default at the moment I would also like to have readwrite as default

> - readwrite - I think should be *our* default
Yes.

> - global - (is the default, I agree)
Yes should be default.

> - noglobal - any use case where we may want this?
Maybe when users run into performance issues?

That's all of my thoughts about what setting should be possible.

Best,
Marius

> I would like to have the possibility to add different options when I create a THttpServer.

I would like to know your "use case". What different options do you want to use in addition to what's already there?

I must ask because some options are insecure (i.e. exposing the webserver to external connections) and
while I have no problem with others shooting themselves in the foot, I think they should be warned
before they do it and I do not want to read it in the news that they did it using a gun I built.

TLDR follows:

> At the moment the manalyzer.cxx only adds the port:
> 2840: sprintf(str, "http:127.0.0.1:%d?cors", httpPort);

this is by design. it is only safe to bind thttpserver to localhost, exposing to external connections is not safe. (until somebody
can review the security situation with the version of civetweb, an antique clone of mongoose, inside ROOT).

same situation with mhttpd, it is not safe to expose it to external connections, it should always live
behind a password protected https proxy.

> But there are more options which one can use https://github.com/root-project/jsroot/blob/master/docs/HttpServer.md.

yes, I see there is more options now that there used to be before.

which ones do you want to use? and which ones you think are generally useful in what we do?

- threads - I am not sure what the latest thread-safe situation is on ROOT. is it useful to increase number of threads beyond 10?
- top=name - yes, useful, but I think jsroot overrides that.
- auth_file, auth_domain - digest authentication .htdigest file stores passwords effectively as plain text, if you steal the .htdigest file, you 
can login into the web server (as opposed to stealing /etc/passwd, you cannot login anywhere with it, not until you crack the hashes).
- loopback - this is what I want to enforce
- debug - could be useful, but is it?
- websocket - does this do anything useful for us?
- cors=domain - same as in midas mhttpd, I think we respond with CORS "*", is some other reponse useful?
- readonly - (is the default?)
- readwrite - I think should be *our* default
- global - (is the default, I agree)
- noglobal - any use case where we may want this?
- anything else?

For general use, I want maximum security configuration: bind to localhost, cors enabled, user can specify tcp port number, plus anything else from 
the list above?

For maximum flexibility we should probably have "--insecure-thttpd" to specify
the complete THttpServer() constructor argument. But it would be silly to force everybody
to use that just to set the "debug" option.

K.O.

Dear all,

I would like to have the possibility to add different options when I create a THttpServer. At the moment the manalyzer.cxx only adds the port:

2840: sprintf(str, "http:127.0.0.1:%d?cors", httpPort);

But there are more options which one can use https://github.com/root-project/jsroot/blob/master/docs/HttpServer.md.

Best regards,
Marius

In preparation to including the manalyzer into MIDAS, I split it into a separate 
repository and included it into ROOTANA as a git submodule.

https://bitbucket.org/tmidas/manalyzer/src/master/

This will cause a minor difficulty and require a manual adjustment next time you 
do a "git pull":
- check that (a) manalyzer directory is empty or (b) it only has .o and .exe 
files, in both cases delete it
- run "git submodule init; git submodule update", you should see it clone the 
manalyzer repository

K.O.