We have an experiment that's been running for a long time and has some ODB keys that haven't been touched in ages. Mostly related to features that we don't use like the elog and lazylogger, or things that don't change often (like the logger data directory).

When we start any program, we now got dozens of error messages in the log with lines like:
hkey 297088, path "/Elog/Display run number", invalid pkey->last_written time 1377040124

That timestamp is reasonable though, as the experiment was set up in 2013!

What's the best way to make these messages go away? 
- Change the logic in db_validate_and_repair_key_wlocked() to not worry if keys are 10+ years old? 
- Write a script to "touch" all the old keys so they've been modified recently?
- Something else?

> What's the best way to make these messages go away? 
> - Change the logic in db_validate_and_repair_key_wlocked() to not worry if keys are 10+ years old? 
> - Write a script to "touch" all the old keys so they've been modified recently?
> - Something else?

The function db_validate_and_repair_key_wlocked() has been written by KO so he should reply here.

In my opinion, I would go with the first one. Changing the function is easier than to write a script
and teach everybody how to use it. This would be one more thing not to forget.

Now changing the function is not so obvious. We could extend the check to let's say 20 years, but
then we meet here again in ten years. Maybe the best choice would be to just check that the time
is not in the future.

Anyhow, most people don't realize, but we all will have fun on Jan 19, 2038, when the Unix time
overflows in 32-bit signed integers. I don't know if midas will be around by then (I will be 74 years),
but before that date one has to worry about many places in midas where we use Unix time. At that time
your date stamps from 2013 would be 25 years old, so we either remove the date check (just keep
the check of not being in the future), or extend it to 26 years.

Stefan

Hi,

> What's the best way to make these messages go away? 
1.
> - Change the logic in db_validate_and_repair_key_wlocked() to not worry if keys are 10+ years old? 
2.
> - Write a script to "touch" all the old keys so they've been modified recently?
3.
> - Something else?

I wondered about this just under a year ago, and Konstantin forwarded my query here:
https://daq00.triumf.ca/elog-midas/Midas/2470

I am now of the opinion that 2 is not a good approach since it removes potentially
useful information.

I think some version of 1. is the correct choice. Whatever it fix is, I think it
should not care that timestamps of when variables are set are "old" (or at least
it should be user configurable via some odb setting).

Nick.

I don't immediately see a reason for saying that if a DB key is older than 10 yrs, it may not be valid.

However, it would be worth learning what was the logic behind choosing 10 yrs as a threshold. 
If 10 is just a more or less arbitrary number, changing 10 --> 100 seems to be the way to go.

-- regards, Pasha 

> I don't immediately see a reason for saying that if a DB key is older than 10 yrs, it may not be valid.
> 
> However, it would be worth learning what was the logic behind choosing 10 yrs as a threshold. 
> If 10 is just a more or less arbitrary number, changing 10 --> 100 seems to be the way to go.

Please run "git blame" to find out who added that check.

If I remember right, it was added to complain/correct dates in the future.

I think the oldest experiment at TRIUMF where we still can load an odb into current MIDAS is TWIST,
now about 25 years old. the purpose of loading odb would be to test the history function
to see if we can look at 10-15 year old histories. (TWIST history is in the latest FILE format,
so it will load).

I think this age check should be removed, but there must be *some* check for invalid/bogus timestamps. Or 
not, we should check if MIDAS cares about timestamps at all, if ODB functions never use/look at timestamp, 
maybe we are okey with bogus timestamps. They may look funny in the odb editor, but that's it.

K.O.

> Please run "git blame" to find out who added that check.

OK ok, was me. But actually 2003. I hope that this being more than 20y ago excuses me not remembering it ;-)

> I think this age check should be removed, but there must be *some* check for invalid/bogus timestamps. Or 
> not, we should check if MIDAS cares about timestamps at all, if ODB functions never use/look at timestamp, 
> maybe we are okey with bogus timestamps. They may look funny in the odb editor, but that's it.

I changed the code to only check for timestamps more than 1h in the future and then complain. This should
avoid glitches when switching daylight savings time.

Stefan

Today, I encountered the bug shown in the attached video. The value of the plotted curve does not match the mouseover number.

When trying to understand it better, I stopped being able to replicate. Has anyone else observed a similar problem? 

Version 1.9.4 of midas has been released today. It is mainly a maintenance
update, for all the little things which have been fixed since the last
release, and does not contain major new functionality.

A warning, in case others have the same problem I had.

In the past you could configure mhttpd so that the 'Elog' button would redirect to an external ELOG server; to do this you only needed to create and set the ODB variable '/Elog/URL' to the URL of your external ELOG server.

But with the newer MIDAS you need to set two ODB variables:

* "/Elog/URL" needs to be set to the URL of the external ELOG.
* "/Elog/External Elog" needs to be set to 'y'

I hadn't noticed this and was confused why my Elog button wasn't working after upgrading MIDAS.

MIDAS documentation was updated to reflect this change:

https://midas.triumf.ca/MidasWiki/index.php/Electronic_Logbook_(ELOG)

Hi everyone,

I am currently trying to access a ROOT file produced by manalyzer. By calling JSROOT.openFile("MIDAS_DOMAIN/outputRUN.root"). I can download the rootfile via MIDAS_DOMAIN/outputRUN.root. Using JSROOT.openFile results in an 501 error, 
since the request feature is not provided. Using a simple API and uploading outputRUN.root there worked fine (when the run finised). 

Is there a way to use JSROOT.openFile with the current analyzed root file in Midas (so during the run)? I know that one can access histograms of the THttpServer via JSON but I need to get the full root tree.

Cheers,
Marius

> 
> I am currently trying to access a ROOT file produced by manalyzer. By calling JSROOT.openFile("MIDAS_DOMAIN/outputRUN.root"). I can download the rootfile via MIDAS_DOMAIN/outputRUN.root. Using JSROOT.openFile results in an 501 error, 
> since the request feature is not provided. Using a simple API and uploading outputRUN.root there worked fine (when the run finised). 
> 
> Is there a way to use JSROOT.openFile with the current analyzed root file in Midas (so during the run)? I know that one can access histograms of the THttpServer via JSON but I need to get the full root tree.
> 

Good questions. Right now in manalyzer I do not do anything more than starting the ROOT web server (so whatever
they support should work) and providing two "standard" location: one in the output file for histograms
and other permanent output and one in memory for transient objects, such as waveform plots, etc.

At some point I would like to provide a function to "get" TAFlowEvent objects so you can do things
like event displays in javascript. But I need a c++ to json serializer and standard c++ does not have it.
So I will have to use the clang serializer (also used by ROOT) and it will take me a few days
to figure it out.

Back to "openFile".

If you figure out the missing bits that need to be added to our code,
please post them here or submit them as a pull request or a bug report in bitbucket.

Also it would be good if you can provide a code example of "openFile" working elsewhere
but not with manalyzer, if I can run it, maybe I can figure out what's missing. But lacking
some example code, there is nothing for me to hack at.

K.O.

Thank you for the answer :)

> At some point I would like to provide a function to "get" TAFlowEvent objects so you can do things
> like event displays in javascript. But I need a c++ to json serializer and standard c++ does not have it.
> So I will have to use the clang serializer (also used by ROOT) and it will take me a few days
> to figure it out.

That sounds exactly what I was searching for. Because I wanted to create an interface between rootana and 
an event display build in javascript. Since all I tried did not really worked with the current rootana
I have now a "solution". I use the MIDAS python client, read directly events from the MIDAS buffer and provided
the events in a JSON format with the python flask API. Since the rendering of the event display is the bottleneck 
and I only need a view events to display this solution worked really well for me. Maybe having such a JSON API of 
the event buffer in MIDAS directly would also work for most of the event display applications or other simple javescript 
applications (my opinion).

> If you figure out the missing bits that need to be added to our code,
> please post them here or submit them as a pull request or a bug report in bitbucket.

One of the problems I had was the CORS domain of the THttpServer. In manalyzer:1894 you do 
sprintf(str, "http:127.0.0.1:%d", httpPort); but there are additional options for the THttpServer (like "?cors=DOMAIN").
So maybe a flag while starting manalyzer passing such options would be nice. I will create a pull request passing them later.

> Also it would be good if you can provide a code example of "openFile" working elsewhere
> but not with manalyzer, if I can run it, maybe I can figure out what's missing. But lacking
> some example code, there is nothing for me to hack at.

The problem is that it was not even running on simple THttpServer using interactive root:

    serv = new THttpServer("http:8088?cors=*");
    TFile *_file0 = TFile::Open("example_root.root")
    serv->Register("File", _file0);

So I tried just saving the file in the $MIDAS_DIR and tried to use mserver with JSROOT.openFile. I attached the html file and 
a test root file.

Cheers,
Marius

well, if this is something in ROOT, perhaps you can pursue it with the ROOT crowd,
they are quite friendly.

on my side, if all you need is to pull event data banks, this is easy to add
in mhttpd.

the jsonrpc request will look something like this:

get_event {
"buffer":"system",
"get_type":"GET_LATEST", (or whatever bm_receive_event() can do)
"include_banks":["AAAA","BBBB"],
"exclude_banks":["CCCC","DDDD"]
}

and return something like this:

event {
"header":{"event_id":1,...},
"banks":{
"AAAA":[1,2,3,4],
"BBBB":NULL (you asked for it, so you always get it, but it is NULL if bank does not exist)
}
}

would this work for what you are doing?

(this is not good enough if data has to be pre-digested by c++ analysis in rootana)

K.O.

I also need midas events going back to the

I also need midas events going back to the browser for single event display, so put +1 for me.

Please also consider to use JavaScript typed arrays instead of JSON. For large midas banks, type 
arrays are 5-10 times faster than JSON encoding/decoding.

Best,
Stefan

> would this work for what you are doing?

Yes, having such a function would be perfect for the applications I have a the moment.

> (this is not good enough if data has to be pre-digested by c++ analysis in rootana)

Also agree, if one wants to have a more sophisticated applications it is definitely needed to preprocess the data.

Cheers,
Marius