A few months ago, definitions of SYNC and ASYNC in midas.h have been changed away from "0" and "1", 
and this caused problems with some event buffer management functions bm_xxx().

For example, when event buffers are getting full, bm_send_event(SYNC) unexpectedly started returning 
BM_ASYNC_RETURN instead of waiting for free space, causing unexpected crashes of frontend programs.

Part of the problem was confusion between SYNC/ASYNC used by buffer management (bm_xxx) and by run 
transition (cm_transition()) functions. Adding to confusion, documentation of bm_send_event() & co used 
FALSE/TRUE while most actual calls used SYNC/ASYNC.

To sort this out, an executive decision was made to abolish the SYNC/ASYNC defines:

For buffer management calls bm_send_event(), bm_receive_event(), etc, please use:
SYNC -> BM_WAIT
ASYNC -> BM_NO_WAIT

For run transitions, please use:
SYNC -> TR_SYNC
ASYNC -> TR_ASYNC
MTHREAD -> TR_MTHREAD
DETACH -> TR_DETACH

K.O.

• write a custom parser in C++ using the instructions contained in the Mhformat page;
  
• call the mhist program from within my application;
  
• call the mhdump program from within my application;
  

<p>[quote=&quot;Pintaudi Giorgio&quot;]Dear MIDAS people, I need to borrow your 
wisdom for a bit. I am developing a piece of software that should read the history data 
stored in a [FONT=Times New Roman].midas[/FONT] file (MIDAS format) and integrate it 
into the WAGASCI data quality output. In other words, I need to read some temperature 
values stored in a [FONT=Times New Roman].midas[/FONT] file and compare them with 
the MPPC gains and check for temperature/gain dependence. I see three possibilities: 
[LIST] [*] write a custom parser in C++ using the instructions contained in the 
[URL=https://midas.triumf.ca/MidasWiki/index.php/Mhformat]Mhformat page[/URL]; [*] call 
the mhist program from within my application; [*] call the mhdump program from within my 
application; [/LIST] [B]Which solution do you think is the best?[/B] Because there is no 
need for raw performance, if possible, I would like to write my application in Python3 but 
C++ is also an option. [/quote]</p>

(Please write messages in plain text format, thank you)

The format of .hst midas history files is pretty simple and mhdump.cxx is an easy to read 
illustration on how to read it from basic principles (without going through the midas library, 
which can be somewhat complicated). The newer "FILE" format for history is even simpler 
to read because it is just fixed-record-size binary data prepended by a text header.

You can also use the mh2sql program to import history data into an sql database (mysql 
and sqlite should work) or to convert .hst files to "FILE" format files. This works well
for "archiving" history data, because the "FILE" format works better for looking at old data,
and for looking at data in "months" or "years" timescale.

Back to your question, you can certainly use "mhdump" as is, using a pipe (popen()), or 
you can package mhdump.cxx as a c++ class and use it in your application. If you go this 
route, your contribution of such a c++ class back to midas would be very welcome.

You can also use mhist, but the mhist code cannot be trivially packaged as a c++ class
to use in your application.

You can also suggest that we write an easier to use history utility, we are always open to 
suggested improvements.

Let us know how it works out for you. Good luck!

K.O.

> The format of .hst midas history files is pretty simple and mhdump.cxx is an easy to read 
> illustration on how to read it from basic principles (without going through the midas library, 
> which can be somewhat complicated). The newer "FILE" format for history is even simpler 
> to read because it is just fixed-record-size binary data prepended by a text header.
> 
> You can also use the mh2sql program to import history data into an sql database (mysql 
> and sqlite should work) or to convert .hst files to "FILE" format files. This works well
> for "archiving" history data, because the "FILE" format works better for looking at old data,
> and for looking at data in "months" or "years" timescale.
> 
> Back to your question, you can certainly use "mhdump" as is, using a pipe (popen()), or 
> you can package mhdump.cxx as a c++ class and use it in your application. If you go this 
> route, your contribution of such a c++ class back to midas would be very welcome.
> 
> You can also use mhist, but the mhist code cannot be trivially packaged as a c++ class
> to use in your application.
> 
> You can also suggest that we write an easier to use history utility, we are always open to 
> suggested improvements.
> 
> Let us know how it works out for you. Good luck!
> 
> K.O.

Dear Konstantin,
thank you very much for the wealth of information you provided.
I have thought about it and I see two options:

- One is to convert to SQL format and then use a SQLite library to import the data in my 
application.

- The other is to encapsulate the mhdump.cxx code into a C++ class, as you say.

I am leaning towards the first option for three reasons.
1. I have never used a SQLite database so it is a good learning opportunity for me.
2, The SQLite database format is very well known and widespread, so there are tons of tools to 
handle it
3. I have taken a look at the mhdump.cxx source code and I think it is a beautiful piece of code, 
but has a very "functional" taste with little encapsulation. Basically, all the fun is happening 
inside the readHstFile function and there is no trivial way to get the data out of it. I don't mean 
that it would be difficult to wrap it around a C++ class, but I feel that I can learn more by going 
the SQL way.

PS some time ago, I don't remember if you or Stefan, recommended CLion as C++ IDE. I have tried it 
(together with PyCharm) and I must admit that it is really good. It took me years to configure Emacs 
as a IDE, while it took me minutes to have much better results in CLion. Thank you very much for 
your recommendation.

>
> - One is to convert to SQL format and then use a SQLite library to import the data in my 
> application.
> 

You can also configure midas to write history directly to an SQLITE database. I have not used
it recently, but it should still work. In terms of efficiency, sqlite file size is about the same
as .hst files. sqlite file and table naming is similar to the SQL and FILE implementation.

(But note that back when I implemented the SQLITE history writer, sqlite database corruption
recovery instructions were "delete the file, restore from backup". And indeed in every test
experiment I tried, the sqlite history databases eventually corrupted themselves. You see
same thing with google-chrome, lots of sqlite errors (bad locking, corrupted table, etc)
in it's terminal output).

>
> - The other is to encapsulate the mhdump.cxx code into a C++ class, as you say.
> 

If I were to write this today, there would be a c++ class that takes a history file,
iterates over all records and calls "callback" classlets. You can see this in the history.h
(HistoryBufferInterface) and in the tmfe.h (RpcHandlerInterface, etc).

I think this style of OO programming originally comes from java. If you so desire,
an "mhdump" class could be a nice way to learn it.

> 
> PS some time ago, I don't remember if you or Stefan, recommended CLion as C++ IDE. I have tried it 
> (together with PyCharm) and I must admit that it is really good. It took me years to configure Emacs 
> as a IDE, while it took me minutes to have much better results in CLion. Thank you very much for 
> your recommendation.
>

I remember, years ago, the Borland TurboC IDE was like a gift from Gods. But today, I think IDEs have
declined in quality and usefulness. They clog the screen with too much eye candy and fluff, use hard
to read fonts and silly colours, insist on using tabs where I want spaces, reformat the text even as I type it,
and detract from productive work with distracting popups ("try this new function!", "let's upgrade now!").

For serious programming, I use emacs with minimal decorations. I can easily open 3 or 4 windows at the same
time and still have enough screen space left for a terminal to run "make". And it is the only editor that can
edit the same file in two or more windows at the same time. You do not know you need this until
you work on odb.cxx.

K.O.

> PS some time ago, I don't remember if you or Stefan, recommended CLion as C++ IDE. I have tried it 
> (together with PyCharm) and I must admit that it is really good. It took me years to configure Emacs 
> as a IDE, while it took me minutes to have much better results in CLion. Thank you very much for 
> your recommendation.

Was probably me. I use it as my standard IDE and am quite happy with it. All the things KO likes with emacs, plus much 
more. Especially the CMake integration is nice, since you don't have to leave the IDE for editing, compiling and debugging. 
The tooltips the IDE gave me in the past months made me write code much better. So quite an opposite opinion compared 
with KO, but luckily this planet has space for all kinds of opinions. I made myself the cheat sheet attached, which lets me 
do things much faster. Maybe you can use it.

Stefan

> (But note that back when I implemented the SQLITE history writer, sqlite database corruption
> recovery instructions were "delete the file, restore from backup". And indeed in every test
> experiment I tried, the sqlite history databases eventually corrupted themselves. You see
> same thing with google-chrome, lots of sqlite errors (bad locking, corrupted table, etc)
> in it's terminal output).

Thank you for the info. But I do not quite understand the comment above.
Do you mean that there is something wrong with the SQLite library itself or with the way that MIDAS creates the SQLite 
database?

Eventually, I have settled for the SQLite format.
I could convert the MIDAS history files .hst to SQLite
database .sqlite3 using the utility mh2sql.
It worked out nicely, thank you for the advice.

However, as Konstantine predicted I did notice some
database corruption when a couple of problematic .hst
files were read. I solved the issue by just deleting
those .hst files (I think they were empty anyway).

Now I am developing a piece of code to read the
database using the SOCI library and integrate it
into a TTree but this is not relevant for MIDAS I think.

Thank you again for the discussion.

I recently needed to watch the Midas messages for a particular error - and 
thus needed a command to "get all the messages since a time t".

The documentation (https://midas.triumf.ca/MidasWiki/index.php/AJAX#jmsg) 
documents a way to "get the most recent n messages" - but when I dug into the 
code, I was delighted to find that the existing Midas code also supports the 
"get all messages since t" query.

For the "get all messages since t" query, the parameter t should be the unix 
timestamp in seconds, and the parameter n should be zero: curl -X GET 
"http://localhost:8081/?cmd=jmsg&n=0&t=1473437918".

Pretty useful!  Perhaps this should be added to the AJAX documentation?

> I recently needed to watch the Midas messages for a particular error - and 
> thus needed a command to "get all the messages since a time t".
> 
> The documentation (https://midas.triumf.ca/MidasWiki/index.php/AJAX#jmsg) 
> documents a way to "get the most recent n messages" - but when I dug into the 
> code, I was delighted to find that the existing Midas code also supports the 
> "get all messages since t" query.
> 
> For the "get all messages since t" query, the parameter t should be the unix 
> timestamp in seconds, and the parameter n should be zero: curl -X GET 
> "http://localhost:8081/?cmd=jmsg&n=0&t=1473437918".
> 
> Pretty useful!  Perhaps this should be added to the AJAX documentation?

Thank you - I have added it to the documentation.

> I recently needed to watch the Midas messages for a particular error - and 
> thus needed a command to "get all the messages since a time t".
> 
> The documentation (https://midas.triumf.ca/MidasWiki/index.php/AJAX#jmsg) 
> documents a way to "get the most recent n messages" - but when I dug into the 
> code, I was delighted to find that the existing Midas code also supports the 
> "get all messages since t" query.
> 
> For the "get all messages since t" query, the parameter t should be the unix 
> timestamp in seconds, and the parameter n should be zero: curl -X GET 
> "http://localhost:8081/?cmd=jmsg&n=0&t=1473437918".
> 
> Pretty useful!  Perhaps this should be added to the AJAX documentation?

The "jmsg" methods are obsolete - please use the JSON-RPC method "cm_msg_retrieve" as shown in resources/example.html. It takes all the same parameters as the midas.h 
cm_msg_retrieve() function, see the snipped from example.html below.

To see the full list of JSON-RPC methods, go to the "help" page and press the button for "json-rpc schema in text table format".

The entry for "cm_msg_retrieve" has this:

------------------------------------------------------------------------------------
cm_msg_retrieve?      | Retrieve midas messages using cm_msg_retrieve2()
                      | ------------------------------------------------------------
                      | params   | facility?           | string         | message facility, default is "midas"
                      |          | min_messages?       | integer        | get at least this many messages, default is 1
                      |          | time?               | number         | start from given timestamp, value 0 means give me newest messages, default is 0
                      | ------------------------------------------------------------
                      | result   | num_messages        | integer        | number of messages returned
                      |          | messages            | string         | messages separated by \n
                      |          | status              | integer        | return status of cm_msg_retrieve2()
------------------------------------------------------------------------------------

Snippet from resources/example.html: (to add "time" parameter, put "time":12345 next to "min_messages").

<input type=button value='Get last 10 midas messages'
          onClick='mjsonrpc_call("cm_msg_retrieve", { "min_messages": 10 })
                   .then(function(rpc) {
                   document.getElementById("cm_msg_retrieve_num_messages").innerHTML = JSON.stringify(rpc.result.num_messages);
                   document.getElementById("cm_msg_retrieve_messages").innerHTML = JSON.stringify(rpc.result.messages);
                   //mjsonrpc_debug_alert(rpc);
                   })
                   .catch(function(error) {
                   mjsonrpc_error_alert(error);
                   });'></input>

When writing custom webpages, it would be nice to be able to write code such as

<odb src="/Equipment/TITAN_ACQ/ppg cycle/trans3/time offset (ms)" edit=1>

from Javascript, e.g.
<script  type="text/javascript">
if ( flag != 3)
   document.write('<odb src="/Equipment/TITAN_ACQ/ppg cycle/trans3/time offset
(ms)" edit=1>ms');
else
   document.write('<odb src="/Equipment/TITAN_ACQ/ppg cycle/trans4/time offset
(ms)" edit=1>ms');
</script>

This is not translated correctly by mhttpd; the final quote and bracket get
stripped off, and it gives Javascript error

 Error: unterminated string literal
Source File: http://titan04:8089/CS/ppg_cycle?cmd=Edit&index=11
Line: 477, Column: 18
Source Code:
   document.write('<input type=text size=10 maxlength=80 name=value value="1">

I can get round this by using an input box and a combination of ODBGet and
ODBSet, but it would be easier if the edit=1 form above worked correctly, or
there was a command like ODBSet that would accept input from the user.

Thanks.

 would be nice is there was a command such as ODBGet or ODBSet that would work
with javascript to 

> When writing custom webpages, it would be nice to be able to write code such as
> 
> <odb src="/Equipment/TITAN_ACQ/ppg cycle/trans3/time offset (ms)" edit=1>
> 
> from Javascript, e.g.
> <script  type="text/javascript">
> if ( flag != 3)
>    document.write('<odb src="/Equipment/TITAN_ACQ/ppg cycle/trans3/time offset
> (ms)" edit=1>ms');
> else
>    document.write('<odb src="/Equipment/TITAN_ACQ/ppg cycle/trans4/time offset
> (ms)" edit=1>ms');
> </script>
> 
> This is not translated correctly by mhttpd; the final quote and bracket get
> stripped off, and it gives Javascript error
> 
>  Error: unterminated string literal
> Source File: http://titan04:8089/CS/ppg_cycle?cmd=Edit&index=11
> Line: 477, Column: 18
> Source Code:
>    document.write('<input type=text size=10 maxlength=80 name=value value="1">
> 
> I can get round this by using an input box and a combination of ODBGet and
> ODBSet, but it would be easier if the edit=1 form above worked correctly, or
> there was a command like ODBSet that would accept input from the user.
> 
> Thanks.
> 
>  would be nice is there was a command such as ODBGet or ODBSet that would work
> with javascript to 

Actually that won't work, even if I would fix it. The <odb> tag is evaluated on the
server side (mhttpd), where is gets replaced by the actual ODB value. But if you
use JavaScript to generate the <odb> tag dynamically, this only happens on the
client side, so the server has no chance to substitute them. So you have to go with
ODBGet's I'm afraid. Nevertheless, I changed the code such that any ODB tags inside
a JavaScript is not interpreted by mhttpd.

Hi,

I've been doing some preliminary work to use at least the MIDAS
SQL history component for a new CERN experiment (Aegis). I wonder
whether there is any plan to support 64-bit signed/unsigned integer data types
in MIDAS. time_t on 64-bit architectures is actually signed 64-bit
(the 'easy' way to work around the 2038 crisis), and this may be enough to
cause problems.

Thanks.
Francesco Prelz
INFN Milano

I've seen that a similar question has been asked in 2011 but I'll ask again in 
case there are any updates. Is there any way to write 64-bit data words to MIDAS 
banks (other than breaking them up in to two 32-bit words, such as 2 DWORDs) 
currently? And if not, is there any plan to introduce this feature in the future?

Many thanks,
Tom