I will work today on the odbxx API to make sure there are no memory leaks when you switch form one file to another. I talked to KO so he agreed that yo then commit your proposed change of manalyzer 

Best,
Stefan

> I will work today on the odbxx API to make sure there are no memory leaks when you switch form one file to another. I talked to KO so he agreed that yo then commit your proposed change of manalyzer 

That, and add a "clear()" method that resets odbxx state to "empty". I will call odbxx.clear() everywhere where I call "delete fOdb;" (TARunInfo::dtor and other places).

K.O.

> > ....Before commit of this patch, can you confirm the RunInfo destructor
> > deletes this ODB stuff from odbxx? manalyzer takes object life times very seriously.
> 
> The call stores the ODB string in static members of the midas::odb class. So these will have a lifetime of the process or until they're replaced by another 
> call. When a midas::odb is instantiated it reads from these static members and then that data has the lifetime of that instance.

this is the behavious we need to modify.

> > Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.
> Yes, I hadn't realised that was an option.

It is an option I would like to keep open. Not too many use cases, but imagine a "split brain" experiment
that has two MIDAS instances record data into two separate midas files. (if LIGO were to use MIDAS,
consider LIGO Hanford and LIGO Livingston).

Assuming data in these two data sets have common precision timestamps,
our task is to assemble data from two input files into single physics events. The analyzer will need
to read two input files, each file with it's run number, it's own ODB dump, etc, process the midas
events (unpack, calibrate, filter, etc), look at the timestamps, assemble the data into physics events.

This trivially generalizes into reading 2, 3, or more input files.

> For that to work I guess the aforementioned static members could be made thread local storage, and 
> processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
> class member or something.

manalyzer is already multithreaded, if you will need to keep track of which thread should see which odbxx global object,
seems like abuse of the thread-local storage idea and intent.

> Note that I missed doing the same for the end of run event, which should probably also be added.

Ideally, the memory sanitizer will flag this for us, complain about anything that odbxx.clear() failes to free.

K.O.

> > I will work today on the odbxx API to make sure there are no memory leaks when you switch form one file to another. I talked to KO so he agreed that yo then commit your proposed change of manalyzer 
> 
> That, and add a "clear()" method that resets odbxx state to "empty". I will call odbxx.clear() everywhere where I call "delete fOdb;" (TARunInfo::dtor and other places).

No need for clear(), since no memory gets allocated by midas::odd::set_odb_source(). All it does is to remember the file name. When you instantiate a midas::odd object, the file gets loaded, and the midas::odd object gets initialized from the file contents. Then the buffer 
gets deleted (actually it's a simple local variable). Of course this causes some overhead (each midas::odd() constructor reads the whole file), but since the OS will cache the file, it's probably not so bad.

Stefan

> > > Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.
> > Yes, I hadn't realised that was an option.
> 
> It is an option I would like to keep open. Not too many use cases, but imagine a "split brain" experiment
> that has two MIDAS instances record data into two separate midas files. (if LIGO were to use MIDAS,
> consider LIGO Hanford and LIGO Livingston).
> 
> Assuming data in these two data sets have common precision timestamps,
> our task is to assemble data from two input files into single physics events. The analyzer will need
> to read two input files, each file with it's run number, it's own ODB dump, etc, process the midas
> events (unpack, calibrate, filter, etc), look at the timestamps, assemble the data into physics events.
> 
> This trivially generalizes into reading 2, 3, or more input files.
> 
> > For that to work I guess the aforementioned static members could be made thread local storage, and 
> > processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
> > class member or something.
> 
> manalyzer is already multithreaded, if you will need to keep track of which thread should see which odbxx global object,
> seems like abuse of the thread-local storage idea and intent.

I made the global variables storing the file name of type "thread_local", so each thread gets it's own copy. This means however that each thread must then call midas::odb::set_odb_source() individually before 
creating any midas::odb objects. Interestingly enough I just learned that thread_local (at least under linux) is almost zero overhead, since these variable are placed by the linker into a separate memory space which is 
separate for each thread, so accessing them only means to add a memory offset.

Let's see how far we get with this...

Stefan

Documenting a discussion I had with Konstantin a while ago.

One aspect of the MIDAS history plotting I find frustrating is the sequence for adding a new history 
variable and then plotting them.  At least for me ,the sequence for a new variable is:

1) Add a new named variable to a MIDAS bank; compile the frontend and restart it; check that the new 
variable is being displayed correctly in MIDAS equipment pages.

2) Stop and restart programs; usually I do
- stop and restart mlogger
- stop and restart mhttpd
- stop and restart mlogger
- stop and restart mhttpd

3) Start adding the new variable to the history plots.

My frustration is with step 2, where the logger and web server need to be restarted multiple times.  
I think that only one of those programs actually needs to be restart twice, but I can never remember 
which one, so I restart both programs twice just to be sure.

I don't entirely understand the sequence of what happens with these restarts so that the web server 
becomes aware of what new variables are in the history system; so I can't make a well-motivated 
suggestion.

Ideally it would be nice if step 2 only required restarting the mlogger and mhttpd automatically 
became aware of what new variables were in the system.

But even just having a single restart of mlogger, then mhttpd would be an improvement on the current 
situation and easier to explain to users.

Hopefully this would not be a huge amount of work.

> ...I talked to KO so he agreed that yo then commit your proposed change of manalyzer 

Merged and pushed.

Thanks,

Mark.

> > ...I talked to KO so he agreed that yo then commit your proposed change of manalyzer 
> Merged and pushed.

negative. as I already 

• ODB over Frontend precedence.
  When starting a FE client, the equipment settings are taken from the ODB if this equipment already existed. This meant the ODB has precedence over the EQUIPEMENT structure and whatever change you apply to the C-Structure, it will NOT be taken in consideration until you clean (remove) the equipment tree in ODB.
  
  
• Revived 64 bit support. This was required as more OS are already supporting such architecture. Originally Midas did support Alpha/OSF/1 which operated on 64 bit machine. This new code has been tested on SL4.2 with Dual-Core 64-bit AMD Opterons.
  
  
• Multi-threading in Slow Control equipments.
  Check entry 289 in Midas Elog from Stefan.
  
  
• mhttpd using external Elog.
  The standalone ELOG package can be coupled to an existing experiment and therefore supersede the internal elog functionality from mhttpd.
  This requires a particular configuration which is described in the documentation.
  
  
• MySQL test in mlogger
  A reminder that mlogger can generate entries in a MySQL database as long as the pre-compilation flag -HAVE_MYSQL is enabled during system built. The access and form filling is then defined from the ODB under Logger/SQL once the logger is running, see documentation.
  
  
• Directory destination for midas.log and odb dump files
  It is now possible to specify an individual directory to the default midas.log file as well as to the "ODB Dump file" destination. If either of these fields contains a preceding directory, it will take the string as an absolute path to the file.
  
  
• User defined "event Data buffer size" (ODB)
  The event buffer size has been until now defined at the system level in midas.h. It is now possible to optimize the memory allocation specific to the event buffer with an entry in the ODB under /experiment, see documentation.
  
  
• History group display
  It is now possible to display an individual group of history plots. No documentation on that topics as it should be self explanatory.
  
  
• History export option
  From the History web page, it is possible to export to a ASCII .csv file the history content. This file can later be imported into excel for example. No documentation on that topics as it should be self explanatory.
  
  
• Multiple "minor" corrections:
  - Alarm reset for multiple experiment (return directly to the experiment).
  - mdump -b option bug fixed.
  - Alarm evaluation function fixed.
  - mlogger/SQL boolean handling fixed.
  - bm_get_buffer_level() was returning a wrong value which has been fixed now.
  - Event buffer bug traced and exterminated (Thanks to Konstantin).
  

I committed an updated lazylogger with updated documentation. The new version supports subruns and 
can save to external storage arbitrary files (i.e. odb dump files). It also moves most book keeping out of 
odb to permit handling more files on bigger storage disks.

Example lazylogger scripts for castor (CERN) and dcache (TRIUMF) are in the directory "utils".

The lazylogger documentation was updated to remove obsolete information and to describe the new 
functions. As usual "make dox; cd doxfiles/html; firefox index.html" or see my copy at:

http://ladd00.triumf.ca/~olchansk/midas/Utilities.html#lazylogger_task

svn rev 4615, 4616.
K.O.

The general Midas documentation has been rejuvenated by Suzannah Daviel through 
a proof reading and with a collection of custom perl scripts to improve the 
Doxygen capabilities for the document itself. In particular, a contents list and 
alphabetical index of the documentation is generated automatically.

The new content is based on the previous version but with more cross-references, 
examples and descriptive images where necessary. Many of the previously 
undocumented features are now included.

The layout and organization is slightly different and requires getting used to, 
but hopefully will be an improvement.
Some of the changes are:
- The main topics are maintained, but we try to regroup all the aspects
  related to a particular topic in the same section.
- The Yellow icons provide navigation within the index section.
- The Blue icons provide navigation within the section content.

The full documentation is included under the midas/doc/src directory in the 
Midas distribution (SVN) and can be generated with the Doxygen tool.
The midasdoc-images.tar.gz from either https://midas.psi.ch/download/ or 
http://ladd00.triumf.ca/~daqweb/ftp/ needs to be extracted to the midas 
directory under doc/images for complete local web pages generation.

There are a few "ToDo" items which hopefully will be ironed out soon.
Feel free to contact us for pointing out omissions or improvements.

We hope this online documentation will serve as a better tool for your 
understanding of the Midas capabilities.

The final bits of the JSON-RPC API to MIDAS are committed. The API uses the Javascript Promise mechanism (supported on all 
supported platforms - MacOS, Windows, Linux Ubuntu, el5, el6, el7).

Simple example for pasting the current run number into an html element:

mjsonrpc_db_get_values(["/runinfo/run number"]).then(function(rpc) {
   document.getElementById("run_number").innerHTML = rpc.response.data[0];
}).catch(function(error) {
   mjsonrpc_error_alert(error);
});

The documentation for the JSON-RPC API, including special quirks in JSON encoding of ODB data is here:
https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc

Documentation (with examples) for the related Javascript functions in mhttpd.js is here (via Doxygen):
https://daq.triumf.ca/~daqweb/doc/midas-devel/html/group__mjsonrpc__js.html

Examples of using all mhttpd.js functions is in .../examples/javascript1/example.html

The experimental git branch feature/mhttpd_js implements the MIDAS "programs" page purely in html and javascript,
go there to see all this new JSON and RPC stuff in action. See .../resources/programs.html.

K.O.

> The final bits of the JSON-RPC API to MIDAS are committed.

Here is example conversion of the function "generate midas message" from old-style AJAX to JSON-RPC:

before (mhttpd.cxx):

   /* process "jgenmsg" command */
   if (equal_ustring(getparam("cmd"), "jgenmsg")) {

      if (getparam("facility") && *getparam("facility"))
         strlcpy(facility, getparam("facility"), sizeof(facility));
      else
         strlcpy(facility, "midas", sizeof(facility));
      
      if (getparam("user") && *getparam("user"))
         strlcpy(user, getparam("user"), sizeof(user));
      else
         strlcpy(user, "javascript_commands", sizeof(user));
      
      if (getparam("type") && *getparam("type"))
         type = atoi(getparam("type"));
      else
         type = MT_INFO;

      if (getparam("msg") && *getparam("msg")) {
         cm_msg1(type, __FILE__, __LINE__, facility, user, "%s", getparam("msg"));
      }

      show_text_header();
      rsputs("Message successfully created\n");
      return;
   }

after: (mjsonrpc.cxx)

static MJsonNode* js_cm_msg1(const MJsonNode* params)
{
   if (!params) {
      MJSO *doc = MJSO::I();
      doc->D("Generate a midas message using cm_msg1()");
      doc->P("facility?", MJSON_STRING, "message facility, default is \"midas\"");
      doc->P("user?", MJSON_STRING, "message user, default is \"javascript_commands\"");
      doc->P("type?", MJSON_INT, "message type, MT_xxx from midas.h, default is MT_INFO");
      doc->P("message", MJSON_STRING, "message text");
      doc->R("status", MJSON_INT, "return status of cm_msg1()");
      return doc;
   }

   MJsonNode* error = NULL;

   const char* facility = mjsonrpc_get_param(params, "facility", &error)->GetString().c_str();
   const char* user = mjsonrpc_get_param(params, "user", &error)->GetString().c_str();
   int type = mjsonrpc_get_param(params, "type", &error)->GetInt();
   const char* message = mjsonrpc_get_param(params, "message", &error)->GetString().c_str(); if (error) return error;

   if (strlen(facility)<1)
      facility = "midas";
   if (strlen(user)<1)
      user = "javascript_commands";
   if (type == 0)
      type = MT_INFO;

   int status = cm_msg1(type, __FILE__, __LINE__, facility, user, "%s", message);

   return mjsonrpc_make_result("status", MJsonNode::MakeInt(status));
}

With the corresponding javascript-side stabs:

before:

function ODBGenerateMsg(type,facility,user,msg)
{
   var request = XMLHttpRequestGeneric();

   var url = ODBUrlBase + '?cmd=jgenmsg';
   url += '&type='+type;
   url += '&facility='+facility;
   url += '&user='+user;
   url += '&msg=' + encodeURIComponent(msg);
   request.open('GET', url, false);
   request.send(null);
   return request.responseText;
}

after:

function mjsonrpc_cm_msg(message, type, id) {
   /// \ingroup mjsonrpc_js
   /// Get values of ODB variables
   ///
   /// RPC method: "cm_msg1"
   ///
   /// \code
   /// mjsonrpc_cm_msg("this is a new message").then(function(rpc) {
   ///    var req    = rpc.request; // reference to the rpc request
   ///    var id     = rpc.id;      // rpc response id (should be same as req.id)
   ///    var status = rpc.result.status;  // return status of MIDAS cm_msg1()
   ///    ...
   /// }).catch(function(error) {
   ///    mjsonrpc_error_alert(error);
   /// });
   /// \endcode
   /// @param[in] message Text of midas message (string)
   /// @param[in] type optional message type, one of MT_xxx. Default is MT_INFO (integer)
   /// @param[in] id optional request id (see JSON-RPC specs) (object)
   /// @returns new Promise
   ///
   var req = new Object();
   req.message = message;
   if (type)
      req.type = type;
   return mjsonrpc_call("cm_msg1", req, id);
}

K.O

> > The final bits of the JSON-RPC API to MIDAS are committed.

JSON-RPC methods are now provided for all old ODBxxx() javascript functions, except ODBGetMsg().

The currently present RPC methods are sufficient to write the MIDAS "programs" and "alarms" pages
purely in HTML+Javascript (see the git branch feature/mhttpd_js). These pages can be served i.e. by apache httpd
with midas mhttpd only required to service the RPC requests.

Please see .../examples/javascript1/example.html on how to use the new RPC methods.

K.O.

P.S. Note how many examples use the generic mjsonrpc_call() because I did not write the corresponding
javascript functions - I wore out the cut-and-paste button on my keyboard. All are welcome to contribute
the missing functions, post them here or email them to me, I will commit them to midas git.

The code for the new pure html and javascript web pages was merged into main midas.

In this release, the "programs" and "alarms" pages are implemented as html files, see 
resources/programs.html and alarms.html.

Eventually we hope to implement all midas web pages in html, so this is just a start.

If you see problems with the new html code, you can revert to the old mhttpd-generated web 
pages by removing the files programs.html and alarms.html.

The new code for starting and stopping runs (start.html and transition.html) is also merged, but not 
yet enabled, pending a few more tests.

K.O.

We are readying a new release of midas and it is almost here except for a few buglets on the new html status page.

The current release candidate branch is "feature/midas-2017-10" and if you have problems with the older versions
of midas, I recommend that you try this release candidate to check if your problem is already fixed. If the problem
still exists, please file a bug report on this forum or on the bitbucket issue tracker
https://bitbucket.org/tmidas/midas/issues?status=new&status=open

Highlights of the new release include
- new and improved web pages done in html and javascript
- many bug fixes and improvements for json and json-rpc support, including improvements in handling of long strings in odb
- locked (protected) operation of odb, where odb shared memory is not writable outside of odb operations
- improved multithead support for odb
- fixes for odb corruption when odb becomes 100% full

For the next release we hope to switch midas from C to fully C++ (building everything with C++ already works). To support el6 we avoid use of 
c++11 language constructs.

K.O.

the midas release 2019-03 is ready for general use.

main changes from previous releases (midas-2017-10, midas-2018-12 and midas-2019-02):

- change to the midas URL scheme
- removal of cm_watchdog()
- rewrite of event buffer code (and fix of hard to trigger event buffer corruption bug)
- fully thread safe odb and event buffer code (except for rpc_send_event())
- corrected compatibility problems wrt older versions of midas when serving custom web pages via odb /custom/path

To obtain this release, either checkout the top of branch feature/midas-2019-03 (recommended)
or checkout the tag midas-2019-03-f.

K.O.

> the midas release 2019-03 is ready for general use.

first ever bug fix release on a git release branch.

fixed a crash if frontend built against this midas is connected to mserver from old (pre-db_watch) midas (size mismatch of MSG_ODB 
message).

to use this update:

# recommended:
git pull
git checkout feature/midas-2019-03
git pull
make ...

# or checkout "detached HEAD"
git pull
git checkout midas-2019-03-g
make ...

odbedit "ver" should report:

GIT revision:       Wed May 22 07:35:11 2019 -0700 - midas-2019-03-g on branch feature/midas-2019-03

K.O.

P.S. Thanks for finding this bug go to Greg Hackman on TIGRESS and EMMA experiments at TRIUMF.

K.O.

> > the midas release 2019-03 is ready for general use.

A bug fix update for midas-2019-03:

- fix broken expand_env() in mhttpd
- fix "Invalid name passed to db_create_key: should not be an empty string" in midas.log when loading the MIDAS status page if one of the alarms has empty 
class name.

odbedit "ver" should report: Thu Jun 6 18:02:14 2019 -0700 - midas-2019-03-h on branch feature/midas-2019-03

K.O.

> > > the midas release 2019-03 is ready for general use.
> A bug fix update for midas-2019-03:
> odbedit "ver" should report: Thu Jun 6 18:02:14 2019 -0700 - midas-2019-03-h on branch feature/midas-2019-03

For building this release of MIDAS, please use mxml branch feature/midas-2019-03, tag mxml-2019-03-a:

cd .../mxml
git fetch
git checkout feature/midas-2019-03

Going forward, I will try to remember to tag the mxml version that corresponds to specific midas versions.

K.O.