Back Midas Rome Roody Rootana
  Midas DAQ System, Page 1 of 122  Not logged in ELOG logo
ID Date Author Topic Subject
  2450   17 Nov 2022 Konstantin OlchanskiBug Fix"Detected duplicate or non-monotonous data" in history files
> > serious (but rare) bug was fixed in the history reader.
> previous fix was incomplete. please update to git commit
> https://bitbucket.org/tmidas/midas/commits/b343c3c98e4e6fd00a00cf686c74c7ccc6da0c63

a race condition between reading history file in mhttpd and writing history file in 
mlogger was accidentally introduced. mhttpd would file spurious errors about "timestamp 
is after last timestamp".

fixed, please update to git commit
https://bitbucket.org/tmidas/midas/commits/7a9f6e0c58ffddcacb9ee19934ce3e2033a805ef

fix race condition in history file reader - a race condition was added accidentally - 
first the reader remembers the history file size and the time of the last entry, then it 
goes to read the file and bombs if at the same time mlogger added more entries - their 
time is after the remembered time of last entry and error "timestamp is after last 
timestamp" is triggered.

K.O.
  2449   17 Nov 2022 Konstantin OlchanskiBug FixO_CREAT in open in split.cxx
> > midas currently does not compile on linux
> > fix is to give open in midas/examples/lowlevel/split.cxx a default mode, e.g. 006600

I got more warnings from split.cxx, looked at the code and see so many problems that it is easier
to delete it than it is to fix it.

Check for end of file is done incorrectly (check for read() return 0, -1 or short read),
memory overrun if given file name is longer than 80 bytes, no check for valid event length
read from the file, and so on and so on.

A better example for reading and writing midas files is in midasio/test_midasio.cxx. Proper c++ coding, and can read compressed files.

K.O.
  2448   12 Nov 2022 Stefan RittBug FixO_CREAT in open in split.cxx
> midas currently does not compile on linux
> 
> /usr/include/x86_64-linux-gnu/bits/fcntl2.h:50:24: error: call to ‘__open_missing_mode’ declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments
>    50 |    __open_missing_mode ();
> 
> giving the mode is mandatory: https://man7.org/linux/man-pages/man2/open.2.html
> 
> fix is to give open in midas/examples/lowlevel/split.cxx a default mode, e.g. 006600

Thanks. Fixed.

Stefan
  2447   11 Nov 2022 Frederik WautersBug FixO_CREAT in open in split.cxx
midas currently does not compile on linux

/usr/include/x86_64-linux-gnu/bits/fcntl2.h:50:24: error: call to ‘__open_missing_mode’ declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments
   50 |    __open_missing_mode ();

giving the mode is mandatory: https://man7.org/linux/man-pages/man2/open.2.html

fix is to give open in midas/examples/lowlevel/split.cxx a default mode, e.g. 006600
  2446   05 Nov 2022 Zaher SalmanSuggestionhistories capture 'ruy'
The histories capture key events from 'r' 'u' 'y' and 'Escape' for various functions like rescaling etc. However, this also means that if we include an editable modbvalue and a history in the same custom page then changing the modbvalue to something that includes 'ruy' is not possible.

In mhistory.js we have

// Keyboard event handler (has to be on the window!)
window.addEventListener("keydown", this.keyDown.bind(this));

I am not sure why it "has to be on the window". For now, I am bypassing this issue by changing the event listener to "keyup" but maybe there is a more elegant solution for this. Adding the event listener to the div element that includes the history does not seem to work.
  2445   29 Oct 2022 Stefan RittSuggestionread_only odbxx?
Ok, I implemented and committed that. Just call

o.set_write_protect(true)

on a key you don't want to modify. If you do so, an exception gets thrown.

Best,
Stefan
  2444   26 Oct 2022 Lars MartinSuggestionread_only odbxx?
> Having a "const midas::odb" probably does not work (at least I would not know how to implement that).
> 
> But I could make an internal flag analog to the auto refresh flags. So you would have
> 
>     o.set_write_protect(true);
> 
> to turn on write protection. Would that work for you?

Absolutely. Looking at the underlying code I was also at a loss how const would work.
I'm mostly just interested in having small clients that only read from the odb (for whatever reason) without risking corrupting it by messing something 
up in the code, especially since such small clients are almost by definition hacked together quickly on the fly.
  2443   24 Oct 2022 Stefan RittSuggestionread_only odbxx?
> I really like the concept of the odbxx interface.
> I think it would be a nice feature if one could have a read_only connection, e.g. by declaring a "const midas::odb".
> Just for fun I tried if this already works, but the compiler doesn't allow const midas::odb for e.g. the [] operator. I'm guessing this would be non-trivial to implement, but I like the idea of certain Midas clients being able to read the odb without risking corruption.

Having a "const midas::odb" probably does not work (at least I would not know how to implement that).

But I could make an internal flag analog to the auto refresh flags. So you would have

    o.set_write_protect(true);

to turn on write protection. Would that work for you?

Best,
Stefan
  2442   22 Oct 2022 Lars MartinInfoAllow onchange to refer to arbitrary js function
Actually, now that I look again, there is a mistake in the instructions:
you say

onchange="this.dataset.color=check_therm(this)"

but check_therm doesn't return anything and instead changes the color itself. So you either want the function to return the string and use the above assignment, or use the function you provide and use

onchange="check_therm(this)"
  2441   22 Oct 2022 Lars MartinSuggestionread_only odbxx?
I really like the concept of the odbxx interface.
I think it would be a nice feature if one could have a read_only connection, e.g. by declaring a "const midas::odb".
Just for fun I tried if this already works, but the compiler doesn't allow const midas::odb for e.g. the [] operator. I'm guessing this would be non-trivial to implement, but I like the idea of certain Midas clients being able to read the odb without risking corruption.
  2440   22 Oct 2022 Lars MartinInfoAllow onchange to refer to arbitrary js function
I figured I wasn't the first to have this idea.
Works great, thanks!
  2439   14 Oct 2022 Ben SmithInfoAllow onchange to refer to arbitrary js function
> I would like to use a function defined in the <script> block of my custom page as an onchange callback.
>
> Is this a possibility?

Yes, this is already possible. An example was shown in the "modb" section of the custom page documentation, but not in the "Changing properties of controls dynamically" section. I've updated the wiki with an example.

https://daq00.triumf.ca/MidasWiki/index.php/Custom_Page#Changing_properties_of_controls_dynamically
  2438   14 Oct 2022 Lars MartinSuggestionAllow onchange to refer to arbitrary js function
Maybe this is already possible, I have a hard time understanding the mhttpd source code.

I would like to use a function defined in the <script> block of my custom page as an onchange callback.

Specific example:
I have an modbthermo that I would like to change to three different colours for "too cold", "just right", and "too hot" (measuring porridge, presumably). The examples only show the explicit (condition)?(val1):(val2) syntax, which doesn't allow more than two values, so I had hoped to replace

onchange="this.dataset.color=this.value > 40?'red':'blue';"

with something like

onchange="this.dataset.color=check_Temp(this.value);"

or

onchange="check_Temp(this.value, this.dataset.color);"

if that's easier somehow. The function itself would then return the colour string, or set the color argument to that string (I'm not sure if JS passes references or just values.)

Is this a possibility?
  2437   10 Oct 2022 Zaher SalmanSuggestionJSON-RPC function to read files
Hello ,

The midas sequencer uses the function js_seq_list_files to get a list of files in the /Sequencer/State/Path with extension *.msl. It would be nice to generalize this function to be able to read files with other (or any) extension.

Based on the js_seq_list_files I added a function in js_any_list_files mjsonrpc_user.cxx (attached) which does the job. Maybe a better/safer implementation can be made in midas. Are there any plans to do this?

thanks.
Attachment 1: mjsonrpc_user.cxx
/********************************************************************\

  Name:         mjsonrpc_user.cxx
  Created by:   Konstantin Olchanski

  Contents:     handler of user-provided and experimental JSON-RPC requests

\********************************************************************/

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <map>

#include "mjson.h"
#include "midas.h"
#include "msystem.h"

#include "mjsonrpc.h"

#include <mutex> // std::mutex

//
// example 1: extract request parameters, return up to 3 results
//

static MJsonNode* user_example1(const MJsonNode* params)
{
   if (!params) {
      MJSO* doc = MJSO::I();
      doc->D("example of user defined RPC method that returns up to 3 results");
      doc->P("arg", MJSON_STRING, "example string argment");
      doc->P("optional_arg?", MJSON_INT, "optional example integer argument");
      doc->R("string", MJSON_STRING, "returns the value of \"arg\" parameter");
      doc->R("integer", MJSON_INT, "returns the value of \"optional_arg\" parameter");
      return doc;
   }

   MJsonNode* error = NULL;

   std::string arg  = mjsonrpc_get_param(params, "arg", &error)->GetString(); if (error) return error;
   int optional_arg = mjsonrpc_get_param(params, "optional_arg", NULL)->GetInt();

   if (mjsonrpc_debug)
      printf("user_example1(%s,%d)\n", arg.c_str(), optional_arg);

   return mjsonrpc_make_result("string", MJsonNode::MakeString(arg.c_str()), "integer", MJsonNode::MakeInt(optional_arg));
}

//
// example 2: extract request parameters, return more than 3 results
//

static MJsonNode* user_example2(const MJsonNode* params)
{
   if (!params) {
      MJSO* doc = MJSO::I();
      doc->D("example of user defined RPC method that returns more than 3 results");
      doc->P("arg", MJSON_STRING, "example string argment");
      doc->P("optional_arg?", MJSON_INT, "optional example integer argument");
      doc->R("string1", MJSON_STRING, "returns the value of \"arg\" parameter");
      doc->R("string2", MJSON_STRING, "returns \"hello\"");
      doc->R("string3", MJSON_STRING, "returns \"world!\"");
      doc->R("value1", MJSON_INT, "returns the value of \"optional_arg\" parameter");
      doc->R("value2", MJSON_NUMBER, "returns 3.14");
      return doc;
   }

   MJsonNode* error = NULL;

   std::string arg  = mjsonrpc_get_param(params, "arg", &error)->GetString(); if (error) return error;
   int optional_arg = mjsonrpc_get_param(params, "optional_arg", NULL)->GetInt();

   if (mjsonrpc_debug)
      printf("user_example2(%s,%d)\n", arg.c_str(), optional_arg);

   MJsonNode* result = MJsonNode::MakeObject();

   result->AddToObject("string1", MJsonNode::MakeString(arg.c_str()));
   result->AddToObject("string2", MJsonNode::MakeString("hello"));
   result->AddToObject("string3", MJsonNode::MakeString("world!"));
   result->AddToObject("value1", MJsonNode::MakeInt(optional_arg));
   result->AddToObject("value2", MJsonNode::MakeNumber(3.14));

   return mjsonrpc_make_result(result);
}

//
// example 3: return an error
//

static MJsonNode* user_example3(const MJsonNode* params)
{
   if (!params) {
      MJSO* doc = MJSO::I();
      doc->D("example of user defined RPC method that returns an error");
      doc->P("arg", MJSON_INT, "integer value, if zero, throws a JSON-RPC error");
      doc->R("status", MJSON_INT, "returns the value of \"arg\" parameter");
      return doc;
   }

   MJsonNode* error = NULL;

   int arg  = mjsonrpc_get_param(params, "arg", &error)->GetInt(); if (error) return error;

   if (mjsonrpc_debug)
      printf("user_example3(%d)\n", arg);

   if (arg)
      return mjsonrpc_make_result("status", MJsonNode::MakeInt(arg));
   else
      return mjsonrpc_make_error(15, "example error message", "example error data");
}

static MJsonNode* js_any_list_files(const MJsonNode* params)
{
   if (!params) {
      MJSO* doc = MJSO::I();
      doc->D("js_any_list_files");
      doc->P("subdir", MJSON_STRING, "List files in /Seq/State/Path/subdir");
      doc->R("status", MJSON_INT, "return status of midas library calls");
      doc->R("path", MJSON_STRING, "Search path");
      doc->R("fileext", MJSON_STRING, "Filename extension");
      doc->R("subdirs[]", MJSON_STRING, "list of subdirectories");
      doc->R("files[].filename", MJSON_STRING, "script filename");
      doc->R("files[].description", MJSON_STRING, "script description");
      return doc;
   }

   MJsonNode* error = NULL;

   std::string subdir = mjsonrpc_get_param(params, "subdir", &error)->GetString(); if (error) return error;
   std::string fileext  = mjsonrpc_get_param(params, "fileext", &error)->GetString(); if (error) return error;
   std::string path  = mjsonrpc_get_param(params, "path", NULL)->GetString(); 

   if (subdir.find("..") != std::string::npos) {
      return mjsonrpc_make_result("status", MJsonNode::MakeInt(DB_INVALID_PARAM));
   }

   int status;
   HNDLE hDB;

   status = cm_get_experiment_database(&hDB, NULL);

   if (status != DB_SUCCESS) {
      return mjsonrpc_make_result("status", MJsonNode::MakeInt(status));
   }

   //   std::string path;
   // If path is not provided get from ODB
   if (path == "") {
      status = db_get_value_string(hDB, 0, "/Sequencer/State/Path", 0, &path, FALSE);
   }
   
   if (status != DB_SUCCESS) {
      return mjsonrpc_make_result("status", MJsonNode::MakeInt(status));
   }

   path = cm_expand_env(path.c_str());

   if (subdir.length() > 0) {
      if (path[path.length()-1] != DIR_SEPARATOR) {
         path += DIR_SEPARATOR_STR;
      }
      path += subdir;
   }

   char* flist = NULL;

   //printf("path: [%s]\n", path.c_str());

   MJsonNode* s = MJsonNode::MakeArray();
   
   /*---- go over subdirectories ----*/
   int n = ss_dir_find(path.c_str(), "*", &flist);

   for (int i=0 ; i<n ; i++) {
      if (flist[i*MAX_STRING_LENGTH] != '.') {
         //printf("subdir %d: [%s]\n", i, flist+i*MAX_STRING_LENGTH);
         ss_repair_utf8(flist+i*MAX_STRING_LENGTH);
         s->AddToArray(MJsonNode::MakeString(flist+i*MAX_STRING_LENGTH));
      }
   }
   
   MJsonNode* f = MJsonNode::MakeArray();

   /*---- go over files in path ----*/
   n = ss_file_find(path.c_str(), fileext.c_str(), &flist);
   for (int i=0 ; i<n ; i++) {
      //printf("file %d: [%s]\n", i, flist+i*MAX_STRING_LENGTH);
      MJsonNode* o = MJsonNode::MakeObject();
      ss_repair_utf8(flist+i*MAX_STRING_LENGTH);
      o->AddToObject("filename", MJsonNode::MakeString(flist+i*MAX_STRING_LENGTH));
      o->AddToObject("description", MJsonNode::MakeString("description"));
      f->AddToArray(o);
#if 0
      char comment[512];
      comment[0] = 0;
      strlcpy(str, path, sizeof(str));
      if (strlen(str)>1 && str[strlen(str)-1] != DIR_SEPARATOR)
         strlcat(str, DIR_SEPARATOR_STR, sizeof(str));
      strlcat(str, flist+i*MAX_STRING_LENGTH, sizeof(str));
      
      if (msl_parse(str, error, sizeof(error), &error_line)) {
         if (strchr(str, '.')) {
            *strchr(str, '.') = 0;
            strlcat(str, ".xml", sizeof(str));
         }
         comment[0] = 0;
         if (pnseq) {
            mxml_free_tree(pnseq);
            pnseq = NULL;
         }
         pnseq = mxml_parse_file(str, error, sizeof(error), &error_line);
         if (error[0]) {
            strlcpy(comment, error, sizeof(comment));
         } else {
            if (pnseq) {
               pn = mxml_find_node(pnseq, "RunSequence/Comment");
               if (pn)
                  strlcpy(comment, mxml_get_value(pn), sizeof(comment));
               else
                  strcpy(comment, "<No description in XML file>");
            }
         }
         if (pnseq) {
            mxml_free_tree(pnseq);
            pnseq = NULL;
         }
      } else {
         sprintf(comment, "Error in MSL: %s", error);
      }
      
      strsubst(comment, sizeof(comment), "\"", "\\\'");
      r->rsprintf("<option onClick=\"document.getElementById('cmnt').innerHTML='%s'\"", comment);
      r->rsprintf(" onDblClick=\"load();\">%s</option>\n", flist+i*MAX_STRING_LENGTH);
#endif
   }

   free(flist);
   flist = NULL;

   MJsonNode* r = MJsonNode::MakeObject();
   r->AddToObject("status", MJsonNode::MakeInt(SUCCESS));
   ss_repair_utf8(path);
   r->AddToObject("path", MJsonNode::MakeString(path.c_str()));
   r->AddToObject("subdirs", s);
   r->AddToObject("files", f);
   
   return mjsonrpc_make_result(r);

}

//
// to create your own rpc method handler, copy one of the examples here, register it in user_init below
//


//
// user_init function is called at startup time to register user rpc method handlers
//

void mjsonrpc_user_init()
{
   if (mjsonrpc_debug) {
      printf("mjsonrpc_user_init!\n");
   }

   // add user functions to the rpc list

   mjsonrpc_add_handler("user_example1", user_example1);
   mjsonrpc_add_handler("user_example2", user_example2);
   mjsonrpc_add_handler("user_example3", user_example3);
   mjsonrpc_add_handler("any_list_files", js_any_list_files, true);
}



/* emacs
 * Local Variables:
 * tab-width: 8
 * c-basic-offset: 3
 * indent-tabs-mode: nil
 * End:
 */

  2436   23 Aug 2022 Konstantin OlchanskiBug Fix"Detected duplicate or non-monotonous data" in history files
> serious (but rare) bug was fixed in the history reader.

previous fix was incomplete. please update to git commit
https://bitbucket.org/tmidas/midas/commits/b343c3c98e4e6fd00a00cf686c74c7ccc6da0c63

K.O.
  2435   22 Aug 2022 Stefan RittSuggestionmvodb functionality to get the 'LastWritten' property of a key
> I want to read data from the ODB with the mvodb interface in one of my frontends, it's useful to know how old that data is, so I prototyped functionality in a pull request to mvodb:
> 
> https://bitbucket.org/tmidas/mvodb/pull-requests/2/add-readkeylastwritten-function-to-extract

Thanks for raising that point. I realized that the odbxx API was also missing that functionality, so I added it: 
https://bitbucket.org/tmidas/midas/commits/6991a92c19292eaf67721cb80f182c61db077f45

Best,
Stefan
  2434   21 Aug 2022 Joseph McKennaSuggestionmvodb functionality to get the 'LastWritten' property of a key


I want to read data from the ODB with the mvodb interface in one of my frontends, it's useful to know how old that data is, so I prototyped functionality in a pull request to mvodb:

https://bitbucket.org/tmidas/mvodb/pull-requests/2/add-readkeylastwritten-function-to-extract
  2433   19 Aug 2022 Konstantin OlchanskiBug Fix"Detected duplicate or non-monotonous data" in history files
serious (but rare) bug was fixed in the history reader. unlucky experiment would see 
errors about "Detected duplicate or non-monotonous data" in some history file, fixed by 
removing/renaming the offending file. (reported by MEG experiment)

it turns out there was nothing wrong with the data files (good), but there
was a nasty bug in the history reader. it did not ensure that we read history
files in chronological order. under some conditions order of files could be
reversed, older files would be read after newer files and trip the built-in
protection against returning non-monotonically increasing history data to the user.

fixed commit 
https://bitbucket.org/tmidas/midas/commits/9893f85ebe33e96cc63f501a0f89e1f8932c894d

for more details, see https://bitbucket.org/tmidas/midas/issues/350/file-history-non-
monotonic-time

K.O.
  2432   17 Aug 2022 Stefan RittBug Reportfirefox hangs due to mhistory
> Some of my histories are placed in an IFrame object. I eventually realized that my code fails 
> when it tries to resize a history which is placed in an invisible IFrame. I resolved the issue 
> by making sure that I am resizing plots only if they are in a visible IFrame.

Just to be clear: You could resolve everything on your side, or do you need to change anything in mhistory.js?

Just a tip: IFrames are not good to put anything in. I recommend just to dynamically crate a <div> element, 
append it to the document body, make it floating and initially invisible. Then put all inside that div. Have
a look how control.js do it. This takes less resources than a complete IFrame and is much easier to handle.

Stefan
  Draft   17 Aug 2022 Stefan RittBug Reportfirefox hangs due to mhistory
> Some of my histories are placed in an IFrame object. I eventually realized that my code fails 
> when it tries to resize a history which is placed in an invisible IFrame. I resolved the issue 
> by making sure that I am resizing plots only if they are in a visible IFrame.

Just to be clear: You could resolve everything on your side, or do you need to change anything in mhistory.js?

Just a tip: IFrames are not good to put anything in. I recommend just to dynamically crate a <div> element, 
append it to the document body, make it floating and initially invisible. Then put all inside that div. Have
a look how control.js do it. T
ELOG V3.1.4-2e1708b5