Back Midas Rome Roody Rootana
  Midas DAQ System, Page 121 of 122  Not logged in ELOG logo
IDup Date Author Topic Subject
  2423   08 Aug 2022 Stefan RittInfoInformation for midas updates though git
> after I set "submodule.recurse true", I still have to type "git submodule update --
> init --recursive", without --recursive, mscb/mxml is empty and the build bombs.

Indeed, doesn't work for me either. If some git guru has some more insight, please post 
here!

 
> P.S. the underlying issue is that the mxml submodule is now included twice 
> (midas/mxml and midas/mscb/mxml) and there is nothing to enforce that both copies are 
> the same. (No idea what happens if the two mxml's are different).

The version of each mxml is defined by last commit of the parent repository, which contains 
the hash of the submodule version. If we have to update mxml for some reason, we have to 
commit also mscb with the new version, and then midas with the same version of mxml. If one 
checks out midas then with 

git clone https://bitbucket.org/tmidas/midas --recursive

one gets the same versions for mxml.
  2424   15 Aug 2022 Zaher SalmanBug Reportfirefox hangs due to mhistory
Firefox is hanging/becoming unresponsive due to javascript code. After stopping the script manually to get firefox back in control I have the following message in the console

17:21:28.821 Script terminated by timeout at:
MhistoryGraph.prototype.drawTAxis@http://lem03.psi.ch:8081/mhistory.js:2828:7
MhistoryGraph.prototype.draw@http://lem03.psi.ch:8081/mhistory.js:1792:9
mhistory.js:2828:7

Any ideas how to resolve this??
  2425   15 Aug 2022 Stefan RittBug Reportfirefox hangs due to mhistory
> Firefox is hanging/becoming unresponsive due to javascript code. After stopping the script manually to get firefox back in control I have the following message in the console
> 
> 17:21:28.821 Script terminated by timeout at:
> MhistoryGraph.prototype.drawTAxis@http://lem03.psi.ch:8081/mhistory.js:2828:7
> MhistoryGraph.prototype.draw@http://lem03.psi.ch:8081/mhistory.js:1792:9
> mhistory.js:2828:7
> 
> Any ideas how to resolve this??

I have to reproduce the problem. Can you send me the full URL from your browser when you see that problem? Probably you have some "special" axis limits, so we don't see that 
problem anywhere else.

Stefan
  2426   16 Aug 2022 Zaher SalmanBug Reportfirefox hangs due to mhistory
> > Firefox is hanging/becoming unresponsive due to javascript code. After stopping the script manually to get firefox back in control I have the following message in the console
> > 
> > 17:21:28.821 Script terminated by timeout at:
> > MhistoryGraph.prototype.drawTAxis@http://lem03.psi.ch:8081/mhistory.js:2828:7
> > MhistoryGraph.prototype.draw@http://lem03.psi.ch:8081/mhistory.js:1792:9
> > mhistory.js:2828:7
> > 
> > Any ideas how to resolve this??
> 
> I have to reproduce the problem. Can you send me the full URL from your browser when you see that problem? Probably you have some "special" axis limits, so we don't see that 
> problem anywhere else.
> 
> Stefan

Hi Stefan and Konstantin,

The URL (reachable only within PSI) is http://lem03.psi.ch:8081/?cmd=custom&page=Mudas
Firefox is version 91.12.0esr (64-bit), but I had similar issues with chrome/chromium too.
The hangs seem to happen randomly so I have not been able to reproduce it yet. 
I have histories here  http://lem03.psi.ch:8081/?cmd=custom&page=Mudas&tab=3 (30 minutes each), but I have also histories popping up in modals though they do not cause any issues. 
I'll try to reproduce it in the coming few days and report again.

thanks,
Zaher
  2427   16 Aug 2022 Zaher SalmanBug Reportfirefox hangs due to mhistory
I found the bug. The problem is triggered by changing the firefox window. This calls a function that is supposed to change the size of the history plot and it works well when the history plots are visible but not if the history plots are hidden in a javascript tab (not another firefox tab).

Is there a clean way to resize the history plot if the parent div changes size?? The offending code is
mhist[i].mhg = new MhistoryGraph(mhist[i]);
mhist[i].mhg.initializePanel(i);
mhist[i].mhg.resize();
mhist[i].resize = function () {
   mhis.mhg.resize();
};
  2428   16 Aug 2022 Konstantin OlchanskiBug Reportfirefox hangs due to mhistory
> > > Firefox is hanging/becoming unresponsive due to javascript code.
> 
> The URL (reachable only within PSI) is http://lem03.psi.ch:8081/?cmd=custom&page=Mudas

so malfunction is not in the midas history page, but in a custom page. I could help you debug it,
but you would have to provide the complete source code (javascript and html).

> Firefox is version 91.12.0esr (64-bit), but I had similar issues with chrome/chromium too.

my firefox is 103.something. when you say google-chrome has "similar issues",
I read it as "google-chrome does not show this same bug, but shows some other
bug somewhere else". (if I misread you, you have to write better).

but this gives you a front to attack your bugs. basically all browsers should render your
custom page exactly the same (unless you use some obscure or experimental feature, which I
recommend against).

so you tweak your page to identify the source of different rendering results, and try to eliminate it,
hopefully by the time you get your page render exactly the same everywhere, all the real bugs
have gotten shaken out, too. (this is similar to debugging a c++ program by compiling
it on linux, mac, windows, vax, raspbery pi, etc and checking that you get the same result everywhere).

> The hangs seem to happen randomly so I have not been able to reproduce it yet.

I find that javascript debuggers are not setup to debug hangs. I think debugger runs partially
inside the same javascript engine you are debugging, so both hang and debugging is impossible.

(latest google-chrome has another improvement, all pages from the same computer run in the same
javascript engine, so if one midas page stops (on exception or because I debug it), all midas pages
stop and I have to run two different browsers if I want to debug (i.e.) a history page crash
and look at odb at the same time. fun).

K.O.
  2429   17 Aug 2022 Stefan RittBug Reportfirefox hangs due to mhistory
The problem lies in your function mhistory_init_one() in Mudas.js:1965. You can only call "new MhistoryGraph(e)" with an element "e" which is something like

<div class="mjshistory" data-group="..." data-panel="..." data-base-u-r-l="https://host.psi.ch/?cmd=history" title="">

Please note the "data-base-u-r-l". This gets automatically added by the function mhistory_init() in mhistory.js:48. The URL is necessary sot that the upper right button in a history graph works which goes to a history page only showing the current graph.

In you function mhistory_init_one() you forgot the call

   mhist.dataset.baseURL = baseURL; 

where baseURL has to come from the current address bar like

   let baseURL = window.location.href;
   if (baseURL.indexOf("?cmd") > 0)
      baseURL = baseURL.substr(0, baseURL.indexOf("?cmd"));
   baseURL += "?cmd=history";

If you duplicate some functionality from mhistory.js, please make sure to duplicate it completely.

Best,
Stefan
  2430   17 Aug 2022 Zaher SalmanBug Reportfirefox hangs due to mhistory
> The problem lies in your function mhistory_init_one() in Mudas.js:1965. You can only call "new MhistoryGraph(e)" with an element "e" which is something like
> 
> <div class="mjshistory" data-group="..." data-panel="..." data-base-u-r-l="https://host.psi.ch/?cmd=history" title="">
> 
> Please note the "data-base-u-r-l". This gets automatically added by the function mhistory_init() in mhistory.js:48. The URL is necessary sot that the upper right button in a history graph works which goes to a history page only showing the current graph.
> 
> In you function mhistory_init_one() you forgot the call
> 
>    mhist.dataset.baseURL = baseURL; 
> 
> where baseURL has to come from the current address bar like
> 
>    let baseURL = window.location.href;
>    if (baseURL.indexOf("?cmd") > 0)
>       baseURL = baseURL.substr(0, baseURL.indexOf("?cmd"));
>    baseURL += "?cmd=history";
> 
> If you duplicate some functionality from mhistory.js, please make sure to duplicate it completely.
> 

Thanks Stefan, but this was not the problem since I am setting the baseURL. You may have looked at the code during my debugging.

Some of my histories are placed in an IFrame object. I eventually realized that my code fails when it tries to resize a history which is placed in an invisible IFrame. I resolved the issue by making sure that I am resizing plots only if they are in a visible IFrame.
  
  Draft   17 Aug 2022 Stefan RittBug Reportfirefox hangs due to mhistory
> Some of my histories are placed in an IFrame object. I eventually realized that my code fails 
> when it tries to resize a history which is placed in an invisible IFrame. I resolved the issue 
> by making sure that I am resizing plots only if they are in a visible IFrame.

Just to be clear: You could resolve everything on your side, or do you need to change anything in mhistory.js?

Just a tip: IFrames are not good to put anything in. I recommend just to dynamically crate a <div> element, 
append it to the document body, make it floating and initially invisible. Then put all inside that div. Have
a look how control.js do it. T
  2432   17 Aug 2022 Stefan RittBug Reportfirefox hangs due to mhistory
> Some of my histories are placed in an IFrame object. I eventually realized that my code fails 
> when it tries to resize a history which is placed in an invisible IFrame. I resolved the issue 
> by making sure that I am resizing plots only if they are in a visible IFrame.

Just to be clear: You could resolve everything on your side, or do you need to change anything in mhistory.js?

Just a tip: IFrames are not good to put anything in. I recommend just to dynamically crate a <div> element, 
append it to the document body, make it floating and initially invisible. Then put all inside that div. Have
a look how control.js do it. This takes less resources than a complete IFrame and is much easier to handle.

Stefan
  2433   19 Aug 2022 Konstantin OlchanskiBug Fix"Detected duplicate or non-monotonous data" in history files
serious (but rare) bug was fixed in the history reader. unlucky experiment would see 
errors about "Detected duplicate or non-monotonous data" in some history file, fixed by 
removing/renaming the offending file. (reported by MEG experiment)

it turns out there was nothing wrong with the data files (good), but there
was a nasty bug in the history reader. it did not ensure that we read history
files in chronological order. under some conditions order of files could be
reversed, older files would be read after newer files and trip the built-in
protection against returning non-monotonically increasing history data to the user.

fixed commit 
https://bitbucket.org/tmidas/midas/commits/9893f85ebe33e96cc63f501a0f89e1f8932c894d

for more details, see https://bitbucket.org/tmidas/midas/issues/350/file-history-non-
monotonic-time

K.O.
  2434   21 Aug 2022 Joseph McKennaSuggestionmvodb functionality to get the 'LastWritten' property of a key


I want to read data from the ODB with the mvodb interface in one of my frontends, it's useful to know how old that data is, so I prototyped functionality in a pull request to mvodb:

https://bitbucket.org/tmidas/mvodb/pull-requests/2/add-readkeylastwritten-function-to-extract
  2435   22 Aug 2022 Stefan RittSuggestionmvodb functionality to get the 'LastWritten' property of a key
> I want to read data from the ODB with the mvodb interface in one of my frontends, it's useful to know how old that data is, so I prototyped functionality in a pull request to mvodb:
> 
> https://bitbucket.org/tmidas/mvodb/pull-requests/2/add-readkeylastwritten-function-to-extract

Thanks for raising that point. I realized that the odbxx API was also missing that functionality, so I added it: 
https://bitbucket.org/tmidas/midas/commits/6991a92c19292eaf67721cb80f182c61db077f45

Best,
Stefan
  2436   23 Aug 2022 Konstantin OlchanskiBug Fix"Detected duplicate or non-monotonous data" in history files
> serious (but rare) bug was fixed in the history reader.

previous fix was incomplete. please update to git commit
https://bitbucket.org/tmidas/midas/commits/b343c3c98e4e6fd00a00cf686c74c7ccc6da0c63

K.O.
  2437   10 Oct 2022 Zaher SalmanSuggestionJSON-RPC function to read files
Hello ,

The midas sequencer uses the function js_seq_list_files to get a list of files in the /Sequencer/State/Path with extension *.msl. It would be nice to generalize this function to be able to read files with other (or any) extension.

Based on the js_seq_list_files I added a function in js_any_list_files mjsonrpc_user.cxx (attached) which does the job. Maybe a better/safer implementation can be made in midas. Are there any plans to do this?

thanks.
Attachment 1: mjsonrpc_user.cxx
/********************************************************************\

  Name:         mjsonrpc_user.cxx
  Created by:   Konstantin Olchanski

  Contents:     handler of user-provided and experimental JSON-RPC requests

\********************************************************************/

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <map>

#include "mjson.h"
#include "midas.h"
#include "msystem.h"

#include "mjsonrpc.h"

#include <mutex> // std::mutex

//
// example 1: extract request parameters, return up to 3 results
//

static MJsonNode* user_example1(const MJsonNode* params)
{
   if (!params) {
      MJSO* doc = MJSO::I();
      doc->D("example of user defined RPC method that returns up to 3 results");
      doc->P("arg", MJSON_STRING, "example string argment");
      doc->P("optional_arg?", MJSON_INT, "optional example integer argument");
      doc->R("string", MJSON_STRING, "returns the value of \"arg\" parameter");
      doc->R("integer", MJSON_INT, "returns the value of \"optional_arg\" parameter");
      return doc;
   }

   MJsonNode* error = NULL;

   std::string arg  = mjsonrpc_get_param(params, "arg", &error)->GetString(); if (error) return error;
   int optional_arg = mjsonrpc_get_param(params, "optional_arg", NULL)->GetInt();

   if (mjsonrpc_debug)
      printf("user_example1(%s,%d)\n", arg.c_str(), optional_arg);

   return mjsonrpc_make_result("string", MJsonNode::MakeString(arg.c_str()), "integer", MJsonNode::MakeInt(optional_arg));
}

//
// example 2: extract request parameters, return more than 3 results
//

static MJsonNode* user_example2(const MJsonNode* params)
{
   if (!params) {
      MJSO* doc = MJSO::I();
      doc->D("example of user defined RPC method that returns more than 3 results");
      doc->P("arg", MJSON_STRING, "example string argment");
      doc->P("optional_arg?", MJSON_INT, "optional example integer argument");
      doc->R("string1", MJSON_STRING, "returns the value of \"arg\" parameter");
      doc->R("string2", MJSON_STRING, "returns \"hello\"");
      doc->R("string3", MJSON_STRING, "returns \"world!\"");
      doc->R("value1", MJSON_INT, "returns the value of \"optional_arg\" parameter");
      doc->R("value2", MJSON_NUMBER, "returns 3.14");
      return doc;
   }

   MJsonNode* error = NULL;

   std::string arg  = mjsonrpc_get_param(params, "arg", &error)->GetString(); if (error) return error;
   int optional_arg = mjsonrpc_get_param(params, "optional_arg", NULL)->GetInt();

   if (mjsonrpc_debug)
      printf("user_example2(%s,%d)\n", arg.c_str(), optional_arg);

   MJsonNode* result = MJsonNode::MakeObject();

   result->AddToObject("string1", MJsonNode::MakeString(arg.c_str()));
   result->AddToObject("string2", MJsonNode::MakeString("hello"));
   result->AddToObject("string3", MJsonNode::MakeString("world!"));
   result->AddToObject("value1", MJsonNode::MakeInt(optional_arg));
   result->AddToObject("value2", MJsonNode::MakeNumber(3.14));

   return mjsonrpc_make_result(result);
}

//
// example 3: return an error
//

static MJsonNode* user_example3(const MJsonNode* params)
{
   if (!params) {
      MJSO* doc = MJSO::I();
      doc->D("example of user defined RPC method that returns an error");
      doc->P("arg", MJSON_INT, "integer value, if zero, throws a JSON-RPC error");
      doc->R("status", MJSON_INT, "returns the value of \"arg\" parameter");
      return doc;
   }

   MJsonNode* error = NULL;

   int arg  = mjsonrpc_get_param(params, "arg", &error)->GetInt(); if (error) return error;

   if (mjsonrpc_debug)
      printf("user_example3(%d)\n", arg);

   if (arg)
      return mjsonrpc_make_result("status", MJsonNode::MakeInt(arg));
   else
      return mjsonrpc_make_error(15, "example error message", "example error data");
}

static MJsonNode* js_any_list_files(const MJsonNode* params)
{
   if (!params) {
      MJSO* doc = MJSO::I();
      doc->D("js_any_list_files");
      doc->P("subdir", MJSON_STRING, "List files in /Seq/State/Path/subdir");
      doc->R("status", MJSON_INT, "return status of midas library calls");
      doc->R("path", MJSON_STRING, "Search path");
      doc->R("fileext", MJSON_STRING, "Filename extension");
      doc->R("subdirs[]", MJSON_STRING, "list of subdirectories");
      doc->R("files[].filename", MJSON_STRING, "script filename");
      doc->R("files[].description", MJSON_STRING, "script description");
      return doc;
   }

   MJsonNode* error = NULL;

   std::string subdir = mjsonrpc_get_param(params, "subdir", &error)->GetString(); if (error) return error;
   std::string fileext  = mjsonrpc_get_param(params, "fileext", &error)->GetString(); if (error) return error;
   std::string path  = mjsonrpc_get_param(params, "path", NULL)->GetString(); 

   if (subdir.find("..") != std::string::npos) {
      return mjsonrpc_make_result("status", MJsonNode::MakeInt(DB_INVALID_PARAM));
   }

   int status;
   HNDLE hDB;

   status = cm_get_experiment_database(&hDB, NULL);

   if (status != DB_SUCCESS) {
      return mjsonrpc_make_result("status", MJsonNode::MakeInt(status));
   }

   //   std::string path;
   // If path is not provided get from ODB
   if (path == "") {
      status = db_get_value_string(hDB, 0, "/Sequencer/State/Path", 0, &path, FALSE);
   }
   
   if (status != DB_SUCCESS) {
      return mjsonrpc_make_result("status", MJsonNode::MakeInt(status));
   }

   path = cm_expand_env(path.c_str());

   if (subdir.length() > 0) {
      if (path[path.length()-1] != DIR_SEPARATOR) {
         path += DIR_SEPARATOR_STR;
      }
      path += subdir;
   }

   char* flist = NULL;

   //printf("path: [%s]\n", path.c_str());

   MJsonNode* s = MJsonNode::MakeArray();
   
   /*---- go over subdirectories ----*/
   int n = ss_dir_find(path.c_str(), "*", &flist);

   for (int i=0 ; i<n ; i++) {
      if (flist[i*MAX_STRING_LENGTH] != '.') {
         //printf("subdir %d: [%s]\n", i, flist+i*MAX_STRING_LENGTH);
         ss_repair_utf8(flist+i*MAX_STRING_LENGTH);
         s->AddToArray(MJsonNode::MakeString(flist+i*MAX_STRING_LENGTH));
      }
   }
   
   MJsonNode* f = MJsonNode::MakeArray();

   /*---- go over files in path ----*/
   n = ss_file_find(path.c_str(), fileext.c_str(), &flist);
   for (int i=0 ; i<n ; i++) {
      //printf("file %d: [%s]\n", i, flist+i*MAX_STRING_LENGTH);
      MJsonNode* o = MJsonNode::MakeObject();
      ss_repair_utf8(flist+i*MAX_STRING_LENGTH);
      o->AddToObject("filename", MJsonNode::MakeString(flist+i*MAX_STRING_LENGTH));
      o->AddToObject("description", MJsonNode::MakeString("description"));
      f->AddToArray(o);
#if 0
      char comment[512];
      comment[0] = 0;
      strlcpy(str, path, sizeof(str));
      if (strlen(str)>1 && str[strlen(str)-1] != DIR_SEPARATOR)
         strlcat(str, DIR_SEPARATOR_STR, sizeof(str));
      strlcat(str, flist+i*MAX_STRING_LENGTH, sizeof(str));
      
      if (msl_parse(str, error, sizeof(error), &error_line)) {
         if (strchr(str, '.')) {
            *strchr(str, '.') = 0;
            strlcat(str, ".xml", sizeof(str));
         }
         comment[0] = 0;
         if (pnseq) {
            mxml_free_tree(pnseq);
            pnseq = NULL;
         }
         pnseq = mxml_parse_file(str, error, sizeof(error), &error_line);
         if (error[0]) {
            strlcpy(comment, error, sizeof(comment));
         } else {
            if (pnseq) {
               pn = mxml_find_node(pnseq, "RunSequence/Comment");
               if (pn)
                  strlcpy(comment, mxml_get_value(pn), sizeof(comment));
               else
                  strcpy(comment, "<No description in XML file>");
            }
         }
         if (pnseq) {
            mxml_free_tree(pnseq);
            pnseq = NULL;
         }
      } else {
         sprintf(comment, "Error in MSL: %s", error);
      }
      
      strsubst(comment, sizeof(comment), "\"", "\\\'");
      r->rsprintf("<option onClick=\"document.getElementById('cmnt').innerHTML='%s'\"", comment);
      r->rsprintf(" onDblClick=\"load();\">%s</option>\n", flist+i*MAX_STRING_LENGTH);
#endif
   }

   free(flist);
   flist = NULL;

   MJsonNode* r = MJsonNode::MakeObject();
   r->AddToObject("status", MJsonNode::MakeInt(SUCCESS));
   ss_repair_utf8(path);
   r->AddToObject("path", MJsonNode::MakeString(path.c_str()));
   r->AddToObject("subdirs", s);
   r->AddToObject("files", f);
   
   return mjsonrpc_make_result(r);

}

//
// to create your own rpc method handler, copy one of the examples here, register it in user_init below
//


//
// user_init function is called at startup time to register user rpc method handlers
//

void mjsonrpc_user_init()
{
   if (mjsonrpc_debug) {
      printf("mjsonrpc_user_init!\n");
   }

   // add user functions to the rpc list

   mjsonrpc_add_handler("user_example1", user_example1);
   mjsonrpc_add_handler("user_example2", user_example2);
   mjsonrpc_add_handler("user_example3", user_example3);
   mjsonrpc_add_handler("any_list_files", js_any_list_files, true);
}



/* emacs
 * Local Variables:
 * tab-width: 8
 * c-basic-offset: 3
 * indent-tabs-mode: nil
 * End:
 */

  2438   14 Oct 2022 Lars MartinSuggestionAllow onchange to refer to arbitrary js function
Maybe this is already possible, I have a hard time understanding the mhttpd source code.

I would like to use a function defined in the <script> block of my custom page as an onchange callback.

Specific example:
I have an modbthermo that I would like to change to three different colours for "too cold", "just right", and "too hot" (measuring porridge, presumably). The examples only show the explicit (condition)?(val1):(val2) syntax, which doesn't allow more than two values, so I had hoped to replace

onchange="this.dataset.color=this.value > 40?'red':'blue';"

with something like

onchange="this.dataset.color=check_Temp(this.value);"

or

onchange="check_Temp(this.value, this.dataset.color);"

if that's easier somehow. The function itself would then return the colour string, or set the color argument to that string (I'm not sure if JS passes references or just values.)

Is this a possibility?
  2439   14 Oct 2022 Ben SmithInfoAllow onchange to refer to arbitrary js function
> I would like to use a function defined in the <script> block of my custom page as an onchange callback.
>
> Is this a possibility?

Yes, this is already possible. An example was shown in the "modb" section of the custom page documentation, but not in the "Changing properties of controls dynamically" section. I've updated the wiki with an example.

https://daq00.triumf.ca/MidasWiki/index.php/Custom_Page#Changing_properties_of_controls_dynamically
  2440   22 Oct 2022 Lars MartinInfoAllow onchange to refer to arbitrary js function
I figured I wasn't the first to have this idea.
Works great, thanks!
  2441   22 Oct 2022 Lars MartinSuggestionread_only odbxx?
I really like the concept of the odbxx interface.
I think it would be a nice feature if one could have a read_only connection, e.g. by declaring a "const midas::odb".
Just for fun I tried if this already works, but the compiler doesn't allow const midas::odb for e.g. the [] operator. I'm guessing this would be non-trivial to implement, but I like the idea of certain Midas clients being able to read the odb without risking corruption.
  2442   22 Oct 2022 Lars MartinInfoAllow onchange to refer to arbitrary js function
Actually, now that I look again, there is a mistake in the instructions:
you say

onchange="this.dataset.color=check_therm(this)"

but check_therm doesn't return anything and instead changes the color itself. So you either want the function to return the string and use the above assignment, or use the function you provide and use

onchange="check_therm(this)"
ELOG V3.1.4-2e1708b5