Back Midas Rome Roody Rootana
  Midas DAQ System, Page 2 of 136  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
ID Date Authorup Topic Subject
  1140   19 Nov 2015 Amy RobertsInfosynchronous ajax deprecated
> Right now the JSON-RPC client library does not check the return status of MIDAS calls themselves, i.e. ODBGet("/nonexistant") will go to Promise.resolve() with
> the MIDAS db_find_key() status DB_NO_KEY instead of Promise.reject(). So some error handling in Promise.resolve() is still required.

> I am thinking how to make these calls go to the error handler automatically, but there is no obvious solution for ODBMGet(["/runinfo", "/nonexistant"]) - the first path
> is a success, the second is a failure, do I fail the entire transaction (i.e. with a JSON-RPC error)? Same for JSON-RPC batch transactions - if one transaction
> in the batch fails, do I Promise.reject() all of them?


Generally, I'd prefer a grouped-request like ODBMGet to return an array of
Promises.  This way, I get to decide how to handle the request responses.  While
I do have cases where I use Promise.all(...), most of my current code would want
to treat each response individually.

But as you point out, my approach differs from the Midas approach significantly.
While I've set up my queries to reject on responses like DB_NO_KEY, the function
mjsonrpc_send_request in mhttpd.js tests purely for a successful http
request.

Since ODBMGet makes a *single* http call, I'd naively lean toward having it
return a single Promise.  Presumably, one that resolves if the http request
"goes through" and rejects if the http request fails. 

My perspective may not be the useful one to consider, though.  If other users
expect an array of promises returned from ODBMGet, definitely feel free to
ignore my thoughts on the matter.

If people really want ODBMGet to return an array of promises, one way to do it
would be to have a 'get' function that only cares about the success of the http
request, and a separate 'response checking' function that validates the
response.  ODBMGet can use these two functions together to return an array of
Promises:

#########
mhttpd.js
#########
function get(url) { 
  // return a promise that resolves if the http request returns status=200
  // reject if the http request does anything else
}

function checkResponse(response) {
  // return a promise that resolves for "good" responses
  // and rejects for "bad response"
}

function ODBMGet(path_arr) {
  var url = // build url from path_arr
  
  // syntax get().then(A).catch(B) means
  // if the http request goes through, A is executed
  // if the http request fails, B is executed
  // ;
  // for get, failure means no successful reply at all
  // so ODBMGet should return an array of rejected Promises 
  get(url).then(function(response) {
    response_arr = // split the response
    return response_arr.map(checkResponse)
  }).catch(function(err) {
    return path_arr.forEach(function() {
      return Promise.reject(err)
    })
  })
}

#########
user code
#########
// here the Promises are treated individually
var response_arr = ODBMGet([path1, path2, path3])

response_arr.forEach(function(thisResponse) {
  thisResponse.then( /* do something */ )
              .catch( /* do something else */ )
})

// and here the failure of a single promise in the array
// determines the code that's executed next
var required_arr = ODBMGet([pathA, pathB, pathC])

Promise.all(required_arr).then( /* do something */ )
                         .catch( /* do something else */ )
  1150   10 Dec 2015 Amy RobertsSuggestionscript command limited to 256 characters; remove limit?
Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
script via Midas - which silently truncates command strings longer than 
256 characters.

I'd prefer that Midas place no limit on string length.  Failing that, it would be
helpful to have character limits called out in the documentation 
(https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).

As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
(Although key *names* are restricted to 256 characters.)  I've submitted one 
possible version of an arbitrary-length exec_script() as a pull request 
(https://bitbucket.org/tmidas/midas/pull-requests/).

Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
strings in the ODB as limited to 256 characters?
  1158   28 Jan 2016 Amy RobertsSuggestionscript command limited to 256 characters; remove limit?
Using low-level memory allocation routines in higher-level programs like mhttpd makes me nervous.

We could use vector arrays to allow variable-sized allocation, and use the data() member function to access the char* needed for functions like strlcat,
db_get_data, and db_sprintf.

This conforms to the c++ standard, but doesn't require explicit freeing by the user - at least, not when you're allocating std::vector<char>.

Amy

> Thank you for reporting this problem:
> 
> a) ODB key *names* are restricted to 31 characters (32 bytes, last byte is a NUL), not 256 characters.
> b) ODB string length is unlimited (32-bit length field)
> c) ODB C API "db_get_value" & co require fixed length buffer and most users of this API provide a 256-byte fixed buffer for strings, some of them also do not 
> check the status code, resulting in silent truncation. (I think the ODB functions themselves report truncation to midas.log, so not completely silent).
> 
> We try to fix this where we must - but it is cumbersome with the current ODB API - as in your fix on has to:
> - get the ODB key, extract size
> - allocate buffer
> - call db_get_value() & co
> - use the data
> - remember to free the buffer on each and every return path
> 
> The first three steps could become one if we had an ODB "get_data" function that automatically allocated the data buffer.
> 
> But the main source of bugs will be the last step - remember to free the buffer, always.
> 
> P.S.
> 
> We are not alone in pondering how to do this best. If you want to see it "done right",
> read the fresh-off-the-presses book "Go Programming Language" by Alan Donovan and Brian Kernighan,
> http://www.gopl.io/
> 
> Brian Kernighan is the "K" in K&R "C programming language", still around and kicking, now at Google.
> Sadly the "R" passed away in 2011 - http://www.nytimes.com/2011/10/14/technology/dennis-ritchie-programming-trailblazer-dies-at-70.html
> 
> K.O.
> 
> > Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
> > script via Midas - which silently truncates command strings longer than 
> > 256 characters.
> > 
> > I'd prefer that Midas place no limit on string length.  Failing that, it would be
> > helpful to have character limits called out in the documentation 
> > (https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
> > https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).
> > 
> > As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
> > (Although key *names* are restricted to 256 characters.)  I've submitted one 
> > possible version of an arbitrary-length exec_script() as a pull request 
> > (https://bitbucket.org/tmidas/midas/pull-requests/).
> > 
> > Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
> > strings in the ODB as limited to 256 characters?
  1196   08 Sep 2016 Amy RobertsBug Reportcontrol characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail
I've recently run into issues when using JSON.parse on ODB keys containing 
8-bit data.

For JSON.parse to successfully parse a string, (A) the string must be valid 
UTF-8, (B) several whitespace characters, control characters, and the 
characters " and \ must be escaped, and (C) you've got to follow the key-
value rules laid out in http://www.json.org/.

The web browser takes care of (A), and I verified that for this key Midas 
handled (C) correctly.  In principle, the function json_write in odb.c 
handles (B) - but json_write does not escape control characters.

To manage this problem, I modified json_write (in odb.c) to replace any 
control character with the more-inocuous character, 'C'.  My default case 
now looks like:

default:
         {
           // if a char is a control character,
           // print 'C' in its place
           // note that this loses data:
           // a more-correct method would be to print
           // \uXXXX, where XXXX is the character in hex
           if(iscntrl(*s)){
             (*buffer)[(*buffer_end)++] = 'C';
             s++;
           } else {
             (*buffer)[(*buffer_end)++] = *s++;
           }
         }
      
Where the call to iscntrl(*s) requires the addition of the ctype.h header 
file.

I'm guessing a blanket replacement of control characters with 'C' isn't 
something all Midas users would want to do.  Replacing the control character 
with its hex value seems like a good choice - but not without adding bounds 
checking!

An alternative to changing odb.c could be to add a regex to Midas response 
text which removes all control characters (U+0000 - U+001F): 

var resp_lint = req.response.replace(/[\u{0000}-\u{001F}]/gmu, '');
var json_obj = JSON.parse(resp_lint);

Unfortunately, the 'u' regex flax doesn't work on the Firefox version 
included in Scientific Linux 6.8.  
  1197   09 Sep 2016 Amy RobertsSuggestionAJAX jmsg "get messages since t" ability - add to docs?
I recently needed to watch the Midas messages for a particular error - and 
thus needed a command to "get all the messages since a time t".

The documentation (https://midas.triumf.ca/MidasWiki/index.php/AJAX#jmsg) 
documents a way to "get the most recent n messages" - but when I dug into the 
code, I was delighted to find that the existing Midas code also supports the 
"get all messages since t" query.

For the "get all messages since t" query, the parameter t should be the unix 
timestamp in seconds, and the parameter n should be zero: curl -X GET 
"http://localhost:8081/?cmd=jmsg&n=0&t=1473437918".

Pretty useful!  Perhaps this should be added to the AJAX documentation?
  1339   16 Feb 2018 Amy RobertsSuggestionrespect capitalization option in db_get_values mjsonrpc method?
I'd like to use the mjsonrpc db_get_values method, but (as indicated in the
documentation) it returns all ODB keys as lowercase.

This breaks quite a lot of my code - it was written with the old AJAX commands,
and these did respect the capitalization of the ODB keys.

Would it be possible to add a capitalization-preserve option to db_get_values? 
  1340   17 Feb 2018 Amy RobertsSuggestionrespect capitalization option in db_get_values mjsonrpc method?
It appears I needed to read the documentation more closely - the method db_save
does respect key-name capitalization and solves my problem.

Is db_save considered a deprecated method?  If so, I'd reiterate my suggestion for
a capitalize-preserve option for db_get_values.

Otherwise, I'll plan on using db_save.

> I'd like to use the mjsonrpc db_get_values method, but (as indicated in the
> documentation) it returns all ODB keys as lowercase.
> 
> This breaks quite a lot of my code - it was written with the old AJAX commands,
> and these did respect the capitalization of the ODB keys.
> 
> Would it be possible to add a capitalization-preserve option to db_get_values? 
  1796   28 Jan 2020 Amy RobertsSuggestionMIDAS tested with MariaDB?
We're using the History Logger MIDAS feature and writing to mySQL tables, but 
in some cases have run into issues installing mySQL on centos7 systems.

Has anyone ever tried running this MIDAS feature with MariaDB rather than 
mySQL?
  2000   29 Sep 2020 Amy RobertsForumusing python client to start and stop run
I'm using a python client to start and stop runs, and the following code *appears* 
to set the MIDAS state to "Run"

client.odb_set("/Runinfo/State", 3)

However, it doesn't seem to do other things associated with a run, like start 
accumulating events.

Is there a different way I should start the run from the python client?

Thanks!
  2021   24 Nov 2020 Amy RobertsSuggestionODBSET wildcards with array keys in Sequencer files
I'm interested in using the matching feature for ODBSET explained on 
https://midas.triumf.ca/MidasWiki/index.php/Sequencer for settings that are in an 
array, like:

COMMENT "Ground the detectors"
ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[?]" 0

Currently I get an error when I try to run this script.  Is this expected?  Would it 
be possible to implement matching for array values?

Thanks!
  2023   25 Nov 2020 Amy RobertsSuggestionODBSET wildcards with array keys in Sequencer files
The following all fail with "Cannot find ODB key "<key>""

ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[*]" 0
ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[0-9]" 0
ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[1]" 0
ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)*" 0
ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)" 0


> Hi,
> I guess the issue is in the "[?]" part of the command, the indexing is handled differently from the odb path and does not 
> support "?".
> Are you trying to set only the first 9 channels?
> Could you try with "[*]" or "[0-9]" instead?
> 
> Marco
> 
> > I'm interested in using the matching feature for ODBSET explained on 
> > https://midas.triumf.ca/MidasWiki/index.php/Sequencer for settings that are in an 
> > array, like:
> > 
> > COMMENT "Ground the detectors"
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[?]" 0
> > 
> > Currently I get an error when I try to run this script.  Is this expected?  Would it 
> > be possible to implement matching for array values?
> > 
> > Thanks!
  2025   25 Nov 2020 Amy RobertsSuggestionODBSET wildcards with array keys in Sequencer files
I think the issue may be the version of MIDAS I'm using.  Mine is current as of February 4, 2020.  

But since then there have been changes to the sequencer code, specifically parts that handle indexing.

I'll try this out with an updated version of MIDAS and report back if there are still any issues after updating.

> I created some keys in my ODB to try to match yours.
> The ODBSET commands you wrote are all working fine (of course with different results), except only for the "/Detectors/Det*/Settings/Charge/Bias (V)*" which I will have to 
> look into.
> In any case the error message I'm getting is "could not match ay key" and not the one you are reporting.
> 
> Now I'm a bit puzzled:
> Are you sure your ODB contains those keys?
> Are you testing the ODBSET inside a more complex sequencer or on its own?
> 
> Maybe I can try to reproduce it using your ODB setup.
> Could you send an ODB dump of the "/Detectors" folder using the "save" command of odbedit ("cd /Detectors" and then "save detector.odb")?
> 
> Best,
> 
> Marco
> 
> 
> > The following all fail with "Cannot find ODB key "<key>""
> > 
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[*]" 0
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[0-9]" 0
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[1]" 0
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)*" 0
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)" 0
> > 
> > 
> > > Hi,
> > > I guess the issue is in the "[?]" part of the command, the indexing is handled differently from the odb path and does not 
> > > support "?".
> > > Are you trying to set only the first 9 channels?
> > > Could you try with "[*]" or "[0-9]" instead?
> > > 
> > > Marco
> > > 
> > > > I'm interested in using the matching feature for ODBSET explained on 
> > > > https://midas.triumf.ca/MidasWiki/index.php/Sequencer for settings that are in an 
> > > > array, like:
> > > > 
> > > > COMMENT "Ground the detectors"
> > > > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[?]" 0
> > > > 
> > > > Currently I get an error when I try to run this script.  Is this expected?  Would it 
> > > > be possible to implement matching for array values?
> > > > 
> > > > Thanks!
  2061   17 Dec 2020 Amy RobertsSuggestionImproving variable functionality in Sequencer?
We're using the sequencer to manage runs, and this typically looks something like:

1. save ODB keys to variables via ODBGET
2. set ODB keys to new values for a "pre-run" process
3. return ODB keys to values created in line 1
4. take data

The problem I'm running into is that the list of ODB keys to save is pretty 
unwieldy.  I'm wondering if there are sequencer features that exist or that I could 
request that might make this easier.

For example, having a way to list ODB keys, save ODB directories, and load ODB 
directories would be much more concise way for me to write my script.

Another option might be to have some version of the ODBSET wildcards for ODBGET.  
Although for this, setting the variable names might be tricky.  

In any case, even being able to ODBGET an array and set that to one variable name 
would be a big improvement.
  2065   05 Jan 2021 Amy RobertsSuggestionImproving variable functionality in Sequencer?
Hello, just wanted to re-ping on this question now that folks are starting to get back from 
the holidays.
  2289   14 Oct 2021 Amy RobertsSuggestionAdding (or improving discoverability) of TID for odbset
Creating an ODB key requires users to know the Type ID that are defined in 
https://bitbucket.org/tmidas/midas/src/develop/include/midas.h starting at line 320.

I can't find any information on the Midas Wiki about these values or how to find 
them.

Am I missing something obvious?  Is there a way to improve how to find these values?  
Or is this not the best way to interact with the ODB?
  967   23 Feb 2014 Andre FrankenthalBug ReportInstallation failing on Mac OS X 10.9 -- related to strlcat and strlcpy
Hi,

I don't know if this actually fits the Bug Report category. I've been trying to install Midas on my Mac OS 
Mavericks and I keep getting errors like "conflicting types for '___builtin____strlcpy_chk' ..." and similarly for 
strlcat. I googled a bit and I think the problem might be that in Mavericks strlcat and strlcpy are already 
defined in string.h, and so there might be a redundant definition somewhere. I'm not sure what the best 
way to fix this would be though. Any help would be appreciated.

Thanks,

Andre
  972   27 Feb 2014 Andre FrankenthalBug ReportInstallation failing on Mac OS X 10.9 -- related to strlcat and strlcpy
> > 
> > I don't know if this actually fits the Bug Report category. I've been trying to install Midas on my Mac OS 
> > Mavericks and I keep getting errors like "conflicting types for '___builtin____strlcpy_chk' ..." and similarly for 
> > strlcat. I googled a bit and I think the problem might be that in Mavericks strlcat and strlcpy are already 
> > defined in string.h, and so there might be a redundant definition somewhere. I'm not sure what the best 
> > way to fix this would be though. Any help would be appreciated.
> > 
> 
> We have run into this problem - MacOS 10.9 plays funny games with definitions of strlcpy() & co - and it has been fixed since last Summer.
> 
> For the record, current MIDAS builds just fine on MacOS 10.9.2.
> 
> For a pure test, try the instructions posted at midas.triumf.ca:
> 
> cd $HOME
> mkdir packages
> cd packages
> git clone https://bitbucket.org/tmidas/midas
> git clone https://bitbucket.org/tmidas/mscb
> git clone https://bitbucket.org/tmidas/mxml
> cd midas
> make
> 
> K.O.

Thanks, it works like a charm now! I must have obtained an outdated version of Midas.

Andre
  1321   15 Nov 2017 Andreas KnechtSuggestionFeature request: Separate ODB flag to show programs on "Programs page"
Currently one has to set the required flag in the ODB (e.g., /Programs/Logger/Required) to "y" for the program 
to appear on the "Programs page" and being able to start and stop the program easily.

However, if one wants to run with the "Prevent start on required progs" in /Experiment enabled, all the 
programs in the "Programs page" need to be running and one cannot have one of them stopped while still 
taking a run.

It would be nice to separate these two functionalities: Have a flag that makes the program appear on the 
"Programs page" and have a flag that controls the "Prevent start on required frogs" functionality.
  487   11 Jun 2008 Andreas SuterSuggestionmlogger is flooding the message queue
The current versions of mlogger SVN 4215 is flooding our message system with
stuff like

> Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events

This is fatal to us and blowing up the midas.log like hell. I would prefer if
one could flag these kind of messages (ODB /Logger/..), i.e. enable and disable
it. At the moment I have to comment it out in the source code since we cannot
work with it.

Cheers,
  Andreas 
  557   21 Jan 2009 Andreas SuterBug Reportmhttpd, mlogger updates
There is an obvious "unwanted feature" in this version of the mhttpd. It writes the
"plot time" into the gif (mhttpd, if-statement starting in line 8853). 

Please check this obvious things more carefully in the future before submitting code. ;-)

> mhttpd and mlogger have been updated with potentially troublesome changes.
> Before using these latest versions, please make a backup of your ODB. This is
> svn revisions 4434 (mhttpd.c) and 4435 (mlogger.c).
> 
> These new features are now available:
> - a "feature complete" implementation of "history in an SQL database". We use
> this new code to write history data from the T2K test setup in the TRIUMF M11
> beam line to a MySQL database (mlogger) and to make history plots directly from
> this database (mhttpd). We still write normal midas history files and we have a
> utility to import midas .hst files into an SQL database (utils/mh2sql). The code
> is functional, but incomplete. For best SQL database data layout, you should
> enable the "per variable history" (but backup your ODB before you do this!). All
> are welcome to try it, kick the tires, report any problems. Documentation TBW.
> - experimental implementation of "ODBRpc" added to the midas javascript library
> (ODBSet, ODBGet & co). This permits buttons on midas "custom" web pages to
> invoke RPC calls directly into user frontend programs, for example to turn
> things on or off. Documentation TBW.
> - the mlogger/mhttpd implementation of /History/Tags has proved troublesome and
> we are moving away from it. The SQL database history implementation already does
> not use it. During the present transition period:
> - mlogger and mhttpd will now work without /History/Tags. This implementation
> reads history tags directly from the history files themselves. Two downsides to
> this: it is slower and tags become non-persistent: if some frontends have not
> been running for a while, their variables may vanish from the history panel
> editor. To run in this mode, set "/History/DisableTags" to "y". Existing
> /History/Tags will be automatically deleted.
> - for the above 2 reasons, I still recommend using /History/Tags, but the format
> of the tags is now changed to simplify management and reduce odb size. mlogger
> will automatically convert the tags to this new format (this is why you should
> make a backup of your ODB).
> - using old mlogger with new mhttpd is okey: new mhttpd understands both formats
> of /History/Tags.
> - using old mhttpd with new mlogger is okey: please set ODB
> "/History/CreateOldTags" to "y" (type TID_BOOL/"boolean") before starting mlogger.
> 
> K.O.
ELOG V3.1.4-2e1708b5