Back Midas Rome Roody Rootana
  Midas DAQ System, Page 38 of 136  Not logged in ELOG logo
ID Date Author Topic Subjectdown
  1443   24 Jan 2019 Konstantin OlchanskiSuggestionjson rpc API for history data
> For us it would be a handy feature if history data could be requested directly
> from a custom page (time range or run based intervals) . Here I am not talking
> about history plots but I am talking about recorded time series data. This way
> we could easily generate useful graphs. For instance, if we measure the voltage
> (constant current) while cooling, we could instantly get the resistance versus
> temperature. Often we would like to 'correlate' recorded slow control data.
> 
> Is it already possible to extract history data the way suggested?

There are sundry hs_read() json rpc methods already implemented in preparation
of writing a javascript based history viewer (did not happen yet).

You can try to use them, they should work, but have not been tested extensively.

To find this stuff:
a) go to the mhttpd "help" page, open the "json rpc schema in text table format", look for the "hs_xxx" methods.
b) also from the "help" page, open "javascript examples- example.html", scroll down to "hs_get_active_events". Press the buttons, they should 
work, look at the source code to see how to call the rpc methods from your own page.

K.O.
  1993   08 Sep 2020 Zaher SalmanForumjson parser error
I am getting the following error alert in a custom page whenever a run starts

json parser exception: SyntaxError: Unexpected token < in JSON at position 985, batch request: method: "db_get_values", params: [object Object], id: 1598691925697 method: "get_alarms", params: null, id: 1598691925697 method: "cm_msg_retrieve", params: [object Object], id: 1598691925697 method: "cm_msg_retrieve", params: [object Object], id: 1598691925697

Does anyone know why and what causes this? This does not affect anything and things seem to continue running fine.

thanks.
  1994   08 Sep 2020 Konstantin OlchanskiForumjson parser error
> I am getting the following error alert in a custom page whenever a run starts
> json parser exception: SyntaxError: Unexpected token < in JSON at position 985, batch request: method: "db_get_values", params: [object Object], id: 1598691925697 method: "get_alarms", params: null, id: 1598691925697 method: "cm_msg_retrieve", params: [object Object], id: 1598691925697 method: "cm_msg_retrieve", params: [object Object], id: 1598691925697
> Does anyone know why and what causes this? This does not affect anything and things seem to continue running fine.

this is bug #242, https://bitbucket.org/tmidas/midas/issues/242/mjsonrpc-calls-should-return-valid-utf8

we read stuff from midas.log and push it to the web browser. we have seen this stuff
contain arbitrary binary data (both intentionally written into midas.log by cm_msg() and
file content corruption/truncation from computer crashes), the json decoder in the web browser
does not like that stuff - it is invalid utf-8 unicode - and throws an exception.

since we cannot ensure content of midas.log (and other files on disk) are always valid utf-8,
we have to sanitize it before sending it to the browser.

right now I am not sure of the best way to do this sanitizing. we do have a function to check
for valid utf-8 unicode, perhaps it should be extended to replace invalid unicode with spaces
or Xes or "?" or whatever, I am open to suggestions and ideas.

BTW, this is a new recent change to how strings generally work. C NUL-terminated strings are
permitted to contain arbitrary binary data (except for NUL char, of course). C++ std::string
are permitted to contain arbitrary binary data. but javascript strings are only permitted
to contain valid unicode, and the json standard was recently amended to require that json
strings are valid utf-8 unicode. So there is a disconnect between C/C++ code written in the
last 50 years where strings can contain binary data and the javascript world requiring
valid utf-8 unicode pretty much everywhere.

K.O.
  1072   16 Jul 2015 Thomas LindnerBug Reportjset/ODBSet using true/false for booleans
MIDAS does not seem to be consistent (or at least convenient) with how it
handles booleans in AJAX functions.

When you request an ODB value that is a boolean with AJAX call like

http://neut14.triumf.ca:8081/?cmd=jcopy&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys

then you get

{ "Hidden/last_written" : 1437065425, "Hidden" : false }

This seems correct, since the JSON convention has booleans encoded as true/false.

But this convention does not work when trying to set the boolean value. For instance

http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=true

does not set the variable to true. To make this work you need to use the
characters y/n

http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y

I tested this with ajax/jset, but the same problem seems to occur when using the
javascript function ODBSet. The documentation doesn't say what sort of encoding
to use when using these functions, so I guess the idea is that these functions
use MIDAS encoding for booleans. But it seems to me that it would be more
convenient if jset/ODBSet allowed the option to use json/javascript encoding for
boolean values; or at least had that as a format option for jset/ODBSet.  That
way my javascript could look like

var mybool = true;
URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=" + mybool;

instead of

var mybool = true;
URI_command = ""
if(mybool){
  URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y";
else
  URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=n";


__________________________________________________________
Cross-posting from bitbucket issue tracker:

https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans
  1082   29 Jul 2015 Stefan RittBug Reportjset/ODBSet using true/false for booleans
See bitbucket for the solution.

https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans#comment-20550474
  1118   25 Sep 2015 Konstantin OlchanskiBug Reportjset/ODBSet using true/false for booleans
> MIDAS does not seem to be consistent (or at least convenient) with how it
> handles booleans in AJAX functions.

MIDAS documentation does not say that arguments to "jset" (ODBSet) should be JSON-encoded:
https://midas.triumf.ca/MidasWiki/index.php/AJAX#jset

You found undocumented behavior.

As solution, we plan to replace "jset/ODBSet" with a JSON-RPC function, where all parameters will be JSON-encoded 
(obviously).

K.O.


> 
> When you request an ODB value that is a boolean with AJAX call like
> 
> http://neut14.triumf.ca:8081/?cmd=jcopy&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys
> 
> then you get
> 
> { "Hidden/last_written" : 1437065425, "Hidden" : false }
> 
> This seems correct, since the JSON convention has booleans encoded as true/false.
> 
> But this convention does not work when trying to set the boolean value. For instance
> 
> http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=true
> 
> does not set the variable to true. To make this work you need to use the
> characters y/n
> 
> http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y
> 
> I tested this with ajax/jset, but the same problem seems to occur when using the
> javascript function ODBSet. The documentation doesn't say what sort of encoding
> to use when using these functions, so I guess the idea is that these functions
> use MIDAS encoding for booleans. But it seems to me that it would be more
> convenient if jset/ODBSet allowed the option to use json/javascript encoding for
> boolean values; or at least had that as a format option for jset/ODBSet.  That
> way my javascript could look like
> 
> var mybool = true;
> URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=" + mybool;
> 
> instead of
> 
> var mybool = true;
> URI_command = ""
> if(mybool){
>   URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y";
> else
>   URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=n";
> 
> 
> __________________________________________________________
> Cross-posting from bitbucket issue tracker:
> 
> https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans
  1135   11 Nov 2015 Konstantin OlchanskiBug Reportjset/ODBSet using true/false for booleans
> > MIDAS does not seem to be consistent (or at least convenient) with how it
> > handles booleans in AJAX functions.

The JSON-RPC functions have been merged into main midas and you can now use the new function mjsonrpc_db_paste(paths, values, id, 
callback, error_callback);

For example:

mjsonrpc_db_paste(["/foo","/bar","/baz"],[1,2,3]);

the target items should already exist (for this example, not in general).

All data is JSON encoded, success/failure is returned via callbacks.

K.O.
  607   08 Jul 2009 Konstantin OlchanskiForumjorway73a.c, Data taking hangs in the middle of run
> > Could you give more info on your setup:
> > - CAMAC controller model
> Jorway 73A, we have three in hand and the problem doesn't depend on which controller
> we were using.

Dawei sent me a copy of his jorway73a.c scsi-camac driver. It is quite different from the
file in the MIDAS distribution. Dawei tells me that the file from the MIDAS distribution
does not compile. Stack traces from Dawei indicate a hang in this modified jorway73a.c
scsi-camac driver.

K.O.
  608   18 Jul 2009 Exaos LeeForumjorway73a.c, Data taking hangs in the middle of run
> > > Could you give more info on your setup:
> > > - CAMAC controller model
> > Jorway 73A, we have three in hand and the problem doesn't depend on which controller
> > we were using.
> 
> Dawei sent me a copy of his jorway73a.c scsi-camac driver. It is quite different from the
> file in the MIDAS distribution. Dawei tells me that the file from the MIDAS distribution
> does not compile. Stack traces from Dawei indicate a hang in this modified jorway73a.c
> scsi-camac driver.
> 
> K.O.

I encountered too that the jorway73a.c cannot work for my SCM-301 CAMAC driver. The
"jorway73a.c" distributed with MIDAS seems to work with Jorway 73a reversion > 300. But my
module has the reversion number 203. :-( I hope you can paste the modified version here so
that I can try it if I have spare time.
Regards.
  753   28 Feb 2011 Konstantin OlchanskiInfojavascript example experiment
I just committed a MIDAS example for using most mhttpd html and javascript functions: ODBGet(), 
ODBSet(), ODBRpc() & co.

Please checkout .../midas/examples/javascript1, and follow instructions in the README file.

For your enjoyment, the example html file is attached to this message.

(to use the function ODBRpc_rev1 - the midas rpc call that returns data to the web page - you need 
mhttpd svn rev 4994 committed today. Other functions should work with any revision of mhttpd)

svn rev 4992.
K.O.
Attachment 1: example.html
  1127   29 Oct 2015 Konstantin OlchanskiInfojavascript docs, midas JSON-RPC interface
> JSON-RPC interface

For interfacing to MIDAS just from browser javascript, the user does not need to know anything about JSON-RPC, all the user-level mjsonrpc_xxx() functions provided by 
mhttpd.js work the same as the old ODBxxx() functions.

As usual, the functions are documented using Doxygen, so here there is no difference between old and new interfaces.

To generate the documentation, run "make dox" (doxygen and graphviz "dot" packages should be installed). (it will take some time to generate everything), then open 
html/index.html and navigate to "files" to "mhttpd.js" and you will see the list of all RPC funcrions (old functions are ODBxxx, new functions are mjsonrpc_xxx).

There was a possibility to implement the mjsonrpc javascript client interface as a javascript class, but older versions of doxygen seem to work incorrectly for such code making it 
impossible to document the code. So it remains implemented as traditional functions with a few globals, but the design an implementation are done with a view to convert the 
code to a javascript class some time in the future.

As ever, the examples/javascript1 experiment provides examples of using all available javascript functions supported by midas. (except for functions that are hard to example or 
hard to document).

K.O.
  1733   15 Nov 2019 Andreas SuterSuggestionjavascript comunication
I am currently testing the new history system on the mhttpd side and stumbled over the following issue: typically our user open a lot of midas web-page tabs and keep them open. With the current version this leads after a night typically to a state where the browser is busy with itself and not reacting anymore.

One important reason seems to be that ALL tabs trying to communicate all the time which is totally unnecessary, since I think a hidden tab should stay in a sleeping mode. 

I was browsing if there is a way to find out if a tab is active or not, and found the following API which exactly does this:

https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API

Furthermore, the simple

document.hidden 

tag, could be used to find out if the page is currently active.

Wouldn't it a good idea to send all midas tabs which are not active into a sleep mode and only reactivate them if they come into focus?

I had a quick look at the JavaScript libs of midas, but I am not quite certain where to best inject this. 
  1735   15 Nov 2019 Stefan RittSuggestionjavascript comunication
Very good idea. And thanks for finding the document.hidden solution. I put it in, so give it a try.

Best,
Stefan


> I am currently testing the new history system on the mhttpd side and stumbled over the following issue: typically our user open a lot of midas web-page tabs and keep them open. With the current version this leads after a night typically to a state where the browser is busy with itself and not reacting anymore.
> 
> One important reason seems to be that ALL tabs trying to communicate all the time which is totally unnecessary, since I think a hidden tab should stay in a sleeping mode. 
> 
> I was browsing if there is a way to find out if a tab is active or not, and found the following API which exactly does this:
> 
> https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API
> 
> Furthermore, the simple
> 
> document.hidden 
> 
> tag, could be used to find out if the page is currently active.
> 
> Wouldn't it a good idea to send all midas tabs which are not active into a sleep mode and only reactivate them if they come into focus?
> 
> I had a quick look at the JavaScript libs of midas, but I am not quite certain where to best inject this. 
  1737   17 Nov 2019 Konstantin OlchanskiSuggestionjavascript comunication
> I am currently testing the new history system on the mhttpd side and stumbled over the following issue:
> typically our user open a lot of midas web-page tabs and keep them open. With the current version this leads after a night typically to a state where the browser is busy with itself and not reacting anymore.
> 
> One important reason seems to be that ALL tabs trying to communicate all the time which is totally unnecessary, since I think a hidden tab should stay in a sleeping mode. 

I am looking at two more problems with inactive tabs:

a) google chrome slows down the execution of javascript in inactive tabs, leading to trouble
with memory management - midas pages poll at 1/sec, each poll allocates memory for processing RPC messages,
and (until recently) allocates memory for new DOM objects to update the web page - but the garbage collector
gets slowed down and does not keep up - leading to huge memory use (up to 200 Mbytes) for inactive midas pages
that normally consume 50-ish Mbytes.

b) the playing of the alarm sound is throttled by "user has not interacted with the document" thing, but there is a bug -
instead of canceling the playing of the alarm sound, the sound file is still loaded, (but not played). (this is hard to debug
because I do not know how to manually trigger the "user has not interacted..." condition, I have to wait for many days for it.
Then, for inactive tabs, the loading of the sound files is slowed down, leading to many of them getting queued up,
and eventually they all try to load and play at the same time, again leading to huge memory and cpu use in inactive tabs.
(this sounds incredible because we play the alarm sound at most 1/minute, for sure the previous sound file must have
finished playing by then, but no, it is easy to see it happen - add a few console.log messages and wait for a few days).

>
> I was browsing if there is a way to find out if a tab is active or not, and found the following API which exactly does this:
> https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API
> 

From looking at the inactive tab business, I see that javascript in inactive tabs runs quite differently from javascript
in active tabs (i.e. timers do not work the same) and I see how the "visibility api" had to be invented to counter that.

> 
> Wouldn't it a good idea to send all midas tabs which are not active into a sleep mode and only reactivate them if they come into focus?
> I had a quick look at the JavaScript libs of midas, but I am not quite certain where to best inject this.
>

most midas web pages poll in two places - mhttpd_refresh() updates the current date timestamp, alarms, currently active midas.log message;
and each page has it's own loop for updating it's own data (i.e. "alarms" page, "programs" page).

we should be careful to not completely disable all polling as some experiments do use and do rely on the midas producing
loud alarm messages ("program logged is not running!!!", "program mhttpd aborted!!!"). Even if all midas tabs are inactive,
some javascript is some tab still has to run frequently enough to poll midas and to sound the alarm sounds (even though
I am not sure how to 100% reliably counteract the google-chrome not playing sound files because
of the "user did not interact with the site..." thing).

K.O.
  1738   17 Nov 2019 Konstantin OlchanskiSuggestionjavascript comunication
> Very good idea. And thanks for finding the document.hidden solution. I put it in, so give it a try.

Hi, Stefan - I did not look at your code, if all midas tabs are inactive, will the alarm sound still play?

K.O.


> 
> Best,
> Stefan
> 
> 
> > I am currently testing the new history system on the mhttpd side and stumbled over the following issue: typically our user open a lot of midas web-page tabs and keep them open. With the current version this leads after a night typically to a state where the browser is busy with itself and not reacting anymore.
> > 
> > One important reason seems to be that ALL tabs trying to communicate all the time which is totally unnecessary, since I think a hidden tab should stay in a sleeping mode. 
> > 
> > I was browsing if there is a way to find out if a tab is active or not, and found the following API which exactly does this:
> > 
> > https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API
> > 
> > Furthermore, the simple
> > 
> > document.hidden 
> > 
> > tag, could be used to find out if the page is currently active.
> > 
> > Wouldn't it a good idea to send all midas tabs which are not active into a sleep mode and only reactivate them if they come into focus?
> > 
> > I had a quick look at the JavaScript libs of midas, but I am not quite certain where to best inject this. 
  1739   18 Nov 2019 Stefan RittSuggestionjavascript comunication
> Hi, Stefan - I did not look at your code, if all midas tabs are inactive, will the alarm sound still play?

Nope. All updates are done in mhhtpd_refresh(), and I changed it such that nothing is updated if hidden.

I agree however that this is bad. You want to hear alarms always. So I added some code to do ONLY alarm updates if in the background. 
No GUI changes, only audio playing. Checking is reduced from 1 Hz to 0.1 Hz (once every 10 seconds). I don't have however a solution 
to your problem "not enough interaction with page" which I have never seen so far.

I wonder if we should try push notifications: https://developers.google.com/web/fundamentals/codelabs/push-notifications
which seems a bit complicated to me. It might also be too subtle if someone is sleeping in front of the computer.

Stefan
  1740   18 Nov 2019 Stefan RittSuggestionjavascript comunication
> a) google chrome slows down the execution of javascript in inactive tabs, leading to trouble
> with memory management - midas pages poll at 1/sec, each poll allocates memory for processing RPC messages,
> and (until recently) allocates memory for new DOM objects to update the web page - but the garbage collector
> gets slowed down and does not keep up - leading to huge memory use (up to 200 Mbytes) for inactive midas pages
> that normally consume 50-ish Mbytes.

Try my latest change, which drops any update if hidden, except the alarm sound. At the moment this only works on the main "status" page.

> most midas web pages poll in two places - mhttpd_refresh() updates the current date timestamp, alarms, currently active midas.log message;
> and each page has it's own loop for updating it's own data (i.e. "alarms" page, "programs" page).

That's correct. I changed mhttpd_refresh() now and the main loop for the "status" page. If that works for everybody, we can do that also for "programs" and other pages. The code is here:

      if (document.hidden) {
         // don't update page if hidden
         setTimeout(update_page, 500);
         return;
      }

where "update_page" has to be replaced with the proper function.

Stefan
  1741   18 Nov 2019 Konstantin OlchanskiSuggestionjavascript comunication
> > Hi, Stefan - I did not look at your code, if all midas tabs are inactive, will the alarm sound still play?
> I added some code to do ONLY alarm updates if in the background (once every 10 seconds)

Ok, good.

> I don't have however a solution to your problem "not enough interaction with page" which I have never seen so far.

When it happens, you should see my messages about it in the javascript console. (rejected promise from audio.play()).

How to make it happen so one can see it and be sure it is handled correctly (audio.src has to be set to blank, at the least), I have no idea.

What I have been looking at for the huge memory and cpu use problem, I would call "a bad interaction between two kludges",
the first kludge is the throttling javascript in inactive tabs, the second kludge is the "not interacted" rejection of audio.play() promise.
(audio.play() returning a promise, and the strange sequencing between this promise success/reject and
the audio events loaded/canplay/done/etc smells of a kludge, too).

> I wonder if we should try push notifications: https://developers.google.com/web/fundamentals/codelabs/push-notifications
> which seems a bit complicated to me. It might also be too subtle if someone is sleeping in front of the computer.

I did not read all the explanation, but if it requires use of 3rd party services, I think we cannot use it, we do not want
to miss "experiment is on fire" alarms just because google is down.

K.O.
  46   14 Jul 2004 Exaos Lee install problem of Makefile on MacOS X (Darwin 7.4.0, gcc 3.3)
I have compiled the sources on Darwin 7.4.0 with gcc 3.3. After the compilation of source codes, I 
try to execute "gmake install". I got the following message:
-------
Nothing to be done for "install".
-------

The install target could not be executed. Then I add the following line to the Makefile:
------
.PHONY: install
------

The install target can be executed. But when it is tring to copy "dio" to the proper directory, it 
cannot find the file. Then I found that the "utils" target isn't built. 
I try to build the target: darwin/bin/dio, I got the following error:
-------
cc -g -O2 -Wall -Iinclude -Idrivers -Ldarwin/lib -DINCLUDE_FTPLIB   -DOS_LINUX -DOS_DARWIN 
-DHAVE_STRLCPY -fPIC -Wno-unused-function -o darwin/bin/dio utils/dio.c
utils/dio.c:39:20: sys/io.h: No such file or directory
utils/dio.c: In function `main':
utils/dio.c:46: warning: implicit declaration of function `iopl'
gmake: *** [darwin/bin/dio] Error 1
--------
So, the include file "sys/io.h" may be changed under Darwin. I don't know how. I will try later. I 
hope somebody can notice this. 
Best regards.
  47   14 Jul 2004 Exaos Lee install problem of Makefile on MacOS X (Darwin 7.4.0, gcc 3.3)
There are not such a file "io.h" inside my MacOS X. In fact, I didn't find any file containing function iopl().
So what is the equivalent function of iopl() under MacOS X? The utility dio should be modified in order to be compiled under Darwin-
gcc platform. 
ELOG V3.1.4-2e1708b5