12 Sep 2023, Stefan Ritt, Suggestion, Syntax highlighting for sequencer scripts
|
I like the idea of syntax highlighting, but your solution is just for one editor which not everybody
is using. It would be better if the editor built into mhttpd for MSL files would have the possibility.
I looked at highlighting in an HTML <textarea> tag, and found that we can do it with a
<div contenteditable="true" style="font-family: monospace"> ... </div>
tag where we can change the color of individual words. If you translate your existing rules of syntax
highlighting into JavaScript, I'm happy to put that into the mhttpd sequencer editor. So I would need
a function which receives a MSL text, then replaces all keywords with some color tagging, like
ODBSET -> <span style="color:red">ODBSET</span>
Best,
Stefan |
03 Apr 2007, Stefan Ritt, Info, Switch to Visual C++ 2005 under Windows
|
I had to switch to Visual C++ 2005 under Windows. This required the upgrade of
all project files under \midas\nt\ and fixing a few warnings, since the new
compiler is more picky.
Note that in order to use most C RTL funcitons, you have to define two
preprocessor statements:
#define _CRT_SECURE_NO_DEPRECATE
#define _CRT_NONSTDC_NO_DEPRECATE
either at the beginning of a file (before you include stdio.h), or via the
project property page under C/C++ / Preprocessor / Preprocessor Definitions,
where you also have the WIN32 and the _CONSOLE definitions. I adapted all
project files in the distribution, but for all local projects this has to be
done additionally. |
20 Jan 2009, Stefan Ritt, Info, Subrun scheme implemented
|
A new "subrun" scheme has been implemented in mlogger to split a big data file into several individual data files. This feature might be helpful if a data file from a single run gets too large (>4 GB for example) and if shorter runs are not wanted for efficiency reasons. The scheme works as follows:
Each subrun will contain an ODB dump if this is turned on via /Channels/x/Settings/ODB dump. The stopping of the "main" run (after four subruns in the above example) can be done in the usual way (event limit in the front-end, manually through odbedit, etc.).
The code has been tested in two test environments, but not yet in a real experiment. So please test it before going into production. The modification in mlogger requires SVN revision 4440 of mlogger.c and 4441 of odb.c.
Please note that the lazylogger cannot be used with this scheme at the moment since it does not recognize the subruns. That will be fixed in a future version and announced in this forum.
- Stefan |
23 Jan 2009, Renee Poutissou, Info, Subrun scheme implemented
|
Hi Stefan,
My colleague Tobi Raufer (tobi.raufer@stfc.ac.uk) has tested this new implementation and
sent me the following questions:
-------- Original Message --------
Subject: Re: [Fwd: [Midas] Subrun scheme implemented]
Date: Fri, 23 Jan 2009 01:52:37 +0000
From: Tobias Raufer <tobi.raufer@stfc.ac.uk>
To: Renee Poutissou <renee@triumf.ca>
Hi Renee
I have tested the new subrun functionality a bit more and I have two observations. First, it seems to work on a basic level, i.e. subruns are created, which are equal in size. However, I can't relate their size to the byte limit set in the ODB.
Here is an example. The settings in the ODB are the following:
[local:testExp:S]/>ls /Logger/Channels/0/Settings/
Active y
Type Disk
Filename run%05d_%02d.mid
Format MIDAS
Compression 0
ODB dump n
Log messages 0
Buffer SYSTEM
Event ID -1
Trigger mask -1
Event limit 0
Byte limit 0
Subrun Byte limit 10000
Tape capacity 0
Subdir format
Current filename run00005_07.mid
As you can see, I set the subrun byte limit to 10000. Here are the subrun files which were created:
-rw-r--r-- 1 raufer 32800 Jan 23 01:36 run00005_00.mid
-rw-r--r-- 1 raufer 32800 Jan 23 01:36 run00005_01.mid
-rw-r--r-- 1 raufer 32800 Jan 23 01:36 run00005_02.mid
-rw-r--r-- 1 raufer 32800 Jan 23 01:36 run00005_03.mid
-rw-r--r-- 1 raufer 32800 Jan 23 01:36 run00005_04.mid
-rw-r--r-- 1 raufer 32800 Jan 23 01:36 run00005_05.mid
-rw-r--r-- 1 raufer 32800 Jan 23 01:36 run00005_06.mid
-rw-r--r-- 1 raufer 4960 Jan 23 01:36 run00005_07.mid
The file size seems to be 32800 bytes. Any idea what's going on? I first thought this might have to do with the ODB dump not being accounted for but as you can see from the configuration above, I turned it off for this run.
When I run with the ODB dump on but with the same byte limit, things become even more strange. I get the following sizes:
bash-3.2$ ls -l run00006_*.mid
-rw-r--r-- 1 raufer 53798 Jan 23 01:46 run00006_00.mid
-rw-r--r-- 1 raufer 53804 Jan 23 01:46 run00006_01.mid
-rw-r--r-- 1 raufer 53793 Jan 23 01:46 run00006_02.mid
-rw-r--r-- 1 raufer 53781 Jan 23 01:46 run00006_03.mid
-rw-r--r-- 1 raufer 53781 Jan 23 01:46 run00006_04.mid
-rw-r--r-- 1 raufer 53781 Jan 23 01:46 run00006_05.mid
-rw-r--r-- 1 raufer 53802 Jan 23 01:46 run00006_06.mid
-rw-r--r-- 1 raufer 53833 Jan 23 01:46 run00006_07.mid
-rw-r--r-- 1 raufer 71557 Jan 23 01:46 run00006_08.mid
-rw-r--r-- 1 raufer 20999 Jan 23 01:46 run00006_09.mid
As you can see, now the sizes are larger and they don't even seem to be consistent between the different subruns. Renee, could you forward this to the MIDAS developers?
Thanks much,
Tobi
Quote: |
The code has been tested in two test environments, but not yet in a real experiment. So please test it before going into production. The modification in mlogger requires SVN revision 4440 of mlogger.c and 4441 of odb.c.
Please note that the lazylogger cannot be used with this scheme at the moment since it does not recognize the subruns. That will be fixed in a future version and announced in this forum.
- Stefan |
|
25 Jan 2009, Stefan Ritt, Info, Subrun scheme implemented
|
Renee Poutissou wrote: | I have tested the new subrun functionality a bit more and I have two observations. First, it seems to work on a basic level, i.e. subruns are created, which are equal in size. However, I can't relate their size to the byte limit set in the ODB. |
What you describe is expected. The logger process maintains a write cache, which is 32 kB under linux and 1 MB under Windows. The size is controlled through the constant TAPE_BUFFER_SIZE defined in midas.h. The reason for this buffer is to optimize writes to disks and tapes and has been carefully optimized to give maximum performance. It means however that data gets written only in 32 kB chunks to disk. That's the reason why your run size is 32kB plus a few bytes. You can change this by modifying TAPE_BUFFER_SIZE, but be aware that this will then slow down your logging of data. |
12 Jun 2019, Marius Koeppel, Forum, Strange JS array creation
|
Hello everybody,
I have a strange JS behavior. In one of my frontends I create a key in the ODB with:
db_create_key(hDB, 0, "Equipment/Switching/Variables/DATA_WRITE", TID_INT);
In my custom page I have a JS function which loops over an array and sets the
value of this key with:
for (i = 0; i < lines.length; i++) {
modbset("/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]",
parseInt(lines[i]));
}
After calling this function I have an array in the ODB now. For my understanding
calling an INT like an array shouldn't be possible. So is this dangerous to do?
Best regards,
Marius |
17 Jun 2019, Konstantin Olchanski, Forum, Strange JS array creation
|
> db_create_key(hDB, 0, "Equipment/Switching/Variables/DATA_WRITE", TID_INT);
you can also do this from javascript, too, using the db_create rpc call, see mjsonrpc_db_create() and
example.html
> for (i = 0; i < lines.length; i++) {
> modbset("/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]", parseInt(lines[i]));
> }
this is wrong.
a) you are programming javascript as if it were C/C++. You think this code wrote lines.length() values
to ODB, when what the code actually did is queued lines.length() RPC requests for later execution.
Eventually some time later, each RPC request will open a connection to mhttpd, send a request, wait
for mhtttpd to process it, etc. Where do you wait for the completion of all these RPCs before
proceeding as if all the data has been successfully written to ODB? (answer: you cannot, javascript
cannot "wait for things", instead you have to make chains of event handlers. javascript != C/C++.
They are completely different).
b) you should write the whole array in one operation instead of looping over each element. see
mjsonrpc_db_paste() and example.html.
> After calling this function I have an array in the ODB now. For my understanding
> calling an INT like an array shouldn't be possible. So is this dangerous to do?
I do not understand your question about "calling an INT like an array". You are not calling anything
called "INT". Your code has a loop, a call to parseInt() (defined where?) and a call modbset()
(defined in mhttpd.js). It looks like correct javascript (it does not do what I think you expected it to
do), what do you think is dangerous?
K.O. |
24 Jun 2019, Marius Koeppel, Forum, Strange JS array creation
|
> > for (i = 0; i < lines.length; i++) {
> > modbset("/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]", parseInt(lines[i]));
> > }
>
> this is wrong.
>
> a) you are programming javascript as if it were C/C++. You think this code wrote lines.length() values
> to ODB, when what the code actually did is queued lines.length() RPC requests for later execution.
> Eventually some time later, each RPC request will open a connection to mhttpd, send a request, wait
> for mhtttpd to process it, etc. Where do you wait for the completion of all these RPCs before
> proceeding as if all the data has been successfully written to ODB? (answer: you cannot, javascript
> cannot "wait for things", instead you have to make chains of event handlers. javascript != C/C++.
> They are completely different).
--> Following your discussion about async. functions I will change this part of the code and make chains of
event handlers.
> b) you should write the whole array in one operation instead of looping over each element. see
> mjsonrpc_db_paste() and example.html.
--> In the midas back-end I never created an array. I created an INT in the ODB with db_create_key(hDB, 0,
"Equipment/Switching/Variables/DATA_WRITE", TID_INT). By using modset in javascript and parsing the string
"/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]" I call it like an array and it shows up like an
array in the ODB. So for explaining it a bit better how the value changes in the ODB take this pseudo code
example:
// midas part //
> int a = 1; // this is more or less what I think db_create_key is doing in the ODB
// midas part //
// ODB //
> print(a) // this prints me 1 and this is also the value what I see in the ODB
// ODB //
// javascript part //
> for int i in [1,2,3,4] do
> modset(a[i], i) // for simplification I don't use event handlers here
> end for
// javascript part //
// ODB //
> print(a) // now I see [1,2,3,4]
// ODB //
This example violates type safety. I know that javascript is not type safe. According to this I would like
to know if this behavior is wanted or why there is no bounds checking?
> I do not understand your question about "calling an INT like an array".
--> Here I mean that I call the variable in the ODB via string passing, like I would call a variable, which is
an array. I don't speak about function calls.
> parseInt() (defined where?)
--> This is a global JavaScript function (https://www.w3schools.com/jsref/jsref_obj_global.asp)
Cheers,
Marius |
25 Jun 2019, Konstantin Olchanski, Forum, Strange JS array creation
|
> --> In the midas back-end I never created an array. I created an INT in the ODB with db_create_key(hDB, 0,
> "Equipment/Switching/Variables/DATA_WRITE", TID_INT). By using modset in javascript and parsing the string
> "/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]" I call it like an array and it shows up like an
> array in the ODB.
I think you are good. In ODB, a TID_INT is actually an array of size 1. Writing to an array index automatically
extends the array. I am not sure where this is written down, but this is how most ODB array index access functions
have always worked.
You do have a performance bug with your loop, though, by writing a[1], a[2], a[3], you cause
the array to grow from size 1 to size 2. Then grow it from size 2 to size 3, etc. Of course
this only happens the first time you run the thing. Afterwards, the array has the correct size
and does not need to be grown.
K.O. |
07 Nov 2024, Lukas Gerritzen, Suggestion, Stop run and sequencer button
|
Due to popular demand among our students, I added a button to the sequencer that stops the run and the sequence. If you find it useful, please consider merging this upstream.
$ git diff sequencer.html
diff --git a/resources/sequencer.html b/resources/sequencer.html
index e7f8a79d..95c7e3d8 100644
--- a/resources/sequencer.html
+++ b/resources/sequencer.html
@@ -115,6 +115,7 @@
<img src="icons/play.svg" title="Start" class="seqbtn Stopped" onclick="startSeq();">
<img src="icons/debug.svg" title="Debug" class="seqbtn Stopped" onclick="debugSeq();">
<img src="icons/square.svg" title="Stop" class="seqbtn Running Paused" onclick="stopSeq();">
+ <img src="icons/x-octagon.svg" title="Stop Run and Sequencer immediately" class="seqbtn Running Paused" onclick="stopRunAndSeq();">
<img src="icons/pause.svg" title="Pause" class="seqbtn Running" onclick="modbset('/Sequencer/Command/Pause script',true);">
<img src="icons/resume.svg" title="Resume" class="seqbtn Paused" onclick="modbset('/Sequencer/Command/Resume script',true);">
<img src="icons/step-over.svg" title="Step Over" class="seqbtn Running Paused" onclick="modbset('/Sequencer/Command/Step over',true);">
[gac-megj@pc13513 resources]$ git diff sequencer.js
diff --git a/resources/sequencer.js b/resources/sequencer.js
index cc5398ef..b75c926c 100644
--- a/resources/sequencer.js
+++ b/resources/sequencer.js
@@ -1582,6 +1582,23 @@ function stopSeq() {
});
}
+function stopRunAndSeq() {
+ const message = `Are you sure you want to stop the run and sequence?`;
+ dlgConfirm(message,function(resp) {
+ if (resp) {
+ modbset('/Sequencer/Command/Stop immediately',true);
+
+ mjsonrpc_call("cm_transition", {"transition": "TR_STOP"}).then(function (rpc) {
+ if (rpc.result.status !== 1) {
+ throw new Error("Cannot stop run, cm_transition() status " + rpc.result.status + ", see MIDAS messages");
+ }
+ }).catch(function (error) {
+ mjsonrpc_error_alert(error);
+ });
+ }
+ });
+}
+
// Show or hide parameters table
function showParTable(varContainer) {
let e = document.getElementById(varContainer);
|
07 Nov 2024, Stefan Ritt, Suggestion, Stop run and sequencer button
|
I don't find this very useful. Some experiments do not only want to stop the run, but also do other cleanup things. To do that, I proposed and "atexit" function like C has it. Then the user can put a run stop there, plus any other cleanup. This will be much more flexible. Think about the "reset" script we have to manually run if we abort a sequencer. The atexit function will come next week, so you should consider to use it instead your additional button.
Stefan |
09 Aug 2023, Konstantin Olchanski, Bug Fix, Stefan's improved ODB flush to disk
|
This is an important improvement, should have a post of it's own. K.O.
> > > RFE filed:
> > > https://bitbucket.org/tmidas/midas/issues/367/odb-should-be-saved-to-disk-
periodically
> >
> > Implemented and closed: https://bitbucket.org/tmidas/midas/issues/367/odb-
should-be-saved-to-disk-periodically
> >
> > Stefan
>
> Stefan's comments from the closed bug report:
>
> Ok I implemented some periodic flushing. Here is what I did:
>
> Created
>
> /System/Flush/Flush period : TID_UINT32 /System/Flush/Last flush : TID_UINT32
>
> which control the flushing to disk. The default value for “Flush period” is 60
seconds or one minute.
>
> All clients call db_flush_database() through their cm_yield() function
> db_flush_database() checks the “Last flush” and only flushes the ODB when the
period has expired. This test is
> done inside the ODB semaphore so that we don’t get a race condigiton
> If the period has expired, db_flush_database() calls ss_shm_flush()
> ss_shm_flush() tries to allocate a buffer of the shared memory. If the
allocation is not successful (out of
> memory), ss_shm_flush() writes directly to the binary file as before.
> If the allocation is successful, ss_shm_flush() copies the share memory to a
buffer and passes this buffer to a
> dedicated thread which writes the buffer to the binary file. This causes
ss_shm_flush() to return immediately and
> not block the calling program during the disk write operation.
> Added back the “if (destroy_flag) ss_shm_flush()” so that the ODB is flushed
for sure before the shared memory
> gets deleted.
> This means now that under normal circumstances, exiting programs like odbedit
do NOT flush the ODB. This allows to
> call many “odbedit -c” in a row without the flush penalty. Nevertheless, the
ODB then gets flushed by other
> clients latest 60 seconds (or whatever the flush period is) after odbedit
exits.
>
> Please note that ODB flushing has two purposes:
>
> When all programs exit, we need a persistent storage for the ODB. In most
experiments this only happens very
> seldom. Maybe at the end of a beam time period.
> If the computer crashes, a recent version of the ODB is kept on disk to
simplify recovery after the crash.
> Since crashes are not so often (during production periods we have maybe one
hardware failure every few years) the
> flushing of the ODB too often does not make sense and just consumes resources.
Flushing does also not help from
> corrupted ODBs, since the binary image will also get corrupted. So the only
reason for periodic flushes is to ease
> recovery after a total crash. I put the default to 60 seconds, but if people
are really paranoid they can decrease
> it to 10 seconds or so. Or increase it to 600 seconds if their system does not
crash every week and disks are
> slow.
>
> I made a dedicated branch feature/periodic_odb_flush so people can test the
new functionality. If there are no
> complaints within the next few days, I will merge that into develop.
>
> Stefan |
28 Jun 2019, Thorsten Lux, Bug Report, Status page reloads every second
|
Hello,
We observed a strange behavior, from our point of view:
After some issues with with a 100% full database and recovering from this by
creating a new odb file from a previous copy, the Midas status page started to
reload/refresh every second while on the other hand it solved the 100% full
issue.
There are no error messages and the pages looks like normal but it is impossible
to start a new run due to the permanent reloading.
It is possible to click for example on odb and check the settings.
Any idea what could be the problem and what the solution?
Thanks
Thorsten |
28 Jun 2019, Konstantin Olchanski, Bug Report, Status page reloads every second
|
> We observed a strange behavior, from our point of view:
> ... the Midas status page started to reload/refresh every second
What version of midas is this? Run the odbedit "ver" command please. Also which
browser on what OS is this? (chrome->about google chrome, firefox->about firefox).
The current versions of midas do not reload the status page ever, and I think
all the page-reload code has been removed and they cannot reload automatically.
Old versions of the midas status page were designed to reload every 60 seconds or so.
The reload interval is adjustable, but I do not think it was stored in ODB. It was
accessed from the status page "config" button and I think it stored the reload
period is a browser cookie.
This reload value may have gotten confused, and in this case, to fix it,
you can try to clear all the web cookies from the web page. Another test for this
would be to try an alternate web browser, which would presumable not have the bad cookie
and will not suffer from the reload problem.
You can also open the web page debugger (google chrome -> right click menu -> inspect ->>
console & etc) and see if anything shows up there. I think you can set a break point
on the page reload function and catch the place that causes the reload.
K.O. |
29 Jun 2019, Thorsten Lux, Bug Report, Status page reloads every second
|
I am sorry, yesterday evening I must have been a bit tired after a long day with a lot of
problems and error messages, so that I did not realize that yes, the frontend was finally
starting well again but by recovering the odb file from an old one, it was stuck in the
transition "stopping run" and this caused the continuous reloading of the status page.
A "obdedit -C stop" solved the problem.
Sorry for this!
> We observed a strange behavior, from our point of view:
> > ... the Midas status page started to reload/refresh every second
>
> What version of midas is this? Run the odbedit "ver" command please. Also which
> browser on what OS is this? (chrome->about google chrome, firefox->about firefox).
>
> The current versions of midas do not reload the status page ever, and I think
> all the page-reload code has been removed and they cannot reload automatically.
>
> Old versions of the midas status page were designed to reload every 60 seconds or so.
> The reload interval is adjustable, but I do not think it was stored in ODB. It was
> accessed from the status page "config" button and I think it stored the reload
> period is a browser cookie.
>
> This reload value may have gotten confused, and in this case, to fix it,
> you can try to clear all the web cookies from the web page. Another test for this
> would be to try an alternate web browser, which would presumable not have the bad cookie
> and will not suffer from the reload problem.
>
> You can also open the web page debugger (google chrome -> right click menu -> inspect ->>
> console & etc) and see if anything shows up there. I think you can set a break point
> on the page reload function and catch the place that causes the reload.
>
> K.O. |
03 Mar 2015, Zaher Salman, Forum, Starting program from custom page
|
I am trying to start a program (fronend) from a custom page. What is the best
way to do that? Would ODBRpc() do this? if so can anyone give me an example of
how to do this. Thanks. |
03 Mar 2015, Stefan Ritt, Forum, Starting program from custom page
|
> I am trying to start a program (fronend) from a custom page. What is the best
> way to do that? Would ODBRpc() do this? if so can anyone give me an example of
> how to do this. Thanks.
You have a look at the documentation:
http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/RC_mhttpd_defining_script_buttons.html
Cheers,
Stefan |
03 Mar 2015, Zaher Salman, Forum, Starting program from custom page
|
> > I am trying to start a program (fronend) from a custom page. What is the best
> > way to do that? Would ODBRpc() do this? if so can anyone give me an example of
> > how to do this. Thanks.
>
> You have a look at the documentation:
>
> http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/RC_mhttpd_defining_script_buttons.html
>
> Cheers,
> Stefan
Hi Stefan, thanks for the quick reply. I guess my question was not clear enough.
My aim is to create a button which mimics the "Start/Stop" button functionality in the
"Programs" page where we start all the front-ends for the various equipment. The idea is that
the user will use a simple interface in a custom page (not the status page) which sets up the
equipment needed for a specific type of measurement.
thanks
Zaher |
03 Mar 2015, Stefan Ritt, Forum, Starting program from custom page
|
> Hi Stefan, thanks for the quick reply. I guess my question was not clear enough.
>
> My aim is to create a button which mimics the "Start/Stop" button functionality in the
> "Programs" page where we start all the front-ends for the various equipment. The idea is that
> the user will use a simple interface in a custom page (not the status page) which sets up the
> equipment needed for a specific type of measurement.
All functions in midas are controlled through special URLs. So the URL
http://<host:port>/?cmd=Start&value=10
will start run #10. Similarly with ?cmd=Stop. Now all you need is to set up a custom button, and use the
OnClick="" JavaScript method to fire off an Ajax request with the above URL.
To send an Ajax request, you can use the function XMLHttpRequestGeneric which ships as part of midas in the
mhttpd.js file. Then the code would be
<input type="button" onclick="start()">
and in your JavaScript code:
...
function start()
{
var request = XMLHttpRequestGeneric();
url = '?cmd=Start&value=10';
request.open('GET', url, false);
request.send(null);
}
...
Cheers,
Stefan |
03 Mar 2015, Zaher Salman, Forum, Starting program from custom page
|
Thank you very much, this is exactly what I need and it works.
Zaher
> All functions in midas are controlled through special URLs. So the URL
>
> http://<host:port>/?cmd=Start&value=10
>
> will start run #10. Similarly with ?cmd=Stop. Now all you need is to set up a custom button, and use the
> OnClick="" JavaScript method to fire off an Ajax request with the above URL.
>
> To send an Ajax request, you can use the function XMLHttpRequestGeneric which ships as part of midas in the
> mhttpd.js file. Then the code would be
>
> <input type="button" onclick="start()">
>
> and in your JavaScript code:
>
> ...
> function start()
> {
> var request = XMLHttpRequestGeneric();
>
> url = '?cmd=Start&value=10';
> request.open('GET', url, false);
> request.send(null);
> }
> ...
>
>
> Cheers,
> Stefan |
|