ID |
Date |
Author |
Topic |
Subject |
2754
|
03 May 2024 |
Stefan Ritt | Suggestion | Possible addition to IF Statements | The tinyexpr library I use to evaluate expressions does not support boolean operations. I would have to switch to the newer
tineyexpr-plusplus version, which also has much richer functionality:
https://github.com/Blake-Madden/tinyexpr-plusplus/blob/tinyexpr%2B%2B/TinyExprChanges.md
Unfortunately it requires C++17, and at the moment we limit MIDAS to C++11, meaning we would break this requirement. I
believe at the moment there are still some experiments (mainly at TRIUMF) which are stuck to older OS and therefore cannot
switch to C++17, but hopefully this will change over time.
Stefan |
2753
|
03 May 2024 |
Thomas Senger | Suggestion | Possible addition to IF Statements | Hello there,
in our setup we use many variables with many different exceptions. Would it be possible to implement something like an
IF or/and IF statement? I believe that this is currently not possible.
Best regards,
Thomas Senger |
2752
|
03 May 2024 |
Luigi Vigani | Bug Report | Params not initialized when starting sequencer | It is pretty much the same as Stefan, I attach here the screenshots. Also in my case it works sometimes, and sometimes partially (one or 2 params, like in
attachment 3).
> Could you please export and send me the /Sequencer ODB tree (or just /Sequencer/Param and /Sequencer/Variables) in both cases while the sequence is running.
>
> thanks,
> Zaher
>
>
> > Good afternoon,
> >
> > After updating Midas to the latest develop commit
> > (0f5436d901a1dfaf6da2b94e2d87f870e3611cf1) we found out a bug when starting
> > sequencer. If we have a simple loop from start value to stop value and step
> > size, just printing the value at each iteration, we see everything good (see
> > first attachment). Then we included another script though, which contains
> > several subroutines we defined for our detector, and we try to run the same
> > script. Unfortunately after this the parameters seem uninitialized, and the
> > value at each loop does not make sense (see second attachment). Also, sometimes
> > when pressing run the set parameter window would pop-up, but sometimes not.
> >
> > The script is this one:
> >
> > >>>
> > COMMENT Test script to check for a specific bug
> >
> > INCLUDE global_basic_functions
> >
> > #CALL setup_paths
> > #CALL generate_DUT_params
> >
> > PARAM lv_start, "Start of LV", 1.8
> > PARAM lv_stop, "Stop of LV", 2.1
> > PARAM lv_step, "Step of LV", 0.02
> >
> > n_iterations = (($lv_stop - $lv_start)/$lv_step)
> >
> > MSG "Parameters:"
> > MSG $lv_start
> > MSG $lv_stop
> > MSG $lv_step
> > MSG $n_iterations
> >
> > MSG "Start of looping"
> >
> > LOOP n, $n_iterations
> > lv_now = $lv_start + $n * $lv_step
> > MSG $lv_now
> > WAIT SECONDS, 1
> > ENDLOOP
> > <<<
> >
> > and the only difference comes from commenting the line:
> >
> > >>>
> > INCLUDE global_basic_functions
> > <<<
> >
> > as global_basic_functions is defined as a LIBRARY and it includes 75 (!)
> > subroutines...
> >
> > Is it possible that when loading a large script it messes up the loading of
> > parameters?
> >
> > Thank you very much,
> > Regards,
> > Luigi. |
2751
|
03 May 2024 |
Stefan Ritt | Bug Report | Params not initialized when starting sequencer | Ok, here is the complete code to reproduce the problem. Load parameter_test.msl which includes functions.msl. From the screenshot you see the variables containing
garbage, and you also see that from the ODB screenshot. For completeness, I added Sequencer.json which contains the whole sequencer tree.
The interesting thing is that this works sometimes, and sometimes not. I'm not sure if this in the GUI or in the sequencer program, so we have to sort out who can
fix it ;-)
Best,
Stefan |
2750
|
03 May 2024 |
Zaher Salman | Bug Report | Params not initialized when starting sequencer | Could you please export and send me the /Sequencer ODB tree (or just /Sequencer/Param and /Sequencer/Variables) in both cases while the sequence is running.
thanks,
Zaher
> Good afternoon,
>
> After updating Midas to the latest develop commit
> (0f5436d901a1dfaf6da2b94e2d87f870e3611cf1) we found out a bug when starting
> sequencer. If we have a simple loop from start value to stop value and step
> size, just printing the value at each iteration, we see everything good (see
> first attachment). Then we included another script though, which contains
> several subroutines we defined for our detector, and we try to run the same
> script. Unfortunately after this the parameters seem uninitialized, and the
> value at each loop does not make sense (see second attachment). Also, sometimes
> when pressing run the set parameter window would pop-up, but sometimes not.
>
> The script is this one:
>
> >>>
> COMMENT Test script to check for a specific bug
>
> INCLUDE global_basic_functions
>
> #CALL setup_paths
> #CALL generate_DUT_params
>
> PARAM lv_start, "Start of LV", 1.8
> PARAM lv_stop, "Stop of LV", 2.1
> PARAM lv_step, "Step of LV", 0.02
>
> n_iterations = (($lv_stop - $lv_start)/$lv_step)
>
> MSG "Parameters:"
> MSG $lv_start
> MSG $lv_stop
> MSG $lv_step
> MSG $n_iterations
>
> MSG "Start of looping"
>
> LOOP n, $n_iterations
> lv_now = $lv_start + $n * $lv_step
> MSG $lv_now
> WAIT SECONDS, 1
> ENDLOOP
> <<<
>
> and the only difference comes from commenting the line:
>
> >>>
> INCLUDE global_basic_functions
> <<<
>
> as global_basic_functions is defined as a LIBRARY and it includes 75 (!)
> subroutines...
>
> Is it possible that when loading a large script it messes up the loading of
> parameters?
>
> Thank you very much,
> Regards,
> Luigi. |
2749
|
02 May 2024 |
Scott Oser | Forum | Midas Sequencer with less than 1 second wait | > Ok, I implemented the float second wait function. Internally it works in ms, so the maximum resolution is 0.001 s.
>
> Best,
> Stefan
Thank you, we will test this soon and let you know if we see any issues (but we're not expecting any). |
2748
|
02 May 2024 |
Stefan Ritt | Forum | Midas Sequencer with less than 1 second wait | Ok, I implemented the float second wait function. Internally it works in ms, so the maximum resolution is 0.001 s.
Best,
Stefan |
2747
|
30 Apr 2024 |
Scott Oser | Forum | Midas Sequencer with less than 1 second wait | > While I will do it, i'm not sure if this is what you want. If I understand correctly, some process gets triggered and then writes some values to the ODB, then the sequencer
> should continue. Putting a wait there is dangerous. Maybe your process always takes like 10-20 ms, so you put a wait of let's say 100ms, and things are fine with you. Your
> script runs many days, but then once in a while your machine is on heavy load because someone starts a web browser, and your process takes 110ms, and you script crashes.
>
> I would rather go following path: put a "done" flag in the ODB, which is the last one which gets set by your process. Then the sequencer does a
>
> WAIT ODBvalue, /path/value, =, 1
>
> which will work always, independend of the delay of your process.
>
> Stefan
Our use case is pretty simple and I don't think is affected by the scenario you envision. We want to turn on a setting in our equipment, and turn it off again some 0.2 s later. We don't need msec timing. So something like:
ODBSET /somekey 1 # this will cause a front-end to flip a bit in our hardware
WAIT seconds, 0.2
ODBSET /somekey 0 # this will cause a front-end to reset a bit in our hardware
It is true that if the load is high there could be a little delay, and the time that the bit is set will not be 0.2 seconds, but on average it should work,
and it should be good enough we think.
Yes, we could also check an ODB key to see that something is done, but we'd still need the ability to wait for time intervals less than 1 second, which
right now doesn't exist. |
2746
|
30 Apr 2024 |
Stefan Ritt | Forum | Midas Sequencer with less than 1 second wait | While I will do it, i'm not sure if this is what you want. If I understand correctly, some process gets triggered and then writes some values to the ODB, then the sequencer
should continue. Putting a wait there is dangerous. Maybe your process always takes like 10-20 ms, so you put a wait of let's say 100ms, and things are fine with you. Your
script runs many days, but then once in a while your machine is on heavy load because someone starts a web browser, and your process takes 110ms, and you script crashes.
I would rather go following path: put a "done" flag in the ODB, which is the last one which gets set by your process. Then the sequencer does a
WAIT ODBvalue, /path/value, =, 1
which will work always, independend of the delay of your process.
Stefan |
2745
|
30 Apr 2024 |
Scott Oser | Forum | Midas Sequencer with less than 1 second wait | > > This would work fine in principle, but isn't implemented in the current MIDAS sequencer as we understand it. (We tried!) Is your proposal to rewrite the sequencer
> > to allow fractional waits? Right now the code seems to store the start_time as a DWORD and uses atoi to parse the wait time, and uses ss_time (which seems only get
> > the time to the nearest second) to fetch the time.
>
> No it's not implemented, was just my idea. If it would work for you, I can implement it in the next couple of days.
>
> Stefan
Yes, please! Something like WAIT seconds, 0.01 would be perfect. |
2744
|
30 Apr 2024 |
Stefan Ritt | Forum | Midas Sequencer with less than 1 second wait | > This would work fine in principle, but isn't implemented in the current MIDAS sequencer as we understand it. (We tried!) Is your proposal to rewrite the sequencer
> to allow fractional waits? Right now the code seems to store the start_time as a DWORD and uses atoi to parse the wait time, and uses ss_time (which seems only get
> the time to the nearest second) to fetch the time.
No it's not implemented, was just my idea. If it would work for you, I can implement it in the next couple of days.
Stefan |
2743
|
30 Apr 2024 |
Scott Oser | Forum | Midas Sequencer with less than 1 second wait | > I guess the simplest way to do that without breaking with existing code is to change the
> second number to a float. So a
>
> WAIT SECONDS, 1
>
> will still work, and you can then write
>
> WAIT SECONDS, 0.01
>
> to get a 10 ms wait. Would that work for you?
This would work fine in principle, but isn't implemented in the current MIDAS sequencer as we understand it. (We tried!) Is your proposal to rewrite the sequencer
to allow fractional waits? Right now the code seems to store the start_time as a DWORD and uses atoi to parse the wait time, and uses ss_time (which seems only get
the time to the nearest second) to fetch the time. |
2742
|
30 Apr 2024 |
Luigi Vigani | Bug Report | Params not initialized when starting sequencer | Good afternoon,
After updating Midas to the latest develop commit
(0f5436d901a1dfaf6da2b94e2d87f870e3611cf1) we found out a bug when starting
sequencer. If we have a simple loop from start value to stop value and step
size, just printing the value at each iteration, we see everything good (see
first attachment). Then we included another script though, which contains
several subroutines we defined for our detector, and we try to run the same
script. Unfortunately after this the parameters seem uninitialized, and the
value at each loop does not make sense (see second attachment). Also, sometimes
when pressing run the set parameter window would pop-up, but sometimes not.
The script is this one:
>>>
COMMENT Test script to check for a specific bug
INCLUDE global_basic_functions
#CALL setup_paths
#CALL generate_DUT_params
PARAM lv_start, "Start of LV", 1.8
PARAM lv_stop, "Stop of LV", 2.1
PARAM lv_step, "Step of LV", 0.02
n_iterations = (($lv_stop - $lv_start)/$lv_step)
MSG "Parameters:"
MSG $lv_start
MSG $lv_stop
MSG $lv_step
MSG $n_iterations
MSG "Start of looping"
LOOP n, $n_iterations
lv_now = $lv_start + $n * $lv_step
MSG $lv_now
WAIT SECONDS, 1
ENDLOOP
<<<
and the only difference comes from commenting the line:
>>>
INCLUDE global_basic_functions
<<<
as global_basic_functions is defined as a LIBRARY and it includes 75 (!)
subroutines...
Is it possible that when loading a large script it messes up the loading of
parameters?
Thank you very much,
Regards,
Luigi. |
2741
|
29 Apr 2024 |
Stefan Ritt | Forum | Midas Sequencer with less than 1 second wait | I guess the simplest way to do that without breaking with existing code is to change the
second number to a float. So a
WAIT SECONDS, 1
will still work, and you can then write
WAIT SECONDS, 0.01
to get a 10 ms wait. Would that work for you?
Stefan |
2740
|
29 Apr 2024 |
Musaab Al-Bakry | Forum | Midas Sequencer with less than 1 second wait | Hello there,
I am working on a task that involves some ODB changes that happen within 20-500
ms. The wait command for Midas Sequencer only works for > 1 second. As a
workaround, I tried calling a python script that has a time.sleep() command, but
the sequencer doesn't wait for the python script to terminate before moving to the
next command. Obviously, I could just move the entire script to python, but that
would cause further issues to us. Is there a way to have a wait that has precision
in order of milliseconds from within the Midas Sequencer? If there is no Midas-
native methods for doing this, what workaround will you suggest to get this to
work? |
2739
|
24 Apr 2024 |
Konstantin Olchanski | Info | MIDAS RPC add support for std::string and std::vector<char> | I now fully understand the MIDAS RPC code, had to add some debugging printfs,
write some test code (odbedit test_rpc), catch and fix a few bugs.
Fixes for the bugs are now committed.
Small refactor of rpc_execute() should be committed soon, this removes the
"goto" in the memory allocation of output buffer. Stefan's original code used a
fixed size buffer, I later added allocation "as-neeed" but did not fully
understand everything and implemented it as "if buffer too small, make it
bigger, goto start over again".
After that, I can implement support for std::string and std::vector<char>.
The way it looks right now, the on-the-wire data format is flexible enough to
make this change backward-compatible and allow MIDAS programs built with old
MIDAS to continue connecting to the new MIDAS and vice-versa.
MIDAS RPC support for std::string should let us improve security by removing
even more uses of fixed-size string buffers.
Support for std::vector<char> will allow removal of last places where
MAX_EVENT_SIZE is used and simplify memory allocation in other "give me data"
RPC calls, like RPC_JRPC and RPC_BRPC.
K.O. |
2738
|
24 Apr 2024 |
Konstantin Olchanski | Info | MIDAS RPC data format | > 4.5) TID_IN32|RPC_VARARRAY does not work, corrupts following parameters. MIDAS only uses TID_ARRAY|RPC_VARARRAY
fixed in commit 0f5436d901a1dfaf6da2b94e2d87f870e3611cf1, TID_ARRAY|RPC_VARARRAY was okey (i.e. db_get_value()), bug happened only if rpc_tid_size()
is not zero.
>
> 4.6) TID_STRING|RPC_IN|RPC_OUT does not seem to work.
>
> 4.7) RPC_VARARRAY does not work is there is preceding TID_STRING|RPC_OUT that returned a short string. memmove() moves stuff in the send buffer,
> this makes prpc_param[] pointers into the send buffer invalid. subsequent RPC_VARARRAY parameter refers to now-invalid prpc_param[i] pointer to
> get param_size and gets the wrong value. MIDAS does not use this sequence of RPC parameters.
>
> 4.8) same bug is in the processing of TID_STRING|RPC_OUT parameters, where it refers to invalid prpc_param[i] to get the string length.
fixed in commits e45de5a8fa81c75e826a6a940f053c0794c962f5 and dc08fe8425c7d7bfea32540592b2c3aec5bead9f
K.O. |
2737
|
15 Apr 2024 |
Stefan Ritt | Bug Report | open MIDAS RPC ports | One thing coming to my mind is the interface binding. If you have a midas host with two networks
("global" and "local"=192.168.x.x), you can tell to which interface a socket should bind.
By default it binds both interfaces, but we could restrict the socket only to bind to the local
interface 192.168.x.x. This way the open port would be invisible from the outside, but from
your local network everybody can connect easily without the need of a white list.
Stefan |
2736
|
15 Apr 2024 |
Konstantin Olchanski | Bug Report | open MIDAS RPC ports | we had a bit of trouble with open network ports recently and I now think security of MIDAS RPC
ports needs to be tightened.
TL;DR, this is a non-trivial network configuration problem, TL required, DR up to you.
as background, right now we have two settings in ODB, "/expt/security/enable non-localhost
RPC" set to "no" (the default) and set to "yes". Set to "no" is very secure, all RPC sockets
listen only on the "localhost" interface (127.0.0.1) and do not accept connections from other
computers. Set to "yes", RPC sockets accept connections from everywhere in the world, but
immediately close them without reading any data unless connection origins are listed in ODB
"/expt/security/RPC hosts" (white-listed).
the problem, one. for security and robustness we place most equipments on a private network
(192.168.1.x). MIDAS frontends running on these equipments must connect to MIDAS running on
the main computer. This requires setting "enable non-localhost RPC" to "yes" and white-listing
all private network equipments. so far so good.
the problem, one, continued. in this configuration, the MIDAS main computer is usually also
the network gateway (with NAT, IP forwarding, DHCP, DNS, etc). so now MIDAS RPC ports are open
to all external connections (in the absence of restrictive firewall rules). one would hope for
security-through-obscurity and expect that "external threat actors" will try to bother them,
but in reality we eventually see large numbers of rejected unwanted connections logged in
midas.log (we log the first 10 rejected connections to help with maintaining the RPC
connections white-list).
the problem, two. central IT do not like open network ports. they run their scanners, discover
the MIDAS RPC ports, complain about them, require lengthy explanations, etc.
it would be much better if in the typical configuration, MIDAS RPC ports did not listen on
external interfaces (campus network). only listen on localhost and on private network
interfaces (192.168.1.x).
I am not yet of the simplest way to implement this. But I think this is the direction we
should go.
P.S. what about firewall rules? two problems: one: from statistic-of-one, I make mistakes
writing firewall rules, others also will make mistakes, a literally fool-proof protection of
MIDAS RPC ports is needed. two: RHEL-derived Linuxes by-default have restrictive firewall
rules, and this is good for security, except that there is a failure mode where at boot time
something can go wrong and firewall rules are not loaded at all. we have seen this happen.
this is a complete disaster on a system that depends on firewall rules for security. better to
have secure applications (TCP ports protected by design and by app internals) with firewall
rules providing a secondary layer of protection.
P.P.S. what about MIDAS frontend initial connection to the mserver? this is currently very
insecure, but the vulnerability window is very small. Ideally we should rework the mserver
connection to make it simpler, more secure and compatible with SSH tunneling.
P.P.S. Typical network diagram:
internet - campus firewall - campus network - MIDAS host (MIDAS) - 192.168.1.x network - power
supplies, digitizers, MIDAS frontends.
P.P.S. mserver connection sequence:
1) midas frontend opens 3 tcp sockets, connections permitted from anywhere
2) midas frontend opens tcp socket to main mserver, sends port numbers of the 3 tcp sockets
3) main mserver forks out a secondary (per-client) mserver
4) secondary mserver connects to the 3 tcp sockets of the midas frontend created in (1)
5) from here midas rpc works
6) midas frontend loads the RPC white-list
7) from here MIDAS RPC sockets are secure (protected by the white-list).
(the 3 sockets are: RPC recv_sock, RPC send_sock and event_sock)
P.P.S. MIDAS UDP sockets used for event buffer and odb notifications are secure, they bind to
localhost interface and do not accept external connections.
K.O. |
2735
|
04 Apr 2024 |
Konstantin Olchanski | Info | MIDAS RPC data format | I am not sure I have seen this documented before. MIDAS RPC data format.
1) RPC request (from client to mserver), in rpc_call_encode()
1.1) header:
4 bytes NET_COMMAND.header.routine_id is the RPC routine ID
4 bytes NET_COMMAND.header.param_size is the size of following data, aligned to 8 bytes
1.2) followed by values of RPC_IN parameters:
arg_size is the actual data size
param_size = ALIGN8(arg_size)
for TID_STRING||TID_LINK, arg_size = 1+strlen()
for TID_STRUCT||RPC_FIXARRAY, arg_size is taken from RPC_LIST.param[i].n
for RPC_VARARRAY|RPC_OUT, arg_size is pointed to by the next argument
for RPC_VARARRAY, arg_size is the value of the next argument
otherwise arg_size = rpc_tid_size()
data encoding:
RPC_VARARRAY:
4 bytes of ALIGN8(arg_size)
4 bytes of padding
param_size bytes of data
TID_STRING||TID_LINK:
param_size of string data, zero terminated
otherwise:
param_size of data
2) RPC dispatch in rpc_execute
for each parameter, a pointer is placed into prpc_param[i]:
RPC_IN: points to the data inside the receive buffer
RPC_OUT: points to the data buffer allocated inside the send buffer
RPC_IN|RPC_OUT: data is copied from the receive buffer to the send buffer, prpc_param[i] is a pointer to the copy in the send buffer
prpc_param[] is passed to the user handler function.
user function reads RPC_IN parameters by using the CSTRING(i), etc macros to dereference prpc_param[i]
user function modifies RPC_IN|RPC_OUT parameters pointed to by prpc_param[i] (directly in the send buffer)
user function places RPC_OUT data directly to the send buffer pointed to by prpc_param[i]
size of RPC_VARARRAY|RPC_OUT data should be written into the next/following parameter.
3) RPC reply
3.1) header:
4 bytes NET_COMMAND.header.routine_id contains the value returned by the user function (RPC_SUCCESS)
4 bytes NET_COMMAND.header.param_size is the size of following data aligned to 8 bytes
3.2) followed by data for RPC_OUT parameters:
data sizes and encodings are the same as for RPC_IN parameters.
for variable-size RPC_OUT parameters, space is allocated in the send buffer according to the maximum data size
that the user code expects to receive:
RPC_VARARRAY||TID_STRING: max_size is taken from the first 4 bytes of the *next* parameter
otherwise: max_size is same as arg_size and param_size.
when encoding and sending RPC_VARARRAY data, actual data size is taken from the next parameter, which is expected to be
TID_INT32|RPC_IN|RPC_OUT.
4) Notes:
4.1) RPC_VARARRAY should always be sent using two parameters:
a) RPC_VARARRAY|RPC_IN is pointer to the data we are sending, next parameter must be TID_INT32|RPC_IN is data size
b) RPC_VARARRAY|RPC_OUT is pointer to the data buffer for received data, next parameter must be TID_INT32|RPC_IN|RPC_OUT before the call should
contain maximum data size we expect to receive (size of malloc() buffer), after the call it may contain the actual data size returned
c) RPC_VARARRAY|RPC_IN|RPC_OUT is pointer to the data we are sending, next parameter must be TID_INT32|RPC_IN|RPC_OUT containing the maximum
data size we are expected to receive.
4.2) during dispatching, RPC_VARARRAY|RPC_OUT and TID_STRING|RPC_OUT both have 8 bytes of special header preceeding the actual data, 4 bytes of
maximum data size and 4 bytes of padding. prpc_param[] points to the actual data and user does not see this special header.
4.3) when encoding outgoing data, this special 8 byte header is removed from TID_STRING|RPC_OUT parameters using memmove().
4.4) TID_STRING parameters:
TID_STRING|RPC_IN can be sent using oe parameter
TID_STRING|RPC_OUT must be sent using two parameters, second parameter should be TID_INT32|RPC_IN to specify maximum returned string length
TID_STRING|RPC_IN|RPC_OUT ditto, but not used anywhere inside MIDAS
4.5) TID_IN32|RPC_VARARRAY does not work, corrupts following parameters. MIDAS only uses TID_ARRAY|RPC_VARARRAY
4.6) TID_STRING|RPC_IN|RPC_OUT does not seem to work.
4.7) RPC_VARARRAY does not work is there is preceding TID_STRING|RPC_OUT that returned a short string. memmove() moves stuff in the send buffer,
this makes prpc_param[] pointers into the send buffer invalid. subsequent RPC_VARARRAY parameter refers to now-invalid prpc_param[i] pointer to
get param_size and gets the wrong value. MIDAS does not use this sequence of RPC parameters.
4.8) same bug is in the processing of TID_STRING|RPC_OUT parameters, where it refers to invalid prpc_param[i] to get the string length.
K.O. |
|