ID |
Date |
Author |
Topic |
Subject |
2722
|
08 Mar 2024 |
Konstantin Olchanski | Info | MIDAS frontend for WIENER L.V. P.S. and VME crates | Our MIDAS frontend for WIENER power supplies is now available as a standalone git repository.
https://bitbucket.org/ttriumfdaq/fewienerlvps/src/master/
This frontend use the snmpwalk and snmpset programs to talk to the power supply.
Also included is a simple custom web page to display power supply status and to turn things on and off.
This frontend was originally written for the T2K/ND280 experiment in Japan.
In addition to controlling Wiener low voltage power supplies, it was also used to control the ISEG MPOD high
voltage power supplies.
In Japan, ISEG MPOD was (still is) connected to the MicroMegas TPC and is operated in a special "spark counting"
mode. This spark counting code is still present in this MIDAS frontend and can be restored with a small amount of
work.
K.o. |
2731
|
01 Apr 2024 |
Konstantin Olchanski | Info | xz-utils bomb out, compression benchmarks | you may have heard the news of a major problem with the xz-utils project, authors of the popular "xz" file compression,
https://nvd.nist.gov/vuln/detail/CVE-2024-3094
the debian bug tracker is interesting reading on this topic, "750 commits or contributions to xz by Jia Tan, who backdoored it",
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024
and apparently there is problems with the deisng of the .xz file format, making it vulnerable to single-bit errors and unreliable checksums,
https://www.nongnu.org/lzip/xz_inadequate.html
this moved me to review status of file compression in MIDAS.
MIDAS does not use or recommend xz compression, MIDAS programs to not link to xz and lzma libraries provided by xz-utils.
mlogger has built-in support for:
- gzip-1, enabled by default, as the most safe and bog-standard compression method
- bzip2 and pbzip2, as providing the best compression
- lz4, for high data rate situations where gzip and bzip2 cannot keep up with the data
compression benchmarks on an AMD 7700 CPU (8-core, DDR5 RAM) confirm the usual speed-vs-compression tradeoff:
note: observe how both lz4 and pbzip2 compress time is the time it takes to read the file from ZFS cache, around 6 seconds.
note: decompression stacks up in the same order: lz4, gzip fastest, pbzip2 same speed using 10x CPU, bzip2 10x slower uses 1 CPU.
note: because of the fast decompression speed, gzip remains competitive.
no compression: 6 sec, 270 MiBytes/sec,
lz4, bpzip2: 6 sec, same, (pbzip2 uses 10 CPU vs lz4 uses 1 CPU)
gzip -1: 21 sec, 78 MiBytes/sec
bzip2: 70 sec, 23 MiBytes/sec (same speed as pbzip2, but using 1 CPU instead of 10 CPU)
file sizes:
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ ls -lSr test.mid*
-rw-r--r-- 1 dsdaqdev users 483319523 Apr 1 14:06 test.mid.bz2
-rw-r--r-- 1 dsdaqdev users 631575929 Apr 1 14:06 test.mid.gz
-rw-r--r-- 1 dsdaqdev users 1002432717 Apr 1 14:06 test.mid.lz4
-rw-r--r-- 1 dsdaqdev users 1729327169 Apr 1 14:06 test.mid
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$
actual benchmarks:
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time cat test.mid > /dev/null
0.00user 6.00system 0:06.00elapsed 99%CPU (0avgtext+0avgdata 1408maxresident)k
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time gzip -1 -k test.mid
14.70user 6.42system 0:21.14elapsed 99%CPU (0avgtext+0avgdata 1664maxresident)k
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time lz4 -k -f test.mid
2.90user 6.44system 0:09.39elapsed 99%CPU (0avgtext+0avgdata 7680maxresident)k
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time bzip2 -k -f test.mid
64.76user 8.81system 1:13.59elapsed 99%CPU (0avgtext+0avgdata 8448maxresident)k
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time pbzip2 -k -f test.mid
86.76user 15.39system 0:09.07elapsed 1125%CPU (0avgtext+0avgdata 114596maxresident)k
decompression benchmarks:
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time lz4cat test.mid.lz4 > /dev/null
0.68user 0.23system 0:00.91elapsed 99%CPU (0avgtext+0avgdata 7680maxresident)k
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time zcat test.mid.gz > /dev/null
6.61user 0.23system 0:06.85elapsed 99%CPU (0avgtext+0avgdata 1408maxresident)k
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time bzcat test.mid.bz2 > /dev/null
27.99user 1.59system 0:29.58elapsed 99%CPU (0avgtext+0avgdata 4656maxresident)k
(vslice) dsdaqdev@dsdaqgw:/zdata/vslice$ /usr/bin/time pbzip2 -dc test.mid.bz2 > /dev/null
37.32user 0.56system 0:02.75elapsed 1377%CPU (0avgtext+0avgdata 157036maxresident)k
K.O. |
2732
|
02 Apr 2024 |
Zaher Salman | Info | Sequencer editor | Dear all,
Stefan and I have been working on improving the sequencer editor to make it look and feel more like a standard editor. This sequencer v2 has been finally merged into the develop branch earlier today.
The sequencer page has now a main tab which is used as a "console" to show the loaded sequence and it's progress when running. All other tabs are used only for editing scripts. To edit a currently loaded sequence simply double click on the editing area of the main tab or load the file in a new tab. A couple of screen shots of the new editor are attached.
For those who would like to stay with the older sequencer version a bit longer, you may simply copy resources/sequencer_v1.html to resources/sequencer.html. However, this version is not being actively maintained and may become obsolete at some point. Please help us improve the new version instead by reporting bugs and feature requests on bitbucket or here.
Best regards,
Zaher
|
2733
|
02 Apr 2024 |
Konstantin Olchanski | Info | Sequencer editor | > Stefan and I have been working on improving the sequencer editor ...
Looks grand! Congratulations with getting it completed. The previous version was
my rewrite of the old generated-C pages into html+javascript, nothing to write
home about, I even kept the 1990-ies-style html formatting and styling as much as
possible.
K.O. |
2735
|
04 Apr 2024 |
Konstantin Olchanski | Info | MIDAS RPC data format | I am not sure I have seen this documented before. MIDAS RPC data format.
1) RPC request (from client to mserver), in rpc_call_encode()
1.1) header:
4 bytes NET_COMMAND.header.routine_id is the RPC routine ID
4 bytes NET_COMMAND.header.param_size is the size of following data, aligned to 8 bytes
1.2) followed by values of RPC_IN parameters:
arg_size is the actual data size
param_size = ALIGN8(arg_size)
for TID_STRING||TID_LINK, arg_size = 1+strlen()
for TID_STRUCT||RPC_FIXARRAY, arg_size is taken from RPC_LIST.param[i].n
for RPC_VARARRAY|RPC_OUT, arg_size is pointed to by the next argument
for RPC_VARARRAY, arg_size is the value of the next argument
otherwise arg_size = rpc_tid_size()
data encoding:
RPC_VARARRAY:
4 bytes of ALIGN8(arg_size)
4 bytes of padding
param_size bytes of data
TID_STRING||TID_LINK:
param_size of string data, zero terminated
otherwise:
param_size of data
2) RPC dispatch in rpc_execute
for each parameter, a pointer is placed into prpc_param[i]:
RPC_IN: points to the data inside the receive buffer
RPC_OUT: points to the data buffer allocated inside the send buffer
RPC_IN|RPC_OUT: data is copied from the receive buffer to the send buffer, prpc_param[i] is a pointer to the copy in the send buffer
prpc_param[] is passed to the user handler function.
user function reads RPC_IN parameters by using the CSTRING(i), etc macros to dereference prpc_param[i]
user function modifies RPC_IN|RPC_OUT parameters pointed to by prpc_param[i] (directly in the send buffer)
user function places RPC_OUT data directly to the send buffer pointed to by prpc_param[i]
size of RPC_VARARRAY|RPC_OUT data should be written into the next/following parameter.
3) RPC reply
3.1) header:
4 bytes NET_COMMAND.header.routine_id contains the value returned by the user function (RPC_SUCCESS)
4 bytes NET_COMMAND.header.param_size is the size of following data aligned to 8 bytes
3.2) followed by data for RPC_OUT parameters:
data sizes and encodings are the same as for RPC_IN parameters.
for variable-size RPC_OUT parameters, space is allocated in the send buffer according to the maximum data size
that the user code expects to receive:
RPC_VARARRAY||TID_STRING: max_size is taken from the first 4 bytes of the *next* parameter
otherwise: max_size is same as arg_size and param_size.
when encoding and sending RPC_VARARRAY data, actual data size is taken from the next parameter, which is expected to be
TID_INT32|RPC_IN|RPC_OUT.
4) Notes:
4.1) RPC_VARARRAY should always be sent using two parameters:
a) RPC_VARARRAY|RPC_IN is pointer to the data we are sending, next parameter must be TID_INT32|RPC_IN is data size
b) RPC_VARARRAY|RPC_OUT is pointer to the data buffer for received data, next parameter must be TID_INT32|RPC_IN|RPC_OUT before the call should
contain maximum data size we expect to receive (size of malloc() buffer), after the call it may contain the actual data size returned
c) RPC_VARARRAY|RPC_IN|RPC_OUT is pointer to the data we are sending, next parameter must be TID_INT32|RPC_IN|RPC_OUT containing the maximum
data size we are expected to receive.
4.2) during dispatching, RPC_VARARRAY|RPC_OUT and TID_STRING|RPC_OUT both have 8 bytes of special header preceeding the actual data, 4 bytes of
maximum data size and 4 bytes of padding. prpc_param[] points to the actual data and user does not see this special header.
4.3) when encoding outgoing data, this special 8 byte header is removed from TID_STRING|RPC_OUT parameters using memmove().
4.4) TID_STRING parameters:
TID_STRING|RPC_IN can be sent using oe parameter
TID_STRING|RPC_OUT must be sent using two parameters, second parameter should be TID_INT32|RPC_IN to specify maximum returned string length
TID_STRING|RPC_IN|RPC_OUT ditto, but not used anywhere inside MIDAS
4.5) TID_IN32|RPC_VARARRAY does not work, corrupts following parameters. MIDAS only uses TID_ARRAY|RPC_VARARRAY
4.6) TID_STRING|RPC_IN|RPC_OUT does not seem to work.
4.7) RPC_VARARRAY does not work is there is preceding TID_STRING|RPC_OUT that returned a short string. memmove() moves stuff in the send buffer,
this makes prpc_param[] pointers into the send buffer invalid. subsequent RPC_VARARRAY parameter refers to now-invalid prpc_param[i] pointer to
get param_size and gets the wrong value. MIDAS does not use this sequence of RPC parameters.
4.8) same bug is in the processing of TID_STRING|RPC_OUT parameters, where it refers to invalid prpc_param[i] to get the string length.
K.O. |
2738
|
24 Apr 2024 |
Konstantin Olchanski | Info | MIDAS RPC data format | > 4.5) TID_IN32|RPC_VARARRAY does not work, corrupts following parameters. MIDAS only uses TID_ARRAY|RPC_VARARRAY
fixed in commit 0f5436d901a1dfaf6da2b94e2d87f870e3611cf1, TID_ARRAY|RPC_VARARRAY was okey (i.e. db_get_value()), bug happened only if rpc_tid_size()
is not zero.
>
> 4.6) TID_STRING|RPC_IN|RPC_OUT does not seem to work.
>
> 4.7) RPC_VARARRAY does not work is there is preceding TID_STRING|RPC_OUT that returned a short string. memmove() moves stuff in the send buffer,
> this makes prpc_param[] pointers into the send buffer invalid. subsequent RPC_VARARRAY parameter refers to now-invalid prpc_param[i] pointer to
> get param_size and gets the wrong value. MIDAS does not use this sequence of RPC parameters.
>
> 4.8) same bug is in the processing of TID_STRING|RPC_OUT parameters, where it refers to invalid prpc_param[i] to get the string length.
fixed in commits e45de5a8fa81c75e826a6a940f053c0794c962f5 and dc08fe8425c7d7bfea32540592b2c3aec5bead9f
K.O. |
2739
|
24 Apr 2024 |
Konstantin Olchanski | Info | MIDAS RPC add support for std::string and std::vector<char> | I now fully understand the MIDAS RPC code, had to add some debugging printfs,
write some test code (odbedit test_rpc), catch and fix a few bugs.
Fixes for the bugs are now committed.
Small refactor of rpc_execute() should be committed soon, this removes the
"goto" in the memory allocation of output buffer. Stefan's original code used a
fixed size buffer, I later added allocation "as-neeed" but did not fully
understand everything and implemented it as "if buffer too small, make it
bigger, goto start over again".
After that, I can implement support for std::string and std::vector<char>.
The way it looks right now, the on-the-wire data format is flexible enough to
make this change backward-compatible and allow MIDAS programs built with old
MIDAS to continue connecting to the new MIDAS and vice-versa.
MIDAS RPC support for std::string should let us improve security by removing
even more uses of fixed-size string buffers.
Support for std::vector<char> will allow removal of last places where
MAX_EVENT_SIZE is used and simplify memory allocation in other "give me data"
RPC calls, like RPC_JRPC and RPC_BRPC.
K.O. |
2766
|
14 May 2024 |
Konstantin Olchanski | Info | ROOT v6.30.6 requires libtbb-dev | root_v6.30.06.Linux-ubuntu22.04-x86_64-gcc11.4 the libtbb-dev package.
This is a new requirement and it is not listed in the ROOT dependancies page (I left a note on the ROOT forum, hopefully it will be
fixed quickly). https://root.cern/install/dependencies/
Symptom is when starting ROOT, you get an error:
cling::DynamicLibraryManager::loadLibrary(): libtbb.so.12: cannot open shared object file: No such file or directory
and things do not work.
Fix is to:
apt install libtbb-dev
K.O. |
2780
|
24 May 2024 |
Konstantin Olchanski | Info | added ubuntu 22 arm64 cross-compilation | Ubuntu 22 has almost everything necessary to cross-build arm64 MIDAS frontends:
# apt install g++-12-aarch64-linux-gnu gcc-12-aarch64-linux-gnu-base libstdc++-12-dev-arm64-cross
$ aarch64-linux-gnu-gcc-12 -o ttcp.aarch64 ttcp.c -static
to cross-build MIDAS:
make arm64_remoteonly -j
run programs from $MIDASSYS/linux-arm64-remoteonly/bin
link frontends to libraries in $MIDASSYS/linux-arm64-remoteonly/lib
Ubuntu 22 do not provide an arm64 libz.a, as a workaround, I build a fake one. (we do not have HAVE_ZLIB anymore...). or you
can link to libz.a from your arm64 linux image, assuming include/zlib.h are compatible.
K.O. |
2781
|
29 May 2024 |
Konstantin Olchanski | Info | MIDAS RPC add support for std::string and std::vector<char> | This is moving slowly. I now have RPC caller side support for std::string and
std::vector<char>. RPC server side is next. K.O. |
2782
|
02 Jun 2024 |
Konstantin Olchanski | Info | MIDAS RPC data format | > MIDAS RPC data format.
> 3) RPC reply
> 3.1) header:
> 3.2) followed by data for RPC_OUT parameters:
>
> data sizes and encodings are the same as for RPC_IN parameters.
Correction:
RPC_VARARRAY data encoding for data returned by RPC is different from data sent to RPC:
4 bytes of arg_size (before 8-byte alignement), (for data sent to RPC, it's 4 bytes of param_size, after 8-byte alignment)
4 bytes of padding
param_size of data
K.O.
P.S. bug/discrepancy caught by GCC/LLVM address sanitizer. |
2803
|
08 Aug 2024 |
Stefan Ritt | Info | mana.cxx | We are considering to remove the analyzer framework mana.cxx from MIDAS. It
currently has some compiler warnings and we wonder if we should fix them which
would take some time or just remove the file. We have now to much more modern
analyzer frameworks "manalyzer" and "ROOTANA" which should be used instead.
Is anybody still using the mana.cxx framework?
/Stefan |
2810
|
23 Aug 2024 |
Stefan Ritt | Info | mana.cxx | Ok, no relevant complains so far, so I removed mana and rmana from the CMake build
process, but left the file mana.cxx still in the repository for educational
purposes ;-)
Stefan |
2811
|
25 Aug 2024 |
Adrian Fisher | Info | Help parsing scdms_v1 data? | Hi! I'm working on creating a ksy file to help with parsing some data, but I'm having trouble finding some information. Right now, I have it set up very rudimentary - it grabs the event header and then uses the data bank size to grab the size of the data, but then I'm needing additional padding after the data bank to reach the next event.
However, there's some irregularity in the "padding" between data banks that I haven't been able to find any documentation for. For some reason, after the data banks, there's sections of data of either 168 or 192 bytes, and it's seemingly arbitrary which size is used.
I'm just wondering if anyone has any information about this so that I'd be able to make some more progress in parsing the data.
The data I'm working with can be found at https://github.com/det-lab/dataReaderWriter/blob/master/data/07180808_1735_F0001.mid.gz
And the ksy file that I've created so far is at https://github.com/det-lab/dataReaderWriter/blob/master/kaitai/ksy/scdms_v1.ksy
There's also a block of data after the odb that runs for 384 bytes that I'm unsure the purpose of, if anyone could point me to some information about that.
Thank you! |
2812
|
26 Aug 2024 |
Stefan Ritt | Info | Help parsing scdms_v1 data? | The MIDAS event format is described here:
https://daq00.triumf.ca/MidasWiki/index.php/Event_Structure
All banks are aligned on a 8-byte boundary, so that one has effective 64-bit CPU access.
If you have sections of 168 or 192 bytes, this must be something else, like another bank (scaler event, slow control event, ...).
The easiest for you is to check how this events got created using the bk_create() function.
Best,
Stefan |
2813
|
26 Aug 2024 |
Adrian Fisher | Info | Help parsing scdms_v1 data? |
Stefan Ritt wrote: | The MIDAS event format is described here:
https://daq00.triumf.ca/MidasWiki/index.php/Event_Structure
All banks are aligned on a 8-byte boundary, so that one has effective 64-bit CPU access.
If you have sections of 168 or 192 bytes, this must be something else, like another bank (scaler event, slow control event, ...).
The easiest for you is to check how this events got created using the bk_create() function.
Best,
Stefan |
Upon further investigation, the sections I'm looking at appear to be clusters of headers for empty banks.
Thank you! |
2824
|
04 Sep 2024 |
Stefan Ritt | Info | News MSCB++ API | I had two free afternoon and took the opportunity to write a new API for the MSCB
system. I'm not sure if anybody else actually uses MSCB (MIDAS slow control bus),
but anyhow.
The new API is contained in a single header file mscbxx.h, and it's extremely
simple to use. Here is some example code:
#include "mscbxx.h"
...
// connect to node 10 at submaster mscb123
midas::mscb m("mscb123", 10);
// print node info and all variables
std::cout << m << std::endl;
// refresh all variables (read from MSCB device)
m.read_range();
// access individual variables
float f = m[5]; // index access
f = m["In0"]; // name access
// write value to MSCB device
m["In0"] = 1.234;
...
Any feedback is welcome.
Stefan |
2830
|
11 Sep 2024 |
Konstantin Olchanski | Info | News MSCB++ API | > Here is some example code:
>
> #include "mscbxx.h"
> f = m["In0"]; // name access
> m["In0"] = 1.234;
> Any feedback is welcome.
Where is the example of error handling?
K.O. |
2836
|
11 Sep 2024 |
Konstantin Olchanski | Info | Help parsing scdms_v1 data? | Look at the C++ implementation of the MIDAS data file reader, the code is very
simple to follow.
Depending on how old are your data files, you may run into a problem with
misaligned 32-bit data banks. Latest MIDAS creates BANK32A events where all
banks are aligned to 64 bits. old BANK32 format had banks alternating between
aligned and misaligned. old 16-bit BANK format data hopefully you do not have.
If you successfully make a data format description file for MIDAS, please post
it here for the next user.
K.O.
[quote="Adrian Fisher"]Hi! I'm working on creating a ksy file to help with
parsing some data, but I'm having trouble finding some information. Right now, I
have it set up very rudimentary - it grabs the event header and then uses the
data bank size to grab the size of the data, but then I'm needing additional
padding after the data bank to reach the next event.
However, there's some irregularity in the "padding" between data banks that I
haven't been able to find any documentation for. For some reason, after the data
banks, there's sections of data of either 168 or 192 bytes, and it's seemingly
arbitrary which size is used.
I'm just wondering if anyone has any information about this so that I'd be able
to make some more progress in parsing the data.
The data I'm working with can be found at https://github.com/det-
lab/dataReaderWriter/blob/master/data/07180808_1735_F0001.mid.gz
And the ksy file that I've created so far is at https://github.com/det-
lab/dataReaderWriter/blob/master/kaitai/ksy/scdms_v1.ksy
There's also a block of data after the odb that runs for 384 bytes that I'm
unsure the purpose of, if anyone could point me to some information about that.
Thank you![/quote] |
2837
|
11 Sep 2024 |
Konstantin Olchanski | Info | mana.cxx | > Ok, no relevant complains so far, so I removed mana and rmana from the CMake build
> process, but left the file mana.cxx still in the repository for educational
> purposes ;-)
+1
K.O. |
|