02 Apr 2024, Konstantin Olchanski, Info, Sequencer editor
|
> Stefan and I have been working on improving the sequencer editor ...
Looks grand! Congratulations with getting it completed. The previous version was
my rewrite of the old generated-C pages into html+javascript, nothing to write
home about, I even kept the 1990-ies-style html formatting and styling as much as
possible.
K.O. |
04 Apr 2024, Konstantin Olchanski, Info, MIDAS RPC data format
|
I am not sure I have seen this documented before. MIDAS RPC data format.
1) RPC request (from client to mserver), in rpc_call_encode()
1.1) header:
4 bytes NET_COMMAND.header.routine_id is the RPC routine ID
4 bytes NET_COMMAND.header.param_size is the size of following data, aligned to 8 bytes
1.2) followed by values of RPC_IN parameters:
arg_size is the actual data size
param_size = ALIGN8(arg_size)
for TID_STRING||TID_LINK, arg_size = 1+strlen()
for TID_STRUCT||RPC_FIXARRAY, arg_size is taken from RPC_LIST.param[i].n
for RPC_VARARRAY|RPC_OUT, arg_size is pointed to by the next argument
for RPC_VARARRAY, arg_size is the value of the next argument
otherwise arg_size = rpc_tid_size()
data encoding:
RPC_VARARRAY:
4 bytes of ALIGN8(arg_size)
4 bytes of padding
param_size bytes of data
TID_STRING||TID_LINK:
param_size of string data, zero terminated
otherwise:
param_size of data
2) RPC dispatch in rpc_execute
for each parameter, a pointer is placed into prpc_param[i]:
RPC_IN: points to the data inside the receive buffer
RPC_OUT: points to the data buffer allocated inside the send buffer
RPC_IN|RPC_OUT: data is copied from the receive buffer to the send buffer, prpc_param[i] is a pointer to the copy in the send buffer
prpc_param[] is passed to the user handler function.
user function reads RPC_IN parameters by using the CSTRING(i), etc macros to dereference prpc_param[i]
user function modifies RPC_IN|RPC_OUT parameters pointed to by prpc_param[i] (directly in the send buffer)
user function places RPC_OUT data directly to the send buffer pointed to by prpc_param[i]
size of RPC_VARARRAY|RPC_OUT data should be written into the next/following parameter.
3) RPC reply
3.1) header:
4 bytes NET_COMMAND.header.routine_id contains the value returned by the user function (RPC_SUCCESS)
4 bytes NET_COMMAND.header.param_size is the size of following data aligned to 8 bytes
3.2) followed by data for RPC_OUT parameters:
data sizes and encodings are the same as for RPC_IN parameters.
for variable-size RPC_OUT parameters, space is allocated in the send buffer according to the maximum data size
that the user code expects to receive:
RPC_VARARRAY||TID_STRING: max_size is taken from the first 4 bytes of the *next* parameter
otherwise: max_size is same as arg_size and param_size.
when encoding and sending RPC_VARARRAY data, actual data size is taken from the next parameter, which is expected to be
TID_INT32|RPC_IN|RPC_OUT.
4) Notes:
4.1) RPC_VARARRAY should always be sent using two parameters:
a) RPC_VARARRAY|RPC_IN is pointer to the data we are sending, next parameter must be TID_INT32|RPC_IN is data size
b) RPC_VARARRAY|RPC_OUT is pointer to the data buffer for received data, next parameter must be TID_INT32|RPC_IN|RPC_OUT before the call should
contain maximum data size we expect to receive (size of malloc() buffer), after the call it may contain the actual data size returned
c) RPC_VARARRAY|RPC_IN|RPC_OUT is pointer to the data we are sending, next parameter must be TID_INT32|RPC_IN|RPC_OUT containing the maximum
data size we are expected to receive.
4.2) during dispatching, RPC_VARARRAY|RPC_OUT and TID_STRING|RPC_OUT both have 8 bytes of special header preceeding the actual data, 4 bytes of
maximum data size and 4 bytes of padding. prpc_param[] points to the actual data and user does not see this special header.
4.3) when encoding outgoing data, this special 8 byte header is removed from TID_STRING|RPC_OUT parameters using memmove().
4.4) TID_STRING parameters:
TID_STRING|RPC_IN can be sent using oe parameter
TID_STRING|RPC_OUT must be sent using two parameters, second parameter should be TID_INT32|RPC_IN to specify maximum returned string length
TID_STRING|RPC_IN|RPC_OUT ditto, but not used anywhere inside MIDAS
4.5) TID_IN32|RPC_VARARRAY does not work, corrupts following parameters. MIDAS only uses TID_ARRAY|RPC_VARARRAY
4.6) TID_STRING|RPC_IN|RPC_OUT does not seem to work.
4.7) RPC_VARARRAY does not work is there is preceding TID_STRING|RPC_OUT that returned a short string. memmove() moves stuff in the send buffer,
this makes prpc_param[] pointers into the send buffer invalid. subsequent RPC_VARARRAY parameter refers to now-invalid prpc_param[i] pointer to
get param_size and gets the wrong value. MIDAS does not use this sequence of RPC parameters.
4.8) same bug is in the processing of TID_STRING|RPC_OUT parameters, where it refers to invalid prpc_param[i] to get the string length.
K.O. |
24 Apr 2024, Konstantin Olchanski, Info, MIDAS RPC data format
|
> 4.5) TID_IN32|RPC_VARARRAY does not work, corrupts following parameters. MIDAS only uses TID_ARRAY|RPC_VARARRAY
fixed in commit 0f5436d901a1dfaf6da2b94e2d87f870e3611cf1, TID_ARRAY|RPC_VARARRAY was okey (i.e. db_get_value()), bug happened only if rpc_tid_size()
is not zero.
>
> 4.6) TID_STRING|RPC_IN|RPC_OUT does not seem to work.
>
> 4.7) RPC_VARARRAY does not work is there is preceding TID_STRING|RPC_OUT that returned a short string. memmove() moves stuff in the send buffer,
> this makes prpc_param[] pointers into the send buffer invalid. subsequent RPC_VARARRAY parameter refers to now-invalid prpc_param[i] pointer to
> get param_size and gets the wrong value. MIDAS does not use this sequence of RPC parameters.
>
> 4.8) same bug is in the processing of TID_STRING|RPC_OUT parameters, where it refers to invalid prpc_param[i] to get the string length.
fixed in commits e45de5a8fa81c75e826a6a940f053c0794c962f5 and dc08fe8425c7d7bfea32540592b2c3aec5bead9f
K.O. |
24 Apr 2024, Konstantin Olchanski, Info, MIDAS RPC add support for std::string and std::vector<char>
|
I now fully understand the MIDAS RPC code, had to add some debugging printfs,
write some test code (odbedit test_rpc), catch and fix a few bugs.
Fixes for the bugs are now committed.
Small refactor of rpc_execute() should be committed soon, this removes the
"goto" in the memory allocation of output buffer. Stefan's original code used a
fixed size buffer, I later added allocation "as-neeed" but did not fully
understand everything and implemented it as "if buffer too small, make it
bigger, goto start over again".
After that, I can implement support for std::string and std::vector<char>.
The way it looks right now, the on-the-wire data format is flexible enough to
make this change backward-compatible and allow MIDAS programs built with old
MIDAS to continue connecting to the new MIDAS and vice-versa.
MIDAS RPC support for std::string should let us improve security by removing
even more uses of fixed-size string buffers.
Support for std::vector<char> will allow removal of last places where
MAX_EVENT_SIZE is used and simplify memory allocation in other "give me data"
RPC calls, like RPC_JRPC and RPC_BRPC.
K.O. |
14 May 2024, Konstantin Olchanski, Info, ROOT v6.30.6 requires libtbb-dev
|
root_v6.30.06.Linux-ubuntu22.04-x86_64-gcc11.4 the libtbb-dev package.
This is a new requirement and it is not listed in the ROOT dependancies page (I left a note on the ROOT forum, hopefully it will be
fixed quickly). https://root.cern/install/dependencies/
Symptom is when starting ROOT, you get an error:
cling::DynamicLibraryManager::loadLibrary(): libtbb.so.12: cannot open shared object file: No such file or directory
and things do not work.
Fix is to:
apt install libtbb-dev
K.O. |
12 Aug 2006, Pierre-André Amaudruz, Release, Midas updates
|
Midas development:
Over the last 2 weeks (Jul26-Aug09), Stefan Ritt has been at Triumf for the "becoming" traditional Midas development 'brainstorming/hackathon' (every second year).
A list with action items has been setup combining the known problems and the wish list from several Midas users.
The online documentation has been updated to reflect the modifications.
Not all the points have been covered, as more points were added daily but the main issues that have been dealt or at least discussed are:
- ODB over Frontend precedence.
When starting a FE client, the equipment settings are taken from the ODB if this equipment already existed. This meant the ODB has precedence over the EQUIPEMENT structure and whatever change you apply to the C-Structure, it will NOT be taken in consideration until you clean (remove) the equipment tree in ODB.
- Revived 64 bit support. This was required as more OS are already supporting such architecture. Originally Midas did support Alpha/OSF/1 which operated on 64 bit machine. This new code has been tested on SL4.2 with Dual-Core 64-bit AMD Opterons.
- Multi-threading in Slow Control equipments.
Check entry 289 in Midas Elog from Stefan.
- mhttpd using external Elog.
The standalone ELOG package can be coupled to an existing experiment and therefore supersede the internal elog functionality from mhttpd.
This requires a particular configuration which is described in the documentation.
- MySQL test in mlogger
A reminder that mlogger can generate entries in a MySQL database as long as the pre-compilation flag -HAVE_MYSQL is enabled during system built. The access and form filling is then defined from the ODB under Logger/SQL once the logger is running, see documentation.
- Directory destination for midas.log and odb dump files
It is now possible to specify an individual directory to the default midas.log file as well as to the "ODB Dump file" destination. If either of these fields contains a preceding directory, it will take the string as an absolute path to the file.
- User defined "event Data buffer size" (ODB)
The event buffer size has been until now defined at the system level in midas.h. It is now possible to optimize the memory allocation specific to the event buffer with an entry in the ODB under /experiment, see documentation.
- History group display
It is now possible to display an individual group of history plots. No documentation on that topics as it should be self explanatory.
- History export option
From the History web page, it is possible to export to a ASCII .csv file the history content. This file can later be imported into excel for example. No documentation on that topics as it should be self explanatory.
- Multiple "minor" corrections:
- Alarm reset for multiple experiment (return directly to the experiment).
- mdump -b option bug fixed.
- Alarm evaluation function fixed.
- mlogger/SQL boolean handling fixed.
- bm_get_buffer_level() was returning a wrong value which has been fixed now.
- Event buffer bug traced and exterminated (Thanks to Konstantin).
|
30 Oct 2009, Konstantin Olchanski, Release, new lazylogger release
|
I committed an updated lazylogger with updated documentation. The new version supports subruns and
can save to external storage arbitrary files (i.e. odb dump files). It also moves most book keeping out of
odb to permit handling more files on bigger storage disks.
Example lazylogger scripts for castor (CERN) and dcache (TRIUMF) are in the directory "utils".
The lazylogger documentation was updated to remove obsolete information and to describe the new
functions. As usual "make dox; cd doxfiles/html; firefox index.html" or see my copy at:
http://ladd00.triumf.ca/~olchansk/midas/Utilities.html#lazylogger_task
svn rev 4615, 4616.
K.O. |
12 Nov 2010, Pierre-Andre Amaudruz, Release, Documentation
|
The general Midas documentation has been rejuvenated by Suzannah Daviel through
a proof reading and with a collection of custom perl scripts to improve the
Doxygen capabilities for the document itself. In particular, a contents list and
alphabetical index of the documentation is generated automatically.
The new content is based on the previous version but with more cross-references,
examples and descriptive images where necessary. Many of the previously
undocumented features are now included.
The layout and organization is slightly different and requires getting used to,
but hopefully will be an improvement.
Some of the changes are:
- The main topics are maintained, but we try to regroup all the aspects
related to a particular topic in the same section.
- The Yellow icons provide navigation within the index section.
- The Blue icons provide navigation within the section content.
The full documentation is included under the midas/doc/src directory in the
Midas distribution (SVN) and can be generated with the Doxygen tool.
The midasdoc-images.tar.gz from either https://midas.psi.ch/download/ or
http://ladd00.triumf.ca/~daqweb/ftp/ needs to be extracted to the midas
directory under doc/images for complete local web pages generation.
There are a few "ToDo" items which hopefully will be ironed out soon.
Feel free to contact us for pointing out omissions or improvements.
We hope this online documentation will serve as a better tool for your
understanding of the Midas capabilities. |
30 Nov 2015, Konstantin Olchanski, Release, Final MIDAS JSON-RPC API
|
The final bits of the JSON-RPC API to MIDAS are committed. The API uses the Javascript Promise mechanism (supported on all
supported platforms - MacOS, Windows, Linux Ubuntu, el5, el6, el7).
Simple example for pasting the current run number into an html element:
mjsonrpc_db_get_values(["/runinfo/run number"]).then(function(rpc) {
document.getElementById("run_number").innerHTML = rpc.response.data[0];
}).catch(function(error) {
mjsonrpc_error_alert(error);
});
The documentation for the JSON-RPC API, including special quirks in JSON encoding of ODB data is here:
https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc
Documentation (with examples) for the related Javascript functions in mhttpd.js is here (via Doxygen):
https://daq.triumf.ca/~daqweb/doc/midas-devel/html/group__mjsonrpc__js.html
Examples of using all mhttpd.js functions is in .../examples/javascript1/example.html
The experimental git branch feature/mhttpd_js implements the MIDAS "programs" page purely in html and javascript,
go there to see all this new JSON and RPC stuff in action. See .../resources/programs.html.
K.O. |
02 Dec 2015, Konstantin Olchanski, Release, Final MIDAS JSON-RPC API
|
> The final bits of the JSON-RPC API to MIDAS are committed.
Here is example conversion of the function "generate midas message" from old-style AJAX to JSON-RPC:
before (mhttpd.cxx):
/* process "jgenmsg" command */
if (equal_ustring(getparam("cmd"), "jgenmsg")) {
if (getparam("facility") && *getparam("facility"))
strlcpy(facility, getparam("facility"), sizeof(facility));
else
strlcpy(facility, "midas", sizeof(facility));
if (getparam("user") && *getparam("user"))
strlcpy(user, getparam("user"), sizeof(user));
else
strlcpy(user, "javascript_commands", sizeof(user));
if (getparam("type") && *getparam("type"))
type = atoi(getparam("type"));
else
type = MT_INFO;
if (getparam("msg") && *getparam("msg")) {
cm_msg1(type, __FILE__, __LINE__, facility, user, "%s", getparam("msg"));
}
show_text_header();
rsputs("Message successfully created\n");
return;
}
after: (mjsonrpc.cxx)
static MJsonNode* js_cm_msg1(const MJsonNode* params)
{
if (!params) {
MJSO *doc = MJSO::I();
doc->D("Generate a midas message using cm_msg1()");
doc->P("facility?", MJSON_STRING, "message facility, default is \"midas\"");
doc->P("user?", MJSON_STRING, "message user, default is \"javascript_commands\"");
doc->P("type?", MJSON_INT, "message type, MT_xxx from midas.h, default is MT_INFO");
doc->P("message", MJSON_STRING, "message text");
doc->R("status", MJSON_INT, "return status of cm_msg1()");
return doc;
}
MJsonNode* error = NULL;
const char* facility = mjsonrpc_get_param(params, "facility", &error)->GetString().c_str();
const char* user = mjsonrpc_get_param(params, "user", &error)->GetString().c_str();
int type = mjsonrpc_get_param(params, "type", &error)->GetInt();
const char* message = mjsonrpc_get_param(params, "message", &error)->GetString().c_str(); if (error) return error;
if (strlen(facility)<1)
facility = "midas";
if (strlen(user)<1)
user = "javascript_commands";
if (type == 0)
type = MT_INFO;
int status = cm_msg1(type, __FILE__, __LINE__, facility, user, "%s", message);
return mjsonrpc_make_result("status", MJsonNode::MakeInt(status));
}
With the corresponding javascript-side stabs:
before:
function ODBGenerateMsg(type,facility,user,msg)
{
var request = XMLHttpRequestGeneric();
var url = ODBUrlBase + '?cmd=jgenmsg';
url += '&type='+type;
url += '&facility='+facility;
url += '&user='+user;
url += '&msg=' + encodeURIComponent(msg);
request.open('GET', url, false);
request.send(null);
return request.responseText;
}
after:
function mjsonrpc_cm_msg(message, type, id) {
/// \ingroup mjsonrpc_js
/// Get values of ODB variables
///
/// RPC method: "cm_msg1"
///
/// \code
/// mjsonrpc_cm_msg("this is a new message").then(function(rpc) {
/// var req = rpc.request; // reference to the rpc request
/// var id = rpc.id; // rpc response id (should be same as req.id)
/// var status = rpc.result.status; // return status of MIDAS cm_msg1()
/// ...
/// }).catch(function(error) {
/// mjsonrpc_error_alert(error);
/// });
/// \endcode
/// @param[in] message Text of midas message (string)
/// @param[in] type optional message type, one of MT_xxx. Default is MT_INFO (integer)
/// @param[in] id optional request id (see JSON-RPC specs) (object)
/// @returns new Promise
///
var req = new Object();
req.message = message;
if (type)
req.type = type;
return mjsonrpc_call("cm_msg1", req, id);
}
K.O |
28 Jan 2016, Konstantin Olchanski, Release, Final MIDAS JSON-RPC API
|
> > The final bits of the JSON-RPC API to MIDAS are committed.
JSON-RPC methods are now provided for all old ODBxxx() javascript functions, except ODBGetMsg().
The currently present RPC methods are sufficient to write the MIDAS "programs" and "alarms" pages
purely in HTML+Javascript (see the git branch feature/mhttpd_js). These pages can be served i.e. by apache httpd
with midas mhttpd only required to service the RPC requests.
Please see .../examples/javascript1/example.html on how to use the new RPC methods.
K.O.
P.S. Note how many examples use the generic mjsonrpc_call() because I did not write the corresponding
javascript functions - I wore out the cut-and-paste button on my keyboard. All are welcome to contribute
the missing functions, post them here or email them to me, I will commit them to midas git. |
08 Aug 2016, Konstantin Olchanski, Release, Merged - new pure html web pages: programs and alarms.
|
The code for the new pure html and javascript web pages was merged into main midas.
In this release, the "programs" and "alarms" pages are implemented as html files, see
resources/programs.html and alarms.html.
Eventually we hope to implement all midas web pages in html, so this is just a start.
If you see problems with the new html code, you can revert to the old mhttpd-generated web
pages by removing the files programs.html and alarms.html.
The new code for starting and stopping runs (start.html and transition.html) is also merged, but not
yet enabled, pending a few more tests.
K.O. |
21 Nov 2017, Konstantin Olchanski, Release, Pending release of midas
|
We are readying a new release of midas and it is almost here except for a few buglets on the new html status page.
The current release candidate branch is "feature/midas-2017-10" and if you have problems with the older versions
of midas, I recommend that you try this release candidate to check if your problem is already fixed. If the problem
still exists, please file a bug report on this forum or on the bitbucket issue tracker
https://bitbucket.org/tmidas/midas/issues?status=new&status=open
Highlights of the new release include
- new and improved web pages done in html and javascript
- many bug fixes and improvements for json and json-rpc support, including improvements in handling of long strings in odb
- locked (protected) operation of odb, where odb shared memory is not writable outside of odb operations
- improved multithead support for odb
- fixes for odb corruption when odb becomes 100% full
For the next release we hope to switch midas from C to fully C++ (building everything with C++ already works). To support el6 we avoid use of
c++11 language constructs.
K.O. |
28 Mar 2019, Konstantin Olchanski, Release, midas-2019-03-f
|
the midas release 2019-03 is ready for general use.
main changes from previous releases (midas-2017-10, midas-2018-12 and midas-2019-02):
- change to the midas URL scheme
- removal of cm_watchdog()
- rewrite of event buffer code (and fix of hard to trigger event buffer corruption bug)
- fully thread safe odb and event buffer code (except for rpc_send_event())
- corrected compatibility problems wrt older versions of midas when serving custom web pages via odb /custom/path
To obtain this release, either checkout the top of branch feature/midas-2019-03 (recommended)
or checkout the tag midas-2019-03-f.
K.O. |
22 May 2019, Konstantin Olchanski, Release, midas-2019-03-g
|
> the midas release 2019-03 is ready for general use.
first ever bug fix release on a git release branch.
fixed a crash if frontend built against this midas is connected to mserver from old (pre-db_watch) midas (size mismatch of MSG_ODB
message).
to use this update:
# recommended:
git pull
git checkout feature/midas-2019-03
git pull
make ...
# or checkout "detached HEAD"
git pull
git checkout midas-2019-03-g
make ...
odbedit "ver" should report:
GIT revision: Wed May 22 07:35:11 2019 -0700 - midas-2019-03-g on branch feature/midas-2019-03
K.O.
P.S. Thanks for finding this bug go to Greg Hackman on TIGRESS and EMMA experiments at TRIUMF.
K.O. |
06 Jun 2019, Konstantin Olchanski, Release, midas-2019-03-h
|
> > the midas release 2019-03 is ready for general use.
A bug fix update for midas-2019-03:
- fix broken expand_env() in mhttpd
- fix "Invalid name passed to db_create_key: should not be an empty string" in midas.log when loading the MIDAS status page if one of the alarms has empty
class name.
odbedit "ver" should report: Thu Jun 6 18:02:14 2019 -0700 - midas-2019-03-h on branch feature/midas-2019-03
K.O. |
10 Jun 2019, Konstantin Olchanski, Release, mxml-2019-03-a, midas-2019-03-h
|
> > > the midas release 2019-03 is ready for general use.
> A bug fix update for midas-2019-03:
> odbedit "ver" should report: Thu Jun 6 18:02:14 2019 -0700 - midas-2019-03-h on branch feature/midas-2019-03
For building this release of MIDAS, please use mxml branch feature/midas-2019-03, tag mxml-2019-03-a:
cd .../mxml
git fetch
git checkout feature/midas-2019-03
Going forward, I will try to remember to tag the mxml version that corresponds to specific midas versions.
K.O. |
10 Jun 2019, Konstantin Olchanski, Release, bin and lib symlinks, mxml-2019-03-a, midas-2019-03-h
|
> > > > the midas release 2019-03 is ready for general use.
The latest version of MIDAS puts libraries and executables in $MIDASSYS/lib and bin (the "linux" part of pathname is removed).
Some packages (rootana) have been already changed to use this new scheme and they will not build against older versions of midas.
I recommend that you create following symlinks to make old versions of midas compatible with the new scheme:
cd $MIDASSYS # (~/packages/midas)
ln -s linux/bin .
ln -s linux/lib .
K.O. |
11 Jun 2019, Stefan Ritt, Release, bin and lib symlinks, mxml-2019-03-a, midas-2019-03-h
|
> The latest version of MIDAS puts libraries and executables in $MIDASSYS/lib and bin (the "linux" part of pathname is removed).
>
> Some packages (rootana) have been already changed to use this new scheme and they will not build against older versions of midas.
> I recommend that you create following symlinks to make old versions of midas compatible with the new scheme:
>
> cd $MIDASSYS # (~/packages/midas)
> ln -s linux/bin .
> ln -s linux/lib .
If i'm not mistaken the proper commands are
cd $MIDASSYS
ln -s ../bin linux/bin
ln -s ../lib linux/lib
Alternatively, you can change your PATH to point to $MIDASSYS/bin instead of $MIDASSYS/linux/bin and link against $MIDASSYS/lib instead of
$MIDASSYS/linux/lib
Stefan |
17 Jun 2019, Konstantin Olchanski, Release, bin and lib symlinks, mxml-2019-03-a, midas-2019-03-h
|
>
> If i'm not mistaken the proper commands are
>
> cd $MIDASSYS
> mkdir linux
> ln -s ../bin linux/bin
> ln -s ../lib linux/lib
>
This is for making the new midas look like the old midas. My instructions were for making the old midas looking like the new midas.
Old midas:
packages/midas/linux/bin, linux/lib with symlinks for
packages/midas/bin -> linux/bin, etc
New midas:
packages/midas/bin, lib with symlinks for
packages/midas/linux/bin -> ../bin, etc.
K.O. |
|