 06 Oct 2023, Konstantin Olchanski, Info, new history panel editor
|
the new history panel editor has been activated. it is meant to work the same as
the old editor, with some improvements to the history variables selection page.
this new version is written in html+javascript and it will be easier to improve,
|
|
06 Oct 2023, Konstantin Olchanski, Info, default midas history switched to "FILE" and "PerVariable" history
|
We are very happy with the "FILE" implementation of MIDAS history and it is time
to make it the default for new experiments. This history driver works best if
"per variable" history is alos enabled. (SQL history already only works in "per-
|
02 Aug 2023, Stefan Ritt, Bug Report, Error accessing history files 
|
We sporadically (like once per few hours) have an error message when we access the
history plots through mhttpd:
|
 09 Aug 2023, Konstantin Olchanski, Bug Report, Error accessing history files
|
I confirm I see same on the agmini system. Two problems: (a) error message is wrong, it's a
short read, not a read error (clue: read() syscall does not return "no such file"). (b)
mlogger is supposed to write history in record-size blocks, read in the same record size
|
|
16 Aug 2023, Stefan Ritt, Bug Report, Error accessing history files
|
Tonight we got another error of that type after the update:
04:17 - [mhttpd,ERROR] [history_schema.cxx:2913:FileHistory::read_data,ERROR] Cannot read
|
|
17 Aug 2023, Konstantin Olchanski, Bug Report, Error accessing history files
|
Confirmed. The error message is wrong. It is printed after a short read(), but short read() does not
set errno, and errno reported by the error message is from some previous syscall. Corrected error
message is already committed. K.O.
|
|
19 Aug 2023, Stefan Ritt, Bug Report, Error accessing history files
|
Still get the same error with the latest version:
3:28 [mhttpd,ERROR] [history_schema.cxx:2913:FileHistory::read_data,ERROR] Cannot read
|
|
06 Oct 2023, Konstantin Olchanski, Bug Report, Error accessing history files
|
> Still get the same error with the latest version:
> 3:28 [mhttpd,ERROR] [history_schema.cxx:2913:FileHistory::read_data,ERROR] Cannot read
> '/data2/history/mhf_1692391703_20230818_hv_tc.dat', read() errno 2 (No such file or directory)
|
|
06 Oct 2023, Konstantin Olchanski, Bug Report, Error accessing history files
|
> two fixes forthcoming:
> a) check for short read in the 2nd place that I missed
> b) two write() are replaced by 2 memcpy() to a preallocated buffer and 1 write()
|
|
03 Oct 2023, Konstantin Olchanski, Bug Fix, wrong array size after loading xml or json file
|
both the xml and the json decoders have a bug (fix pending). loading saved odb
from xml and json file did not truncate arrays in odb to the size of arrays in
the file. for example, if /example/double_array has size 20 in odb, but size 5
|
|
30 Sep 2023, Gennaro Tortone, Bug Report, ODB page and hex values
|
Hi,
|
|
01 Oct 2023, Stefan Ritt, Bug Report, ODB page and hex values
|
Thanks for reporting this bug, I fixed it in the last commit.
Best,
|
|
26 Sep 2023, Stefan Ritt, Info, mjsonrpc_db_save / mjsonrpc_db_load have been dropped
|
The JavaScript function
mjsonrpc_db_save / mjonrpc_db_load
|
|
24 Sep 2023, Frederik Wauters, Suggestion, scroll when browsing for a link
|
Another small user experience request:
When making a link in the odb (web interface) a nice browser window pop's up. There is however not scrolling possible in the window. As a result, you can |
|
26 Sep 2023, Stefan Ritt, Suggestion, scroll when browsing for a link
|
> When making a link in the odb (web interface) a nice browser window pop's up. There is however not scrolling possible in the window. As a result, you
can not reach a odb key if it is nested to deeply.
>
|
|
19 Sep 2023, Frederik Wauters, Bug Report, epics fe "Start Command"
|
The epics frontend overwrites the "start command" odb after each start:
// set start command in ODB
|
|
20 Sep 2023, Stefan Ritt, Bug Report, epics fe "Start Command"
|
Thanks for reporting this problem. It has been fixed today, so the start command is only written if it's emtpy.
Stefan |
|
08 Sep 2023, Nick Hastings, Forum, Hide start and stop buttons
|
The wiki documents an odb variable to enable the hiding of the Start and Stop buttons on the mhttpd status page
https://daq00.triumf.ca/MidasWiki/index.php//Experiment_ODB_tree#Start-Stop_Buttons
|
|
08 Sep 2023, Nick Hastings, Forum, Hide start and stop buttons
|
> Is there now some other mechanism to hide the start and stop buttons?
> Note that this is for a pure slow control system that does not take runs.
|
|
13 Sep 2023, Stefan Ritt, Forum, Hide start and stop buttons
|
Indeed the ODB settings are obsolete. Now that the status page is fully dynamic
(JavaScript), it's much more powerful to modify the status.html page directly. You
can not only hide the buttons, but also remove the run numbers, the running time,
|
|
13 Sep 2023, Nick Hastings, Forum, Hide start and stop buttons
|
Hi Stefan,
> Indeed the ODB settings are obsolete.
|
|
13 Sep 2023, Stefan Ritt, Forum, Hide start and stop buttons
|
> Hi Stefan,
>
> > Indeed the ODB settings are obsolete.
|
|
14 Sep 2023, Konstantin Olchanski, Forum, Hide start and stop buttons
|
I believe the original "hide run start / stop" was added specifically for ND280 GSC MIDAS. I do not know
why it was removed. "hide pause / resume" is still there. I will restore them. Hiding logger channel
section should probably be automatic of there is no /logger/channels, I can check if it works and what
|
|
14 Sep 2023, Stefan Ritt, Forum, Hide start and stop buttons
|
> I believe the original "hide run start / stop" was added specifically for ND280 GSC MIDAS. I do not know
> why it was removed. "hide pause / resume" is still there. I will restore them. Hiding logger channel
> section should probably be automatic of there is no /logger/channels, I can check if it works and what
|
|
15 Sep 2023, Stefan Ritt, Forum, Hide start and stop buttons
|
> I believe the original "hide run start / stop" was added specifically for ND280 GSC MIDAS. I do not know
> why it was removed. "hide pause / resume" is still there. I will restore them. Hiding logger channel
> section should probably be automatic of there is no /logger/channels, I can check if it works and what
|
|
14 Sep 2023, Nick Hastings, Forum, Hide start and stop buttons
|
Hi
> > > Indeed the ODB settings are obsolete.
|
|
12 Sep 2023, Maia Henriksson-Ward, Suggestion, Syntax highlighting for sequencer scripts
|
Recently I was trying to read sequencer scripts written by a previous student, and realized it would be easier to
quickly read/skim sequencer code with some form of syntax highlighting. I've been using Visual Studio Code as my
editor, so I made myself an extension for VS Code that provides basic syntax highlighting (with help from
|
|
12 Sep 2023, Stefan Ritt, Suggestion, Syntax highlighting for sequencer scripts
|
I like the idea of syntax highlighting, but your solution is just for one editor which not everybody
is using. It would be better if the editor built into mhttpd for MSL files would have the possibility.
|
|
01 Jun 2023, Thomas Lindner, Info, MIDAS Workshop 2023
|
Dear MIDAS users,
We would like to arrange another MIDAS workshop, following on from previous successful workshops in 2015, 2017 and 2019. The
|
|
13 Jun 2023, Thomas Lindner, Info, MIDAS Workshop 2023 - Sept 13
|
Hi All,
Thanks to everyone who filled out the doodle poll.
|
|
17 Aug 2023, Thomas Lindner, Info, MIDAS Workshop 2023 - Sept 12-13
|
Dear All,
A quick update on the MIDAS workshop. Based on the number of planned talks we have made the decision to switch to a two day workshop on Sept 12 and 13 |
|
06 Sep 2023, Thomas Lindner, Info, MIDAS Workshop 2023 - Sept 12-13
|
Dear All,
A final reminder about the MIDAS workshop in 6 days. A (hopefully) finalized agenda is posted here:
|
|
16 May 2023, Konstantin Olchanski, Bug Report, excessive logging of http requests
|
Our default configuration of apache httpd logs every request. MIDAS custom web pages can easily make a huge number of RPC calls creating a
huge log file and filling system disk to 100% capacity
|
|
16 May 2023, Konstantin Olchanski, Bug Report, excessive logging of http requests
|
> Our default configuration of apache httpd logs every request. MIDAS custom web pages can easily make a huge number of RPC calls creating a
> huge log file and filling system disk to 100% capacity
|
|
16 May 2023, Stefan Ritt, Bug Report, excessive logging of http requests
|
Maybe you remember the problems we had with a custom page in Japan loading it from TRIUMF. It took almost one minute since each RPC request took
about 1s round-trip. This got fixed by the modb* scheme where the framework actually collects all ODB variables in a custom page and puts them
into ONE rpc request (making the path an actual array of paths). That reduced the requests from 100 to 1 in the above example. Maybe the same
|
|
02 Aug 2023, Konstantin Olchanski, Bug Report, excessive logging of http requests
|
> > Our default configuration of apache httpd logs every request. MIDAS custom web pages can easily make a huge number of RPC calls creating a
> > huge log file and filling system disk to 100% capacity
> perhaps use existing logrotate, add limit on file size (size) and limit of 2 old log files (rotate).
|
|
03 Aug 2023, Konstantin Olchanski, Bug Report, excessive logging of http requests
|
> > > Our default configuration of apache httpd logs every request. MIDAS custom web pages can easily make a huge number of RPC calls creating a
> > > huge log file and filling system disk to 100% capacity
> > perhaps use existing logrotate, add limit on file size (size) and limit of 2 old log files (rotate).
|
|
14 Aug 2023, Konstantin Olchanski, Bug Report, excessive logging of http requests
|
> Our default configuration of apache httpd logs every request.
> MIDAS custom web pages can easily make a huge number of RPC calls creating a
> huge log file and filling system disk to 100% capacity.
|
|
16 Aug 2023, Konstantin Olchanski, Bug Report, excessive logging of http requests
|
> > Our default configuration of apache httpd logs every request.
> > MIDAS custom web pages can easily make a huge number of RPC calls creating a
> > huge log file and filling system disk to 100% capacity.
|
|
17 Aug 2023, Konstantin Olchanski, Bug Report, excessive logging of http requests
|
> > > Our default configuration of apache httpd logs every request.
> > > MIDAS custom web pages can easily make a huge number of RPC calls creating a
> > > huge log file and filling system disk to 100% capacity.
|
|
16 Aug 2023, Konstantin Olchanski, Bug Report, midas wants to show notification?
|
I started to get web browser popups about "midas wants to show notifications,
block/allow/x". is this a glitch or a new unannounced/undocumented feature?
google chrome on macos. K.O. |
|
16 Aug 2023, Stefan Ritt, Bug Report, midas wants to show notification?
|
> I started to get web browser popups about "midas wants to show notifications,
> block/allow/x". is this a glitch or a new unannounced/undocumented feature?
> google chrome on macos. K.O.
|
|
16 Aug 2023, Stefan Ritt, Bug Report, midas wants to show notification?
|
> > I started to get web browser popups about "midas wants to show notifications,
> > block/allow/x". is this a glitch or a new unannounced/undocumented feature?
> > google chrome on macos. K.O.
|
|
16 Aug 2023, Konstantin Olchanski, Bug Report, midas wants to show notification?
|
> This feature was asked by some people ...
"show notifications" popups are strongly associated with disreputable web sites (presumably to
|
|
17 Aug 2023, Stefan Ritt, Bug Report, midas wants to show notification?
|
> > This feature was asked by some people ...
>
> "show notifications" popups are strongly associated with disreputable web sites (presumably to
|
|
15 Aug 2023, Konstantin Olchanski, Info, mlogger update
|
A bit of update to the mlogger. In preparation for more cleanup when Stefan is
here at TRIUMF.
|
|
09 Aug 2023, Konstantin Olchanski, Bug Fix, Stefan's improved ODB flush to disk
|
This is an important improvement, should have a post of it's own. K.O.
> > > RFE filed:
|
|
31 Mar 2022, Stefan Ritt, Suggestion, Maximum ODB size
|
Anybody some idea what the maximum ODB size can be? In the old days, the linux
kernels had a severe limit on shared memory of usually 8MB, but in the age of
64GB RAM being a standard, we should be able to grow bigger. Tried
|
|
04 Apr 2022, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> Anybody some idea what the maximum ODB size can be?
It turns out ODB size limit is hardwired on db_open_database() at 100 Mbytes.
|
|
27 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size
|
Hi all,
> I agree, I think we can safely bump the limit from 100 Mbytes to 1 Gbyte, maybe 1.5 or
|
|
27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > I agree, I think we can safely bump the limit from 100 Mbytes to 1 Gbyte, maybe 1.5 or
> > 1.99 Gbytes. Above that we run into 32-bit/31-bit cleanliness problems.
>
|
|
27 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size
|
> This is change is wrong. As I wrote, ODB is not 64-bit clean and it is not 32-bit clean. We think is is 31-bit clean, so maximum size would be slightly
less than 2 Gbytes.
|
|
27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> You said the writing into .odb is buggy. Do you mean it’s buggy in general or only in this specific case?
> We save the ODB most of the time in the .odb format.
|
|
27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
my vote is to bump the ODB size limit to 1999*1000*1000 (not quite 2GB). but this needs to be tested. especially save and restore from ODB, XML and JSON
files, including how long it takes to save and load a 1.9GB ODB. K.O. |
|
28 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size
|
> my vote is to bump the ODB size limit to 1999*1000*1000 (not quite 2GB). but this needs to be tested. especially save and restore from ODB, XML and JSON
files, including how long it takes to save and load a 1.9GB ODB. K.O.
|
|
28 Apr 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> Is this maybe related to what Stefan said about the run start - so that odbedit needs some time to load the bigger ODB?
At the run start mlogger writes the ODB to the .mid file. This needs conversion (binary ODB -> XML ASCII) which can take time.
|
|
28 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size
|
> At the run start mlogger writes the ODB to the .mid file. This needs conversion (binary ODB -> XML ASCII) which can take time.
> This does NOT depend on the ODB size, but on the ODB *content*. Every key in the ODB takes time to convert. So if your ODB as 1.5 GB
> but only a few keys, this is still fast. Only if you have 200 million keys int he ODB, then mlogger takes lots of time to convert
|
|
28 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > Is this maybe related to what Stefan said about the run start - so that odbedit needs some time to load the bigger ODB?
>
> At the run start mlogger writes the ODB to the .mid file. This needs conversion (binary ODB -> XML ASCII) which can take time.
|
|
27 Apr 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> Congratulations. created != "it works".
Two other tings to consider:
|
|
28 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > Congratulations. created != "it works".
>
> Two other tings to consider:
|
|
09 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > 1) The ODB shared memory is dumped into a binary file (".ODB.SHM") after the last client finished ...
correction: ODB shared memory is saved to .ODB.SHM each time a client stops, this is db_close_database().
|
|
12 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> correction: ODB shared memory is saved to .ODB.SHM each time a client stops, this is db_close_database().
The original design of the midas shared memory (back in the 1990's) was that the ODB shared memory file gets
|
|
12 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > correction: ODB shared memory is saved to .ODB.SHM each time a client stops, this is db_close_database().
>
> The original design of the midas shared memory (back in the 1990's) was that the ODB shared memory file gets
|
|
13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> I remember the same, but I tracked it down in git to the very first commit, and there is no if() there,
> odb is saved to .ODB.SHM on every client shutdown, not just the last client. I guess we both misremebered.
|
|
13 Jun 2023, Marius Koeppel, Suggestion, Maximum ODB size
|
> BTW, how do I resize the ODB. I remember we discussed this some time ago, and concluded that odbedit needs a resize flag. Has this even been
> done? If not, what is the "official" way to resize the ODB. We had some documentation about that some time ago, but I can't find it anymore.
|
|
13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > BTW, how do I resize the ODB.
ODB cannot be resized "online". Everything has to stop, save content to odb.json, get rid of old ODB.SHM, ensure ODB shared memory is destroyed (SysV or |
|
13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > I remember the same, but I tracked it down in git to the very first commit, and there is no if() there,
> > odb is saved to .ODB.SHM on every client shutdown, not just the last client. I guess we both misremebered.
|
|
13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> small problem. build an experiment, start taking data, observe how ODB is never saved to disk because the "last client" never stops. as bonus, crash
|
|
13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
>
> > small problem. build an experiment, start taking data, observe how ODB is never saved to disk because the "last client" never stops. as bonus, crash
|
|
13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> are you sure? when/how often does "last midas program finishes" happen? it does not happen on a system crash, not on power loss, not on "shutdown -r
now"
|
|
15 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
>
> > are you sure? when/how often does "last midas program finishes" happen? it does not happen on a system crash, not on power loss, not on "shutdown -r
now"
|
|
28 Jul 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> RFE filed:
> https://bitbucket.org/tmidas/midas/issues/367/odb-should-be-saved-to-disk-periodically
|
|
09 Aug 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > RFE filed:
> > https://bitbucket.org/tmidas/midas/issues/367/odb-should-be-saved-to-disk-periodically
>
|
|
06 Mar 2023, Gennaro Tortone, Forum, pull request for PostgreSQL support
|
Hi,
|
|
06 Mar 2023, Konstantin Olchanski, Forum, pull request for PostgreSQL support
|
> some minutes ago I published a PR for PostgreSQL support I developed
> at INFN-Napoli for Darkside experiment...
>
|
|
06 Mar 2023, Gennaro Tortone, Forum, pull request for PostgreSQL support
|
Hi Konstantin,
thanks for this update |
|
|
20 Mar 2023, Gennaro Tortone, Forum, pull request for PostgreSQL support
|
Hi,
I have updated the PR with a new one that includes TimescaleDB support and some
|
|
24 May 2023, Gennaro Tortone, Forum, pull request for PostgreSQL support
|
Hi,
is there any news regarding this pull request ?
|
|
18 Jul 2023, Konstantin Olchanski, Forum, pull request for PostgreSQL support
|
> is there any news regarding this pull request ?
> (https://bitbucket.org/tmidas/midas/pull-requests/30)
|
|
21 Jul 2023, Konstantin Olchanski, Forum, pull request for PostgreSQL support
|
> > is there any news regarding this pull request ?
> > (https://bitbucket.org/tmidas/midas/pull-requests/30)
>
|
|
21 Jul 2023, Gennaro Tortone, Forum, pull request for PostgreSQL support
|
|
|
28 Jul 2023, Stefan Ritt, Forum, pull request for PostgreSQL support
|
The compilation of midas was broken by the last modification. The reason is that
Pgsql *fPgsql = NULL;
|
|
09 Aug 2023, Konstantin Olchanski, Forum, pull request for PostgreSQL support
|
> The compilation of midas was broken by the last modification. The reason is that
> Pgsql *fPgsql = NULL;
> was not protected by #ifdef HAVE_PGSQL
|
|
02 Aug 2023, Caleb Marshall, Forum, Issues with Universe II Driver
|
Hello,
At our lab we are currently in the process of migrating more of our systems over to Midas. However, all of our working systems are dependent
on SBCs with the Tsi-148 chips of which we only have a handful. In order to have some backups and spares for testing, we have been attempting to get Midas |
|
02 Aug 2023, Konstantin Olchanski, Forum, Issues with Universe II Driver
|
I maintain the tsi148 and the universe-II drivers. I confirm -KO6 is my latest
version, last updated for 32-bit Debian-11, and we still use it at TRIUMF.
|
|
03 Aug 2023, Caleb Marshall, Forum, Issues with Universe II Driver
|
Here is the output:
vmic_mmap: Mapped VME AM 0x0d addr 0x00000000 size 0x00ffffff at address 0x80a01000
|
|
03 Aug 2023, Konstantin Olchanski, Forum, Issues with Universe II Driver
|
> Here is the output:
>
> vmic_mmap: Mapped VME AM 0x0d addr 0x00000000 size 0x00ffffff at address 0x80a01000
|
|
03 Aug 2023, Caleb Marshall, Forum, Issues with Universe II Driver
|
I am looking into compiling the 32 bit midas.
In the meantime, here is the kernel info:
|
|
04 Aug 2023, Caleb Marshall, Forum, Issues with Universe II Driver
|
I can compile 32 bit midas. Unless I am interpreting the linking error, I don't
think I can use the driver as built.
|
|
04 Aug 2023, Konstantin Olchanski, Forum, Issues with Universe II Driver
|
> I can compile 32 bit midas. Unless I am interpreting the linking error, I don't
> think I can use the driver as built.
|
|
12 May 2023, Stefan Ritt, Info, New environment variable MIDAS_EXPNAME
|
A new environment variable MIDAS_EXPNAME has been introduced. This must be
used for cases where people use MIDAS_DIR, and is then equivalent for the
experiment name and directory usually used in the exptab file. This fixes
|
|
12 May 2023, Konstantin Olchanski, Info, New environment variable MIDAS_EXPNAME
|
> A new environment variable MIDAS_EXPNAME has been introduced [to be used together with
MIDAS_DIR]
|
|
20 Jun 2023, Stefan Ritt, Info, New environment variable MIDAS_EXPNAME
|
I just realized that we had already MIDAS_EXPT_NAME, and now people get confused with
MIDAS_EXPT_NAME
|
|
28 Jul 2023, Stefan Ritt, Info, New environment variable MIDAS_EXPNAME
|
Concerning naming of shared memories I went one step further due to some requirement of a local experiment.
The experiment needs to change the experiment name shown on the web status page depending on the exact
configuration, but we do not want to change the whole midas experiment each time.
|
|
24 Jul 2023, Nick Hastings, Bug Report, Incompatible data types with mysql odbc interface
|
Hello,
I have recently set up a midas-2022-05-c instance and have been trying to configure
|
|
25 Jul 2023, Nick Hastings, Bug Report, Incompatible data types with mysql odbc interface
|
Hello,
wanted add few things:
|