Dear All,

I have created a custom page (please see the attachment) and imported into 
MIDAS with key name "Control panel&" (without the ""). I have the following 
two questions:

1) I display the status of the run with <odb src="/Runinfo/State">, but it 
returns numbers which is not user friendly. How can I make something 
like "Running" with green background and "Stopped" with red background in the 
default status page?

2) When I click either Start/Stop/Pause/Resume, it can performs the right 
things, but afterward it jumps to the page "http://domain.name:8081/CS/" which 
shows "Invalid custom page: NULL path". How can I make it returns to the 
correct page "http://domain.name:8081/CS/Control%20panel"?

Thank you for your attention.

Best Regards,
Jimmy

Dear All,

I have one more question. I use <odb src="odb field" edit=1> to display an 
editable ODB value, but how can I show this value in hexadecimal?

Thanks.

Best Regards,
Jimmy


> Dear All,
> 
> I have created a custom page (please see the attachment) and imported into 
> MIDAS with key name "Control panel&" (without the ""). I have the following 
> two questions:
> 
> 1) I display the status of the run with <odb src="/Runinfo/State">, but it 
> returns numbers which is not user friendly. How can I make something 
> like "Running" with green background and "Stopped" with red background in the 
> default status page?
> 
> 2) When I click either Start/Stop/Pause/Resume, it can performs the right 
> things, but afterward it jumps to the page "http://domain.name:8081/CS/" which 
> shows "Invalid custom page: NULL path". How can I make it returns to the 
> correct page "http://domain.name:8081/CS/Control%20panel"?
> 
> Thank you for your attention.
> 
> Best Regards,
> Jimmy

> 1) I display the status of the run with <odb src="/Runinfo/State">, but it 
> returns numbers which is not user friendly. How can I make something 
> like "Running" with green background and "Stopped" with red background in the 
> default status page?

Sorry my late reply, I was really busy. You need JavaScript to perform such a 
task. See the attached example.

> 2) When I click either Start/Stop/Pause/Resume, it can performs the right 
> things, but afterward it jumps to the page "http://domain.name:8081/CS/" 
> which shows "Invalid custom page: NULL path". How can I make it returns 
> to the correct page "http://domain.name:8081/CS/Control%20panel"?

You add a hidden redirect statement:

  <input type=hidden name=redir value="CS/Control panel">

Best regards,

  Stefan

> I have one more question. I use <odb src="odb field" edit=1> to display an 
> editable ODB value, but how can I show this value in hexadecimal?

Again with JavaScript:

  var v = ODBGet('/some/path&format=%X');

this will retrieve /some/path and format it in hexadecimal. Then you can set a table 
cell with "v" as I wrote in the last reply. If you want to change this value 
however, you need to encode this yourself in JavaScript.

- Stefan

New version of MIDAS has a feature where it seems to automatically resize the
message page horizontally in order to fix each MIDAS message into one line. 
Some of my MIDAS messages (in particular error messages, where I need details)
are very long.  The result is that the MIDAS page automatically becomes very
wide and I have to scroll a lot left/right in order to read my messages.  This
is annoying.

I would vote to roll-back this new feature.

> New version of MIDAS has a feature where it seems to automatically resize the
> message page horizontally in order to fix each MIDAS message into one line. 
> Some of my MIDAS messages (in particular error messages, where I need details)
> are very long.  The result is that the MIDAS page automatically becomes very
> wide and I have to scroll a lot left/right in order to read my messages.  This
> is annoying.
> 
> I would vote to roll-back this new feature.

Issues should be reported in the bitbucket issue tracker. I moved this issue over there.

/Stefan

> online monitor would show (almost in real-time) the 
> gain, the dark noise, and the pedestal for all the channels, the 2D tracks 
> inside the detectors for each spill and so on.

Hmm... I now realize that the midas distribution does not include an example web page that 
integrates all these elements into one easy to understand html file.

I think an example page that would answer your questions and the questions from the other 
thread about starting/stopping runs, should include following elements:

- the general midas web page framework (the midas left-hand side menu, the top side 
status display, Stefan's new odb tags)
- buttons to start and stop runs (javascript code to call the run transition RPCs)
- embedded images for history display (old style gif and new style canvas)
- embedded images for ROOT histograms (via the ROOT http server and jsroot)
- code to live-update all these elements independantly from each other (to allow history 
plots and ROOT histograms to update at different frequencies).

As for the web page code for showing a mini-event-display, I think we do not know yet how 
to do - the event data lives inside the analyzer as C++ data structures, so somehow it 
needs to be encoded as json (this code is missing - but one can use the ROOT C++ to json 
encoder/streamer), needs to be transported to the web browser (we know how to do this) 
and at the end, plotting the json data on a canvas is the easy part.

I know some experiments have done all of this, and I think we should have such a pipeline 
available as part of the ROOTANA package. Maybe some day...

K.O.

Dear Thomas and Konstantin,

thank you very much for the feedback. I found the ROOTANA javascript display a good source of 
information and references.

As Thomas said, maybe the simplest thing would be to use the ROOT THttpServer. Honestly, I do 
not think that ROOT was ever meant to act as an online monitor due to its wacky memory 
management and abysmal multithread support. In other words, I think that by using ROOT we would 
inevitably lose some performance. 

Perhaps there are better ways of achieving the same goal. For example, I was leaning towards a 
plotly.js based approach where I would encode a series of vectors in base64 strings (for better 
transmission performance), send them to the client through the MJSONRPC mechanism, decode them 
and then feed them to plotly.js. But in this case, I should study many new libraries 
(plotly.js, the library for the base64 encoding, the Gaussian fitting, etc...) and I do not 
have the time to do that now: "beam is coming".

So ROOT it is. I will use the ROOTANA javascript display as a reference. Do you happen to know 
who wrote that part? 

In that example, you have some "static" histograms that you keep always in memory, while in our 
case the number of channels is so big that we have to dynamically generate the histograms only 
when needed (when the user select a single channel).

Best regards
Giorgio

> 
> As Thomas said, maybe the simplest thing would be to use the ROOT THttpServer. Honestly, I do 
> not think that ROOT was ever meant to act as an online monitor due to its wacky memory 
> management and abysmal multithread support. In other words, I think that by using ROOT we would 
> inevitably lose some performance.
>

Yes. The previous data analysis frameworks - PAW/PAW++/CERNlib (CERN), NOVA (TRIUMF) - certainly had
support for online monitoring. In CERNlib/PAW the histograms were stored in shared memory,
the analyzer running in the background was filling them, the PAW/PAW++ display was displaying
them "live". I was very surprised to find this function removed/not implemented in ROOT, given
that the same people were behind both projects (Rene Brun & co).

We tried to roll our own implementation of this in ROOTANA/ROODY, with mixed success.

I am glad the JSROOT project finally gained traction and web based "I can see the data" is now
available in ROOT.

> 
> Perhaps there are better ways of achieving the same goal. For example, I was leaning towards a 
> plotly.js based approach where 
>

There are many web/javascript graphics libraries out there, all have the weak spot - how do you
get your data into them?

Going forward, I see us standardizing on JSROOT: https://root.cern.ch/js/

>
> ... send them to the client through the MJSONRPC mechanism ...
>

I am not sure JSROOT have any support for interacting from the web page to the back-end analyzer. Perhaps
we can use the MIDAS MJSONRPC library for this. Hmm... (Note that the ROOT HTTP server is a derivative
of the mongoose web server library, which we use in mhttpd, so I already know how to work it)

>
> I would encode a series of vectors in base64 strings (for better transmission performance)
>

We looked into this when deciding on the data encoding for the midas history data. There is a tradeoff
between network use and cpu use - to save on the network, you try to reduce the data size by using
compressed binary data - to save on the CPU you try to minimize data encoding.

For history data, we gave up on binary json (extra decoding needed), gave up on text json (extra decoding 
needed), gave up on compression (extra cpu use for decompression) and use javascript native binary processing
("arraybuffer").

Our thinking is that network bandwidth is usually quite big and is getting bigger, but cpu resource is limited
and is expensive. (mobile devices seems to be stuck with ~2 GHz CPUs; cpu use means battery use and
battery capacity is limited, not improving quickly)

> 
> So ROOT it is. I will use the ROOTANA javascript display as a reference. Do you happen to know 
> who wrote that part? 
> 

Yes. See "Contact" at https://root.cern.ch/js/

>
> In that example, you have some "static" histograms that you keep always in memory, while in our 
> case the number of channels is so big that we have to dynamically generate the histograms only 
> when needed (when the user select a single channel).
> 

This requires interaction with the analyzer, requires some kind of RPC mechanism. I am now curious what jsroot 
have, also it would not be too hard to add the mjsonrpc library to rootana. Cooperation from ROOT multithreading
is not required: I can queue the RPC requests in a separate (thread safe, non-ROOT) buffer, then process
them in the ROOT main event loop (this is how the ROOTANA histogram server worked in the days when
ROOT had no multithread support at all).

K.O.

Hi,

I need to install MIDAS on a Windows system (Microsoft Windows Server 2003). 
The computer has the Microsoft Visual C++ 2010 Express version.
I have downloaded the MIDAS packages using the tarball mechanism. I have create 
the environment variables without problems and I have create the file           
%SystemRoot%\system32\exptab 
But when I try to build MIDAS and I do 
nmake -f makefile.nt
I have the following problem:
Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
Copyright (C) Microsoft Corporation.  All rights reserved.

NMAKE : fatal error U1073: don't know how to make 'src/mhttpd.c'
Stop.

I don't understand this problem. Can anybody help me, please?

Thanks in advance!!!

Hi Raquel,

The makefile.nt has been corrected.
Obviously Midas on Windows has not been updated for quite a while.
mhttpd.c has been converted to c++ (mhttpd.cxx) as well as a couple of other 
applications.

Please give a try,  PAA

> Hi,
> 
> I need to install MIDAS on a Windows system (Microsoft Windows Server 2003). 
> The computer has the Microsoft Visual C++ 2010 Express version.
> I have downloaded the MIDAS packages using the tarball mechanism. I have create 
> the environment variables without problems and I have create the file           
> %SystemRoot%\system32\exptab 
> But when I try to build MIDAS and I do 
> nmake -f makefile.nt
> I have the following problem:
> Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
> Copyright (C) Microsoft Corporation.  All rights reserved.
> 
> NMAKE : fatal error U1073: don't know how to make 'src/mhttpd.c'
> Stop.
> 
> I don't understand this problem. Can anybody help me, please?
> 
> Thanks in advance!!!

Hi Pierre-André, 

sorry for the long delay, another things keep me out of this computer.
Thanks a lot for correcting makefile.nt and the other applications!

Now I have try, downloading the MIDAS packages from the tarball mechanism, as
before,
and now it seems that the previous problems are solved. It remains only one small
problem, it is related to the odbedit.

I attach here the figure with the error that is reported by the computer. Is it
possible that is another file that needs to be updated? Can you help me with that?

Thanks a lot in advance!!!!



> Hi Raquel,
> 
> The makefile.nt has been corrected.
> Obviously Midas on Windows has not been updated for quite a while.
> mhttpd.c has been converted to c++ (mhttpd.cxx) as well as a couple of other 
> applications.
> 
> Please give a try,  PAA
> 
> > Hi,
> > 
> > I need to install MIDAS on a Windows system (Microsoft Windows Server 2003). 
> > The computer has the Microsoft Visual C++ 2010 Express version.
> > I have downloaded the MIDAS packages using the tarball mechanism. I have create 
> > the environment variables without problems and I have create the file           
> > %SystemRoot%\system32\exptab 
> > But when I try to build MIDAS and I do 
> > nmake -f makefile.nt
> > I have the following problem:
> > Microsoft (R) Program Maintenance Utility Version 10.00.30319.01
> > Copyright (C) Microsoft Corporation.  All rights reserved.
> > 
> > NMAKE : fatal error U1073: don't know how to make 'src/mhttpd.c'
> > Stop.
> > 
> > I don't understand this problem. Can anybody help me, please?
> > 
> > Thanks in advance!!!

> I have a time series of slow control measurements in an ASCII format - 
> data records in a format (run_number, time, temperature, voltage1, ..., voltageN), 
> and, if possible, would like to convert them into a MIDAS history format. 
> 
> Making MIDAS events out of that data is easy, but is it possible to preserve 
> the time stamps?  - Logically, this boils down to whether it is possible to have  
> the event time set by a user frontend

It looks that the original question was not as naive as I expected and may be pointing to a subtle bug. 
I have implemented a python frontend - essentially a clone of 

https://bitbucket.org/tmidas/midas/src/develop/python/midas/frontend.py

reading the old slow control data and setting the event.header.timestamp's to some dates from the year of 2022. 

When I run MIDAS and read the "old slow control events", one event in 10 seconds, 
the MIDAS Event Dump utility shows the data with the correct event timestamps, from the year of 2022. 

However the history plots show the event parameters with the timestamps from Feb 01 2025 and the adjacent 
data points separated by 10 sec.

Is it possible that the history system uses its own timestamp setting instead of using timestamps from the event headers? 
- Under normal circumstances, the two should be very close, and that could've kept the issue hidden... 

-- thanks, regards, Pasha

UPDATE:  I attached the frontend code and the input data file it is reading. The data file should reside in the local directory
- the frontend code doesn't have everything fully automated for the test, 
  -- an integer field "/Mu2e/Offline/Ops/LastTime" would need to be created manually
  -- the history plots would need to be declared manually

Dear MIDAS experts, 

I confirm that when writing out history files corresponding to the slow control event data, 
MIDAS history system timestamps the data not with the event time coming from the event data, 
but with the current time determined by the program - 

https://bitbucket.org/tmidas/midas/src/293d27fad0c87c80c4ed7b94b5c40ba1e150bea4/progs/mlogger.cxx#lines-5321

where 'now' is defined as  

time_t now = time(NULL);

I'm looking for a way to timestamp the history data with the event time - that is important 
for HEP applications outside the DAQ domain. Yes, MIDAS infrastructure is very well suited for that, 
there could have a number of such applications, and experiments could significantly benefit from that.

So I'm wondering whether the implementation is a design choice made or it could be changed. 

The change itself and especially its validation may require a non-negligible amount of work - I'd be happy to contribute.

Any insight much appreciated. 

-- thanks, regards, Pasha

> I confirm that when writing out history files corresponding to the slow control event data, 
> MIDAS history system timestamps the data not with the event time coming from the event data, 
> but with the current time determined by [mlogger].

This is correct. The timestamp in the history file is the mlogger timestamp.

In theory we could use the ODB "last_written" timestamp, but in practice,
timestamps are 1 second granularity and the difference between the two
timestamps normally would be less than 1 second. (time to react to db_watch()).

But ODB last_written also is not the data timestamp. For remote connected clients
it includes the mserver communication delay.

What is the data timestamp, only the user knows - for some FPGA based equipments,
I can see the data timestamp being read from an FPGA register together with the data.

But back to earth.

For making history plots, 1 second granularity with a small (a few seconds) delay should be okey,
and I think the mserver timestamp is good enough.

For data analysis, you are reading history data from a history data file and you are
not constrained to using the MIDAS timestamp.

You can always include your "true" data timestamp as the first value in your data.

We do this in felaview for writing labview data to midas history in the ALPHA antihydrogen experiment at CERN.

This also anticipates your next request, can we have millisecond, microsecond, nanosecond history timestamps:
since you define your "true" data timestamp, you an make it anything you want. (I use "double" time in seconds,
64-bit IEEE-754 "double" has enough precision for microsecond granularity. FPGA based devices can have timestamps
with 10 ns or 8 ns granularity, in this case a uint64_t clock counter could be more appropriate).

K.O.

> You can always include your "true" data timestamp as the first value in your data.

Are you saying that if the first data word of a history event were a timestamp, 
the MIDAS history system, when plotting the time dependencies, would use that timestamp 
instead of the mlogger timestamp?  

if that is true, what tells MIDAS that the first data word is the timestamp? 

I couldn't find a discussion of that on the page describing the history system - 

 https://daq00.triumf.ca/MidasWiki/index.php/History_System#Frontend_history_event

- perhaps I should be looking at a different page?

-- thanks again, regards, Pasha 

> > You can always include your "true" data timestamp as the first value in your data.
> 
> Are you saying that if the first data word of a history event were a timestamp, 
> the MIDAS history system, when plotting the time dependencies, would use that timestamp 
> instead of the mlogger timestamp?
>

you are correct, midas knows nothing about what you put in the history data.

what I suggested is: if you want your true data timestamp recorded in the history,
you can put it into the history data yourself, and I suggested using the 1st value,
but you can also make it the last value or the 10th value, it is up to you.

for making history plots, the history timestamp is used, as you wrote and I confirmed,
this timestamp is generated by mlogger.

what is not clear to me is why this is a problem? do you see a big difference between the 
true data timestamp and the mlogger data timestamp? bigger than 1 second? (this would change 
the shape of "last 10 minutes" plots (600 seconds). bigger than 1 minute? (this would change 
the shape of "last 1 hour plots" (60 minutes, 3600 seconds).

that said, note that we currently store the timestamp as a DWORD 32-bit UNIX time value 
which will overflow in 2038 and which is quickly becoming incompatible with the ongoing 
switch to 64-bit time_t. Ubuntu-24 already build a large number of system libraries with 64-
bit time_t and building MIDAS with 32-bit time_t may soon become as difficult as building 
32-bit MIDAS for 32-bit i686 VME processors. we have to move with the times.

what it means is that the history system data format will have to be updated to 64-bit 
time_t and at the same time, we may try to change the timestamp from mlogger-generated to 
frontend-generated.

but it is still not clear to me how that helps you, because the frontend-generated timestamp 
is not the true data timestamp that you wanted. (and only you know what the true data 
timestamp is and where it comes from and how to tell it to MIDAS).

K.O.

We have tested operation of MIDAS using a 10GigE network connection. Using a dummy frontend 
generating fake data, we can record MIDAS data to disk at at least 700 Mbytes/sec as reported by 
the MIDAS status page.

Two configurations were tested, both run at at least 700Mbytes/sec sustained:

1) MIDAS mhttpd, mserver, mlogger running on the disk server machine (mlogger writes to local 
disk), frontend running on remote machine (10GigE mserver connection).
2) MIDAS mhttpd, mserver, mlogger, frontend running on remote machine (mlogger writes data to 
an NFS-mounted disk over a 10GigE connection).

In addition, for configuration (2), I simulated online analysis reading fresh midas files at the same 
time as MIDAS writes new data. The resulting observation is that Linux seems to be giving main 
priority to disk write traffic (700 Mbytes/sec) with the remaining disk bandwidth given to read traffic 
(50-100Mbytes/sec). In other words, when running online data analysis on fresh data files, mlogger 
continues to run at full speed (analysis does not slow down data taking).

A few problems with MIDAS were observed during this test:

a) mlogger data compression using gzip-1 has to be turned off (limits data rate to about 
200Mbytes/sec). We plan to implement high speed LZO/LZ4 data compression that we expect to 
keep up with a 10GigE network interface.
b) CPU use by mserver and mlogger is rather high (about 40% CPU)
c) when writing to the NFS disk, mlogger has a pause of 1-2 seconds when closing and reopening 
subrun data files. To avoid a interruption in data taking, the SYSTEM event buffer has to be big 
enough to ride through this pause, but stock MIDAS limits the maximum size of event buffer to 1GB 
(too small), this can be easily increased to 2GB (almost big enough) and with some more work it can 
be increased to 4GB, but no more because the buffer length is a 32-bit integer.
d) when writing to the NFS disk, we also see periodic 3-5 second interruptions ("write operation took 
5123 ms") and we had one death of mlogger by a timeout of 60 sec.

Details of the hardware:

1) the disk server machine CPU is 3.4GHz Intel i7-4770, mobo is ASUS Z87 WS (10 SATA, 2xGigE), 
RAM is 32GB DDR3-1600.
2) disk array is 8x4TB Seagate ST4000VN000-1H4168 NAS disks RAID0 (striped) configuration, raw 
data read/write rate is around 1 GByte/sec, disks are directly attached to mobo (no raid card), linux 
software raid.
3) the frontend machine CPU is 3.7GHz Intel i7-4820, mobo is ASUS P9X79 WS, RAM is 32GB DDR3-
1600.
4) 10GigE network is Solarflare Communications SFC9120 (both machines) with a cross-over fiber 
cable (direct connection,no switches)
5) OS is up-to-date SL6.5 (both machines)

K.O.