Hello,

We have problems analyzing large files under Windows XP. For small file sizes,
everything is ok. We have events of 2.8 MB each, and we can read ~30 events per
second. But if the file gets larger than typically 600-800 MB, then access
becomes very slow, about 1 event per second. This is not the case under Linux,
where it stays at 30 Hz (~90 MB/sec). 

Looking at the low level file access, it is obvious that this has nothing to do
with midas, this problem can be reproduced with a simple program reading chunks
of 3MB from a 1GB file. The Windows XP file system is NTFS, default formatting.
Does anyone else have observed a similar problem or maybe even have some
suggestions? Unfortunately many people here want to analyze midas data under
Windows...

Stefan Ritt

Hello,

Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

thanks,
Zaher

> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.

Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
and sends the numeric data as an ASCII string. The first four chars of the data is the name
of the MIDAS data bank, second number is the data timestamp in seconds.

LCRY 1234567 1.1 2.2 3.3

A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
labview component. Please contact him directly for more information.

I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).

K.O.

Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

Zaher

> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.
> 
> Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
> and sends the numeric data as an ASCII string. The first four chars of the data is the name
> of the MIDAS data bank, second number is the data timestamp in seconds.
> 
> LCRY 1234567 1.1 2.2 3.3
> 
> A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
> labview component. Please contact him directly for more information.
> 
> I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).
> 
> K.O.

> Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

https://bitbucket.org/expalpha/a2daq/src/alpha/src/felabview.cxx

this is code circa 2006, there are now better ways to do some of that coding.

if you want bidirectional communication with labview, read/write odb, etc, simplest is probably
to write the "mjserver" that talks midas json-rpc over plain tcp, without all the http/https
gunk you need to go through mhttpd.

K.O.

> Hello,
> 
> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> thanks,
> Zaher

We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.

Stefan

> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
>

Our parallel experience with the CERN ALPHA anti-hydrogen experiment: they have developed a whole labview empire
to control the cryogenics, the magnets, the positron source, the anti-proton trap, anti-hydrogen trap, etc.

At some point there was a wall of monitors in the counting room - each labview computer controlled one or two things -
so there is very many computers and each had to have a monitor (and mouse and keyboard).

All the data from this labview empire is logged to MIDAS history via felabview and feGEM, and they use
the MIDAS history to look and monitor almost everything. Control is done via Labview and Labview
based FPGA sequencers (National Instruments PXI hardware, $$$$$).

This works reasonably well to publish several papers in Nature.

But not 100%:

1) difficulties with labview source control (cannot be trivially managed by git, I guess)
2) unending fight against Microsoft and CERN IT trying to reboot the computers at the wrong time
3) more recently, forced Microsoft updates require trashing perfectly good machines and buy new ones

At TRIUMF there is very little Labview. All experiments use MIDAS and EPICS for most things.

Based on this experience, I agree with Stefan, today's sweet spot is RaspberryPi machines with USB attached
gizmos to control stuff. On the software side, drive the mess with MIDAS and custom web pages.

K.O.

Thanks Stefan, I would be very interested to see your code. At moment we have magnets and cryostats (3He and dilution) being delivered with Labview.



> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
> 
> Stefan

Hey everyone,

When I run mhttpd I get the following error message:

[mhttpd,ERROR] [odb.cxx:1720:db_open_database,ERROR] Removed ODB client 
'mhttpd', index 0 because process pid 4531 does not exists
[mhttpd,INFO] Removed open record flag from "/Experiment/Security/RPC 
hosts/Allowed hosts"
[mhttpd,INFO] Removed exclusive access mode from "/Experiment/Security/RPC 
hosts/Allowed hosts"
[mhttpd,INFO] Removed open record flag from "/Experiment/Security/mhttpd 
hosts/Allowed hosts"
[mhttpd,INFO] Removed exclusive access mode from "/Experiment/Security/mhttpd 
hosts/Allowed hosts"
[mhttpd,INFO] Removed open record flag from "/Sequencer/State"
[mhttpd,INFO] Removed exclusive access mode from "/Sequencer/State"
[mhttpd,INFO] Corrected 3 ODB entries
[mhttpd,INFO] Deleted entry '/System/Clients/4531' for client 'mhttpd' because 
it is not connected to ODB
[mhttpd,INFO] Client 'mhttpd' on buffer 'SYSMSG' removed by bm_open_buffer 
because process pid 4531 does not exist
Mongoose web server will not use password protection
mongoose web server is listening on the HTTP port 8080

So mhttpd works as I have access to it through my browser but mlogger does not 
work when I try running it (Alarm: Program Logger is not running). I've 
managed to get mlogger working before and I think that the problem might be 
from maybe having another instance of ODB running without me knowing. 

Has anyone ever had this issue?

Thanks so much for your time.

Isaac

Over the last 10 years, the traditional C/C++ data types have been
displaced by a hodgepodge of new data types that promise portability
and generate useful (and not so useful) warnings, for example:

for (int i=0; i<array_of_10_elements.size(); i++)

is now a warning with a promise of crash (in theory, even if "int" is 64 bit).

"int" and "long" are dead, welcome "size_t", "off64_t" & co.

What to do, what to do? This is what I figured out:

1) for data returned from hardware: use uint16_t, uint32_t, uint64_t, uint128_t (u16, u32, u64 in 
the Linux kernel), they have well defined width to match hardware (FPGA, AXI, VME, etc) data 
widths.

2) for variables used with strlen(), array.size(), etc: use size_t, a data type wide enough to 
store the biggest data size possible on this hardware (32-bit on 32-bit machines, 64-bit on 64-bit 
machines). use with printf("%zu").

3) for return values of read() and write() syscalls: use ssize_t and observe an inconsistency, 
read() and write() syscalls take size_t (32/64 bits), return ssize_t (31/63 bits) and the error 
check code cannot be written without having to defeat the C/C++ type system (a cast to size_t):

size_t s = 100;
void* ptr = malloc(s);
ssize_t rd = read(fd, ptr, s);
if (rd < 0) { syscall error }
else if ((size_t)rd != s) { short read, important for TCP sockets }
else { good read }

use ssize_t with printf("%zd")

4) file access uses off64_t with lseek64() and ftruncate64(), this is a signed type (to avoid the 
cast in the error handling code) with max file size 2^63 (at $/GB, storage for a file of max size 
costs $$$$$, you cannot have enough money to afford one). use with printf("%jd", (intmax_t)v). 
intmax_t by definition is big enough for all off64_t values, "%jd" is the corresponding printf() 
format.

5) there is no inconsistency between 32-bit size_t and 64-bit off64_t, on 32-bit systems you can 
only read files in small chunks, but you can lseek64() to any place in the file.

BTW, 64-bit time_t has arrived with Ubuntu LTS 24.04, I will write about this some other time.

> for (int i=0; i<array_of_10_elements.size(); i++)

becomes

for (size_t i=0; i<array.size(); i++)

but for a reverse loop, replacing "int" with "size_t" becomes a bug:

for (size_t i=array.size()-1; i>=0; i--)

explodes, last iteration should be with i set to 0, then i--
wraps it around a very big positive value and loop end condition
is still true (i>=0), the loop never ends. (why is there no GCC warning
that with "size_t i", "i>=0" is always true?

a kludge solution is:

for (size_t i=array.size()-1; ; i--) {
do_stuff(i, array[i]);
if (i==0) break;
}

if you do not need the index variable, you can use a reverse iterator (which is missing from a few 
container classes).

K.O.

Hi all,

I am encountering a Javascript error (TypeError: client.error is undefined) when 
I transition between run states. Does anybody have an idea of what my problem 
might be? I have pasted an example of what MIDAS logs during such sequences.

Thanks for all the help!

Isaac


09:24:08.611 2021/02/10 [mhttpd,INFO] Executing script 
"~/ANIS_20210106/scripts/start_daq.sh" from ODB "/Script/Start DAQ"

09:24:13.833 2021/02/10 [Logger,LOG] Program Logger on host localhost started

09:24:28.598 2021/02/10 [fevme,LOG] Program fevme on host localhost started

09:24:33.951 2021/02/10 [mhttpd,INFO] Run #234 started

09:26:30.970 2021/02/10 [mhttpd,ERROR] [midas.cxx:4260:cm_transition_call,ERROR] 
Client "Logger" transition 2 aborted while waiting for client "fevme": 
"/Runinfo/Transition in progress" was cleared

09:26:31.015 2021/02/10 [mhttpd,ERROR] [midas.cxx:5120:cm_transition,ERROR] 
transition STOP aborted: "/Runinfo/Transition in progress" was cleared

09:27:27.270 2021/02/10 [mhttpd,ERROR] 
[system.cxx:4937:ss_recv_net_command,ERROR] timeout receiving network command 
header

09:27:27.270 2021/02/10 [mhttpd,ERROR] [midas.cxx:12262:rpc_client_call,ERROR] 
call to "fevme" on "localhost" RPC "rc_transition": timeout waiting for reply

> I am encountering a Javascript error (TypeError: client.error is undefined) when 
> I transition between run states. Does anybody have an idea of what my problem 
> might be? I have pasted an example of what MIDAS logs during such sequences.


Not enough information. Can you do this:

a) for the javascript error, if you get it every time, open the javascript debugger 
and capture the stack trace? or at least the file name, function name and line number 
where the javascript exception is thrown?

b) for the run start failure, start the run from odbedit "start now -v" or from 
"mtransition -v -d 1 START" (or "stop" as the case may be). capture the output, email 
to me directly or put in this elog here.

K.O.


> 
> Thanks for all the help!
> 
> Isaac
> 
> 
> 09:24:08.611 2021/02/10 [mhttpd,INFO] Executing script 
> "~/ANIS_20210106/scripts/start_daq.sh" from ODB "/Script/Start DAQ"
> 
> 09:24:13.833 2021/02/10 [Logger,LOG] Program Logger on host localhost started
> 
> 09:24:28.598 2021/02/10 [fevme,LOG] Program fevme on host localhost started
> 
> 09:24:33.951 2021/02/10 [mhttpd,INFO] Run #234 started
> 
> 09:26:30.970 2021/02/10 [mhttpd,ERROR] [midas.cxx:4260:cm_transition_call,ERROR] 
> Client "Logger" transition 2 aborted while waiting for client "fevme": 
> "/Runinfo/Transition in progress" was cleared
> 
> 09:26:31.015 2021/02/10 [mhttpd,ERROR] [midas.cxx:5120:cm_transition,ERROR] 
> transition STOP aborted: "/Runinfo/Transition in progress" was cleared
> 
> 09:27:27.270 2021/02/10 [mhttpd,ERROR] 
> [system.cxx:4937:ss_recv_net_command,ERROR] timeout receiving network command 
> header
> 
> 09:27:27.270 2021/02/10 [mhttpd,ERROR] [midas.cxx:12262:rpc_client_call,ERROR] 
> call to "fevme" on "localhost" RPC "rc_transition": timeout waiting for reply

> > I am encountering a Javascript error (TypeError: client.error is undefined) when 
> > I transition between run states. Does anybody have an idea of what my problem 
> > might be? I have pasted an example of what MIDAS logs during such sequences.
> 
> 
> Not enough information. Can you do this:
> 
> a) for the javascript error, if you get it every time, open the javascript debugger 
> and capture the stack trace? or at least the file name, function name and line number 
> where the javascript exception is thrown?

I've attached a screenshot of the call stack showing the file names and line numbers.

> b) for the run start failure, start the run from odbedit "start now -v" or from 
> "mtransition -v -d 1 START" (or "stop" as the case may be). capture the output, email 
> to me directly or put in this elog here.

I have also attached a screen capture of the output.

Thanks for your help as always.

Isaac

> K.O.
> 
> 
> > 
> > Thanks for all the help!
> > 
> > Isaac
> > 
> > 
> > 09:24:08.611 2021/02/10 [mhttpd,INFO] Executing script 
> > "~/ANIS_20210106/scripts/start_daq.sh" from ODB "/Script/Start DAQ"
> > 
> > 09:24:13.833 2021/02/10 [Logger,LOG] Program Logger on host localhost started
> > 
> > 09:24:28.598 2021/02/10 [fevme,LOG] Program fevme on host localhost started
> > 
> > 09:24:33.951 2021/02/10 [mhttpd,INFO] Run #234 started
> > 
> > 09:26:30.970 2021/02/10 [mhttpd,ERROR] [midas.cxx:4260:cm_transition_call,ERROR] 
> > Client "Logger" transition 2 aborted while waiting for client "fevme": 
> > "/Runinfo/Transition in progress" was cleared
> > 
> > 09:26:31.015 2021/02/10 [mhttpd,ERROR] [midas.cxx:5120:cm_transition,ERROR] 
> > transition STOP aborted: "/Runinfo/Transition in progress" was cleared
> > 
> > 09:27:27.270 2021/02/10 [mhttpd,ERROR] 
> > [system.cxx:4937:ss_recv_net_command,ERROR] timeout receiving network command 
> > header
> > 
> > 09:27:27.270 2021/02/10 [mhttpd,ERROR] [midas.cxx:12262:rpc_client_call,ERROR] 
> > call to "fevme" on "localhost" RPC "rc_transition": timeout waiting for reply

> 
> I have also attached a screen capture of the output.
> 

so the error is gone?


> Thanks for your help as always.
> 
> Isaac
> 
> > K.O.
> > 
> > 
> > > 
> > > Thanks for all the help!
> > > 
> > > Isaac
> > > 
> > > 
> > > 09:24:08.611 2021/02/10 [mhttpd,INFO] Executing script 
> > > "~/ANIS_20210106/scripts/start_daq.sh" from ODB "/Script/Start DAQ"
> > > 
> > > 09:24:13.833 2021/02/10 [Logger,LOG] Program Logger on host localhost started
> > > 
> > > 09:24:28.598 2021/02/10 [fevme,LOG] Program fevme on host localhost started
> > > 
> > > 09:24:33.951 2021/02/10 [mhttpd,INFO] Run #234 started
> > > 
> > > 09:26:30.970 2021/02/10 [mhttpd,ERROR] [midas.cxx:4260:cm_transition_call,ERROR] 
> > > Client "Logger" transition 2 aborted while waiting for client "fevme": 
> > > "/Runinfo/Transition in progress" was cleared
> > > 
> > > 09:26:31.015 2021/02/10 [mhttpd,ERROR] [midas.cxx:5120:cm_transition,ERROR] 
> > > transition STOP aborted: "/Runinfo/Transition in progress" was cleared
> > > 
> > > 09:27:27.270 2021/02/10 [mhttpd,ERROR] 
> > > [system.cxx:4937:ss_recv_net_command,ERROR] timeout receiving network command 
> > > header
> > > 
> > > 09:27:27.270 2021/02/10 [mhttpd,ERROR] [midas.cxx:12262:rpc_client_call,ERROR] 
> > > call to "fevme" on "localhost" RPC "rc_transition": timeout waiting for reply

> 
> I have also attached a screen capture of the output.
> 

so the error is gone?

K.O.

> 
> I have also attached a screen capture of the output.
> 

so the error is gone?

K.O.

> > 
> > I have also attached a screen capture of the output.
> > 
> 
> so the error is gone?
> 
> K.O.

Hi K.O.,

No the error persists, 

I switched mhttpd to use the new javascript based run start and stop pages.

There are two new html pages:

resources/start.html - mimics the old run start page exactly - where you can enter the "edit on 
start" parameters and start the run.
resources/transition.html - monitors the transition progress, shows the status of every transition 
client, their sequence number, waiting list dependency, time spent making rpc calls, etc.

If the new pages do not work for you, please report it here and switch to the old pages
by editing src/mhttpd.cxx - comment-out the line "#define NEW_START_STOP 1"

K.O.

> I switched mhttpd to use the new javascript based run start and stop pages.

One initial complaint: the transition.html page doesn't seem to deal well with a frontend program using
a deferred transition.  Specifically, I find with my simulated frontend ([1]), which has a deferred
end-of-run transition, that two problems happen:

i) the page doesn't give any indication that a frontend has a deferred transition; in fact it says that
the frontend immediately has finished the transition.
ii) once the deferred transition has finished, the page doesn't switch to saying that the run has
stopped.  In fact, even if I reload the transition page it still continues to show that the run is
ongoing; the status page, by contrast, shows that the run has stopped.

I separately still think that the transition page should automatically go away after 5 seconds
(assuming that all the transitions were successful).  I think it is annoying that you need to click
back to the status page.

[1] https://github.com/thomaslindner/fesimdaq