I added now code to the logger so it properly complains if there would be a leading space in a variable name.

Stefan

> By accident we had history variables with leading spaces. The history schema check then decides that this is a new variable (the leading space is not read from the history file) and starts a new file. We found this because the run start became slow due to the many, many history files created. It would be nice to just get an error if one has a malformed variable name like this.
> 
> How to reproduce: Try to put a variable with a leading space in the name into the history, repeatedly start runs.
> Sugested fix: Produce an error if a history variable has a leading space.

Since this memory leak is quite obvious, I pushed the fix to develop.

Stefan

> Perhaps have:
> 
> set_write_cache_size("SYSTEM", 0);
> set_write_cache_size("BUF1", bigsize);
> 
> with an internal std::map<std::string,size_t>; for write cache size for each named buffer

Ok, this is implemented now in mfed.cxx and called from examples/experiment/frontend.cxx

Stefan

> > Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.
> 
> Yes, I hadn't realised that was an option. For that to work I guess the aforementioned static members could be made thread local storage, and 
> processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
> class member or something.

If we want to analyze several runs, I can easily add code to make this possible. In a new call to set_odb_source(), the previously allocated memory in that function can be freed. We can aldo make the memory handling 
thread-specific, allowing several thread to analyze different runs at the same time. But I will only invest work there once it's really needed by someone.

Stefan

I will work today on the odbxx API to make sure there are no memory leaks when you switch form one file to another. I talked to KO so he agreed that yo then commit your proposed change of manalyzer 

Best,
Stefan

> > I will work today on the odbxx API to make sure there are no memory leaks when you switch form one file to another. I talked to KO so he agreed that yo then commit your proposed change of manalyzer 
> 
> That, and add a "clear()" method that resets odbxx state to "empty". I will call odbxx.clear() everywhere where I call "delete fOdb;" (TARunInfo::dtor and other places).

No need for clear(), since no memory gets allocated by midas::odd::set_odb_source(). All it does is to remember the file name. When you instantiate a midas::odd object, the file gets loaded, and the midas::odd object gets initialized from the file contents. Then the buffer 
gets deleted (actually it's a simple local variable). Of course this causes some overhead (each midas::odd() constructor reads the whole file), but since the OS will cache the file, it's probably not so bad.

Stefan

> > > Of course with this patch extending manalyzer to process two or more runs at the same time becomes impossible.
> > Yes, I hadn't realised that was an option.
> 
> It is an option I would like to keep open. Not too many use cases, but imagine a "split brain" experiment
> that has two MIDAS instances record data into two separate midas files. (if LIGO were to use MIDAS,
> consider LIGO Hanford and LIGO Livingston).
> 
> Assuming data in these two data sets have common precision timestamps,
> our task is to assemble data from two input files into single physics events. The analyzer will need
> to read two input files, each file with it's run number, it's own ODB dump, etc, process the midas
> events (unpack, calibrate, filter, etc), look at the timestamps, assemble the data into physics events.
> 
> This trivially generalizes into reading 2, 3, or more input files.
> 
> > For that to work I guess the aforementioned static members could be made thread local storage, and 
> > processing of each run kept to a specific thread. Although I could imagine user code making assumptions and breaking, like storing a midas::odb as a 
> > class member or something.
> 
> manalyzer is already multithreaded, if you will need to keep track of which thread should see which odbxx global object,
> seems like abuse of the thread-local storage idea and intent.

I made the global variables storing the file name of type "thread_local", so each thread gets it's own copy. This means however that each thread must then call midas::odb::set_odb_source() individually before 
creating any midas::odb objects. Interestingly enough I just learned that thread_local (at least under linux) is almost zero overhead, since these variable are placed by the linker into a separate memory space which is 
separate for each thread, so accessing them only means to add a memory offset.

Let's see how far we get with this...

Stefan

Sorry to intervene there, but the FEs are usually compiled against libmidas.a . Therefore you have to compile midas, usually do a "make install" to update the libmidas.a/so, then recompile the FEs. You probably forgot the "make install".

Stefan

I confirm the problem is there (at least under MacOSX Safari) and I will take care of it.

Stefan

> We would like to write an sc_frontend in Python instead of C++. All our drivers 
> work correctly, as well as the creation of the database in the ODB, including the 
> creation of Commons, Statistics, Variables, and Settings.
> However, we are unable to correctly create the database entries needed to manage 
> History and Display.
> 
> As we understand it, in C++ this is handled by setting the EQ_SLOW flag, which 
> doesn’t seem to be implemented in the Python libraries.
> How can we manually create the necessary variables for Display and History?

I'm not an expert of the Python part of MIDAS (Ben Smith is), but I know that you have
functions to create keys and set values in the ODB, so you should be able to create
these things manually as you need them.

Stefan

This problem was introduced by ZS in March 2023 with these commits:

https://bitbucket.org/tmidas/midas/commits/25b13f875ff1f7e2f4e987273c81d6356dd2ff53
https://bitbucket.org/tmidas/midas/commits/2a9e902e07156e12edecb5c2257e4dbd944f8377

by setting

 d.style.position = "fixed";

which prevents the scrolling. I have no idea why this change was made, so it should be fixed by the original 
author.

Stefan

Hello!

My name is Soichiro Kuribayashi and I am a Ph.D. student at Kyoto University. 
I'm a T2K collaborator and working for Super FGD which is new detector in ND280.

I'm a beginner of MIDAS and I've just started to develop the DAQ software with 
MIDAS for Super FGD.
For the DAQ of Super FGD, we will run remotely front end part of MIDAS on ZYNQ 
which is system on chip.

For this remote control of front end part with mserver, we have to mount home 
directory of DAQ PC(Cent OS8) on that of Linux on ZYNQ.
So I wonder if we should use NFS(Network file system) + NIS(Network information 
service) + autofs for the mounting. Is it correct?

If you have any information or any suggestion for the remote control on chip, 
please let me know.

Best regards,
Soichiro 

Dear Konstantin,

Thank you very much for your reply and detailed information.
I would appreciate if you could help us.

> I can also suggest that on your frontend SOC/FPGA machine, you boot linux
> using the "nfs-root" method. This way, the local flash memory only
> contains a boot loader (and maybe the linux kernel image, depending on
> bootloader limitations). The rest of the linux rootfs can be on your
> central development machine. This way management of flash cards,
> confusion with different contents of local flash and need to make backups
> of frontend machines is much reduced.

As you said, we can run complete Linux (Ubuntu 16) on ZYNQ and I'm using common NFS 
system now. However, I didn't know "nfs-root" method which you mentioned and this method 
seems to be reasonable way to just share linux rootfs.
First of all, I will try this method for simpler system.

> If you use a fast SSD and ZFS with deduplication, you will also have good
> performance gain (NFS over 1gige network to server with fast SSD works
> so much better compared to the very slow SD/MMC/NAND flash).
>
> I can point you to some of my documentation how we do this.

I'm concerned about such performance and I have checked the performance with common NFS 
over gige network and my DAQ PC roughly(data transfer rate ~ O(10) MByte/sec). However, I 
didn't know the ZFS and also how we can have performance gain with a fast SSD and ZFS.
Please let me know your documentation how to do it if possible.

> I would say you are on a good track. For early development on just one board,
> pretty much any way you do it will work, but once you start scaling up
> beyound 3-4-5 frontends, you will start seeing benefits from common NFS-mounted
> home directories, NFS-root booted linux, etc.

I'm developing with just one board and common NFS-mounted now. I'm looking forward to 
seeing such benefits when I will use multiple frontends.
 
> And of course you may want to study the existing ND280/FGD DAQ. I hope you
> have access to the running system at Jparc. If not, I have a copy of
> pretty much everything (except for running hardware, it is stored in the basement, 
> dead) and I can give you access.

I don't have access to the system at Jparc, but Nick has told us where FGD DAQ code is.
Is bellow URL everything of code of FGD DAQ?
https://git.t2k.org/hastings/fgddaq/-/tree/master

Best regards,
Soichiro

Dear Stefan,

Thank you very much for your help.

I have already contacted someone who has used ZYNQ in that order and It's working fine for now.
But, I'll let you know if something goes wrong.

Best regards,
Soichiro 

Hi,

I've written a python script that reads some data from a file and generates a
.png image. I want to have a button on my MIDAS status page that:

- executes the script and waits for it to finish,
- then displays the image

How can I do that? I tried using the sequencer to just execute the script every
30 seconds, but I can't get it to work, and it would be better to only execute
the script on demand anyway. 

I also am having trouble getting image display to work. I have the ODB keys set:

[local:oven1:S]/Custom>ls
Temperature Map&                /home/deap/ovendaq/online/index.html
Images

[local:oven1:S]/Custom>ls Images/temps.png/           
Background                      /home/deap/ovendaq/online/temps.png

And the HTML file is just this:
<img src="temps.png">

But the image won't display. It shows a "broken" picture, and when I try to view
it directly it says: Invalid custom page: Page not found in ODB.

Any help would be appreciated...

Thanks
Shaun