We have similar demands, and we solve it in the following:
We use a run database. In the simplest case, this can be a text file which gets written at the end of the file. The
mlogger has a built in SQL interface, so one can keep that table even inside a SQL interface. The per-run-
information then contains the run number, start/stop time, number of events, some run parameters and a "junk"
flag. So if a run has a problem, one can set the junk flag by accessing the database (or text file) and setting this
flag. In many cases you see that a run had a problem not at the end of the run, but a bit later. You mayby realize
that the last two or three runs had the problem. With the run database approach, you can flag any run as "junk"
later, which we need often, An edit-on-end would not make this possible.
So technically putting and edit-on-end is not a problem, but your life might be much easier if you use a run
database as outlined above.
Best regards,
Stefan |
serious (but rare) bug was fixed in the history reader. unlucky experiment would see
errors about "Detected duplicate or non-monotonous data" in some history file, fixed by
removing/renaming the offending file. (reported by MEG experiment)
it turns out there was nothing wrong with the data files (good), but there
was a nasty bug in the history reader. it did not ensure that we read history
files in chronological order. under some conditions order of files could be
reversed, older files would be read after newer files and trip the built-in
protection against returning non-monotonically increasing history data to the user.
fixed commit
https://bitbucket.org/tmidas/midas/commits/9893f85ebe33e96cc63f501a0f89e1f8932c894d
for more details, see https://bitbucket.org/tmidas/midas/issues/350/file-history-non-
monotonic-time
K.O. |
We also agree and found the problem now. Since we build everything (MIDAS Event Header, Bank Header, Banks etc.) in the FPGA we had some struggle with the MIDAS data format (http://lmu.web.psi.ch/docu/manuals/bulk_manuals/software/midas195/html/AppendixA.html). We thought that only the MIDAS Event needs to be aligned to 64 bit but as it turned out also the bank data (Stefan updated the wiki page already) needs to be aligned. Since we are using the BANK32 it was a bit unclear for us since the bank header is not 64 bit aligned. But we managed this now by adding empty data and the system is running now.
Our setup looks like this:
- mfe.cxx multithread equipment
- mfe readout thread grabs pointer from dma ring buffer
- since the dma buffer is volatile we do copy_n for transforming the data to MIDAS
- the data is already in the MIDAS format so done from our side :)
- mfe readout thread increments the ring buffer
- mfe main thread grabs events from ring buffer, sends them to the mserver
From the firmware side we have an Arria 10 development board and
But now I am curious, which DMA controller you use? The Altera or Xilinx PCIe block with the vendor supplied DMA driver? Or you do DMA on an ARM SoC FPGA? (no PCI/PCIe,
different DMA controller, different DMA driver).
I am curious because we will be implementing pretty much what you do on ARM SoC FPGAs pretty soon, so good to know
if there is trouble to expect.
But I will probably use the tmfe.h c++ frontend and a "pure c++" ring buffer instead of mfe.cxx and the midas "rb" ring buffer.
(I did not look at your code at all, there could be a bug right there, this ring buffer stuff is tricky. With luck there is no bug
in your dma driver. The dma drivers for our vme bridges did do have bugs).
K.O. |