| ID |
Date |
Author |
Topic |
Subject |
|
2120
|
04 Mar 2021 |
Marius Koeppel | Forum | Using JSROOT.openFile with Midas | Thank you for the answer :)
> At some point I would like to provide a function to "get" TAFlowEvent objects so you can do things
> like event displays in javascript. But I need a c++ to json serializer and standard c++ does not have it.
> So I will have to use the clang serializer (also used by ROOT) and it will take me a few days
> to figure it out.
That sounds exactly what I was searching for. Because I wanted to create an interface between rootana and
an event display build in javascript. Since all I tried did not really worked with the current rootana
I have now a "solution". I use the MIDAS python client, read directly events from the MIDAS buffer and provided
the events in a JSON format with the python flask API. Since the rendering of the event display is the bottleneck
and I only need a view events to display this solution worked really well for me. Maybe having such a JSON API of
the event buffer in MIDAS directly would also work for most of the event display applications or other simple javescript
applications (my opinion).
> If you figure out the missing bits that need to be added to our code,
> please post them here or submit them as a pull request or a bug report in bitbucket.
One of the problems I had was the CORS domain of the THttpServer. In manalyzer:1894 you do
sprintf(str, "http:127.0.0.1:%d", httpPort); but there are additional options for the THttpServer (like "?cors=DOMAIN").
So maybe a flag while starting manalyzer passing such options would be nice. I will create a pull request passing them later.
> Also it would be good if you can provide a code example of "openFile" working elsewhere
> but not with manalyzer, if I can run it, maybe I can figure out what's missing. But lacking
> some example code, there is nothing for me to hack at.
The problem is that it was not even running on simple THttpServer using interactive root:
serv = new THttpServer("http:8088?cors=*");
TFile *_file0 = TFile::Open("example_root.root")
serv->Register("File", _file0);
So I tried just saving the file in the $MIDAS_DIR and tried to use mserver with JSROOT.openFile. I attached the html file and
a test root file.
Cheers,
Marius |
| Attachment 1: test.html
|
| Attachment 2: example_root.root
|
|
2124
|
04 Mar 2021 |
Marius Koeppel | Forum | Using JSROOT.openFile with Midas | > would this work for what you are doing?
Yes, having such a function would be perfect for the applications I have a the moment.
> (this is not good enough if data has to be pre-digested by c++ analysis in rootana)
Also agree, if one wants to have a more sophisticated applications it is definitely needed to preprocess the data.
Cheers,
Marius |
|
2186
|
28 May 2021 |
Marius Koeppel | Info | MidasConfig.cmake usage | > Does anybody actually use "find_package(midas)", does it actually work for anybody?
What we do is to include midas as a submodule and than we call find_package:
add_subdirectory(midas)
list(APPEND CMAKE_PREFIX_PATH ${CMAKE_CURRENT_SOURCE_DIR}/midas)
find_package(Midas REQUIRED)
For us it works fine like this but we kind of always compile Midas fresh and don't use a version on our system (keeping the newest version).
Without the find_package the build does not work for us. |
|
2308
|
12 Dec 2021 |
Marius Koeppel | Bug Report | Writting MIDAS Events via FPGAs | Dear all,
in 13 Feb 2020 to 21 Feb 2020 we had a talk about how I try to create MIDAS events directly on a FPGA and
than use DMA to hand the event over to MIDAS. In the thread I also explained how I do it in my MIDAS frontend.
For testing the DAQ I created a dummy frontend which was emulating my FPGA (see attached file). The interesting code is
in the function read_stream_thread and there I just fill a array according to the 32b BANKS which are 64b aligned (more or less
the lines 306-369). And than I do:
uint32_t * dma_buf_volatile;
dma_buf_volatile = dma_buf_dummy;
copy_n(&dma_buf_volatile[0], sizeof(dma_buf_dummy)/4, pdata);
pdata+=sizeof(dma_buf_dummy);
rb_increment_wp(rbh, sizeof(dma_buf_dummy)); // in byte length
to send the data to the buffer.
This summer (Mai - July) everything was working fine but today I did not get the data into MIDAS.
I was hopping around a bit with the commits and everything was at least working until: 3921016ce6d3444e6c647cbc7840e73816564c78.
Thanks,
Marius |
| Attachment 1: dummy_fe.cpp
|
/********************************************************************\
Name: dummy_fe.cxx
Created by: Frederik Wauters
Changed by: Marius Koeppel
Contents: Dummy frontend producing stream data
\********************************************************************/
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
#include <algorithm>
#include <random>
#include <iostream>
#include <unistd.h>
#include <bitset>
#include "midas.h"
#include "msystem.h"
#include "mcstd.h"
#include <thread>
#include <chrono>
#include "mfe.h"
using namespace std;
/*-- Globals -------------------------------------------------------*/
/* The frontend name (client name) as seen by other MIDAS clients */
const char *frontend_name = "Dummy Stream Frontend";
/* The frontend file name, don't change it */
const char *frontend_file_name = __FILE__;
/* frontend_loop is called periodically if this variable is TRUE */
BOOL frontend_call_loop = FALSE;
/* a frontend status page is displayed with this frequency in ms */
INT display_period = 0;
/* maximum event size produced by this frontend */
INT max_event_size = 1 << 25; // 32MB
/* maximum event size for fragmented events (EQ_FRAGMENTED) */
INT max_event_size_frag = 5 * 1024 * 1024;
/* buffer size to hold events */
INT event_buffer_size = 10000 * max_event_size;
/*-- Function declarations -----------------------------------------*/
INT read_stream_thread(void *param);
uint64_t generate_random_pixel_hit(uint64_t time_stamp);
uint32_t generate_random_pixel_hit_swb(uint32_t time_stamp);
BOOL equipment_common_overwrite = TRUE; //true is overwriting the common odb
/* DMA Buffer and related */
volatile uint32_t *dma_buf;
#define MUDAQ_DMABUF_DATA_ORDER 25 // 29, 25 for 32 MB
#define MUDAQ_DMABUF_DATA_LEN (1 << MUDAQ_DMABUF_DATA_ORDER) // in bytes
size_t dma_buf_size = MUDAQ_DMABUF_DATA_LEN;
uint32_t dma_buf_nwords = dma_buf_size/sizeof(uint32_t);
/*-- Equipment list ------------------------------------------------*/
EQUIPMENT equipment[] = {
{"Stream", /* equipment name */
{1, 0, /* event ID, trigger mask */
"SYSTEM", /* event buffer */
EQ_USER, /* equipment type */
0, /* event source */
"MIDAS", /* format */
TRUE, /* enabled */
RO_RUNNING , /* read always and update ODB */
100, /* poll for 100ms */
0, /* stop run after this event limit */
0, /* number of sub events */
0, /* log history every event */
"", "", ""} ,
NULL, /* readout routine */
},
{""}
};
/*-- Dummy routines ------------------------------------------------*/
INT poll_event(INT source, INT count, BOOL test)
{
return 1;
};
INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
return 1;
};
/*-- Frontend Init -------------------------------------------------*/
INT frontend_init()
{
// create ring buffer for readout thread
create_event_rb(0);
// create readout thread
ss_thread_create(read_stream_thread, NULL);
set_equipment_status(equipment[0].name, "Ready for running", "var(--mgreen)");
return CM_SUCCESS;
}
/*-- Frontend Exit -------------------------------------------------*/
INT frontend_exit()
{
return CM_SUCCESS;
}
/*-- Frontend Loop -------------------------------------------------*/
INT frontend_loop()
{
return CM_SUCCESS;
}
/*-- Begin of Run --------------------------------------------------*/
INT begin_of_run(INT run_number, char *error)
{
set_equipment_status(equipment[0].name, "Running", "var(--mgreen)");
return CM_SUCCESS;
}
/*-- End of Run ----------------------------------------------------*/
INT end_of_run(INT run_number, char *error)
{
set_equipment_status(equipment[0].name, "Ready for running", "var(--mgreen)");
return CM_SUCCESS;
}
/*-- Pause Run -----------------------------------------------------*/
INT pause_run(INT run_number, char *error)
{
return CM_SUCCESS;
}
/*-- Resume Run ----------------------------------------------------*/
INT resume_run(INT run_number, char *error)
{
return CM_SUCCESS;
}
uint64_t generate_random_pixel_hit(uint64_t time_stamp)
{
// Bits 63 - 35: TimeStamp (29 bits)
// Bits 34 - 30: Tot (5 bits)
// Bits 29 - 26: Layer (4 bits)
// Bits 25 - 21: Phi (5 bits)
// Bits 20 - 16: ChipID (5 bits)
// Bits 15 - 8: Col (8 bits)
// Bits 7 - 0: Row (8 bits)
uint64_t tot = rand() % 31; // 0 to 31
uint64_t layer = rand() % 15; // 0 to 15
uint64_t phi = rand() % 31; // 0 to 31
uint64_t chipID = rand() % 5; // 0 to 31
uint64_t col = rand() % 250; // 0 to 255
uint64_t row = rand() % 250; // 0 to 255
uint64_t hit = (time_stamp << 35) | (tot << 30) | (layer << 26) | (phi << 21) | (chipID << 16) | (col << 8) | row;
//printf("tot: 0x%2.2x,layer: 0x%2.2x,phi: 0x%2.2x,chipID: 0x%2.2x,col: 0x%2.2x,row: 0x%2.2x\n",tot,layer,phi,chipID,col,row);
//cout << hex << hit << endl;
//cout << hex << time_stamp << endl;
//cout << hex << (hit >> 35 & 0x7FFFFFFFF) << endl;
//cout << hex << (hit >> 32) << endl;
return hit;
}
uint32_t generate_random_pixel_hit_swb(uint32_t time_stamp)
{
uint32_t tot = rand() % 31; // 0 to 31
uint32_t chipID = rand() % 5; // 0 to 5
uint32_t col = rand() % 250; // 0 to 250
uint32_t row = rand() % 250; // 0 to 250
uint32_t hit = (time_stamp << 28) | (chipID << 22) | (row << 13) | (col << 5) | tot << 1;
// if ( print ) {
// printf("ts:%8.8x,chipID:%8.8x,row:%8.8x,col:%8.8x,tot:%8.8x\n", time_stamp,chipID,row,col,tot);
// printf("hit:%8.8x\n", hit);
// std::cout << std::bitset<32>(hit) << std::endl;
// }
return hit;
}
uint64_t generate_random_scifi_hit(uint32_t time_stamp, uint32_t counter1, uint32_t counter2)
{
uint64_t asic = rand() % 8;
uint64_t hit_type = 1;
uint64_t channel_number = rand() % 32;
uint64_t timestamp_bad = 0;
uint64_t coarse_counter_value = (time_stamp >> 5) & 0x7FFF;
uint64_t fine_counter_value = time_stamp & 0x1F;
uint64_t energy_flag = rand() % 2;
uint64_t fpga_id = 10 + rand() % 4;
if (counter1 > 0) {
fpga_id = counter1;
}
return (asic << 60) | (hit_type << 59) | (channel_number << 54) | (timestamp_bad << 53) | (coarse_counter_value << 38) | (fine_counter_value << 33) | (energy_flag << 32) | (fpga_id << 28) | ((time_stamp & 0x0FFFFFFF));
}
uint64_t generate_random_delta_t_exponential(float lambda)
{
float r = rand()/(1.0+RAND_MAX);
return ceil(-log(1.0-r)/lambda);
}
uint64_t generate_random_delta_t_gauss(float mu, float sigma)
{
std::default_random_engine generator;
generator.seed(rand());
std::normal_distribution<float> distribution(mu,sigma);
float r = ceil(distribution(generator));
if (r<0.0) r=0.0;
return r;
}
INT read_stream_thread(void *param)
{
uint32_t* pdata;
// init bank structure - 64bit alignment
uint32_t SERIAL = 0x00000001;
uint32_t TIME = 0x00000001;
uint64_t hit;
// tell framework that we are alive
signal_readout_thread_active(0, TRUE);
// obtain ring buffer for inter-thread data exchange
int rbh = get_event_rbh(0);
int status;
while (is_readout_thread_enabled()) {
// obtain buffer space
status = rb_get_wp(rbh, (void **)&pdata, 10);
// just sleep and try again if buffer has no space
if (status == DB_TIMEOUT) {
set_equipment_status(equipment[0].name, "Buffer full", "var(--myellow)");
continue;
}
if (status != DB_SUCCESS){
cout << "!DB_SUCCESS" << endl;
break;
}
// don't readout events if we are not running
if (run_state != STATE_RUNNING) {
set_equipment_status(equipment[0].name, "Not running", "var(--myellow)");
continue;
}
set_equipment_status(equipment[0].name, "Running", "var(--mgreen)");
int nEvents = 5000;
size_t eventSize=76;
uint32_t dma_buf_dummy[nEvents*eventSize];
uint64_t prev_mutrig_hit = 0;
uint64_t delta_t_1 = 100;
... 212 more lines ...
|
|
2317
|
26 Jan 2022 |
Marius Koeppel | Bug Report | Writting MIDAS Events via FPGAs |
> Any error messages printed by the frontend? any error message in midas.log? core dumps? crashes?
> I do not understand what you mean by "did not get the data into midas". You create events
> and send them to a midas event buffer and you do not see them there? With mdump?
> Do you see this both connected locally and connected remotely through the mserver?
I simply don't see the event counter counting up and I also don't see them using mdump. No logs, no dumps and no crashes - every is quite. I only tested it locally.
> BTW, I see you are using the mfe.c frontend. Event data handling in mfe.c frontends
> is quite convoluted and impossible to straighten out. I recommend that you use
> the tmfe c++ frontend instead. Event data handling is much simplified and is easier to debug
> compared to the mfe.c frontend. There is examples in the midas repository and there are
> tutorials for converting frontends from mfe.c to tmfe posted in this forum here.
I know the code I used is really old that's why I was so surprised that it suddenly did not work. But I am on the way to change it. Also Stefan gave me some comments on how to improve the code. But still changing them did not really change the behavior.
> BTW, the commit you refer to only changed some html files, could not have affected
> your data.
I just hopped around and the commit I send was the first one which worked again. But it's of course not the one where the stuff broke. I did a bit of git-bisect and ended up with this commit as the first one where my frontend is not working anymore: 91582e4172d534bf9b10e661a423c399fd1a69f4
Cheers,
Marius |
|
2325
|
26 Jan 2022 |
Marius Koeppel | Bug Report | Writting MIDAS Events via FPGAs |
> If you are connected locally (no mserver), I want to know the value returned by bm_send_event(). Simplest
> if you edit mfe.c and everywhere it calls bm_send_event() and rpc_send_event(), print the returned value.
>
> It would be very interesting to see if bm_send_event() returns 1 (SUCCESS), but the event vanishes
> without a trace.
I checked bm_send_event(rbh, (EVENT_HEADER*)(&pdata[0]), 0, 20); which gives me back 1. I also check the status of rb_increment_wp which is also 1.
> Before you do that, try something simpler:
> Run "mdump -s -d", it will print some event buffer internals.
> Watch to see if any data pointers change when you send your events ("wp", "rp", etc).
"rp" & "wp" are not counting up.
> But there is some funny logic for event_id and trigger_mask and it is worth checking their
> values. For a good test, set event_id=1 and trigger_mask=0x1. There might be trouble if either is set to zero.
Changing both to 0x1 did not change the behavior.
Cheers,
Marius |
|
2347
|
16 Feb 2022 |
Marius Koeppel | Bug Report | Writting MIDAS Events via FPGAs | I just came back to this and started to use the dummy frontend.
Unfortunately, I have a problem during run cycles:
Starting the frontend and starting a run works fine -> seeing events with mdump and also on the web GUI.
But when I stop the run and try to start the next run the frontend is sending no events anymore.
It get stuck at line 221 (if (status == DB_TIMEOUT)).
I tried to reduce the nEvents to 1 which helped in terms of DB_TIMEOUT but still I don't get any events after I did a stop / start cycle -> no events in mdump and no events counting up at the web GUI.
If I kill the frontend in the terminal (ctrl+c) and restart it, while the run is still running, it starts to send events again.
Cheers,
Marius |
|
2352
|
07 Mar 2022 |
Marius Koeppel | Bug Report | Writting MIDAS Events via FPGAs | > This problem has (likely) been fixed in the current version. Please pull develop and try again. Was a recursive call to the event collection routine which is only triggered if you send events faster than
> the logger can digest, so not many people see it.
I just pulled the current version (d945fa9) but the problem as explained in 2347 stays the same.
Best,
Marius |
|
2375
|
25 Mar 2022 |
Marius Koeppel | Bug Report | Writting MIDAS Events via FPGAs | I finally found the problem why the readout stops after a run transition.
In my dummy frontend the serial number was not reset to zero at run start.
This leads to a mismatch of the serial number in the function receive_trigger_event of mfe.cxx:1247.
Which is than resulting in the problem that the function founds never a new event in all ring buffers and nothing get read out of the buffer.
Nevertheless, it would be nice that the system would tell the user that there is a mismatch in the serial number (printing a warning / error etc.).
Cheers,
Marius |
|
2476
|
27 Apr 2023 |
Marius Koeppel | Suggestion | Maximum ODB size | Hi all,
> I agree, I think we can safely bump the limit from 100 Mbytes to 1 Gbyte, maybe 1.5 or
> 1.99 Gbytes. Above that we run into 32-bit/31-bit cleanliness problems.
We just went in and changed: int odb_size_limit = INT_MAX;//100*1000*1000; in odb.cxx. And we could create ODBs with 1GB and 1.5 GB.
Since the DecodeSize function in odbinit has also foreseen yottabytes ;) (const char units[] = {'k', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y'};) we think going to GB for the maximum ODB size would be create.
> there is a bug in odbinit, if initial odbinit fails, ODB with default size is creates,
> and original rejected ODB size is written to .ODB_SIZE.TXT (an inconsistency).
Can#t we go with the maximum size here if the user inputs a larger size? So just below printf("Checking ODB size...\n"); one could check for the odb_size_limit. In general one could move the odb_size_limit to midas.h so its not only available in odb.cxx.
Best,
Marius |
|
2480
|
27 Apr 2023 |
Marius Koeppel | Suggestion | Maximum ODB size | > This is change is wrong. As I wrote, ODB is not 64-bit clean and it is not 32-bit clean. We think is is 31-bit clean, so maximum size would be slightly less than 2 Gbytes.
I just wanted to show that changing it and creating bigger ODBs is in general possible.
My main intention was to trigger the discussion again. I also think in general 1GB is enough. But for our applications sometimes 100MB is just on the edge.
> Congratulations. created != "it works". for proper test, you should fill it with 1.5 GB of stuff, save to json file, reload from json file, save to a different json file and compare that they have same contents (minus timestamps).
You’re right we did not properly test it. I will run this test with a 1GB ODB.
> We could spend a lot of time making odb 32-bit clean and give you 4GB-max ODB, but would it be useful? For large ODB, "save to .json" already takes a long time ("save to .xml" is slower, "save to .odb" ditto, also buggy). We already have complaints that runs take forever to start because mlogger
> takes a long time to write the ODB save file.
I also agree that going in and making it 32-bit or even 64-bit clean is not worth the effort.
Also concerning the writing speed of the logger etc I am fully with you.
However, having the freedom to choose a bit bigger ODB would be great.
You said the writing into .odb is buggy. Do you mean it’s buggy in general or only in this specific case?
We save the ODB most of the time in the .odb format.
Cheers,
Marius |
|
2485
|
28 Apr 2023 |
Marius Koeppel | Suggestion | Maximum ODB size | > my vote is to bump the ODB size limit to 1999*1000*1000 (not quite 2GB). but this needs to be tested. especially save and restore from ODB, XML and JSON files, including how long it takes to save and load a 1.9GB ODB. K.O.
I had some fun with python and created a test script which can be executed in the MIDASSYS/online folder (test_odb.py). I did not really normalize the time so it will be different at different systems but I guess the trend is important (see create_time.pdf).
What is surprising to me is that even that I only write one STRING key to the time increases. Is this maybe related to what Stefan said about the run start - so that odbedit needs some time to load the bigger ODB?
Second thing is that also the creation / storing and load time is increasing. Should this be or is there a bug in the code I use or again is this related to the previous point?
The test of comparing the ODB after store / load / store already fails for the json format. I know I only test if the dicts are the same, so for timestamps this already fails.
But what is strange here is that sometimes the test works sometimes not and its different from run to run.
I will try to improve the test a bit more but for a short update this is how it looks so fare.
Best,
Marius |
| Attachment 1: create_time.pdf
|
|
| Attachment 2: fails.pdf
|
|
| Attachment 3: test_odb.py
|
import os, time, json, random, string
import matplotlib.pyplot as plt
from matplotlib import colors
import numpy as np
def read_json(path):
with open(path, "r") as f:
return json.load(f)
def clean_odb():
cmd = "rm .*.SHM >/dev/null"
os.system(cmd)
cmd = "rm .SHM_TYPE.TXT >/dev/null"
os.system(cmd)
cmd = "rm .SHM_HOST.TXT >/dev/null"
os.system(cmd)
cmd = "rm /dev/shm/*_SHM >/dev/null"
os.system(cmd)
cmd = "rm /dev/shm/*online_online* >/dev/null"
os.system(cmd)
cmd = "rm .ODB_SIZE.TXT >/dev/null"
os.system(cmd)
def create_odb(arg):
size = arg["size"]
cmd = f"odbinit -s {size}MB >/dev/null"
os.system(cmd)
def time_function(func, **args):
start = time.time()
if not args:
func()
if args:
func(args)
return time.time() - start
def randomword(length=10):
letters = string.ascii_lowercase
return ''.join(random.choice(letters) for i in range(length))
def fill_odb(arg):
size = arg["size"]
num_key_values = arg["num_key_values"]
cmd = "odbedit -c \'mkdir Test\' >/dev/null"
os.system(cmd)
# one string key has 32 bytes for data and 72 bytes for key
# one INT32 key has 38 bytes for data and 72 bytes for key
# 1MB is 8388608 bits so 524288 bytes ODB per key / data
rounds = int(size * 524288 / 72)
for idx in range(num_key_values):
word = randomword()
cmd = f"odbedit -d Test -c \'create STRING {word}\' >/dev/null"
os.system(cmd)
cmd = f"odbedit -c \'set Test/{word} {word}\' >/dev/null"
os.system(cmd)
def store_odb():
cmd = "odbedit -c \'save test.json\' >/dev/null"
os.system(cmd)
def load_odb():
cmd = "odbedit -c \'load test.json\' >/dev/null"
os.system(cmd)
def check_dict(dictA, dictB, value):
if dictA[value] == dictB[value]:
return 0
return 1
create_time = []
fill_time = []
store_time = []
load_time = []
time_random = time_function(randomword)
MBs = 100#1500
num_key_values = 1
key_list = ["/MIDAS version", "/MIDAS git revision", "/ODB path", "/filename", "Experiment", "System", "Programs", "Logger", "Test", "Runinfo", "Alarms"]
fails = np.zeros([len(range(1, MBs, 10)), len(key_list)])
for i, size in enumerate(range(1, MBs, 10)):
clean_odb()
create_time.append(time_function(create_odb, size=size))
cur_fill_time = time_function(fill_odb, size=size, num_key_values=num_key_values)
fill_time.append(cur_fill_time - time_random * num_key_values)
cmd = "odbedit -c mem"
os.system(cmd)
print(f"Test size {size}MB")
store_time.append(time_function(store_odb))
dictA = read_json("test.json")
load_time.append(time_function(load_odb))
store_odb()
dictB = read_json("test.json")
for j, k in enumerate(key_list):
fails[i][j] = check_dict(dictA, dictB, k)
plt.plot(range(1, MBs, 10), create_time, ".", label="create")
plt.plot(range(1, MBs, 10), fill_time, ".", label=f"fill {num_key_values} key")
plt.plot(range(1, MBs, 10), store_time, ".", label="store")
plt.plot(range(1, MBs, 10), load_time, ".", label="load")
plt.xlabel("size in MB")
plt.ylabel("time in sec")
plt.legend()
plt.savefig("create_time.pdf")
plt.close()
fig, ax = plt.subplots()
cmap = colors.ListedColormap(['green','red'])
plt.pcolor(fails[::-1],cmap=cmap, edgecolors='k', linewidths=3)
ax.set_xticks(range(len(key_list)))
ax.set_xticklabels(key_list, rotation=45, horizontalalignment='right')
ax.set_yticks(range(len(range(1, MBs, 10))))
ax.set_yticklabels([str(i)+"MB" for i in range(1, MBs, 10)])
plt.savefig("fails.pdf", bbox_inches="tight")
|
|
2487
|
28 Apr 2023 |
Marius Koeppel | Suggestion | Maximum ODB size | > At the run start mlogger writes the ODB to the .mid file. This needs conversion (binary ODB -> XML ASCII) which can take time.
> This does NOT depend on the ODB size, but on the ODB *content*. Every key in the ODB takes time to convert. So if your ODB as 1.5 GB
> but only a few keys, this is still fast. Only if you have 200 million keys int he ODB, then mlogger takes lots of time to convert
> 200 million values to XML or JSON strings.
This was also my assumption. Is this the same for odbedit -c save FILE?
Because this is what I tested with the script and there one can see in the plot that the time increases to write the file if the ODB size increases.
The content of the ODB is always the same - one STRING key in the directory Test.
Best,
Marius |
|
2530
|
13 Jun 2023 |
Marius Koeppel | Suggestion | Maximum ODB size |
> BTW, how do I resize the ODB. I remember we discussed this some time ago, and concluded that odbedit needs a resize flag. Has this even been
> done? If not, what is the "official" way to resize the ODB. We had some documentation about that some time ago, but I can't find it anymore.
I guess this is still not done and the issue is still open: https://bitbucket.org/tmidas/midas/issues/329/need-odbresize
I guess if we touch this maybe the problem with the wrong size should be also fixed: https://bitbucket.org/tmidas/midas/issues/328/odbinit-s-1024mb-creates-odb-with-wrong
Best,
Marius |
|
2814
|
30 Aug 2024 |
Marius Koeppel | Suggestion | Improve Event Documentation | Hi,
I am writing a Rust based midas file reader however it was kind of hard to understand the full midas file
structure from the documentation.
Only at the end of the page
https://daq00.triumf.ca/MidasWiki/index.php/Event_Structure#MIDAS_Format_Event one finds under the
headline “tape format” that there are special events which mark the start and the end of the run. It would
be better to place this information more prominent maybe we a headline: “Special Events”. Maybe a link to
this section at the top of the page could help. Also at the mlogger page there is no information about this.
Best,
Marius |
|
2817
|
01 Sep 2024 |
Marius Koeppel | Suggestion | Improve Event Documentation | > > Hi,
> >
> > I am writing a Rust based midas file reader however it was kind of hard to understand the full midas file
> > structure from the documentation.
> >
> > Only at the end of the page
> > https://daq00.triumf.ca/MidasWiki/index.php/Event_Structure#MIDAS_Format_Event one finds under the
> > headline “tape format” that there are special events which mark the start and the end of the run. It would
> > be better to place this information more prominent maybe we a headline: “Special Events”. Maybe a link to
> > this section at the top of the page could help. Also at the mlogger page there is no information about this.
> >
> > Best,
> > Marius
>
> Ben was so kind to update the event documentation:
>
> https://daq00.triumf.ca/MidasWiki/index.php/Event_Structure
>
> Please have a look and let us know if that's better now.
>
> Best,
> Stefan
Thank you Ben! Now its super clear! |
|
2819
|
02 Sep 2024 |
Marius Koeppel | Suggestion | Improve Event Documentation | > > I am writing a Rust based midas file reader
>
> You might find this library I wrote useful: https://crates.io/crates/midasio
>
> It should "just work", and if it doesn't, I would be interested to know.
Nice! I did not know about this. I have now also one simple reader but yours looks much more advanced. My
overall idea here is to connect directly to midas so having some frontend features to analyze the data etc. do
you also have already a library for this? I can also extend your stuff.
Best,
Marius |
|
2847
|
16 Sep 2024 |
Marius Koeppel | Bug Report | Crash using ODB watch | This is not the case here. Note that the error message: "Callback received for a midas::odb object which went out of scope" is not called! The segmentation fault happens later line 96.
> The answer is in the error message: „Object went out of scope“. When your frontent_init() exits, the odb objects are destroyed. When you get a callback, it‘s linked to the
> destroyed object. This is like if you have a local string and pass a reference to that string in the return of the function.
>
> Use a global object (bad) or use „new“ (potential memory leak). I would use a global structure which holds all odb objects.
>
> Stefan
>
> >
> > last week I was running MIDAS with the commit 3ad98c5. Today I updated MIDAS and now all my watch functions are crashing. Attached I have a minimal example frontend of the problem.
> >
> > In our software we have two functions one which sets up the ODB values of the frontend and another one which sets up all watch functions. So overall we connect two time to the ODB during fronend_init one time to create the values and one time to create the watch. In the example code a simple version of this setup is shown:
> >
> > INT frontend_init() {
> >
> > cm_msg(MINFO, "frontend_init() setup", "Test FE");
> >
> > odb settings = {
> > {"Test", 123},
> > {"sub", {}}
> > };
> > settings.connect_and_fix_structure("/Equipment/Test FE/Settings");
> > // settings.watch(watch); <-- this works without segmentation fault
> >
> > odb new_settings("/Equipment/Test FE/Settings");
> > new_settings.watch(watch); // <-- here I am getting a segmentation fault
> >
> > return CM_SUCCESS;
> > }
> >
> > When I directly set the watch everything runs fine however, when I create a new ODB object and use this one to set a watch I am getting the following segmentation fault:
> >
> > Process 18474 stopped
> > * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x34)
> > frame #0: 0x000000010004fa38 test_fe`midas::odb::watch_callback(hDB=<unavailable>, hKey=<unavailable>, index=0, info=0x00006000002001c0) at odbxx.cxx:96:25 [opt]
> > 93 if (po->m_data == nullptr)
> > 94 mthrow("Callback received for a midas::odb object which went out of scope");
> > 95 midas::odb *poh = search_hkey(po, hKey);
> > -> 96 poh->m_last_index = index;
> > 97 po->m_watch_callback(*poh);
> > 98 poh->m_last_index = -1;
> > 99 }
> >
> > Best,
> > Marius |
|
2849
|
16 Sep 2024 |
Marius Koeppel | Bug Report | Crash using ODB watch | Okay, but this is then a big issue IMO. For Mu3e we do this in every frontend and I also checked again all of these watches are broken at the moment (with commit 3ad98c5 they worked).
In the old style we did for example (see https://bitbucket.org/tmidas/midas/src/develop/examples/crfe/crfe.cxx):
INT frontend_init()
{
HNDLE hKey;
// create Settings structure in ODB
db_create_record(hDB, 0, "Equipment/Clock Reset/Settings", strcomb1(cr_settings_str).c_str());
db_find_key(hDB, 0, "/Equipment/Clock Reset", &hKey);
assert(hKey);
db_watch(hDB, hKey, cr_settings_changed, NULL);
/*
* Set our transition sequence. The default is 500. Setting it
* to 600 means we are called AFTER most other clients.
*/
cm_set_transition_sequence(TR_START, 600);
return CM_SUCCESS;
}
I thought this will be the same (under the hood) in the current odbxx way via:
odb settings("Equipment/Clock Reset/Settings");
settings.watch(cr_settings_changed);
Best,
Marius
> Well, the object *went* out of scope. For my code it‘s hard to realize this, so the error reporting is poor. Also the first object should have the same
> problem. Just by accident that it does not crash.
>
> Stefan
>
> > This is not the case here. Note that the error message: "Callback received for a midas::odb object which went out of scope" is not called! The segmentation fault happens later line 96.
> >
> > > The answer is in the error message: „Object went out of scope“. When your frontent_init() exits, the odb objects are destroyed. When you get a callback, it‘s linked to the
> > > destroyed object. This is like if you have a local string and pass a reference to that string in the return of the function.
> > >
> > > Use a global object (bad) or use „new“ (potential memory leak). I would use a global structure which holds all odb objects.
> > >
> > > Stefan
> > >
> > > >
> > > > last week I was running MIDAS with the commit 3ad98c5. Today I updated MIDAS and now all my watch functions are crashing. Attached I have a minimal example frontend of the problem.
> > > >
> > > > In our software we have two functions one which sets up the ODB values of the frontend and another one which sets up all watch functions. So overall we connect two time to the ODB during fronend_init one time to create the values and one time to create the watch. In the example code a simple version of this setup is shown:
> > > >
> > > > INT frontend_init() {
> > > >
> > > > cm_msg(MINFO, "frontend_init() setup", "Test FE");
> > > >
> > > > odb settings = {
> > > > {"Test", 123},
> > > > {"sub", {}}
> > > > };
> > > > settings.connect_and_fix_structure("/Equipment/Test FE/Settings");
> > > > // settings.watch(watch); <-- this works without segmentation fault
> > > >
> > > > odb new_settings("/Equipment/Test FE/Settings");
> > > > new_settings.watch(watch); // <-- here I am getting a segmentation fault
> > > >
> > > > return CM_SUCCESS;
> > > > }
> > > >
> > > > When I directly set the watch everything runs fine however, when I create a new ODB object and use this one to set a watch I am getting the following segmentation fault:
> > > >
> > > > Process 18474 stopped
> > > > * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x34)
> > > > frame #0: 0x000000010004fa38 test_fe`midas::odb::watch_callback(hDB=<unavailable>, hKey=<unavailable>, index=0, info=0x00006000002001c0) at odbxx.cxx:96:25 [opt]
> > > > 93 if (po->m_data == nullptr)
> > > > 94 mthrow("Callback received for a midas::odb object which went out of scope");
> > > > 95 midas::odb *poh = search_hkey(po, hKey);
> > > > -> 96 poh->m_last_index = index;
> > > > 97 po->m_watch_callback(*poh);
> > > > 98 poh->m_last_index = -1;
> > > > 99 }
> > > >
> > > > Best,
> > > > Marius |
|
2852
|
18 Sep 2024 |
Marius Koeppel | Bug Report | Crash using ODB watch | I created a PR to fix this issue https://bitbucket.org/tmidas/midas/pull-requests/42.
The crash happened since the change in commit 3ad98c5 always got the ODB via XML.
However, the creation from XML should only be used when a user wants to read fast (and when we are on a remote machine) so I added the flag use_from_xml to explicitly specify this.
> > {
> > odb new_settings("/Equipment/Test FE/Settings");
> > new_settings.watch(watch); // <-- here I am getting a segmentation fault
> > }
>
> this code has a bug. "watch" is attached to object "new_settings" that is deleted
> after the closing curly bracket.
> I would say Stefan's odb API should not allow you to write code like this. an API defect.
As pointed out in the thread this feature is explicitly supported by odbxx.cxx:
void odb::watch(std::function<void(midas::odb &)> f) {
if (m_hKey == 0 || m_hKey == -1)
mthrow("watch() called for ODB key \"" + m_name +
"\" which is not connected to ODB");
// create a deep copy of current object in case it
// goes out of scope
midas::odb* ow = new midas::odb(*this);
ow->m_watch_callback = f;
db_watch(s_hDB, m_hKey, midas::odb::watch_callback, ow);
// put object into watchlist
g_watchlist.push_back(ow);
}
Also in the old way (see for example https://bitbucket.org/tmidas/midas/src/191d13f98626fae533cbca17b00df7ee361edf16/examples/crfe/crfe.cxx#lines-126) it was possible to create a watch in a scope without the user taking care that the "object" does not go out of scope.
I think this feature should be supported by the framework.
Best,
Marius |
|