I think I added the cache size correctly:

  {"Trigger",               /* equipment name */
      {1, 0,                 /* event ID, trigger mask */
         "SYSTEM",           /* event buffer */
         EQ_POLLED,          /* equipment type */
         0,                  /* event source */
         "MIDAS",            /* format */
         TRUE,               /* enabled */
         RO_RUNNING |        /* read only when running */
         RO_ODB,             /* and update ODB */
         100,                /* poll for 100ms */
         0,                  /* stop run after this event limit */
         0,                  /* number of sub events */
         0,                  /* don't log history */
         "", "", "", "", "", // frontend_host, name, file_name, status, status_color
         0, // hidden
         0  // write_cache_size <<--------------------- set this to zero -----------
      },
   }

K.O.

the main purpose of the event buffer write cache is to prevent high contention for the 
event buffer shared memory semaphore in the pathological case of very high rate of very 
small events.

there is a computation for this, I have posted it here several times, please search for 
it.

in the nutshell, you want the semaphore locking rate to be around 10/sec, 100/sec 
maximum. coupled with smallest event size and maximum practical rate (1 MHz), this 
yields the cache size.

for slow control events generated at 1 Hz, the write cache is not needed, 
write_cache_size value 0 is the correct setting.

for "typical" physics events generated at 1 kHz, write cache size should be set to fit 
10 events (100 Hz semaphore locking rate) to 100 events (10 Hz semaphore locking rate).

unfortunately, one cannot have two cache sizes for an event buffer, so typical frontends 
that generate physics data at 1 kHz and scalers and counters at 1 Hz must have a non-
zero write cache size (or semaphore locking rate will be too high).

the other consideration, we do not want data to sit in the cache "too long", so the 
cache is flushed every 1 second or so.

all this cache stuff could be completely removed, deleted. result would be MIDAS that 
works ok for small data sizes and rates, but completely falls down at 10 Gige speeds and 
rates.

P.S. why is high semaphore locking rate bad? it turns out that UNIX and Linux semaphores 
are not "fair", they do not give equal share to all users, and (for example) an event 
buffer writer can "capture" the semaphore so the buffer reader (mlogger) will never get 
it, a pathologic situation (to help with this, there is also a "read cache"). Read this 
discussion: https://stackoverflow.com/questions/17825508/fairness-setting-in-semaphore-
class

K.O.

> the main purpose of the event buffer write cache

how to control the write cache size:

1) in a frontend, all equipments should ask for the same write cache size, both mfe.c and 
tmfe frontends will complain about mismatch

2) tmfe c++ frontend, per tmfe.md, set fEqConfWriteCacheSize in the equipment constructor, in 
EqPreInitHandler() or EqInitHandler(), or set it in ODB. default value is 10 Mbytes or value 
of MIN_WRITE_CACHE_SIZE define. periodic cache flush period is 0.5 sec in 
fFeFlushWriteCachePeriodSec.

3) mfe.cxx frontend, set it in the equipment definition (number after "hidden"), set it in 
ODB, or change equipment[i].write_cache_size. Value 0 sets the cache size to 
MIN_WRITE_CACHE_SIZE, 10 Mbytes.

4) in bm_set_cache_size(), acceptable values are 0 (disable the cache), MIN_WRITE_CACHE_SIZE 
(10 Mbytes) or anything bigger. Attempt to set the cache smaller than 10 Mbytes will set it 
to 10 Mbytes and print an error message.

All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
rate and size values practical on current computers.

In mfe.cxx it looks to be impossible to set the write cache size to 0 (disable it), but 
actually all you need is call "bm_set_cache_size(equipment[0].buffer_handle, 0, 0);" in 
frontend_init() (or is it in begin_of_run()?).

K.O.

> > the main purpose of the event buffer write cache
> how to control the write cache size:

OP provided insufficient information to say what went wrong for them, but do try this:

1) in ODB, for all equipments, set write_cache_size to 0
2) in the frontend equipment table, set write_cache_size to 0

That is how it is done in the example frontend: examples/experiment/frontend.cxx

If this configuration still produces an error, we may have a bug somewhere, so please let us know how it shakes out.

K.O.

 > All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
> disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
> rate and size values practical on current computers.

Indeed KO is correct that only 0 and 10MB make sense, and we cannot mix it. Having the cache setting in the equipment table is 
cumbersome. If you have 10 slow control equipment (cache size zero), you need to add many zeros at the end of 10 equipment 
definitions in the frontend. 

I would rather implement a function or variable similar to fEqConfWriteCacheSize in the tmfe framework also in the mfe.cxx 
framework, then we need only to add one line llike

gEqConfWriteCacheSize = 0;

in the frontend.cxx file and this will be used for all equipments of that frontend. If nobody complains, I will do that in April when I'm 
back from Japan.

Stefan

>  > All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
> > disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
> > rate and size values practical on current computers.
> 
> Indeed KO is correct that only 0 and 10MB make sense, and we cannot mix it. Having the cache setting in the equipment table is 
> cumbersome. If you have 10 slow control equipment (cache size zero), you need to add many zeros at the end of 10 equipment 
> definitions in the frontend. 
> 
> I would rather implement a function or variable similar to fEqConfWriteCacheSize in the tmfe framework also in the mfe.cxx 
> framework, then we need only to add one line llike
> 
> gEqConfWriteCacheSize = 0;
> 
> in the frontend.cxx file and this will be used for all equipments of that frontend. If nobody complains, I will do that in April when I'm 
> back from Japan.

Cache size is per-buffer. If different equipments write into different event buffers, should be possible to set different cache sizes.

Perhaps have:

set_write_cache_size("SYSTEM", 0);
set_write_cache_size("BUF1", bigsize);

with an internal std::map<std::string,size_t>; for write cache size for each named buffer

K.O.

> Perhaps have:
> 
> set_write_cache_size("SYSTEM", 0);
> set_write_cache_size("BUF1", bigsize);
> 
> with an internal std::map<std::string,size_t>; for write cache size for each named buffer

Ok, this is implemented now in mfed.cxx and called from examples/experiment/frontend.cxx

Stefan

Hi everyone,

It has been a long time since my last message. While porting Midas front-end 
programs developed for the T2K experiment in 2008 to a modern version of Midas, 
I noticed that some debug printf remain in mhttpd.cxx.

A number of debug messages are printed on stdout each time a graph is displayed 
in the OldHistory window (which is the style of history plots that will continue 
to be used in the upgraded T2K experiment for some technical reasons).

Here is an example of such debug messages:
Load from ODB History/Display/HA_EP_0/V33: hist plot: 8 variables
timescale: 1h, minimum: 0.000000, maximum: 0.000000, zero_ylow: 0, log_axis: 0, 
show_run_markers: 1, show_values: 1, show_fill: 0, show_factor 0, enable_factor: 
1
var[0] event [HA_TPC_SC][E0M00FEM_V33] formula [], colour [#00AAFF] label 
[Mod_0] show_raw_value 0 factor 1.000000 offset 0.000000 voffset 0.000000 order 
10
var[1] event [HA_TPC_SC][E0M01FEM_V33] formula [], colour [#FF9000] label 
[Mod_1] show_raw_value 0 factor 1.000000 offset 0.000000 voffset 0.000000 order 
20
var[2] event [HA_TPC_SC][E0M02FEM_V33] formula [], colour [#FF00A0] label 
[Mod_2] show_raw_value 0 factor 1.000000 offset 0.000000 voffset 0.000000 order 
30
var[3] event [HA_TPC_SC][E0M03FEM_V33] formula [], colour [#00C030] label 
[Mod_3] show_raw_value 0 factor 1.000000 offset 0.000000 voffset 0.000000 order 
40
var[4] event [HA_TPC_SC][E0M04FEM_V33] formula [], colour [#A0C0D0] label 
[Mod_4] show_raw_value 0 factor 1.000000 offset 0.000000 voffset 0.000000 order 
50
var[5] event [HA_TPC_SC][E0M05FEM_V33] formula [], colour [#D0A060] label 
[Mod_5] show_raw_value 0 factor 1.000000 offset 0.000000 voffset 0.000000 order 
60
var[6] event [HA_TPC_SC][E0M06FEM_V33] formula [], colour [#C04010] label 
[Mod_6] show_raw_value 0 factor 1.000000 offset 0.000000 voffset 0.000000 order 
70
var[7] event [HA_TPC_SC][E0M07FEM_V33] formula [], colour [#807060] label 
[Mod_7] show_raw_value 0 factor 1.000000 offset 0.000000 voffset 0.000000 order 
80
read_history: nvars 10, hs_read() status 1
read_history: 0: event [HA_TPC_SC], var [E0M00FEM_V33], index 0, odb index 0, 
status 1, num_entries 475
read_history: 1: event [HA_TPC_SC], var [E0M01FEM_V33], index 0, odb index 1, 
status 1, num_entries 475
read_history: 2: event [HA_TPC_SC], var [E0M02FEM_V33], index 0, odb index 2, 
status 1, num_entries 475
read_history: 3: event [HA_TPC_SC], var [E0M03FEM_V33], index 0, odb index 3, 
status 1, num_entries 475
read_history: 4: event [HA_TPC_SC], var [E0M04FEM_V33], index 0, odb index 4, 
status 1, num_entries 475
read_history: 5: event [HA_TPC_SC], var [E0M05FEM_V33], index 0, odb index 5, 
status 1, num_entries 475
read_history: 6: event [HA_TPC_SC], var [E0M06FEM_V33], index 0, odb index 6, 
status 1, num_entries 475
read_history: 7: event [HA_TPC_SC], var [E0M07FEM_V33], index 0, odb index 7, 
status 1, num_entries 475
read_history: 8: event [Run transitions], var [State], index 0, odb index -1, 
status 1, num_entries 0
read_history: 9: event [Run transitions], var [Run number], index 0, odb index 
-2, status 1, num_entries 0


Looking at the source code of mhttpd, these messages originate from:

[mhttpd.cxx line 10279] printf("Load from ODB %s: ", path.c_str());
[mhttpd.cxx line 10280] PrintHistPlot(*hp);
...
[mhttpd.cxx line  8336] int read_history(...
...
[mhttpd.cxx line  8343] int debug = 1;
...

Although seeing many debug messages in the mhttpd does not hurt, these can hide 
important error messages. I would rather suggest to turn off these debug 
messages by commenting out the relevant lines of code and setting the debug 
variable to 0 in read_history().

That is a minor detail and it is always a pleasure to use Midas.

Best regards,
Denis.
 

These debug statements were added by K.O. on June 24, 2021. He should remove it.

https://bitbucket.org/tmidas/midas/commits/21f7ba89a745cfb0b9d38c66b4297e1aa843cffd

Best,
Stefan

Hi,

We are using midas to read ADC. It sometimes hung in the middle of data taking.
We tried to disable analyzer and only run with frontend. The problem still
exists. We tried to use different crate, different CAMAC controller and
different ADC module. All these did not solve the problem. We use polled method
to read data. We have dataway display unit so we know that it hung always after
it executed CAMAC command F9, which is after finishing one data taking and clear
the ADC for the next data taking. The data rate is about 1 KHz. It is random for
how long it takes for the system to hang.

Any ideas ?

Thanks,

Dawei Liu

Hi Dawei,

Could you give more info on your setup:
- CAMAC controller model
- ADC model
- LAM setting
- Mode of polling (on module or on CC)
- Are you still going through the poll_event() after hang up?
- Do you have the same problem at low rate (100Hz)?

Pierre-André
> Hi,
> 
> We are using midas to read ADC. It sometimes hung in the middle of data taking.
> We tried to disable analyzer and only run with frontend. The problem still
> exists. We tried to use different crate, different CAMAC controller and
> different ADC module. All these did not solve the problem. We use polled method
> to read data. We have dataway display unit so we know that it hung always after
> it executed CAMAC command F9, which is after finishing one data taking and clear
> the ADC for the next data taking. The data rate is about 1 KHz. It is random for
> how long it takes for the system to hang.
> 
> Any ideas ?
> 
> Thanks,
> 
> Dawei Liu

Hi Pierr-Andre,


> Hi Dawei,
> 
> Could you give more info on your setup:
> - CAMAC controller model

Jorway 73A, we have three in hand and the problem doesn't depend on which controller
we were using.

> - ADC model

LeCroy 2249W. We also tried two other modules LeCroy 2249A. Same problem.

> - LAM setting

The poll and ADC reading codes are basically from Midas distribution.

> - Mode of polling (on module or on CC)

Polling on CC. I also tried to add a timeout code reading ADC, didn't solve the problem.

> - Are you still going through the poll_event() after hang up?

That's I don't know. I believe the problem happens between finishing reading one event
and passing the control back to poll_event.

> - Do you have the same problem at low rate (100Hz)?

The rate we are currently running is about 400 Hz, it has the same problem. We will
try lower rate more.

Thanks,

Dawei

> 
> Pierre-André
> > Hi,
> > 
> > We are using midas to read ADC. It sometimes hung in the middle of data taking.
> > We tried to disable analyzer and only run with frontend. The problem still
> > exists. We tried to use different crate, different CAMAC controller and
> > different ADC module. All these did not solve the problem. We use polled method
> > to read data. We have dataway display unit so we know that it hung always after
> > it executed CAMAC command F9, which is after finishing one data taking and clear
> > the ADC for the next data taking. The data rate is about 1 KHz. It is random for
> > how long it takes for the system to hang.
> > 
> > Any ideas ?
> > 
> > Thanks,
> > 
> > Dawei Liu

I keep on getting messages like this:

16:25:35 [fecaen,ERROR] [odb.c:4567:db_get_data,ERROR] data for key
"/DAQ/params/VX1730/custom/Board 0/Channel 0/Input range" truncated

whenever I start my frontend. Input range is defined to be a BOOL and using
odbedit to read it shows:

Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Input range                     BOOL    1     4     75h  0   RWD  y

without any error message. The entry is read using

         size = sizeof(fInputRange);
         db_get_data(hDb, hSubKey, &fInputRange, &size, TID_BOOL);

where fInputRange is a bool.

Where does this message come from and how can I resolve this?

> I keep on getting messages like this:
> 16:25:35 [fecaen,ERROR] [odb.c:4567:db_get_data,ERROR] data for key
> "/DAQ/params/VX1730/custom/Board 0/Channel 0/Input range" truncated
>
>  [  bool fInputRange... ]
>          size = sizeof(fInputRange);
>          db_get_data(hDb, hSubKey, &fInputRange, &size, TID_BOOL);
>

The error is correct. size of TID_BOOL is 4 byte (uint32_t) and you give is sizeof(bool) instead which is probably not 4.

Note that sizeof(bool) is not well defined, sometimes it is 1 (you need 4), sometimes something else, see
https://stackoverflow.com/questions/4897844/is-sizeofbool-defined-in-the-c-language-standard

A good fix would be to change fInputRange from bool to uint32_t (which is always 4 byte size).

#include <stdint.h>
...
uint32_t fInputRange;

K.O.

I have an application where I want to (optionally) send my own debugging messages to the midas.log file, but am having some problems with this: 

* Messages with type MT_DEBUG don't show up in midas.log or on the messages page (calling cm_msg from python)
* messages.html is missing the DEBUG filter option
* Messages sent to other log files (not midas.log) don't get message banners on the web page. Is this intentional? 

So I think either there is a bug, or I need to start MIDAS with some flag to enable debugging. Looking at the source, I don't see why these messages wouldn't get logged.

Any insight would be appreciated!

MT_DEBUG messages are there for debugging, not logging. They only go into the SYSMSG buffer and NOT to the log file. If you want anything logged, just use MT_INFO.

Not sure if that's missing in the documentation. Anyhow, there are my original ideas (from 1995 ;-) )

MT_ERROR  
Error message, to be displayed in red

MT_INFO    
 Info or status message

MT_DEBUG 
Only sent to SYSMSG buffer, not to midas.log file. Handy if you produce lots of message and don't want to flood the message file. Plus it does not change the timing of your app, since the SYSMSG buffer is much faster than writing 
to a file.

MT_USER
Message generated interactively by a user, like in the chat window or via the odbedit "msg" command

MT_LOG
Messages with are only logged but not put into the SYSMSG buffer

MT_TALK
Messages which should go through the speech synthesis in the browser and are "spoken"

MT_CALL
Message which would be forwarded to the user via a messaging app (historically this was an actual analog telephone call via a modem ;-) )

If that is missing in the documentation, please feel free to copy/paste it to the appropriate place.


Stefan

Ok thanks for the quick and clear response!

Derek

> MT_DEBUG messages are there for debugging, not logging. They only go into the SYSMSG buffer and NOT to the log file. If you want anything logged, just use MT_INFO.
> 
> Not sure if that's missing in the documentation. Anyhow, there are my original ideas (from 1995 ;-) )
> 
> MT_ERROR  
> Error message, to be displayed in red
> 
> MT_INFO    
>  Info or status message
> 
> MT_DEBUG 
> Only sent to SYSMSG buffer, not to midas.log file. Handy if you produce lots of message and don't want to flood the message file. Plus it does not change the timing of your app, since the SYSMSG buffer is much faster than writing 
> to a file.
> 
> MT_USER
> Message generated interactively by a user, like in the chat window or via the odbedit "msg" command
> 
> MT_LOG
> Messages with are only logged but not put into the SYSMSG buffer
> 
> MT_TALK
> Messages which should go through the speech synthesis in the browser and are "spoken"
> 
> MT_CALL
> Message which would be forwarded to the user via a messaging app (historically this was an actual analog telephone call via a modem ;-) )
> 
> If that is missing in the documentation, please feel free to copy/paste it to the appropriate place.
> 
> 
> Stefan

Dear All,

After clicking the "Start" button, there is a page for the operator to change some 
ODB values. I have created "/Experiment/Edit on start" and added some links there. 
If the link is pointed to a boolean type key, a check box will appear in the 
"Start" page, which is great. But how about if I want to have some radio buttons 
or pull-down menus for the operator to select among different calibration sources 
or running modes?

Thanks,

Jimmy

I am having difficulty getting the custom scripts to work within the updated MIDAS. Before the 
update I was using something like : 

<input type=submit name=customscript value="test">

on my custom page to run a script under /CustomScript/test, however, with the update to 
MIDAS this is no longer working. I can't find any information about this functionality being 
updated in the latest version - has this changed? Or should it still work? 

Thanks,
Becky (g-2 DAQ)

> I am having difficulty getting the custom scripts to work within the updated MIDAS. Before the 
> update I was using something like : 
> 
> <input type=submit name=customscript value="test">
> 
> on my custom page to run a script under /CustomScript/test, however, with the update to 
> MIDAS this is no longer working. I can't find any information about this functionality being 
> updated in the latest version - has this changed? Or should it still work? 
> 
> Thanks,
> Becky (g-2 DAQ)

I do not see any messages about anybody changing this function. I hope it did not break by accident.

Right now I am working on the event buffer code, and did not plan to look at mhttpd, but it looks like
your problem is important and there is at least on more problem (but it has a work-around),
so I may look at it sooner than later...

K.O.