I agree that the binary compatibility checks are crucial. But I kind of find it strange if one gets an assert failure some where if one tries to change MAX_CLIENTS. It is then not straight 
forward to relate both things and understand the consequences. That's why I put a comment next to the definition of MAX_CLIENTS saying:

/* note that if you change any of the following items, the ODB and the event shared memory buffers 
   become binary incopatible and one has to recompile ALL programs which are locally connected to the 
   ODB and to event buffers */

I think this is more descriptive than just a failing assert. 

If you look carefully in my proposal below, you will see that I rather used

sizeof(INT)

and not 

sizeof(int)

since as KO stated correctly sizeof(int) can change between different architectures. The derived type INT (all uppercase) has been carefully designed to have 32 bits on all architectures. So 
it will NOT change between them. If it does change, then we have a principal problem and many more things will break down. We should therefore have something like

if (sizeof(INT) != 4) then severe_error_and_stop_all_programs()

Now given that sizeof(INT) is everywhere the same, we can use it in the test

sizeof(BUFFER_HEADER) == NAME_LENGTH + 7*sizeof(INT) + MAX_CLIENTS *(NAME_LENGTH+13*sizeof(INT)+sizeof(float)+2*sizeof(DWORD)+MAX_EVENT_REQUESTS*4*sizeof(INT))

which then basically tests the structure byte alignment and padding. The comment above should warn users to change MAX_CLIENTS without thinking. 

Another strategy would be to put sizeof(BUFFER_HEADER) as the first two byes of the structure itself. We can the dynamically test the size of each bm_open_buffer(), and if the local size 
differs from the one saved in the buffer header, the program refuses to start, so we know exactly which program should have to be recompiled. The downside of this would be that the 
header structure has to be changed and we break binary compatibility with all existing programs. But maybe we should do this step once and be safe in the future.

Stefan


> The checks for byte sizes of critical data structures have been added to ensure (enforce) binary compatibility
> of midas with itself on different platforms (32-bit and 64-bit intel, on PPC, on ARM, etc).
> 
> This has worked well in the past and helped avoid problems and subtle bugs in the transition
> from 32-bit to 64-bit machines a few years ago. Of course now 32-bit machines are back
> as ARM CPUs and FPGA synthetic CPUs.
> 
> Replacing the checks with "computed" values will defeat this purpose because the values may be computed
> differently on different machines.
> 
> Specifically as proposed by Stefan, sizeof(int) can change depending on the target machine and depending
> on the compiler settings.
> 
> Of course this needs to be balanced against flexibility to adjust important settings like MAX_CLIENTS and MAX_EVENT_REQUESTS.
> 
> I would say the present system is just fine. You can change MAX_CLIENTS, rebuild MIDAS and it will not run (assert failure) giving
> you an indication that you are doing something non-trivial that will cause problems if you do it without thinking about it.
> 
> For example, one may think nothing of changing midas.h and recompiling MIDAS. But having to change odb.c
> may ring the little bell to tell you that you *also* have to rebuild *all* of your frontends. Even one unrebuilt frontend
> will corrupt all shared memory and crash everything.
> 
> I guess one other way to look at this is as a balance between something a few people do rarely against
> a function that protects everybody all the time.
> 
> That said, I think the checks should be reworked, instead of an assert failure they should give the error message
> and tell the user exactly what number to adjust in the size test. Also some checks are obsolete, there is no longer
> need to check the size of many ODB structures (equipment, etc). Once we are done with the db_get_record() rework,
> only checks for data structures in shared memory shall remain.
> 
> As the bottom line, to change MAX_CLIENTS, you already have to edit midas.h, asking you to also edit odb.c does
> not add much to the burden.
> 
> P.S. We are thinking how to make all these values dynamically changable, but basically it requires rolling out
> a new binary-incompatible version of MIDAS with added bugs. Maybe some day.
> 
> K.O.
> 
> 
> > The sizeof checks were originally invented by KO to check for binary compatibility between processes attached to the same ODB and event buffers. So if a 
> > compiler generates different structure sizes due to different padding, one would see that immediately. I wonder however if the absolute numbers make sense 
> > here. We could replace the 16444 by
> > 
> > NAME_LENGTH + 7*sizeof(INT) + MAX_CLIENTS *(NAME_LENGTH+13*sizeof(INT)+sizeof(float)+2*sizeof(DWORD)+MAX_EVENT_REQUESTS*4*sizeof(INT))
> > 
> > which makes this value automatically scale when one changes MAX_CLIENTS.
> > 
> > People of course have to be aware that if one changes MAX_CLIENTS, then all programs connected to the same ODB or event buffer need to be re-compiled 
> > and the ODB needs to be re-created from an ASCII file, but at least this would avoid tedious manual calculations.
> > 
> > Any opinion?
> > 
> > Stefan
> > 
> > 
> > > Below are the steps we used to increase the maximum number of frontends that we could run.
> > > 
> > > In midas.h
> > > 
> > > #define MAX_CLIENTS            64
> > > 
> > > changed to
> > > 
> > > #define MAX_CLIENTS            128
> > > 
> > > In msystem.h:
> > > 
> > > #define MAX_RPC_CONNECTION     64
> > > 
> > > changed to
> > > 
> > > #define MAX_RPC_CONNECTION     128
> > > 
> > > In odb.c:
> > > 
> > > assert(sizeof(BUFFER_HEADER) == 16444); 
> > > 
> > > GUESS: 256*64+60 = 16444, so change 64 to 128
> > > 
> > > changed to:                                                                                                                         
> > > 
> > > assert(sizeof(BUFFER_HEADER) == 32828); //256*128+60
> > > 
> > >  
> > > 
> > > DATABASE_HEADER = 64 + 64*DATABASE_CLIENT = 64 + 64*8256 = 528448
> > > 
> > > changed to:
> > > 
> > > DATABASE_HEADER = 64 + 128*DATABASE_CLIENT = 64 + 128*8256 = 1056832.

The type INT has been defined in 1989 when I for the first time sent data between a 16-bit MS-DOS computer and a 32-bit VAX computer (good old 
days!). At that time, uint32_t was not available at all. So much for the historical background.

I agree that switching from INT to int32_t is getting closer to standards and might help new people better understand things. This means however to 
touch all midas files and change about 5000 (!) locations:

BYTE -> uint8_t
WORD -> uint16_t
DWORD -> uint32_t
INT -> int32_t

Next we have the midas data types TID_xxx?

The nice thing now is that for example WORD and TID_WORD belong together and this is obvious. For uint16_t and TID_WORD is is not so obvious 
any more, so I guess we should rename TID_WORD to TID_UINT16_t. The same fore 

TID_BYTE -> TID_UINT8_T
TID_SBYTE -> TID_INT8_T
TID_WORD -> TID_UINT16_T
TID_DWORD -> TID_UINT32_T
TID_INT -> TID_INT32_T

But if we changer TID_XXX, the ASCII representations of the ODB break compatibility! Right now we have for example

[/Experiment]
midas http port = INT : 8080

which will become

[/Experiment]
midas http port = INT32_T : 8080

so one cannot load old ODB files any more!

With JSON encoding it's better because only the type number is stored, not the string. So INT -> 7 could stay, although in my opinion encoding the 
type in an integer number is not good for readability. Nobody knows what "7" means as a type. You always have to do a look-up in midas.c and count 
array indices manually.

I'm not sure how many experiments use the ASCII ODB format in one way or the other in some custom scripts. It might be that changing the format 
might have severe side effects for some experiments, so before we undertake this endeavor I would like to get some feedback here on the forum 
about people from other experiments and see what they think.

Stefan

> > if (sizeof(INT) != 4) then severe_error_and_stop_all_programs()
> 
> Quick reply.
> 
> Today, for fixed size data types one should use uint32_t & co, see
> stdint.h
> https://en.wikipedia.org/wiki/C_data_types#stdint.h
> https://en.wikipedia.org/wiki/C99 (scroll down and click to open "implementation -> compiler support"
> 
> The other popular convention is "u32" used by the Linux kernel, you will see it in the linux kernel drivers.
> 
> If I remember right, WORD and DWORD grow legs from the 16-bit Motorolla 68xxx processors,
> VxWorks and the VME bus. At some point the data buses were 16-bit wide and that we the WORD.
> 
> (I do not think UNIX ever used the WORD/DWORD names, i.e. MacOS has int32_t and u_int32_t).
> 
> K.O.

> bug in init
> -----------
> 
> I used the lv.c class driver, combined with a custom device driver, to control 
> our Keithley2611B source meter. This to set negative voltage on Si detectors.
> 
> In the 'init' routing, the class driver sets the hv:
> 
>   hv_info->demand_mirror[i] = MIN(hv_info->demand[i], hv_info->voltage_limit[i]);
> 
> This fails for negative voltage, as it sets the (negative) voltage limit, instead 
> of the demand voltage. A simple 'fabs' solves this.
> 
> suggestion for 'idle'
> ---------------------
> 
> I let the device do the ramping, not the driver. This also means I have to reset 
> the state of the device (current limit) after ramping. The easiest way to to 
> this, is using CMD_IDLE of the device driver. This is currently not done in the 
> hv.c class driver.

I can't find the line you quote in the class driver. Why don't you make a git pull request
and I will approve it.

The original idea behind the hv driver is that all voltages in the ODB and the class driver are
positive. If you have a negative power supply, then the voltage is inverted at the device
driver level. That's why you have MIN and MAX in the class driver.

Stefan

> > Currently one has to set the required flag in the ODB (e.g., /Programs/Logger/Required) to "y" for the program 
> > to appear on the "Programs page" and being able to start and stop the program easily.
> > 
> > However, if one wants to run with the "Prevent start on required progs" in /Experiment enabled, all the 
> > programs in the "Programs page" need to be running and one cannot have one of them stopped while still 
> > taking a run.
> > 
> > It would be nice to separate these two functionalities: Have a flag that makes the program appear on the 
> > "Programs page" and have a flag that controls the "Prevent start on required frogs" functionality.
> 
> I agree. All the programs should be always visible on the "programs" page, there should be /Programs/xxx/hidden to 
> hide them, and /Programs/xxx/required should be used for "Prevent start on required progs".

Konstantin, since you wrote the current "Programs" page, can you add that feature to the display (well, when you have time). I guess we 
event don't have to change the subdirectory structure (which might lead to incopatibilities), but just show a program if the "Start command" 
is non-null. If there is no start command, it does not make sense to start that program, so it can be hidden.

Stefan

> There is a small bug in the mfe.c initialization for the EQ_POLLED mode. There 
> is a routine where the number of polls fitting in eq_info->period is counted:
> 
> 
>          count = 1;
>          do {
>             if (display_period)
>                printf(".");
> 
>             start_time = ss_millitime();
> 
>             poll_event(equipment[idx].info.source, (INT)count, TRUE);
> 
>             delta_time = ss_millitime() - start_time;
> 
>             ...
> 
>             if (delta_time > 0)
>                count = count * eq_info->period / delta_time;
>             else
>                count *= 100;
>             
>             // avoid overflows
>             if (count > 2147483647.0) {
>                count = 2147483647.0;
>                break;
>             }
>             
>          } while (delta_time > eq_info->period * 1.2 || delta_time < eq_info-
> >period * 0.8);
> 
> As "start_time = ss_millitime();" resets "delta_time" each time, only the 
> "avoid overflows" addition saves the day. 
> 
> start_time = ss_millitime(); show be out of the loop.

Nope.

What I want is to determine how often I have to call poll_event to stay there for a certain time (usually 100ms). So I iterate "count" until I roughly get to my 100ms. Each call to 
poll_event with a different count is a new measurement, therefore I initialize start_time before each measurement. If i do it outside the loop, and kind of incrementally increase 
it, then the whole code inside the loop is added to the measurement which makes it (slightly) wrong.

The whole loop optimization has some background. Polling can be sow (think of talking to a device via Ethernet which can easily take milli seconds). So how often do we poll 
before we do other things in the main look (like looking if a run has been started). If I only poll once, then the average front-end response time would be poor, because I mostly 
look if a run has been started in the main loop. This is not effective. If I poll too often inside the poll_event loop, then the front-end does not react on run stops any more. So 
there is some optimum, and this is set by the polling time of usually 100ms. This ensures that the front-end does optimal polling - without ANYTHING in between - for about 
100ms. But how can I know how often I should poll for 100 ms? As said above, polling can be very fast (reading a memory cell) or very slow (network). The the best method I 
found is to do a calibration at the startup, and this is what the code above does. Maybe there are better ways today, but that code worked nicely in the last 25 years.

Stefan

> Thanks, I misunderstood the loop then. If poll_event(equipment[idx].info.source, (INT)count, TRUE); doesn`t do anything with "count", the loop becomes infinite except for the overflow 
> check. 

Well, the function poll_event() is _supposed_ to use "count" in a for loop as written in the example frontend:

   for (i = 0; i < count; i++) {
      /* poll hardware and set flag to TRUE if new event is available */
      flag = TRUE;

      if (flag)
         if (!test)
            return TRUE;
   }

where "flag = TRUE" must be replaced with the proper hardware check. This can be a VME access, a network TCP exchange with some Ethernet based hardware, or even a mutex check if the events are collected by a 
separate thread in the frontend.

The idea of having the for (i=0 ; i<count ; i++) loop _inside_ the poll_event() function and not outside is the fact that each function call to poll_event() takes time, and we want the minimal possible response time to new 
events. It might be just a micro-second, but having an experiment running at 100 Hz for one year (like Mu3e), this adds up to about one hour per year, which is a considerable amount of precious beam time.

Stefan

> In any case, IMO, mhttpd has no business serving the contents of /root,
> or serving any files outside of the mhttpd user $HOME directory. (but also
> should not serve files from ~user/.ssh, or any other "secret" files, good
> luck making a complete axhuastive list of all secret files that should not be
> served).

I fully agree with Konstantin. mhttpd should only serve files under certain directories. One is the 
midas/resources directory, another is the one defined in the ODB under /Custom/Path. I plan to modify 
mhttpd to only serve these files (and also prevent tricks like putting "../../../" into the URL). This will then 
also fix Andreas' problem.

Stefan

> There is a bug in the string handling when changing ODB string entries via the
> mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
> 
> For instance I create a string with length 32 and set it with odbedit to 'a'.
> Then the string length stays 32, as expected. If the same is done through the
> web-interface, the string length will be truncated to 2.
> 
> This can lead to problems if some frontend has a hotlink to a structure
> containing this string since it will complain about structure size mismatch. 

I know about this problem since last summer. I mentioned it to KO, since it's deep down in his 
JSONRPC code. We had a long discussion, where he kind of insisted that this is not a bug but a 
feature. The ODB should store strings with variable lengths, and thus adapt it according to the 
current string length. This makes some sense, since in the future we plan to put C++ string 
support for the ODB, where strings have dynamically varying lengths. But this will take a while, so 
I asked KO to change the truncation of the strings though the web interface, because this breaks 
many experiments. He did not react so far. Several people complained. Maybe your request will 
help now.

Stefan

> A feature I always missed (or just missed to find in the docu) is the following:
> 1) It would be nice to have a command in odbedit which allows to check how full
> the ODB currently is.
> 2) Even more important: I would like to have an ODB routine which allows me to
> check the fill level of the ODB, and/or a routine which tells me if I would
> create a structure of given size that it still fits in the current ODB or not. 
> The use case is that some clients create on the fly ODB entries and I would like
> to make sure before hand the ODB's remaining space in order not to crash things
> by overfilling the ODB. 

If you do "mem" in odbedit, you see the currently free areas, one for the keys themselves, one for 
the data of the keys. The corresponding C function is db_show_mem. At the moment it outputs the 
list of free blocks as a long ASCII string, but if necessary I can write a variant which returns the 
number of free bytes.

Stefan

> > > A feature I always missed (or just missed to find in the docu) is the following:
> > > 1) It would be nice to have a command in odbedit which allows to check how full
> > > the ODB currently is.
> > > 2) Even more important: I would like to have an ODB routine which allows me to
> > > check the fill level of the ODB, and/or a routine which tells me if I would
> > > create a structure of given size that it still fits in the current ODB or not. 
> > > The use case is that some clients create on the fly ODB entries and I would like
> > > to make sure before hand the ODB's remaining space in order not to crash things
> > > by overfilling the ODB. 
> > 
> > If you do "mem" in odbedit, you see the currently free areas, one for the keys themselves, one for 
> > the data of the keys. The corresponding C function is db_show_mem. At the moment it outputs the 
> > list of free blocks as a long ASCII string, but if necessary I can write a variant which returns the 
> > number of free bytes.
> > 
> > Stefan
> 
> Thanks for the info, and yes a variant of db_show_mem returning the number of free bytes would be just
> prefect!

I made you db_get_free_mem(HNDLE hDB, INT *key_size, INT *data_size)

The first return gets you the number of free bytes for the key area, the second one for the data area (values of keys).

Committed to develop

Stefan

It might have worked some ~5 years ago, but it never really showed the target value of a link, just the 
link itself. I reworked the code now to show both the link and the target of the link, so you can change 
both in the mhttpd ODB page. Should be consistent now with odbedit. Have a look if it works for you.

Stefan

The function mhttpd_init() scans the custom page and installs handlers etc. for each modbxxx 
element. If you create an modbvalue dynamically in your code, this is probably done after you 
called mhttpd_init(), so the function has no chance to modify the dynamically created element.

To fix that, I separated mhttpd_init() into the old init function which installs the header and sidebar, 
and a function mhttpd_scan() which scans the custom page and processes all modbxxx elements. 
Next, I tapped the error you reported, and added an automatic call to mhttp_scan() in case that 
happens. I tried it on a test page and it worked for me. Please give it a try (commit 090394e8).

Stefan

Two options:

1) Do NOT send your events into the ODB. This is controlled via the flag RO_ODB in your frontend setting. For simple experiments with small events, it might make sense to copy each 
event into the ODB for debugging, but if you have large events, this does not make sense. Use the "mdump" utility to check your events instead.

2) Increase the size of the ODB. See the first FAQ here: https://midas.triumf.ca/MidasWiki/index.php/FAQ

Stefan


> Dear expert,
>       I'm developing a frontend and I'm getting this kind of error at each event:
> 
> 10:14:56.564 2018/05/04 [Sample Frontend,ERROR] [odb.c:5911:db_set_data1,ERROR]
> online database full
> 
> If I run the mem command in odbedit I get the result at the end of this post.
> 
> Notice that I need to use an event size which is significantly larger than the
> default one. I don't know if it is relevant for this error. I have in the ODB:
> 
>   /Experiment/MAX_EVENT_SIZE = 900000000
> 
> and in the frontend code:
> 
>   /* maximum event size produced by this frontend */
>   INT max_event_size = 300000000;
> 
>   /* maximum event size for fragmented events (EQ_FRAGMENTED) */
>   INT max_event_size_frag = 5 * 1024 * 1024;
> 
>   /* buffer size to hold events */
>   INT event_buffer_size = 600000000;
> 
> Events seem to be properly stored in the output files, but I'm afraid I could
> get some other problem.
> 
> Thank you for your help,
>          Francesco
> 
> -------------------------------------------------------------------------
> 
> Database header size is 0x21040, all following values are offset by this!
> Key area  0x00000000 - 0x0007FFFF, size 524288 bytes
> Data area 0x00080000 - 0x00100000, size 524288 bytes
> 
> Keylist:
> --------
> Free block at 0x00000B58, size 0x00000008, next 0x000053E0
> Free block at 0x000053E0, size 0x00000008, next 0x00006560
> Free block at 0x00006560, size 0x00079AA0, next 0x00000000
> 
> Free Key area: 498352 bytes out of 524288 bytes
> 
> Data:
> -----
> Free block at 0x000847F0, size 0x0007B810, next 0x00000000
> 
> Free Data area: 505872 bytes out of 524288 bytes
> 
> Free: 498352 (95.1%) keylist, 505872 (96.5%) data

> Uncaught ReferenceError: XMLHttpRequestGeneric is not defined
>     at cs_button (Trend:165)
>     at HTMLInputElement.onclick (Trend:90)  

That code was not written by me, so I'm must guessing here.

Probably the XMLHttpRequestGeneric() is some function hiding browser specialities to create 
AJAX requests. These days most browser understand the standard request

XMLHttpRequest()

so why don't you try to just remove the "Generic"

Stefan

> I would like to store the address of 1-Wire temperature sensors in a device
> driver. However, the supportet data types (as definded around
> include/midas.h:311) do not foresee a type large enough.
> 
> Is there a good reason against this?
> 
> I know that other experiments use this kind of sensor, how do you store the
> addresses? I've noticed that most of the address is just zeroes, but I wouldn't
> like to store just half the address, assuming that half the address is always
> zeroes.

Well, when this code was written, computers had 640kB  and operating systems had 16 bit. What 
you  can do for your 1-wire sensor is to store the address in two values, one 32-bit LSB and one 
32-bit MSB. Or store it in a string with hex representation.

Stefan 

> > Asume you have a variable foo and a link bar -> foo. When you go to the ODB in
> > mhttpd, click "Delete" and select bar, it actually deletes foo. bar stays,
> > stating "<cannot resolve link>". Trying the same in odbedit with rm gives the
> > expected result (bar is gone, foo is still there).
> > 
> > I'm on the develop branch.
> 
> I think I can confirm this. Created a bug report on bitbucket:
> 
> https://bitbucket.org/tmidas/midas/issues/148/mhttpd-odb-editor-deletes-wrong-symlink
> 
> K.O.

I fixed this and committed the change. Took me a while since it was in KO's code.

Stefan

> > Dear experts,
> >               is there any way to launch an executable script on the host computer from the MIDAS 
> > sequencer? If not, is there any interest to develop such a feature?
> > 
> > Thank you,
> >          Francesco
> 
> The SCRIPT command will do that (on the machine running MIDAS). I know it works with either python or
> bash scripts. I tried without success to pass the parameters and I went around by setting ODB entries
> prior to running the script and then access to them within the script.

Passing parameters should work. If it's confirmed to be broken, I'm willing to fix it.

Stefan

> If you run an external script anyway, you can also call "odbedit -c alarm" to
> reset all alarms. Or you could try to set the "Triggered" entry of that certain
> alarm to 0 (again, with odbedit), that could also work.

That would not really help, because you cannot trigger a script AFTER an alarm occurred. Having 
"self-resetting" alarms is actually not a bad idea. I could add a flag "Auto reset" which is false by 
default and can be set to true for this functionality. Will keep that in mind for the next 
development cycle.

Stefan

> I was planning to use the alarm system to display an information banner when a
> certain valve is open, but I would like it to go away again when the valve is
> closed.
> Is there a way to achieve that? Maybe reset the alarm from an alarm script?
> (Seems like a hack...)
> Maybe this could be a useful feature, to be able to define an alarm class that
> resets itself once the condition is no longer met?

Actually you can implement such a thing already now pretty quickly using custom javascript on 
the status page. Just read the valve state regularly from the ODB and dynamically modify the 
status page to show or hide a banner. Look how custom pages work in midas and try to apply 
this to the status page status.html which you find in the resources directory.

Stefan