> A feature I always missed (or just missed to find in the docu) is the following:
> 1) It would be nice to have a command in odbedit which allows to check how full
> the ODB currently is.
> 2) Even more important: I would like to have an ODB routine which allows me to
> check the fill level of the ODB, and/or a routine which tells me if I would
> create a structure of given size that it still fits in the current ODB or not. 
> The use case is that some clients create on the fly ODB entries and I would like
> to make sure before hand the ODB's remaining space in order not to crash things
> by overfilling the ODB. 

If you do "mem" in odbedit, you see the currently free areas, one for the keys themselves, one for 
the data of the keys. The corresponding C function is db_show_mem. At the moment it outputs the 
list of free blocks as a long ASCII string, but if necessary I can write a variant which returns the 
number of free bytes.

Stefan

> > > A feature I always missed (or just missed to find in the docu) is the following:
> > > 1) It would be nice to have a command in odbedit which allows to check how full
> > > the ODB currently is.
> > > 2) Even more important: I would like to have an ODB routine which allows me to
> > > check the fill level of the ODB, and/or a routine which tells me if I would
> > > create a structure of given size that it still fits in the current ODB or not. 
> > > The use case is that some clients create on the fly ODB entries and I would like
> > > to make sure before hand the ODB's remaining space in order not to crash things
> > > by overfilling the ODB. 
> > 
> > If you do "mem" in odbedit, you see the currently free areas, one for the keys themselves, one for 
> > the data of the keys. The corresponding C function is db_show_mem. At the moment it outputs the 
> > list of free blocks as a long ASCII string, but if necessary I can write a variant which returns the 
> > number of free bytes.
> > 
> > Stefan
> 
> Thanks for the info, and yes a variant of db_show_mem returning the number of free bytes would be just
> prefect!

I made you db_get_free_mem(HNDLE hDB, INT *key_size, INT *data_size)

The first return gets you the number of free bytes for the key area, the second one for the data area (values of keys).

Committed to develop

Stefan

It might have worked some ~5 years ago, but it never really showed the target value of a link, just the 
link itself. I reworked the code now to show both the link and the target of the link, so you can change 
both in the mhttpd ODB page. Should be consistent now with odbedit. Have a look if it works for you.

Stefan

The function mhttpd_init() scans the custom page and installs handlers etc. for each modbxxx 
element. If you create an modbvalue dynamically in your code, this is probably done after you 
called mhttpd_init(), so the function has no chance to modify the dynamically created element.

To fix that, I separated mhttpd_init() into the old init function which installs the header and sidebar, 
and a function mhttpd_scan() which scans the custom page and processes all modbxxx elements. 
Next, I tapped the error you reported, and added an automatic call to mhttp_scan() in case that 
happens. I tried it on a test page and it worked for me. Please give it a try (commit 090394e8).

Stefan

Two options:

1) Do NOT send your events into the ODB. This is controlled via the flag RO_ODB in your frontend setting. For simple experiments with small events, it might make sense to copy each 
event into the ODB for debugging, but if you have large events, this does not make sense. Use the "mdump" utility to check your events instead.

2) Increase the size of the ODB. See the first FAQ here: https://midas.triumf.ca/MidasWiki/index.php/FAQ

Stefan


> Dear expert,
>       I'm developing a frontend and I'm getting this kind of error at each event:
> 
> 10:14:56.564 2018/05/04 [Sample Frontend,ERROR] [odb.c:5911:db_set_data1,ERROR]
> online database full
> 
> If I run the mem command in odbedit I get the result at the end of this post.
> 
> Notice that I need to use an event size which is significantly larger than the
> default one. I don't know if it is relevant for this error. I have in the ODB:
> 
>   /Experiment/MAX_EVENT_SIZE = 900000000
> 
> and in the frontend code:
> 
>   /* maximum event size produced by this frontend */
>   INT max_event_size = 300000000;
> 
>   /* maximum event size for fragmented events (EQ_FRAGMENTED) */
>   INT max_event_size_frag = 5 * 1024 * 1024;
> 
>   /* buffer size to hold events */
>   INT event_buffer_size = 600000000;
> 
> Events seem to be properly stored in the output files, but I'm afraid I could
> get some other problem.
> 
> Thank you for your help,
>          Francesco
> 
> -------------------------------------------------------------------------
> 
> Database header size is 0x21040, all following values are offset by this!
> Key area  0x00000000 - 0x0007FFFF, size 524288 bytes
> Data area 0x00080000 - 0x00100000, size 524288 bytes
> 
> Keylist:
> --------
> Free block at 0x00000B58, size 0x00000008, next 0x000053E0
> Free block at 0x000053E0, size 0x00000008, next 0x00006560
> Free block at 0x00006560, size 0x00079AA0, next 0x00000000
> 
> Free Key area: 498352 bytes out of 524288 bytes
> 
> Data:
> -----
> Free block at 0x000847F0, size 0x0007B810, next 0x00000000
> 
> Free Data area: 505872 bytes out of 524288 bytes
> 
> Free: 498352 (95.1%) keylist, 505872 (96.5%) data

> Uncaught ReferenceError: XMLHttpRequestGeneric is not defined
>     at cs_button (Trend:165)
>     at HTMLInputElement.onclick (Trend:90)  

That code was not written by me, so I'm must guessing here.

Probably the XMLHttpRequestGeneric() is some function hiding browser specialities to create 
AJAX requests. These days most browser understand the standard request

XMLHttpRequest()

so why don't you try to just remove the "Generic"

Stefan

> I would like to store the address of 1-Wire temperature sensors in a device
> driver. However, the supportet data types (as definded around
> include/midas.h:311) do not foresee a type large enough.
> 
> Is there a good reason against this?
> 
> I know that other experiments use this kind of sensor, how do you store the
> addresses? I've noticed that most of the address is just zeroes, but I wouldn't
> like to store just half the address, assuming that half the address is always
> zeroes.

Well, when this code was written, computers had 640kB  and operating systems had 16 bit. What 
you  can do for your 1-wire sensor is to store the address in two values, one 32-bit LSB and one 
32-bit MSB. Or store it in a string with hex representation.

Stefan 

> > Asume you have a variable foo and a link bar -> foo. When you go to the ODB in
> > mhttpd, click "Delete" and select bar, it actually deletes foo. bar stays,
> > stating "<cannot resolve link>". Trying the same in odbedit with rm gives the
> > expected result (bar is gone, foo is still there).
> > 
> > I'm on the develop branch.
> 
> I think I can confirm this. Created a bug report on bitbucket:
> 
> https://bitbucket.org/tmidas/midas/issues/148/mhttpd-odb-editor-deletes-wrong-symlink
> 
> K.O.

I fixed this and committed the change. Took me a while since it was in KO's code.

Stefan

> > Dear experts,
> >               is there any way to launch an executable script on the host computer from the MIDAS 
> > sequencer? If not, is there any interest to develop such a feature?
> > 
> > Thank you,
> >          Francesco
> 
> The SCRIPT command will do that (on the machine running MIDAS). I know it works with either python or
> bash scripts. I tried without success to pass the parameters and I went around by setting ODB entries
> prior to running the script and then access to them within the script.

Passing parameters should work. If it's confirmed to be broken, I'm willing to fix it.

Stefan

> If you run an external script anyway, you can also call "odbedit -c alarm" to
> reset all alarms. Or you could try to set the "Triggered" entry of that certain
> alarm to 0 (again, with odbedit), that could also work.

That would not really help, because you cannot trigger a script AFTER an alarm occurred. Having 
"self-resetting" alarms is actually not a bad idea. I could add a flag "Auto reset" which is false by 
default and can be set to true for this functionality. Will keep that in mind for the next 
development cycle.

Stefan

> I was planning to use the alarm system to display an information banner when a
> certain valve is open, but I would like it to go away again when the valve is
> closed.
> Is there a way to achieve that? Maybe reset the alarm from an alarm script?
> (Seems like a hack...)
> Maybe this could be a useful feature, to be able to define an alarm class that
> resets itself once the condition is no longer met?

Actually you can implement such a thing already now pretty quickly using custom javascript on 
the status page. Just read the valve state regularly from the ODB and dynamically modify the 
status page to show or hide a banner. Look how custom pages work in midas and try to apply 
this to the status page status.html which you find in the resources directory.

Stefan

> Hello Everybody,
> 
> I am a member of a satellite team with a scientific payload and I am considering
> coordinating the payload using MIDAS. This looks to be challenging since MIDAS
> would be implemented on an Xilinx Spartan 6 FPGA with minimal hardware
> resources. The idea would be to install a soft processor on the Spartan 6 and
> run MIDAS through UCLinux either on the FPGA or boot it from SPI Flash. Does
> anybody have any comments on how feasible this would be or perhaps have
> experience implementing a similar system?
> 
> -Devin

While some people successfully implemented a midas *client* in an FPGA softcore, the full midas 
backend would probably not fit into a Spartan 6. Having done some FPGA programming and 
working on satellites, I doubt that midas would be well suited for such an environment. It's 
probably some kind of overkill. The complete GUI is likely useless since you want to minimize your 
communication load on the satellite link.

Stefan

> 
> 
> One can collapse the side panel when looking at history pages with the button in
> the top left, great! We want to see many pages so screen real estate is important
> 
> The issue we face is that when the page refreshes, the side panel expands. Can
> we make the panel state more 'sticky'?
> 
> Many thanks
> Joseph (ALPHA)
> 
> Version: 	2.1
> Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
> on branch feature/midas-2017-10

Hi Joseph,

In principle a page refresh should now not be necessary, since pages should reload automatically 
the contents which changes. If a custom page needs a reload, it is not well designed. If necessary, I 
can explain the details. 

Anyhow I implemented your "stickyness" of the side panel in the last commit to the develop branch.

Best regards,
Stefan

> I apologise for miss using the word refresh. The re-appearing sidebar was also seen with the automatic
> reload, I have implemented your fix here and it now works great!

Still did not get your point. Why is there "automatic reload"? The status page should not "completely reload" any more. 
Instead, all data is fetched in the background using AJAX calls, and only the data on the page is updated once per second. If 
there is a "complete reload", something is wrong.

Stefan

> Joseph's original message says that the problem is with the standard MIDAS history page, which currently use a complete reload 
> when refreshing.  Of course we are planning to update this history pages to only grab what it needs (as well as changing the 
> plotting to use newer HTML plotting). But until that upgrade happens your fix is helpful for the history page.

Ok, now I understand, and of course I agree with you.

Stefan

> In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache 
> httpd, etc).

I guess this is what moste people do anyhow these days. Do I understand correctly that this then rules out the usage of letsencrype certificates, since the 
host needs to be accessed from outside, which is not possible if running behind a password protected firewall.

Stefan

All makes sense to me. I agree to proceed with the refactoring.

One additional comment: In the 90's when I developed this code, locking was expensive. On a decent computer you could do a couple of thousand lock operations per second before you hit the 100% 
CPU limit. Therefore I tried to reduce the number of lock operation as much as possible. Like a db_find_key locks the ODB once and then goes through all keys before it unlocks again. If I would lock for 
every key and have an ODB with ten thousands of keys, that would have taken very long in the old days. 

Now the world has changed, we can do almost a million locks a second. So a db_get_record() does not have to obtain a whole directory in one go, but can get each value separately, and if necessary lock 
the ODB on each key access. This would be slower, but only a negligible amount these days. So in the spirit of making midas more robust, we can even go a step beyond simple refactoring and change the 
locking scheme if it becomes more transparent and stable.

Best,
Stefan

I implemented that fix. Thank you to Andreas. Creating "Custom" directory from the web now does 
not have that problem any more.

Stefan

> I am not sure this is quite true. The CPU can execute 3000 million operations per second (3GHz CPU, assuming 1 op/Hz),
> so 1 lock operation is worth 3000 normal operations. Of course cache misses and branch mispredictions mess up
> this simple arithmetic...

You can try that with "t1" in odbedit. This times the number of db_get_data() calls midas can do per second. On my MacBook Pro I get 470'000 
accesses per second.

> BTW, "the fix" in mhttpd unconditionally creates /Custom/Path and sets it to the value of $MIDASSYS. This path 
> seems to be prepended to all file paths, so this fix also breaks the normal use of /Custom/xxx that contain the full 
> path name of the file to serve...

I just set the /Custom/Path to $MIDASSYS to have something non-zero there. This is only a default which should be changed to the directory 
containing the actual custom pages. If it breaks existing code, just set it manually to an empty string, nothing prevents you from doing that.

> Looks like file serving in mhttpd got messed up and needs to be reviewed. I still strongly believe that mhttpd should 
> be serve arbitrary files (only serve files explicitly listed in ODB) or as next best option, only serve files from 
> subdirectories explicitly listed in ODB.

I'm thinking along the same lines, but figured out that this cannot be done easily. If people have access to the ODB, the can put the directory 
/etc/ into the ODB and again read that way /etc/passwd. We would have to explicitly hard-code some directories to exclude like /etc/ /var/ etc. 
but on macOS that might be different. We could put the list of directories into a physical file, which cannot be edited via the web interface. 

Stefan