I switched mhttpd to use the new javascript based run start and stop pages.

There are two new html pages:

resources/start.html - mimics the old run start page exactly - where you can enter the "edit on 
start" parameters and start the run.
resources/transition.html - monitors the transition progress, shows the status of every transition 
client, their sequence number, waiting list dependency, time spent making rpc calls, etc.

If the new pages do not work for you, please report it here and switch to the old pages
by editing src/mhttpd.cxx - comment-out the line "#define NEW_START_STOP 1"

K.O.

midas wiki at https://midas.triumf.ca/MidasWiki/index.php/Main_Page
was updated to MediaWiki version 1.27.1, the current MediaWiki LTS release.
Everything should work as before, but if you see any problems or anomalies, please report
them on this forum here.
K.O.

> 
> I see you put some switches into the environment ("MIDAS_INVALID_STRING_IS_OK"). Do you think this is a good idea? Most variables are 
> sitting in the ODB (/experiment/xxx), except those which cannot be in the ODB because we need it before we open the ODB, like MIDAS_DIR. 
> Having them in the ODB has the advantage that everything is in one place, and we see a "list" of things we can change. From an empty 
> environment it is not clear that such a thing like "MIDAS_INVALID_STRING_IS_OK" does exist, while if it would be an ODB key it would be 
> obvious. Can I convince you to move this flag into the ODB?
>


Some additional explanation.

Time passed, the world turned, and the current web-compatible standard for text strings is UTF-8 encoded Unicode, see 
https://en.wikipedia.org/wiki/UTF-8
(ObCanadianContent, UTF-8 was invented the Canadian Rob Pike https://en.wikipedia.org/wiki/Rob_Pike)
(and by some other guy https://en.wikipedia.org/wiki/Ken_Thompson).

It turns out that not every combination of 8-bit characters (char*) is valid UTF-8 Unicode.

In the MIDAS world we run into this when MIDAS ODB strings are exported to Javascript running inside web
browsers ("custom pages", etc). ODB strings (TID_STRING) and ODB key names that are not valid UTF-8
make such web pages malfunction and do not work right.

One solution to this is to declare that ODB strings (TID_STRING) and ODB key names *must* be valid UTF-8 Unicode.

The present commits implemented this solution. Invalid UTF-8 is rejected by db_create() & co and by the ODB integrity validator.

This means some existing running experiment may suddenly break because somehow they have "old-style" ODB entries
or they mistakenly use TID_STRING to store arbitrary binary data (use array of TID_CHAR instead).

To permit such experiments to use current releases of MIDAS, we include a "defeat" device - to disable UTF-8 checks
until they figure out where non-UTF-8 strings come from and correct the problem.

Why is this defeat device non an ODB entry? Because it is not a normal mode of operation - there is no use-case where
an experiment will continue to use non-UTF-8 compatible ODB indefinitely, in the long term. For example, as the MIDAS user
interface moves to more and more to HTML+Javascript+"AJAX", such experiments will see that non-UTF-8 compatible ODB entries
cause all sorts of problems and will have to convert.


K.O.

> 
> If you compile with the included Makefile, you will see a 
> 
> -DOS_LINUX -DOS_DARWIN
> 

Moving forward, it looks like I can define these variables in midas.h and remove the need to define them on the compiler command line.

This would be part of the Makefile and header files cleanup to get things working on Windows10.

K.O.

> > I switched mhttpd to use the new javascript based run start and stop pages.
> 
> One initial complaint: the transition.html page doesn't seem to deal well with a frontend program using
> a deferred transition.
>

We now have a test frontend for deferred transitions, and this problem will likely be fixed.

> 
> I separately still think that the transition page should automatically go away after 5 seconds
>

This is a user-interface philosophy issue.

Instead of using personal preferences one should follow established design principles
(there is research done and books written about this).

I did not recently look at current recommendations for this type of interaction, but generally
one expects web pages to "do things" (such as switch to a different page) only when directed
by user input (press a button).

My personal opinion is that half the users will find 5 sec delay too slow, the other half will
find 5 sec too fast and the 3rd half will wonder "what happened, the web page flashed and disappeared,
did I miss something important, how do I get back to whatever is was?!?".

One idea is to implement the transition page as a implant on the state page - after the "start" page
you go back to the status page where you can see the progress of the transition. After the transition
completes, it's progress window "collapses" into a "success/failure" display with a link to the full
transition page to see any details of what happened. Any volunteers? (I would html-ize the status page first).

K.O.

As discussed before, the midas omnibus javascript file mhttpd.js has been split into three pieces:

midas.js - midas "public api" for building web pages that interact with midas
mhttpd.js - javascript functions used by mhttpd web pages
obsolete.js - functions still in use, but not recommended for new designs, mostly because of the deprecated "Synchronous XMLHttpRequest" business.

Consider these use cases:

a) completely standalone web pages served from some other web server (not mhttpd): loading midas.js, set the mhttpd location (base URL) via mjsonrpc_set_url(url) and issue 
midas json-rpc requests as normal. (mhttpd fully supports the cross-site scripting (CORS) function).

b) custom pages loaded from mhttpd without midas styling: same as above, but no need to set the mhttpd base url.

c) custom pages loaded from mhttpd with midas styling: load midas.js, load mhttpd.js, load midas.css or mhttpd.css, see aaa_template.html or example.html to see how it all fits 
together.

d) custom replacement for mhttpd standard web pages: to replace (for example) the standard "alarms" page, copy (or create a new one) alarms.html into the experiment directory 
($MIDAS_DIR, same place as .ODB.SHM) and hack away. You can start from alarms.html, from aaa_template.html or from example.html.

K.O.

P.S. I am also reviewing mhttpd.css - the existing css file severely changes standard html formatting making it difficult to create custom web pages (all online tutorials and examples 
look nothing like that are supposed to look like). The new CSS file midas.css fixes this by only changing formatting of html elements that explicitly ask for "midas styling", without 
contaminating the standard html formatting. midas.css only works for example.html and aaa_template.html for now.

P.P.S. Here is the complete list of javascript functions in all 3 files:

8s-macbook-pro:resources 8ss$ grep ^function midas.js mhttpd.js obsolete.js
midas.js:function mjsonrpc_set_url(url)
midas.js:function mjsonrpc_send_request(req)
midas.js:function mjsonrpc_debug_alert(rpc) {
midas.js:function mjsonrpc_decode_error(error) {
midas.js:function mjsonrpc_error_alert(error) {
midas.js:function mjsonrpc_make_request(method, params, id)
midas.js:function mjsonrpc_call(method, params, id)
midas.js:function mjsonrpc_start_program(name, id) {
midas.js:function mjsonrpc_stop_program(name, unique, id) {
midas.js:function mjsonrpc_cm_exist(name, unique, id) {
midas.js:function mjsonrpc_al_reset_alarm(alarms, id) {
midas.js:function mjsonrpc_al_trigger_alarm(name, message, xclass, condition, type, id) {
midas.js:function mjsonrpc_db_copy(paths, id) {
midas.js:function mjsonrpc_db_get_values(paths, id) {
midas.js:function mjsonrpc_db_ls(paths, id) {
midas.js:function mjsonrpc_db_resize(paths, new_lengths, id) {
midas.js:function mjsonrpc_db_key(paths, id) {
midas.js:function mjsonrpc_db_delete(paths, id) {
midas.js:function mjsonrpc_db_paste(paths, values, id) {
midas.js:function mjsonrpc_db_create(paths, id) {
midas.js:function mjsonrpc_cm_msg(message, type, id) {
mhttpd.js:function ODBFinishInlineEdit(p, path, bracket)
mhttpd.js:function ODBInlineEditKeydown(event, p, path, bracket)
mhttpd.js:function ODBInlineEdit(p, odb_path, bracket)
mhttpd.js:function mhttpd_disable_button(button)
mhttpd.js:function mhttpd_enable_button(button)
mhttpd.js:function mhttpd_hide_button(button)
mhttpd.js:function mhttpd_unhide_button(button)
mhttpd.js:function mhttpd_init_overlay(overlay)
mhttpd.js:function mhttpd_hide_overlay(overlay)
mhttpd.js:function mhttpd_unhide_overlay(overlay)
mhttpd.js:function mhttpd_getParameterByName(name) {
mhttpd.js:function mhttpd_goto_page(page) {
mhttpd.js:function mhttpd_navigation_bar(current_page)
mhttpd.js:function mhttpd_page_footer()
mhttpd.js:function mhttpd_create_page_handle_create(mouseEvent)
mhttpd.js:function mhttpd_create_page_handle_cancel(mouseEvent)
mhttpd.js:function mhttpd_delete_page_handle_delete(mouseEvent)
mhttpd.js:function mhttpd_delete_page_handle_cancel(mouseEvent)
mhttpd.js:function mhttpd_start_run()
mhttpd.js:function mhttpd_stop_run()
mhttpd.js:function mhttpd_pause_run()
mhttpd.js:function mhttpd_resume_run()
mhttpd.js:function mhttpd_cancel_transition()
mhttpd.js:function mhttpd_reset_alarm(alarm_name)
mhttpd.js:function msg_load(f)
mhttpd.js:function msg_prepend(msg)
mhttpd.js:function msg_append(msg)
mhttpd.js:function findPos(obj) {
mhttpd.js:function msg_extend()
mhttpd.js:function alarm_load()
mhttpd.js:function aspeak_click(t)
mhttpd.js:function mhttpd_alarm_speak(t)
mhttpd.js:function chat_kp(e)
mhttpd.js:function rb()
mhttpd.js:function speak_click(t)
mhttpd.js:function chat_send()
mhttpd.js:function chat_load()
mhttpd.js:function chat_format(line)
mhttpd.js:function chat_prepend(msg)
mhttpd.js:function chat_append(msg)
mhttpd.js:function chat_reformat()
mhttpd.js:function chat_extend()
obsolete.js:function XMLHttpRequestGeneric()
obsolete.js:function ODBSetURL(url)
obsolete.js:function ODBSet(path, value, pwdname)
obsolete.js:function ODBGet(path, format, defval, len, type)
obsolete.js:function ODBMGet(paths, callback, formats)
obsolete.js:function ODBGetRecord(path)
obsolete.js:function ODBExtractRecord(record, key)
obsolete.js:function ODBKey(path)
obsolete.js:function ODBCopy(path, format)
obsolete.js:function ODBCall(url, callback)
obsolete.js:function ODBMCopy(paths, callback, encoding)
obsolete.js:function ODBMLs(paths, callback)
obsolete.js:function ODBMCreate(paths, types, arraylengths, stringlengths, callback)
obsolete.js:function ODBMResize(paths, arraylengths, stringlengths, callback)
obsolete.js:function ODBMRename(paths, names, callback)
obsolete.js:function ODBMLink(paths, links, callback)
obsolete.js:function ODBMReorder(paths, indices, callback)
obsolete.js:function ODBMKey(paths, callback)
obsolete.js:function ODBMDelete(paths, callback)
obsolete.js:function ODBRpc_rev0(name, rpc, args)
obsolete.js:function ODBRpc_rev1(name, rpc, max_reply_length, args)
obsolete.js:function ODBRpc(program_name, command_name, arguments_string, callback, max_reply_length)
obsolete.js:function ODBGetMsg(facility, start, n)
obsolete.js:function ODBGenerateMsg(type,facility,user,msg)
obsolete.js:function ODBGetAlarms()
obsolete.js:function ODBEdit(path)
obsolete.js:function getMouseXY(e)
8s-macbook-pro:resources 8ss$

K.O.

> I have problem in event buffer size.
> 
> When run MIDAS, I got error "total event size (1307072) larger than buffer size
> (1048576)", so I guess that the EVENT_BUFFER_SIZE is small.
>

Correct. You have a choice of sending smaller events or increasing the buffer size.

Increasing the buffer size consumes computer memory, how much memory do you have on your machine?

> 
> I change EVENT_BUFFER_SIZE in midas.h from 0x100000 to 0x200000. After compiling
> and run MIDAS, I got other error "Shared memory segment with key 0x4d040761
> already exists, please remove it manually: ipcrm -M 0x4d040761 size0x204a3c" in
> system.C
> 

This is not normal. In recent versions of MIDAS (for the last few years)

a) buffer size is changed via ODB "/Experiment/buffer sizes", no need to edit midas.h
b) shared memory was switched from SYSV shared memory to POSIX shared memory, and you should not see any references to 
SYSV shared memory functions like "ipcrm", "shmget" and "segment key".

Are you using a very old version of MIDAS? Or maybe you have a MIDAS installation that still uses SYSV shared memory. Check 
the contents of .SHM_TYPE.TXT (in the same directory as .ODB.SHM), if would normally say "POSIXv2_SHM". If it says 
something else, it is best to convert to POSIX SHM. Simplest way is to stop everything, save odb to text file, delete 
.SHM_TYPE.TXT, restart odb with odbedit, reload from text file. Now check that .SHM_TYPE.TXT says "POSIXv2_SHM".

>
> I check the shmget() function in system.C and it is said that error come from
> Shared memory segments larger than 16,773,120 bytes and create teraspace shared
> memory segments
> 

What teraspace?!? You changed the size from 1 Mbyte to 2 Mbyte (0x200000), this is still below even the value you have above 
(16,773,120).

At the end, it is not clear what your problem is. After changing the shared memory size (via odb or via midas.h),
the midas *will* complain about the mismatch in size (existing vs expected) and will tell you how to fix it, (run "ipcrm").
After does this, is there still an error? Normally everything will just work. (you might also have to erase .SYSTEM.SHM,
midas will tell you to do so if it is needed).

So what is your final error? (After running ipcrm?)

K.O.

> memory in my PC is 16 GB 

You can safely go to buffer size 100 Mbytes or more.

> I check the contents of .SHM_TYPE.TXT and it is "POSIXv2_SHM".

Good.

> But there is no buffer sizes in "/Experiment" 

This is strange. How old is your midas? What does it say on the "help" page in "Revision"?

> After run "ipcrm -M 0x4d040761 size0x204a3c"

This command is wrong. It probably gave you an error instead of removing the shared memory, that's why
nothing worked afterwards.

My copy of system.c reads this:
cm_msg(MERROR, "ss_shm_open", "Shared memory segment with key 0x%x already exists, please remove it manually: ipcrm -M 0x%x", key, key);

Note how there is no text "size0x..." in my copy? What does your copy say? Did somebody change it?

> remove .SYSTEM.SHM and run MIDAS again, I still get error "Shared memory segment
> with key 0x4d040761 already exists, please remove it manually: ipcrm -M 0x4d040761 size0x204a3c" M.T

Yes, that's because the ipcrm command is wrong and did not work,
it should read "ipcrm -M 0x4d040761" without the spurious "size..." text.

K.O.

> memory in my PC is 16 GB 

You can safely go to buffer size 100 Mbytes or more.

> I check the contents of .SHM_TYPE.TXT and it is "POSIXv2_SHM".

Good.

> But there is no buffer sizes in "/Experiment" 

This is strange. How old is your midas? What does it say on the "help" page in "Revision"?

> After run "ipcrm -M 0x4d040761 size0x204a3c"

This command is wrong. It probably gave you an error instead of removing the shared memory, that's why
nothing worked afterwards.

My copy of system.c reads this:
cm_msg(MERROR, "ss_shm_open", "Shared memory segment with key 0x%x already exists, please remove it manually: ipcrm -M 0x%x", key, 
key);

Note how there is no text "size0x..." in my copy? What does your copy say? Did somebody change it?

> remove .SYSTEM.SHM and run MIDAS again, I still get error "Shared memory segment
> with key 0x4d040761 already exists, please remove it manually: ipcrm -M 0x4d040761 size0x204a3c" M.T

Yes, that's because the ipcrm command is wrong and did not work,
it should read "ipcrm -M 0x4d040761" without the spurious "size..." text.

K.O.

> > memory in my PC is 16 GB 
> 
> You can safely go to buffer size 100 Mbytes or more.
> 
> > I check the contents of .SHM_TYPE.TXT and it is "POSIXv2_SHM".
> 
> Good.


No, wait, this is all wrong. If it says POSIX shared memory, how come it later
complains about SYSV shared memory and tells you to run SYSV shared memory
commands like ipcrm?!?


> > But there is no buffer sizes in "/Experiment" 


Now this kind of makes sense - you are probably running a strange mixture
of very old and recently new MIDAS. Probably you current version is so old
that it does not use .SHM_TYPE.TXT and can only do SYSV shared memory
and so old it does not have "/Experiment/buffer sizes".

But at some point you must have run a recent version of midas, or you would
not have the file .SHM_TYPE.TXT in your experiment directory.

I say:

a) run the correct ipcrm command (without the spurious "size..." text)
b) review your computer contents to identify all the versions of midas
   and to make sure you are using the midas you want to use (old or new,
   whatever), but not some wrong version by accident (incorrect PATH setting, etc)

As MIDAS developers, we usually recommend that you use the latest version of MIDAS,
certainly latest version is simpler to debug.

K.O.

I reworked the alarm sounds in mhttpd - now you can turn off all sounds without disabling the 
alarm system for everybody.

a) new checkbox on the "alarms" page to turn off the alarm buzzer sound
b) fixed a bug where the status page will speak the last alarm even if the "speak" checkbox is 
unchecked on the "alarms" page (was coming through the TALK messages)
c) made sure the chat messages are only spoken if "speak" is enabled on the "chat" page
d) these speech and sounds settings are now stored in the browser "localStorage", which means 
they are shared across all open tabs and windows and are preserved across browser sessions and 
computer reboots.

I hope this is an improvement.

There is still one bug remaining - the first (last?) alarm is always spoken twice - 1st time in the loop 
over all alarms and 2nd time through the TALK messages. I do not know how to fix this.

K.O.

> Confirmed, this is a bug in mlogger. It should be creating *2* files, one with the before-compression checksum and one with the after-compression checksum. At 
> least both checksums are written to midas.log, so you can grep them from there. K.O.

This should be fixed now. Thank you for nudging me.

K.O.



> 
> > Hi
> > 
> > > On one side, such checksums help me confirm that uncompressed data contents is the same as original 
> > > data (compression/decompression is okey).
> > > 
> > 
> > > I can write the computed checksums into midas.log, or into runNNN.crc32, runNNN.sha256, etc files. (or 
> > > both).
> > > 
> > 
> > Just a thought on my side. I have been using a checksum, on data produced  by our experiments via mlogger, the runxxxx.mid.gz, in 
> > the same manner you proposed and I see now implemented. 
> > 
> > I have a slight, objection, if I may call it that, to how the checksum is saved to disk, in 
> > run00007.mid.gz.sha256 as an example.
> > 
> > 
> > $ cat ~/Data/run00007.mid.gz.sha256
> > f315af7caf6ca204cc082132862cb4227d77066cb60c6e2b1039d6dc5b04d1ee 650597 Data/run00007.mid.gz
> > 
> > 
> > It seems a little misleading to have the gzip'd filename paired with the checksum of the uncompressed content.
> > 
> > May I suggest that the pairing should be ,
> > 
> > f315af7caf6ca204cc082132862cb4227d77066cb60c6e2b1039d6dc5b04d1ee  run00007.mid as an example.
> > 
> > As I find, this information will sit in an archive, database in my case for a long period, and it might
> > be confusing later on, when verification of the checksum is required.

> I think there sneaked in a little bug in the mhttpd: when starting an experiment
> from scratch and starting the mhttpd, the Menu Buttons are missing and,
> correctly, I get periodic error messages. I expected that the default ODB entry
> for the Menu Buttons is create if it doesn't exist. As far as I see this happens
> now since the default creation of the 'Menu Buttons' is now tag as an obsolete
> feature. In case this is not a bug but a feature, it should documented.

I think you are right. Will fix.

K.O.

> > I think there sneaked in a little bug in the mhttpd: when starting an experiment
> > from scratch and starting the mhttpd, the Menu Buttons are missing

Ok, the original problem with a small bug in the javascript code for the menu buttons (fixed now),
but I was moved to implement something I wanted to do for a long time.

The menu configuration is now done through a subdirectory /experiment/menu. Each entry corresponds to 
one menu button. Set to "y" to show it, set to "n" to hide it.

Buttons are displayed in the same order as they are in ODB, to change the order of buttons,
change their order in ODB (odbedit command "move").

This fixes the long standing problem with adding new midas pages - they were not automatically added to 
the existing "menu buttons" lists. So for example when the "chat" page was added, I did not know about it 
for a long time (and some people still do not know about it's existence) because it is was not included in 
my  "/experiment/menu buttons" list in all my already existing experiments. When the "start" and 
"transition" pages were added, probably nobody knows that they exist.

Now new buttons for new pages are automatically added to the list (via mhttpd.cxx::init_menu_buttons()), 
the users have an option to hide them by setting their values to "n".

K.O.

> > > > I think there sneaked in a little bug in the mhttpd: when starting an experiment
> > > > from scratch and starting the mhttpd, the Menu Buttons are missing
> > 
> > Ok, the original problem with a small bug in the javascript code for the menu buttons (fixed now),
> > but I was moved to implement something I wanted to do for a long time.
> > 
> 
> Is this change back-wards compatible with an old ODB?  Ie, if I upgrade MIDAS, will it notice that I have the old-style key "/Experiment/Menu Buttons" 
> and replace it equivalently set keys in /Experiment/Menu?  Or will it just continue to use the old-style ODB key?

I am trying to keep some compatibility between the web pages and mhttpd. I think in most cases, old mhttpd should continue to work
against new web pages (assuming matching mhttpd.js & co). But old web pages would probably break against new mhttpd, mostly due
to the rapid pace of their development.

Anyhow, the midas web page forms menu buttons in this order:
/Experiment/Menu, if it does not exist, then:
/Experiment/menu buttons, if it does not exist, then
built in list of menu buttons, which includes all possible buttons, hardcoded in mhttpd.js.

In cooperation with mhttpd: new mhttpd
- will automatically create the tree /experiment/menu with all buttons disabled
- will complain about the existence of /expriment/menu buttons, instruct user to delete it.

So to answer the question:
after git pull, make, restart mhttpd, you will see all possible menu buttons and you will have to go
into the odb editor to disable the buttons you do not want to see (i.e. the mscb button).

I did it this way on purpose, to give old-time midas users an opportunity to discover
some of the newly added buttons and pages, like the "chat" page, or the "example" page. If I migrated
the existing "menu buttons" verbatim, to the new tree, I would not even today know
that the "chat" page exists (I do not think it was ever announced or described on this forum
or anywhere in the documentation).

K.O.

> In my opinion this makes sense. If KO agrees, you should commit your change.

Please go ahead (sorry for slow reply). I have no idea what this change does. A screenshot of "before" 
and "after" would be nice. The reason I ask is:

note that I am getting rid of the css hell in mhttpd.css. all the new pages will be using the simplified css 
rules in midas.css.

the main change is: the new css rules only change the appearance of html elements that request the 
"midas look" and one can still use the normal html formatting if desired. The old css changed all (and I 
do mean *all*) html elements, making it impossible to write custom web pages using common examples 
from the web - the insane formatting from mhttpd.css was applied to everything indiscriminantly, i.e. h1, 
h2, h3 all look the same.

K.O.


> 
> Stefan
> 
> > We use the customHeader to display some useful information. Currently I do not
> > like its style. What about to make it more alike the footer?
> > 
> > I just changed in resources/mhttpd.css
> > 
> > diff --git a/resources/mhttpd.css b/resources/mhttpd.css
> > index fb0070d..f3264c8 100644
> > --- a/resources/mhttpd.css
> > +++ b/resources/mhttpd.css
> > @@ -280,6 +280,15 @@ table.headerTable td{
> >         border: none;
> >  }
> >  
> > +div.headerDiv{
> > +       background-color: #6F6F6F;
> > +       text-align: center;
> > +       padding:1em;
> > +       color:#EEEEEE;
> > +       border-bottom:1px solid #000000;
> > +       height:3em;
> > +}
> > +
> >  div.footerDiv{
> >         background-color: #808080;
> >         text-align: center;
> > 
> > and
> > 
> > diff --git a/resources/mhttpd.js b/resources/mhttpd.js
> > index de8bc6c..972c261 100644
> > --- a/resources/mhttpd.js
> > +++ b/resources/mhttpd.js
> > @@ -172,7 +172,7 @@ function mhttpd_goto_page(page) {
> >  
> >  function mhttpd_navigation_bar(current_page, path)
> >  {
> > -   document.write("<div id=\"customHeader\">\n");
> > +   document.write("<div class=\"headerDiv\" id=\"customHeader\">\n");
> >     document.write("</div>\n");
> >  
> >     document.write("<div class=\"mnavcss\">\n");
> > 
> > What do you think?

> > > I'd liked very much the possibility to hide away Equipment on the main page. It
> > > is also nice to have the '+' to get it quickly back when needed. However, this
> > > seems not to work anymore (git c9d9d604803). Is this a feature or something went
> > > wrong?
> > 
> > The expansion of the equipment list is handled by a Cookie ("expeq" being 1 or 0). When Konstantin 
> > implemented the mongoose server instead of the internal mhttp server, he neglected to evaluate 
> > this cookie. I fixed this now (also renamed the cookie to "midas_expeq") in the current development 
> > branch. Please check if it's working.
> > 
> > Stefan
> 
> Tested it on two machines and expansion is back and working! Thanks a lot!
> 

Confirmed fixed. Thanks. Not sure how this got lost.

K.O.

> > when I try to stop a run from odbedit I get a core dump.
> > 
> > [ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
> > Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
> > 


I am quite puzzled by this situation. We have seen the above error before, tried to track it down, failed. I was 
always thinking this is some kind of strange size mismatch between odb size and shared memory size and 
shared memory save file odb.shm size.

Now with your information, it looks like it is memory corruption.

I always thought there is no length limit to odb strings, except for the odb api problem where you have to 
know the maximum string length for db_get_value() & co otherwise long strings will be corrupted. Today
nobody uses fixed size buffers, either db_get_value() allocates the string of correct size (replacing buffer
overflow errors with memory leak errors), or return std::string.

I shall check on the use of MAX_STRING_SIZE at least in odb itself...

The default value 256 seems to be too small for today's use. (if you want to store json data, web page 
fragments, etc).


K.O.




> > midas commit 53af92a5d0...
> > 
> > -----
> > 
> > I checked what happens if I try to stop a run via the mhttpd web-page: this
> > works! So what is different?
> > 
> > -----
> > 
> > I placed a issue (# 47) on bitbucket as well.
> > 
> > What is the preferred channel to report potential bugs (elog / bitbucket issues)? 
> 
> I think I found the problem. Some ODB String values which are **automatically**
> generated:
> 
> CSS File = STRING : [1024] mhttpd.css
> Sqlite dir = STRING : [1024]
> History dir = STRING : [1024]
> Sound = STRING : [1000] alarm.mp3
> 
> are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)
> 
> It looks as if this screws up quite a bit of the system! When deleting .ODB.SHM and
> afterwards try to reload the ODB via a dump I previously made with odbedit, the
> following is happening:
> 
> 1) I get the error message that some strings are too long (exceeding
> MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
> variables this is.
> 
> 2) After this reload, essentially nothing is working anymore. Any client I tried to
> start just crashed.
> 
> Since it seems that the string length of MAX_STRING_LENGTH is very crucial I would
> suggest that db_create_record (or whatever routine is dealing with it) checks for
> STRING variables and ensures that they cannot exceed MAX_STRING_LENGTH.
> 
> When I shortened in my dump the above variables to MAX_STRING_LENGTH, regenerated the
> ODB, also the 'stop' Problem in odbedit is gone.

> > Hi everyone, 
> > 
> > We have recently been experiencing a lot of lag with our midas control webpage,
> > which is making it very frustrating to use. Has anyone experienced this, and do
> > you have any advice to speed it up? Are there particular web browsers that work
> > better than others, or certain settings that can make respond more quickly?
> > 
> > Thanks!
> > Wes
> 
> We saw this happening as well. In our case, we could track this down to mhttpd
> taking a lot of CPU. A kill/restart of mhttpd is usually doing the trick (without
> disturbing data taking). We did not find an obvious reason for this happening.

One place where mhttpd can be stalled (and even go into infornite loop) is making history plots.

If you ask for a history plot of 10 variables across 1 year, nobody can access any midas web page
until mhttpd finishes grinding through the history data. (with the old .hst history format is was exceedingly 
slow, with the new "file" format, it is pretty quick, but everybody still has to wait). If you leave this page 
open, it will autorefresh every so many minutes ensuring continuing delays for other mhttpd users.

The other place for stalling mhttpd was in the run transitions (mhttpd was unresponsive while executing a 
run transition), this was fixed by the multithreaded transitions.

To fix the unresponsive history requests, you can try to setup a separate "history mhttpd", run a second 
mhttpd on a different port (with "-H" if desired), put this URL of this mhttpd in ODB "/history/url". (if you 
are using my instructions for setting up the apache httpd proxy, you can see provisions for this. 
/history/url will be set to "https://proxy.host.net/history/").

If neither of the above, there is the usual culprits of bad networking somewhere, etc.

Best way to test if delays are in midas or elsewhere is to stand in front of your midas computer, run a 
current version of google-chrome or firefox right on it, there should be no delays.

K.O.

> > > 
> > > We have recently been experiencing a lot of lag with our midas control webpage,
> > > which is making it very frustrating to use. Has anyone experienced this, and do
> > > you have any advice to speed it up? Are there particular web browsers that work
> > > better than others, or certain settings that can make respond more quickly?
> > > 

Wes, you provided excessive information. Who is "we", what is your location (internet in africa is different from internet in canada),
what is your computer (rpi3 is different from mac mini), what is your os (fedora-1 is different from centos-7), what
is your browser (netscape is different from google-chrome).

As to "what browser should work", on MacOS, google-chrome and firefox should be ok (that's what I test), on Linux,
stock firefox (usually an oldish esr version) should work, on el7 and ubuntu google-chrome works. On windows, google-chrome
and firefox should be ok. microsoft browsers probably not ok (no testing). cellphone browsers also not tested (but google-chrome and firefox 
should be ok).

K.O.