ID |
Date |
Author |
Topic |
Subject |
1253
|
28 Mar 2017 |
Konstantin Olchanski | Bug Report | Replaced with /experiment/menu, mhttpd - /Experiment/Menu Buttons - git-sha a350e8db11 | > > > > I think there sneaked in a little bug in the mhttpd: when starting an experiment
> > > > from scratch and starting the mhttpd, the Menu Buttons are missing
> >
> > Ok, the original problem with a small bug in the javascript code for the menu buttons (fixed now),
> > but I was moved to implement something I wanted to do for a long time.
> >
>
> Is this change back-wards compatible with an old ODB? Ie, if I upgrade MIDAS, will it notice that I have the old-style key "/Experiment/Menu Buttons"
> and replace it equivalently set keys in /Experiment/Menu? Or will it just continue to use the old-style ODB key?
I am trying to keep some compatibility between the web pages and mhttpd. I think in most cases, old mhttpd should continue to work
against new web pages (assuming matching mhttpd.js & co). But old web pages would probably break against new mhttpd, mostly due
to the rapid pace of their development.
Anyhow, the midas web page forms menu buttons in this order:
/Experiment/Menu, if it does not exist, then:
/Experiment/menu buttons, if it does not exist, then
built in list of menu buttons, which includes all possible buttons, hardcoded in mhttpd.js.
In cooperation with mhttpd: new mhttpd
- will automatically create the tree /experiment/menu with all buttons disabled
- will complain about the existence of /expriment/menu buttons, instruct user to delete it.
So to answer the question:
after git pull, make, restart mhttpd, you will see all possible menu buttons and you will have to go
into the odb editor to disable the buttons you do not want to see (i.e. the mscb button).
I did it this way on purpose, to give old-time midas users an opportunity to discover
some of the newly added buttons and pages, like the "chat" page, or the "example" page. If I migrated
the existing "menu buttons" verbatim, to the new tree, I would not even today know
that the "chat" page exists (I do not think it was ever announced or described on this forum
or anywhere in the documentation).
K.O. |
1255
|
05 Apr 2017 |
Andreas Suter | Bug Report | Equipment Expand doesn't work anymore | I'd liked very much the possibility to hide away Equipment on the main page. It
is also nice to have the '+' to get it quickly back when needed. However, this
seems not to work anymore (git c9d9d604803). Is this a feature or something went
wrong? |
1257
|
10 Apr 2017 |
Stefan Ritt | Bug Report | Equipment Expand doesn't work anymore | > I'd liked very much the possibility to hide away Equipment on the main page. It
> is also nice to have the '+' to get it quickly back when needed. However, this
> seems not to work anymore (git c9d9d604803). Is this a feature or something went
> wrong?
The expansion of the equipment list is handled by a Cookie ("expeq" being 1 or 0). When Konstantin
implemented the mongoose server instead of the internal mhttp server, he neglected to evaluate
this cookie. I fixed this now (also renamed the cookie to "midas_expeq") in the current development
branch. Please check if it's working.
Stefan |
1258
|
10 Apr 2017 |
Andreas Suter | Bug Report | Equipment Expand doesn't work anymore | > > I'd liked very much the possibility to hide away Equipment on the main page. It
> > is also nice to have the '+' to get it quickly back when needed. However, this
> > seems not to work anymore (git c9d9d604803). Is this a feature or something went
> > wrong?
>
> The expansion of the equipment list is handled by a Cookie ("expeq" being 1 or 0). When Konstantin
> implemented the mongoose server instead of the internal mhttp server, he neglected to evaluate
> this cookie. I fixed this now (also renamed the cookie to "midas_expeq") in the current development
> branch. Please check if it's working.
>
> Stefan
Tested it on two machines and expansion is back and working! Thanks a lot!
Andreas |
1259
|
13 Apr 2017 |
Andreas Suter | Bug Report | stop form odbedit broken | when I try to stop a run from odbedit I get a core dump.
[ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
midas commit 53af92a5d0...
-----
I checked what happens if I try to stop a run via the mhttpd web-page: this
works! So what is different?
-----
I placed a issue (# 47) on bitbucket as well.
What is the preferred channel to report potential bugs (elog / bitbucket issues)? |
1260
|
13 Apr 2017 |
Andreas Suter | Bug Report | stop form odbedit broken | > when I try to stop a run from odbedit I get a core dump.
>
> [ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
> Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
>
> midas commit 53af92a5d0...
>
> -----
>
> I checked what happens if I try to stop a run via the mhttpd web-page: this
> works! So what is different?
>
> -----
>
> I placed a issue (# 47) on bitbucket as well.
>
> What is the preferred channel to report potential bugs (elog / bitbucket issues)?
I think I found the problem. Some ODB String values which are **automatically**
generated:
CSS File = STRING : [1024] mhttpd.css
Sqlite dir = STRING : [1024]
History dir = STRING : [1024]
Sound = STRING : [1000] alarm.mp3
are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)
It looks as if this screws up quite a bit of the system! When deleting .ODB.SHM and
afterwards try to reload the ODB via a dump I previously made with odbedit, the
following is happening:
1) I get the error message that some strings are too long (exceeding
MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
variables this is.
2) After this reload, essentially nothing is working anymore. Any client I tried to
start just crashed.
Since it seems that the string length of MAX_STRING_LENGTH is very crucial I would
suggest that db_create_record (or whatever routine is dealing with it) checks for
STRING variables and ensures that they cannot exceed MAX_STRING_LENGTH.
When I shortened in my dump the above variables to MAX_STRING_LENGTH, regenerated the
ODB, also the 'stop' Problem in odbedit is gone. |
1264
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | Equipment Expand doesn't work anymore | > > > I'd liked very much the possibility to hide away Equipment on the main page. It
> > > is also nice to have the '+' to get it quickly back when needed. However, this
> > > seems not to work anymore (git c9d9d604803). Is this a feature or something went
> > > wrong?
> >
> > The expansion of the equipment list is handled by a Cookie ("expeq" being 1 or 0). When Konstantin
> > implemented the mongoose server instead of the internal mhttp server, he neglected to evaluate
> > this cookie. I fixed this now (also renamed the cookie to "midas_expeq") in the current development
> > branch. Please check if it's working.
> >
> > Stefan
>
> Tested it on two machines and expansion is back and working! Thanks a lot!
>
Confirmed fixed. Thanks. Not sure how this got lost.
K.O. |
1265
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | stop form odbedit broken | > > when I try to stop a run from odbedit I get a core dump.
> >
> > [ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
> > Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
> >
I am quite puzzled by this situation. We have seen the above error before, tried to track it down, failed. I was
always thinking this is some kind of strange size mismatch between odb size and shared memory size and
shared memory save file odb.shm size.
Now with your information, it looks like it is memory corruption.
I always thought there is no length limit to odb strings, except for the odb api problem where you have to
know the maximum string length for db_get_value() & co otherwise long strings will be corrupted. Today
nobody uses fixed size buffers, either db_get_value() allocates the string of correct size (replacing buffer
overflow errors with memory leak errors), or return std::string.
I shall check on the use of MAX_STRING_SIZE at least in odb itself...
The default value 256 seems to be too small for today's use. (if you want to store json data, web page
fragments, etc).
K.O.
> > midas commit 53af92a5d0...
> >
> > -----
> >
> > I checked what happens if I try to stop a run via the mhttpd web-page: this
> > works! So what is different?
> >
> > -----
> >
> > I placed a issue (# 47) on bitbucket as well.
> >
> > What is the preferred channel to report potential bugs (elog / bitbucket issues)?
>
> I think I found the problem. Some ODB String values which are **automatically**
> generated:
>
> CSS File = STRING : [1024] mhttpd.css
> Sqlite dir = STRING : [1024]
> History dir = STRING : [1024]
> Sound = STRING : [1000] alarm.mp3
>
> are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)
>
> It looks as if this screws up quite a bit of the system! When deleting .ODB.SHM and
> afterwards try to reload the ODB via a dump I previously made with odbedit, the
> following is happening:
>
> 1) I get the error message that some strings are too long (exceeding
> MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
> variables this is.
>
> 2) After this reload, essentially nothing is working anymore. Any client I tried to
> start just crashed.
>
> Since it seems that the string length of MAX_STRING_LENGTH is very crucial I would
> suggest that db_create_record (or whatever routine is dealing with it) checks for
> STRING variables and ensures that they cannot exceed MAX_STRING_LENGTH.
>
> When I shortened in my dump the above variables to MAX_STRING_LENGTH, regenerated the
> ODB, also the 'stop' Problem in odbedit is gone. |
1268
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | where to report bugs, stop form odbedit broken | >
> What is the preferred channel to report potential bugs (elog / bitbucket issues)?
>
I prefer that bugs be reported on this forum here. Most bugs affect every midas user, so best to notify the
whole community.
Bitbucket have a nice bug tracking system, but there is a couple of problems:
a) only a couple of people see the bug reports for midas, minimizing probability of fix.
b) bug reports on bitbucket stay on bitbucket, we do not have backups and archives
of bug reports, if tomorrow bitbucket goes belly-up, our bug database goes poof! with them.
c) I can search the bug report on this forum using "grep" (i am sure there is a "find" button
on the bitbucket web page and it finds what I am looking for right away).
So if you have a bug report that others should know about (i.e. the "+" button on the status page does
not work), I say use this forum.
If you have a bug that you think is unique to you, not interesting to others (i.e. my midas crashes when I
do X), file it on bitbucket. If you see no activity on the bitbucket for a week or two, repost it here.
K.O. |
1269
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | badly managed case in history_schema.cxx: dat file empty | > For an unknown reason, Logger died few days ago while writing the history. The
> file mhf_1489577446_20170315_system.dat was created, but was empty.
I ran into same problem installing new midas in the alpha experiment at cern. It should be fixed now:
https://bitbucket.org/tmidas/midas/commits/788021d9cb39a348a40e36f1b35b1440e06aa744
K.O.
>
> When trying to restart Logger, I would get a seg fault without any special error
> message.
>
> I tracked the issue to the "read_file_schema" function in history_schema.cxx
>
> * L4731, a pointer to HsFileSchema *s is declared.
> * L4747, We enter a while(1) loop.
> * L4749, get char on the filename.
> In our case, the file was empty, so the variable "b" gets NULL and the loop breaks.
>
> Problem: the memory allocation for "s" is later in the loop, L4768.
> Upon exiting the loop, L4854, we try to access record_size on a NULL pointer ==>
> SegFault.
>
> It would be nice to at least have a message before breaking the loop... |
1270
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | stop form odbedit broken | > when I try to stop a run from odbedit I get a core dump.
> [ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
> Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
>
I am puzzled. The crash is at the very end of everything (save odb shared memory to odb.shm),
does the run actually stop, or the crash is before the run is fully stopped? (I guess if you want
to run more odbedit commands after stopping the run, so you care about not crashing).
K.O. |
1271
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | MAX_STRING_LENGTH, stop form odbedit broken | >
> I shall check on the use of MAX_STRING_LENGTH at least in odb itself...
>
Ok, I looked at the use of MAX_STRING_LENGTH in ODB (odb.c):
a) it is not used in any critical places for the database itself, so it is not a limit on maximum length of TID_STRING data. good.
b) it is used in the code for saving/loading odb from .odb files (old format), not sure how it works against overlong strings, but probably
truncates/corrupts/crashes.
c) it is used in the code for saving odb to odb.xml files. Overlong strings are truncated (I added a message about it).
d) code for loading/saving to json files handles overlong strings okey.
e) odbedit "ls" truncates overlong strings, mhttpd has some oddities against overlong strings.
f) db_sprintf() truncates string text to MAX_STRING_LENGTH to avoid output buffer overflow (should use db_snprintf() instead).
Conclusion, overlong strings should be okey, but do not use the old .odb and .xml save files. (mlogger saves odb to output .mid file in xml
format, we should switch it to use json format).
> > CSS File = STRING : [1024] mhttpd.css
> > Sqlite dir = STRING : [1024]
> > History dir = STRING : [1024]
> > Sound = STRING : [1000] alarm.mp3
> > are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)
So these should not cause any corruption or problem unless actual content length exceeds 255 bytes,
even then they are okey if odb is only saved and loaded into json files.
> > 1) I get the error message that some strings are too long (exceeding
> > MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
> > variables this is.
this is in db_check_record(), where it compares odb content with user-supplied data descriptions (there is no system-supplied
data descriptions with strings longer than MAX_STRING_LENGTH).
so I think what happened is you created a data structure with overlong strings, passed it to db_paste() or something,
db_check_record() complained about it, and db_paste() corrupted memory.
> >
> > 2) After this reload, essentially nothing is working anymore. Any client I tried to start just crashed.
> >
Somebody corrupted some shared memory, most likely it was db_paste() corrupted odb shared memory.
K.O. |
1272
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | MAX_STRING_LENGTH, stop form odbedit broken | > >
> > I shall check on the use of MAX_STRING_LENGTH at least in odb itself...
> >
>
> Ok, I looked at the use of MAX_STRING_LENGTH in ODB (odb.c):
>
Fixed a small buglet, now saving and reloading odb in the old ".odb" format will silently truncate all overlong strings to 256 bytes. (I think it always did that).
K.O. |
1273
|
18 Apr 2017 |
Andreas Suter | Bug Report | run start/stop oddity | I stumbled over an oddity which I would like to understand better. Here the
boundaries:
- Enable non-localhost RPC -> y
- Disable RPC hosts check -> y
1) I am starting a run from ODBedit (start now -v):
07:13:11.272 2017/04/19 [ODBEdit,INFO] Run #26 started
07:13:25.516 2017/04/19 [Logger,LOG] File '/data/max/dlog/lem17_0026.root'
CRC32C checksum: 0x05ca4e7e, 1523383 bytes
On this little test experiment there is not much running, but it already shows
the effect I wanted to understand.
2) I am stopping the run from ODBedit (stop -v):
07:13:25.519 2017/04/19 [ODBEdit,INFO] Run #26 stopped
So, everything looks perfectly fine up to this point.
3) Now the 'strange' thing happens. To any point in time after this, I will stop
ODBEdit which results in the following messages:
07:13:32.335 2017/04/19 [ODBEdit,INFO] Program ODBEdit on host pc7962 stopped
07:13:32.335 2017/04/19 [Logger,ERROR] [midas.c:14079:rpc_server_receive,ERROR]
rpc check ok, abort canceled
This I do NOT understand! It looks as if the Logger (or any other client which
gets the run state transition) thinks that some Client (here ODBEdit) has a
broken connection. At least this is how I understand the comment in midas.c /
rpc_server_receive(). Is something broken in the de-registration from the RPC
server? By the way, all clients where running on the localhost, i.e. no remote
connection used here.
All this only happens if a run transition took place.
Unfortunately I do not understand the system well enough to suggest any fix to
this :-( and hence would appreciate any help. |
1274
|
19 Apr 2017 |
Stefan Ritt | Bug Report | MAX_STRING_LENGTH, stop form odbedit broken | ODB name lengths (the name of a key) are limited to 256 characters, the length of strings in the ODB should NOT be limited. At some point we wanted to have complete web pages inside the ODB,
which for sure are longer than 256 characters. While this was the idea, I see now that db_paste & co. is hopelessly broken. To fix it, everything should be changed to std::string which is in my opinion
the only 'clean' solution. That would also remove the cumbersome strlcpy and strlcat.
But looking at odb.c, replacing everything with std::string would probably take a brave programmer a couple of weeks. Not sure if we should dive into that adventure right now. The quick fix would be:
a) The strings "CSS File", "Sqlite dir" etc. reported below get reduced to 256 characters (MAX_STRING_LENGTH). The value of 256 characters came from the file system limitation in linux (some many
years ago), where a full path of a file could not exceed 256 characters. Not sure if this limit is still valid today, but having all file names in the ODB limited to 256 characters is maybe not a bad idea
anyhow (who wants to type in file names with more than 256 characters ???).
b) Change the max string length in db_paste to 1024 to cover the few exceptions above.
If we go with a), KO has to change his ODB file names, in case of b) I can do the change.
So what is your opinion?
Best regards,
Stefan
> >
> > I shall check on the use of MAX_STRING_LENGTH at least in odb itself...
> >
>
> Ok, I looked at the use of MAX_STRING_LENGTH in ODB (odb.c):
>
> a) it is not used in any critical places for the database itself, so it is not a limit on maximum length of TID_STRING data. good.
> b) it is used in the code for saving/loading odb from .odb files (old format), not sure how it works against overlong strings, but probably
> truncates/corrupts/crashes.
> c) it is used in the code for saving odb to odb.xml files. Overlong strings are truncated (I added a message about it).
> d) code for loading/saving to json files handles overlong strings okey.
> e) odbedit "ls" truncates overlong strings, mhttpd has some oddities against overlong strings.
> f) db_sprintf() truncates string text to MAX_STRING_LENGTH to avoid output buffer overflow (should use db_snprintf() instead).
>
> Conclusion, overlong strings should be okey, but do not use the old .odb and .xml save files. (mlogger saves odb to output .mid file in xml
> format, we should switch it to use json format).
>
> > > CSS File = STRING : [1024] mhttpd.css
> > > Sqlite dir = STRING : [1024]
> > > History dir = STRING : [1024]
> > > Sound = STRING : [1000] alarm.mp3
> > > are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)
>
> So these should not cause any corruption or problem unless actual content length exceeds 255 bytes,
> even then they are okey if odb is only saved and loaded into json files.
>
> > > 1) I get the error message that some strings are too long (exceeding
> > > MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
> > > variables this is.
>
> this is in db_check_record(), where it compares odb content with user-supplied data descriptions (there is no system-supplied
> data descriptions with strings longer than MAX_STRING_LENGTH).
>
> so I think what happened is you created a data structure with overlong strings, passed it to db_paste() or something,
> db_check_record() complained about it, and db_paste() corrupted memory.
>
> > >
> > > 2) After this reload, essentially nothing is working anymore. Any client I tried to start just crashed.
> > >
>
> Somebody corrupted some shared memory, most likely it was db_paste() corrupted odb shared memory.
>
> K.O. |
1275
|
19 Apr 2017 |
Stefan Ritt | Bug Report | MAX_STRING_LENGTH, stop form odbedit broken | > Fixed a small buglet, now saving and reloading odb in the old ".odb" format will silently truncate all overlong strings to 256 bytes. (I think it always did that).
Not sure that we want that. There might be cases where people want to store long strings. I would remove the truncation completely when saving .odb or .xml files, and fix the load routines to
deal with overlong strings.
Stefan |
1276
|
22 Apr 2017 |
Konstantin Olchanski | Bug Report | MAX_STRING_LENGTH, stop form odbedit broken | > > Fixed a small buglet, now saving and reloading odb in the old ".odb" format will silently truncate all overlong strings to 256 bytes. (I think it always did that).
>
> Not sure that we want that. There might be cases where people want to store long strings. I would remove the truncation completely when saving .odb or .xml files, and fix the load routines to
> deal with overlong strings.
>
Since I just looked at the code for reading/writing .odb format, I see that it uses fixed size buffer for reading lines from a file,
currently 2*MAX_STRING_LENGTH). I am not in the mood to rewrite and retest all that code. Never looked at the xml reader,
probably has same problem (xml writer truncates long strings via truncation in db_sprintf()).
Since we already have the json odb reader/writer that handles unlimited string length correctly (also handles unicode and
unusual odb names), perhaps we should make json as the default and be done with it.
K.O. |
1277
|
22 Apr 2017 |
Konstantin Olchanski | Bug Report | MAX_STRING_LENGTH, stop form odbedit broken | > ODB name lengths (the name of a key) are limited to 256 characters, the length of strings in the ODB should NOT be limited.
Right, I was not ever aware of such limitation until I just now looked at the .odb and .xml writing code. Definitely string length
is truncated to MAX_STRING_LENGTH on writing, chokes or truncates on reading.
The new json reader/writer handles overlength strings correctly. I would say we should deprecate the old formats and go forward
with json. Most current software can work with json data much easier than xml or custom .odb.
> I see now that db_paste & co. is hopelessly broken. To fix it, everything should be changed to std::string which is in my opinion
> the only 'clean' solution. That would also remove the cumbersome strlcpy and strlcat.
Yes, that's the code for reading .odb format.
>
> But looking at odb.c, replacing everything with std::string would probably take a brave programmer a couple of weeks. Not sure if we should dive into that adventure right now.
>
I agree. Too much of an adventure.
Simpler solution could be add a db_get_data(), db_get_value() that allocates a data buffer of correct size (user has to remember to free it).
> a) The strings "CSS File", "Sqlite dir" etc. reported below get reduced to 256 characters (MAX_STRING_LENGTH).
We should fix the inconsistency, my vote is it should be either MAX_STRING_LENGTH or PATH_MAX (from limits.h).
K.O. |
1278
|
24 Apr 2017 |
Stefan Ritt | Bug Report | stop form odbedit broken | > CSS File = STRING : [1024] mhttpd.css
> Sqlite dir = STRING : [1024]
> History dir = STRING : [1024]
> Sound = STRING : [1000] alarm.mp3
After a quick discussion with Konstantin, I changed these strings to a length of 256 chars
(MAX_STRING_LENGTH). Actually all changes I had to made was on code introduced by KO, so I hope I
did everything correctly. He should carefully check my changes (actually I would have preferred if he
could change his code himself...).
I agree with KO that the preferred format for saving the ODB should be JSON, but there might be
experiments with have some old ODB dumps in other formats, so we should not remove the possibility to
read those formats back.
Stefan |
1285
|
02 May 2017 |
Konstantin Olchanski | Bug Report | mhttpd inline-editor and web MAX_STRING_LENGTH, stop form odbedit broken | > > I shall check on the use of MAX_STRING_LENGTH at least in odb itself...
Also tested the web interface:
In the odb editor, overlong strings show truncated to MAX_STRING_LENGTH (via db_sprintf()),
but the odb inline-editor can handle overlong strings correctly.
The inline-editor implementation that uses ODBSet() had a string length limitation to maximum
URL length (ODBSet uses AJAX jset with call parameters encoded into the URL).
I now converted the inline-editor to use the json-rpc api (uses http post) and I confirm that this can handle
arbitrary long strings.
K.O. |
|