ID |
Date |
Author |
Topic |
Subject |
1343
|
01 Mar 2018 |
Andreas Suter | Bug Report | mhttpd / odb set strings -> truncates odb entry |
There is a bug in the string handling when changing ODB string entries via the
mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
For instance I create a string with length 32 and set it with odbedit to 'a'.
Then the string length stays 32, as expected. If the same is done through the
web-interface, the string length will be truncated to 2.
This can lead to problems if some frontend has a hotlink to a structure
containing this string since it will complain about structure size mismatch. |
1344
|
02 Mar 2018 |
Stefan Ritt | Bug Report | mhttpd / odb set strings -> truncates odb entry |
> There is a bug in the string handling when changing ODB string entries via the
> mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
>
> For instance I create a string with length 32 and set it with odbedit to 'a'.
> Then the string length stays 32, as expected. If the same is done through the
> web-interface, the string length will be truncated to 2.
>
> This can lead to problems if some frontend has a hotlink to a structure
> containing this string since it will complain about structure size mismatch.
I know about this problem since last summer. I mentioned it to KO, since it's deep down in his
JSONRPC code. We had a long discussion, where he kind of insisted that this is not a bug but a
feature. The ODB should store strings with variable lengths, and thus adapt it according to the
current string length. This makes some sense, since in the future we plan to put C++ string
support for the ODB, where strings have dynamically varying lengths. But this will take a while, so
I asked KO to change the truncation of the strings though the web interface, because this breaks
many experiments. He did not react so far. Several people complained. Maybe your request will
help now.
Stefan |
1345
|
05 Mar 2018 |
Andreas Suter | Bug Report | mhttpd / odb set strings -> truncates odb entry |
> > There is a bug in the string handling when changing ODB string entries via the
> > mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
> >
> > For instance I create a string with length 32 and set it with odbedit to 'a'.
> > Then the string length stays 32, as expected. If the same is done through the
> > web-interface, the string length will be truncated to 2.
> >
> > This can lead to problems if some frontend has a hotlink to a structure
> > containing this string since it will complain about structure size mismatch.
>
> I know about this problem since last summer. I mentioned it to KO, since it's deep down in his
> JSONRPC code. We had a long discussion, where he kind of insisted that this is not a bug but a
> feature. The ODB should store strings with variable lengths, and thus adapt it according to the
> current string length. This makes some sense, since in the future we plan to put C++ string
> support for the ODB, where strings have dynamically varying lengths. But this will take a while, so
> I asked KO to change the truncation of the strings though the web interface, because this breaks
> many experiments. He did not react so far. Several people complained. Maybe your request will
> help now.
>
> Stefan
Well I appreciate the direction towards more C++ string handling, yet it must not break the hotlink
functionality which is very important at many places. |
1364
|
18 Apr 2018 |
Thomas Lindner | Bug Fix | mhttpd / odb set strings -> truncates odb entry |
I wanted to try to summarize the current situation with regards to the handling of strings through the MIDAS web interface.
During the last year, we started the switch-over to using the new mjson-rpc functions in the MIDAS webpages. As part of this work, changes were made that allowed for
resizing strings through the MIDAS web interface (specifically through the MJSON-RPC calls). This reflected desires from some users that strings could be allowed to
grow larger than the default 256 size that is usually used for MIDAS strings (for instance, see [1]). In this first set of changes the ODB strings would always be resized to
the exact size of the string that it was set to.
The problem with this change was that it breaks the behaviour of db_get_record(), which typically expects strings to be a fixed length of 32 or 256 (for instance, see
[2,3]). Konstantin notes that we can work around this problem by using db_get_record1() function, which automatically resizes strings to expected values; this method
has already been used in the MIDAS core code. But the problem would still remain for some user code that uses db_get_record.
As a short-term compromise solution, Stefan implemented a change where the MJSON-RPC string setting will now expand the size of strings, but not truncate them; ie a
string that is size 256 will stay 256 bytes, unless you set it to something larger. So this should fix most cases of user code that call db_get_record() and hence expects
fixed string lengths. Users who call that db_get_record with strings that exceed the expected length will still have problems; but now you will at least get an explicit
MIDAS error message, rather than just silent string truncation (as was the case before).
In the longer run we still want to develop a new set of ODB methods that more naturally support C++-style variable-length strings. But that's a discussion for the longer
term.
[1] https://midas.triumf.ca/elog/Midas/1150
[2] https://bitbucket.org/tmidas/midas/issues/121/odb-inline-editor-truncates-stings
[3] https://midas.triumf.ca/elog/Midas/1343 |
1351
|
12 Mar 2018 |
Andreas Suter | Forum | mhttpd / javascript - simple check if a client is running |
Is there a simple way from the javascript side to check if a fontend is running?
Currently one would need to go through the /System/Client list to find out if a
frontend/client is running. Wouldn't it be nice to have this centralized, either
in the mhttpd.cxx or mhttpd.js part? |
1353
|
13 Mar 2018 |
Thomas Lindner | Forum | mhttpd / javascript - simple check if a client is running |
> Is there a simple way from the javascript side to check if a fontend is running?
> Currently one would need to go through the /System/Client list to find out if a
> frontend/client is running. Wouldn't it be nice to have this centralized, either
> in the mhttpd.cxx or mhttpd.js part?
Hi,
I think that this option already exists with the cm_exist method for the mjsonrpc calls. For instance, you can use a
call like
curl -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","id":null,"method":"cm_exist","params":
{"name":"Logger"}}' 'http://localhost:8081?mjsonrpc'
to get the status of the logger program. There is a description of the cm_exist parameters on this page:
https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc |
1354
|
13 Mar 2018 |
Andreas Suter | Forum | mhttpd / javascript - simple check if a client is running |
> > Is there a simple way from the javascript side to check if a fontend is running?
> > Currently one would need to go through the /System/Client list to find out if a
> > frontend/client is running. Wouldn't it be nice to have this centralized, either
> > in the mhttpd.cxx or mhttpd.js part?
>
> Hi,
>
> I think that this option already exists with the cm_exist method for the mjsonrpc calls. For instance, you can use a
> call like
>
> curl -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","id":null,"method":"cm_exist","params":
> {"name":"Logger"}}' 'http://localhost:8081?mjsonrpc'
>
> to get the status of the logger program. There is a description of the cm_exist parameters on this page:
>
> https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc
Thanks a lot for the info. I just simply missed it :-| |
1291
|
09 May 2017 |
Andreas Suter | Bug Report | mhttpd / history / export data |
A handy feature of the history of the mhttpd is to export the data. However, this
seems to be broken. It currently only works if the run marker flag is activated by
fails otherwise. |
1292
|
16 May 2017 |
Konstantin Olchanski | Bug Report | mhttpd / history / export data |
> A handy feature of the history of the mhttpd is to export the data. However, this
> seems to be broken. It currently only works if the run marker flag is activated by
> fails otherwise.
imo, it never worked properly. I think the best hope for working "export" button
is an "export as json" which gives you basically the output of hs_read_buffer() in the json
format. With options for "raw data" or "binned, with mean, rms, min, max for each bin".
K.O. |
1499
|
18 Mar 2019 |
Andreas Suter | Bug Report | mhttpd - slowcontrol frontend - multi class driver |
When using a slowcontrol frontend which operates a device using the multi class
driver the current midas version (ec3225902d6) has the following issue:
There is a row labeled with: All Input Output
So far by clicking e.g. on Input, only the Input related part was displayed, etc.
Currently this leads to the following error message:
Error: cannot find key LS336/Variables/Input
where LS336 is my DD. |
1501
|
18 Mar 2019 |
Konstantin Olchanski | Bug Report | mhttpd - slowcontrol frontend - multi class driver |
> When using a slowcontrol frontend which operates a device using the multi class
> driver the current midas version (ec3225902d6) has the following issue:
>
> There is a row labeled with: All Input Output
>
This is the "slow control" page you enter by clicking on the equipment name
on the midas status page, yes?
There is some kid of logic behind that link that send one either to the ODB page
for /Eq/XXX/Variables or to the "slow control" page that displays
/Eq/XXX/Variables in a table.
I just tried it and I cannot get to this page in my test experiment, so it will take
a few minutes for me to reproduce your problem. But I think I know where the breakage
is - with the new URL scheme, the links for selecting which variable to show either
was not converted to the new URL scheme or there is a bug and it send us to the wrong place.
I do not remember testing that code, so I will take a look at it definitely.
> Currently this leads to the following error message:
> Error: cannot find key LS336/Variables/Input
Looks like the /Equipment part is missing from the ODB path...
K.O. |
1506
|
25 Mar 2019 |
Konstantin Olchanski | Bug Report | mhttpd - slowcontrol frontend - multi class driver |
Fixed in https://bitbucket.org/tmidas/midas/commits/e2c4871026121ed1cc44a69b9e3e2d428a6c84d1
The link was pointing to the wrong place - going to ODB instead of staying on the same page.
K.O.
> > When using a slowcontrol frontend which operates a device using the multi class
> > driver the current midas version (ec3225902d6) has the following issue:
> >
> > There is a row labeled with: All Input Output
> >
>
> This is the "slow control" page you enter by clicking on the equipment name
> on the midas status page, yes?
>
> There is some kid of logic behind that link that send one either to the ODB page
> for /Eq/XXX/Variables or to the "slow control" page that displays
> /Eq/XXX/Variables in a table.
>
> I just tried it and I cannot get to this page in my test experiment, so it will take
> a few minutes for me to reproduce your problem. But I think I know where the breakage
> is - with the new URL scheme, the links for selecting which variable to show either
> was not converted to the new URL scheme or there is a bug and it send us to the wrong place.
>
> I do not remember testing that code, so I will take a look at it definitely.
>
> > Currently this leads to the following error message:
> > Error: cannot find key LS336/Variables/Input
>
> Looks like the /Equipment part is missing from the ODB path...
>
> K.O. |
1335
|
10 Jan 2018 |
Andreas Suter | Bug Report | mhttpd - custom page - RHEL/Fedora |
Description of the problem (starting with 61be7a1):
When starting a new experiment, creating a fresh ODB and than adding the
directory '/Custom', the mhttpd runs into a problem on RHEL/Fedora, but not on
Ubuntu and macOS. When trying to open the ODB from within whatever browser I get
the following error message in the midas message queque:
[mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of
4096 returned -1, errno 21 (Is a directory)
and in the browser I get a popup which tries to save a file called 'root'.
I track this down to the following: in mhttpd, interprete (line 18046) it is
check if a custom page file exists (ss_file_exist) and if yes, it tries to 'load'
it. Now, at this stage the variable dec_path contains '/root'.
Here now what goes wrong: ss_file_exist tries to open the given path, and if a
valid file descriptor is returned it assumes the file exists. This is not
perfectly correct since it also will get a valid file descriptor is path is an
accessible directory!
Now for whatever reason, on RHEL/Fedora '/root' will return a valid file
descriptor, but not on macOS and Ubuntu. Others I haven't tested. A possible fix
would be to check explicitly if path is a directory and if yes return 0 in
ss_file_exist (see attached diff).
Perhaps there is cleaner way to deal with this issue?! |
1336
|
11 Jan 2018 |
Konstantin Olchanski | Bug Report | mhttpd - custom page - RHEL/Fedora |
> [mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of
> 4096 returned -1, errno 21 (Is a directory)
On some linux systems, "/root" exists, it is a directory used as the home directory
of user "root" (~root is /root; traditional UNIX has ~root as /).
open() of a directory succeeds on some UNIX systems, on some of them,
read() also works, but on other systems one is supposed
to use opendir() and readdir().
MacOS is of course a BSD system (not SysV like Solaris, not Linux), so things
are different yet again. I think MacOS does not have a /root.
In any case, IMO, mhttpd has no business serving the contents of /root,
or serving any files outside of the mhttpd user $HOME directory. (but also
should not serve files from ~user/.ssh, or any other "secret" files, good
luck making a complete axhuastive list of all secret files that should not be
served).
K.O.
>
> and in the browser I get a popup which tries to save a file called 'root'.
>
> I track this down to the following: in mhttpd, interprete (line 18046) it is
> check if a custom page file exists (ss_file_exist) and if yes, it tries to 'load'
> it. Now, at this stage the variable dec_path contains '/root'.
>
> Here now what goes wrong: ss_file_exist tries to open the given path, and if a
> valid file descriptor is returned it assumes the file exists. This is not
> perfectly correct since it also will get a valid file descriptor is path is an
> accessible directory!
>
> Now for whatever reason, on RHEL/Fedora '/root' will return a valid file
> descriptor, but not on macOS and Ubuntu. Others I haven't tested. A possible fix
> would be to check explicitly if path is a directory and if yes return 0 in
> ss_file_exist (see attached diff).
>
> Perhaps there is cleaner way to deal with this issue?! |
1337
|
12 Jan 2018 |
Stefan Ritt | Bug Report | mhttpd - custom page - RHEL/Fedora |
> In any case, IMO, mhttpd has no business serving the contents of /root,
> or serving any files outside of the mhttpd user $HOME directory. (but also
> should not serve files from ~user/.ssh, or any other "secret" files, good
> luck making a complete axhuastive list of all secret files that should not be
> served).
I fully agree with Konstantin. mhttpd should only serve files under certain directories. One is the
midas/resources directory, another is the one defined in the ODB under /Custom/Path. I plan to modify
mhttpd to only serve these files (and also prevent tricks like putting "../../../" into the URL). This will then
also fix Andreas' problem.
Stefan |
Draft
|
15 Jan 2018 |
Andreas Suter | Bug Report | mhttpd - custom page - RHEL/Fedora |
> > In any case, IMO, mhttpd has no business serving the contents of /root,
> > or serving any files outside of the mhttpd user $HOME directory. (but also
> > should not serve files from ~user/.ssh, or any other "secret" files, good
> > luck making a complete axhuastive list of all secret files that should not be
> > served).
>
> I fully agree with Konstantin. mhttpd should only serve files under certain directories. One is the
> midas/resources directory, another is the one defined in the ODB under /Custom/Path. I plan to modify
> mhttpd to only serve these files (and also prevent tricks like putting "../../../" into the URL). This will then
> also fix Andreas' problem.
>
> Stefan
Yes, I think this is a good idea as well. But just to make sure that you get the problem: it's not the custom page call by itself! If you currently type e.g. <midas-server>/root it will open the ODB html-page. |
1416
|
21 Dec 2018 |
Stefan Ritt | Bug Report | mhttpd - custom page - RHEL/Fedora |
I implemented that fix. Thank you to Andreas. Creating "Custom" directory from the web now does
not have that problem any more.
Stefan |
1417
|
26 Dec 2018 |
Konstantin Olchanski | Bug Report | mhttpd - custom page - RHEL/Fedora |
> > [mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of
> > 4096 returned -1, errno 21 (Is a directory)
>
> On some linux systems, "/root" exists, it is a directory used as the home directory
> of user "root" (~root is /root; traditional UNIX has ~root as /).
>
I just got burned by the same problem on MacOS. mhttpd odb editor cannot open ODB "/System"
because on MacOS there is a subdirectory called "/System".
So the question is why did mhttpd suddenly started serving files from the main URL?
K.O. |
1418
|
26 Dec 2018 |
Konstantin Olchanski | Bug Report | mhttpd - custom page - RHEL/Fedora |
> I implemented that fix. Thank you to Andreas. Creating "Custom" directory from the web now does
> not have that problem any more.
This fix also stops mhttpd from serving the /etc/passwd file.
BTW, "the fix" in mhttpd unconditionally creates /Custom/Path and sets it to the value of $MIDASSYS. This path
seems to be prepended to all file paths, so this fix also breaks the normal use of /Custom/xxx that contain the full
path name of the file to serve...
Looks like file serving in mhttpd got messed up and needs to be reviewed. I still strongly believe that mhttpd should
be serve arbitrary files (only serve files explicitly listed in ODB) or as next best option, only serve files from
subdirectories explicitly listed in ODB.
K.O. |
1424
|
27 Dec 2018 |
Stefan Ritt | Bug Report | mhttpd - custom page - RHEL/Fedora |
> BTW, "the fix" in mhttpd unconditionally creates /Custom/Path and sets it to the value of $MIDASSYS. This path
> seems to be prepended to all file paths, so this fix also breaks the normal use of /Custom/xxx that contain the full
> path name of the file to serve...
I just set the /Custom/Path to $MIDASSYS to have something non-zero there. This is only a default which should be changed to the directory
containing the actual custom pages. If it breaks existing code, just set it manually to an empty string, nothing prevents you from doing that.
> Looks like file serving in mhttpd got messed up and needs to be reviewed. I still strongly believe that mhttpd should
> be serve arbitrary files (only serve files explicitly listed in ODB) or as next best option, only serve files from
> subdirectories explicitly listed in ODB.
I'm thinking along the same lines, but figured out that this cannot be done easily. If people have access to the ODB, the can put the directory
/etc/ into the ODB and again read that way /etc/passwd. We would have to explicitly hard-code some directories to exclude like /etc/ /var/ etc.
but on macOS that might be different. We could put the list of directories into a physical file, which cannot be edited via the web interface.
Stefan |