ID |
Date |
Author |
Topic |
Subject |
1495
|
14 Mar 2019 |
Konstantin Olchanski | Info | Gyrations of custom pages and ODB /Custom/Path | I now understand Stefan's and Thomas's proposal a little bit better.
In my mind only one issue remains - when we say "we will serve files from directory X", how
to we prevent mhttpd from serving files outside this directory by using trick URLs containing ".."
and/or other gimmicks.
The code I currently put in mhttpd, disallows multi-level URL path names (by rejecting names that contain the directory separator "/").
This has the effect of keeping the mhttpd URL space flat (without subdirectories).
http://localhost:8080/Status.html <--- no multi level URLs like we used to have:
http://localhost:8080/CS/Custom.html <--- no more of these
Keeping the URL space restricted to one level is important if we do not want to defeat
the recent change to the mhttpd URL scheme - if mhttpd runs behind a proxy, without
a "Base URL" (which we just removed), we can only use relative URLs to navigate
between midas pages and if we permit multi-level URLs, it becomes hard to get
back to the status page without the ugly counting of ".." URL elements (which ugly and
brittle code we also just recently removed) (n.b. to navigate from CS/Custom.html
to the status page, one must redirect to "../Status.html").
But this whole beautiful cathedral falls apart from one valid use case: we want to serve "jsroot"
from a subdirectory called "jsroot" - this is how this package is packaged and we do not want
to mess with it just to make midas happy.
So at the least we must enable serving of multi-level URL path names to serve 3rd party packages.
The most trivial way out is to replace the URL check "/ is not permitted" with ".. is not permitted".
(One could also have a list of all permitted subdirectories in ODB, but this would be hard to use and
difficult to implement. Not my favourite solution.)
This will break the flatness of the mhttpd URLs (no subdirectories). But maybe it is sufficient
to write down "do not do this!" and close with "wontfix" all bug reports about "my custom page
is at http://localhost:8080/mycustomdir/new/verynew/custom.html, how come the [status] button
does not take me back to the status page?".
K.O.
> Parsing all URL in mhttpd to prevent /etc/passwd etc. to be returned is tricky, because people can use escape sequences etc. Therefore I think it is much better to restrict file access
> on the file system level when opening a file. The only escape there one could have is "..", which can be tested easily.
>
> Therefore, I propose to restrict file access to two well-defined directories, which is one system directory and one user directory. The system directory should be defined via
> $MIDASSYS/resources, and the user directory should be the experiment directory (as defined in exptab) followed by "resources". So if MIDASSYS equals to /usr/local/midas and the
> experiment directory equals to /home/users/exp for example, we would only have these two directories (and of course the subdirectories within these) served by mhttpd:
>
> $MIDASSYS/resource -> /usr/local/midas/resources
> <exptab>/resources -> /home/users/exp/resources
>
> These directories should be hard-wired into mhttpd, and not go through and ODB entry, since otherwise one could manipulate the ODB entries (knowingly or unknowingly) and open a
> back-door.
>
> If users need a more complex structure, they can put soft links into these directories.
>
> The code which opens a resource file should then first evaluate $MIDASSYS, then add "/resources/", then add the requested file name, make sure that there is no ".." in the file name,
> then open the file. If not existing, do the same for the <exptab>/resources/ directory.
>
> This change will break most experiments, and forces people to move their custom pages to different directories, but I think it's the only clean solution and we just have to bite the
> bullet.
>
> Comments are welcome.
>
> Stefan |
1496
|
14 Mar 2019 |
Konstantin Olchanski | Info | Gyrations of custom pages and ODB /Custom/Path | > In my mind only one issue remains - when we say "we will serve files from directory X", how
> to we prevent mhttpd from serving files outside this directory by using trick URLs containing ".."
> and/or other gimmicks.
>
> Disallow multi-level URL path names (by rejecting names that contain the directory separator "/").
> Replace the URL check "/ is not permitted" with ".. is not permitted".
>
https://en.wikipedia.org/wiki/Directory_traversal_attack
K.O. |
1497
|
14 Mar 2019 |
Konstantin Olchanski | Info | Gyrations of custom pages and ODB /Custom/Path | > > In my mind only one issue remains - when we say "we will serve files from directory X", how
> > to we prevent mhttpd from serving files outside this directory by using trick URLs containing ".."
> > and/or other gimmicks.
> >
> > Disallow multi-level URL path names (by rejecting names that contain the directory separator "/").
> > Replace the URL check "/ is not permitted" with ".. is not permitted".
>
> https://en.wikipedia.org/wiki/Directory_traversal_attack
and, from Mitre's "Common Weakness Enumeration", with examples:
http://cwe.mitre.org/data/definitions/22.html
K.O. |
1503
|
21 Mar 2019 |
Konstantin Olchanski | Info | Gyrations of custom pages and ODB /Custom/Path | > In my mind only one issue remains - when we say "we will serve files from directory X", how
> to we prevent mhttpd from serving files outside this directory by using trick URLs containing ".."
> and/or other gimmicks.
>
> So at the least we must enable serving of multi-level URL path names to serve 3rd party packages.
>
> The most trivial way out is to replace the URL check "/ is not permitted" with ".. is not permitted".
>
This change is "in". commit https://bitbucket.org/tmidas/midas/commits/b231d10b5816c14428a69ee97b16f6fee7819367
mhttpd should be able to serve "jsroot" and other 3rd packages now.
K.O. |
1504
|
21 Mar 2019 |
Konstantin Olchanski | Info | Gyrations of custom pages and ODB /Custom/Path | > Before the days of javascript and ajax and web 2.0, MIDAS introduced "custom pages" for
> building graphical display that could show "live" data from MIDAS and that could
> have buttons and controls to operate slow controls equipment, etc.
As summary of latest gyrations, this is how mhttpd can be used to serve custom pages:
a) old custom pages path:
?cmd=custom&page=XXX serves filename contained in ODB /Custom/XXX if it exists. Value of ODB /Custom/Path is prepended to the filename unless it already starts with a "/"
b) alternate custom pages path:
if ODB /Custom/URL or /Custom/URL& or /Custom/URL! exist, serves filename contained in corresponding ODB entry. Again value of ODB /Custom/Path is prepended to the filename
unless it already starts with a "/".
In both cases, ".." is not permitted in the custom page name to avoid ODB path traversal attack (escape from /Custom subdirectory by using custom page names like "../System/blah").
c) new custom page path:
if ODB /Custom/Path exists and is not empty, it is prepended to the URL and this forms the filename (ODB[/Custom/Path] + "/" + URL). If this file exists, it is served. To prevent directory
traversal attacks, ".." is not permitted in the URL.
d) resource search path:
file given by the URL is searched in the resource search path (see "resource paths" on the mhttpd help page, typically $MIDASSYS/resources, etc), e.g.
http://localhost:8080/status.html -> serves $MIDASSYS/resources/status.html.
this is the normal way to serve all standard midas web pages.
to (a) prevent directory traversal attack and (b) enforce flat namespace (no URL subdirectories), send_resource() disallows "/" (and "\" on Windows) anywhere in the filename.
Notes:
1) path traversal attacks are detailed here, MIDAS is subject to both filesystem and ODB path traversal attacks.
http://cwe.mitre.org/data/definitions/22.html
2) methods (c) and (d) are duplicative. In the next rework of mhttpd (update of mongoose library, update of multithreading,
update of web server configuration in ODB, etc), we will probably change serving of custom files along the lines
proposed by Stefan and Thomas.
3) the "old custom pages" code will most likely remain as is: it works with the new url scheme, it does not suffer from path traversal attacks and it is still used by some experiments.
K.O. |
435
|
18 Feb 2008 |
Exaos Lee | Bug Report | Great! But I failed to run it. :( | I encountered the error message as the following:
Traceback (most recent call last):
File "runtest.py", line 42, in <module>
import midas
File "/opt/MIDAS.PSI/Resources/PyMIDAS/pymidas/midas/__init__.py", line 140, in
<module>
cmidas = ctypes.cdll.LoadLibrary('libmidas.so')
File "/usr/lib/python2.5/site-packages/PIL/__init__.py", line 431, in LoadLibrary
File "/usr/lib/python2.5/site-packages/PIL/__init__.py", line 348, in __init__
OSError: /opt/MIDAS.PSI/Versions/Current/lib/libmidas.so: undefined symbol:
cam16i_rq
Compiling the MIDAS library using NEED_SHLIB=1 causes the same "undefined
reference" error. But it can be fixed by adding "-shared" to CFLAGS in the
Makefile. Though the libmidas.so can be successfully created, the above error is
still there. Can anybody help me?
Environment:
Platform: Ubuntu Linux 7.10 with gcc 4.1
MIDAS version: 2.0.0, svn-4106
Python version: 2.5.1 |
1574
|
27 Jun 2019 |
Hassan | Bug Report | Getting an error when trying to compile a frontend file | When we run the following commands on the hostname(DAQ machine) and the remote
frontend(Rpi):
cd $HOME/online
cp $MIDASSYS/examples/experiment/* .
make
We get errors such as
=================
On Rpi:
pi@raspberrypi:~/online/fe_test $ make
...
Missing definition of environment variable 'ROOTSYS' !
=================
On host machine
inking CXX executable frontend
/usr/bin/ld: cannot find -lmfe
/usr/bin/ld: cannot find -lmidas
collect2: error: ld returned 1 exit status
make[2]: *** [frontend] Error 1
make[1]: *** [CMakeFiles/frontend.dir/all] Error 2
make: *** [all] Error 2
The Rpi(32bit) doesn't have root installed but the host machine(64bit) does.
What can we do to fix this?
Thank you this forum has been of great help. |
1576
|
27 Jun 2019 |
Konstantin Olchanski | Bug Report | Getting an error when trying to compile a frontend file | If the latest midas does not work, try the previous release versions. "git tags" and "git branch -
a" will show you what exists. Look for branch and tag names in the form "midas-YYYY-MM".
As shortcut, the latest release candidate is midas-2019-06, the latest release branch is midas-
2019-03, latest release tag midas-2019-03-h.
Read the messages in this thread for more information:
https://midas.triumf.ca/elog/Midas/1513
>
> When we run the following commands ...
> make[1]: *** [CMakeFiles/frontend.dir/all] Error 2
>
I do not understand cmake well enough to debug this. Falling back to midas-2019-03 may help
you as it uses normal make and with luck you know how to debug normal Makefiles if you see
the same problem.
K.O. |
1579
|
27 Jun 2019 |
Stefan Ritt | Bug Report | Getting an error when trying to compile a frontend file | Note that the example experiment compiles a simple example frontend and a root-based analyzer. If you don't have
ROOT installed, you of course cannot compile the analyzer. If you don't need the analyzer, remove it from the
Makefile/CMakeLists.txt
It's not clear to me why the frontend did not compile on our server machine. You did not post the command how you
initiated the build. Note that there are now two parallel build schemes: the traditional Makefile and the new
CMakeFiles.txt. We try to maintain both of them, so you have to specify which one you use when you get an error.
I realize now that the CMakeLists.txt in the experiment example directory builds nicely under midas, but when you move
it to another directory and extract it from the normal build scheme it breaks. I rewrote the CMakeLists.txt now that it
looks for MIDASSYS and also build at different locations. Do
cd $HOME/online
cp $MIDASSYS/examples/experiment/* .
mkdir build
cd build
cmake ..
make
and it should work. Of course first pull the current develop version.
Stefan |
2874
|
11 Oct 2024 |
Denis Calvet | Bug Report | Frontend name must differ from others by more than the last three characters | Hi,
I have developed two Midas front-end programs for different hardware. The frontend_name of the first one is "FSCD_SC" (slow control) and that of the second one is "FSCD_PS" (power supply).
Each front-end program runs fine separately, but when attempting to start FSCD_SC while FSCD_PS is running, FSCD_PS is terminated and Midas indicates "Previous frontend stopped" in the window where it starts FSCD_SC.
The problem is that these two frontend names only differ in their last two characters, and Midas currently does not distinguish them properly.
Looking in mfe.cxx we have:
int main(int argc, char *argv[])
{
...
/* shutdown previous frontend */
status = cm_shutdown(full_frontend_name, FALSE);
...
And looking in midas.cxx we have:
INT cm_shutdown(const char *name, BOOL bUnique) {
...
if (!bUnique)
client_name[strlen(name)] = 0; /* strip number */
...
The above line removes the last 3 characters of the front-end name before the subsequent comparison with other frontend names. Stripping the last 3 characters of the front-end name is correct for frontend programs that use the "-i" command line option to specify an index for that frontend, but all the characters of the front-end name should otherwise be kept for comparison.
I have changed the names of my frontend programs to avoid the interference, but it would be nice that the code that determines if an instance of a frontend program is already running is corrected.
I hope this can help.
Best regards,
Denis. |
2875
|
11 Oct 2024 |
Stefan Ritt | Bug Report | Frontend name must differ from others by more than the last three characters | Hi Denis,
indeed a bug. Will fix it next week.
Best,
Stefan
> Hi,
> I have developed two Midas front-end programs for different hardware. The frontend_name of the first one is "FSCD_SC" (slow control) and that of the second one is "FSCD_PS" (power supply).
>
> Each front-end program runs fine separately, but when attempting to start FSCD_SC while FSCD_PS is running, FSCD_PS is terminated and Midas indicates "Previous frontend stopped" in the window where it starts FSCD_SC.
>
> The problem is that these two frontend names only differ in their last two characters, and Midas currently does not distinguish them properly. |
2878
|
18 Oct 2024 |
Stefan Ritt | Bug Report | Frontend name must differ from others by more than the last three characters | Fixed and committed.
Best,
Stefan
> Hi Denis,
>
> indeed a bug. Will fix it next week.
>
> Best,
> Stefan
>
>
> > Hi,
> > I have developed two Midas front-end programs for different hardware. The frontend_name of the first one is "FSCD_SC" (slow control) and that of the second one is "FSCD_PS" (power supply).
> >
> > Each front-end program runs fine separately, but when attempting to start FSCD_SC while FSCD_PS is running, FSCD_PS is terminated and Midas indicates "Previous frontend stopped" in the window where it starts FSCD_SC.
> >
> > The problem is that these two frontend names only differ in their last two characters, and Midas currently does not distinguish them properly. |
1597
|
08 Jul 2019 |
Vinzenz Bildstein | Bug Report | Frontend killed at stop of run | I wrote a c++ frontend to read data from CAEN VX1730 digitizers which is used in
parallel with the GRIFFIN frontend to read out DESCANT.
After a long overnight run to check that the frontend runs smoothly for a longer
time, I stopped the run and the frontend was killed by midas. I am not sure why
this happened, as the end_of_run function returned successfully (at least the
print statement right before "return SUCCESS;" appeared right away). So
something else must have timed-out and caused it to be killed, I guess?
Any suggestions on where to look to find out what causes this?
Thanks in advance for your help! |
1598
|
08 Jul 2019 |
Konstantin Olchanski | Bug Report | Frontend killed at stop of run | > After a long overnight run to check that the frontend runs smoothly for a longer
> time, I stopped the run and the frontend was killed by midas.
run the frontend inside gdb and post the stack trace after the crash?
if there is no crash (the program is stopped by exit()), you may need
to set a breakpoint in exit() or _exit() (not sure what it's latest name is)
then with luck your stack trace will show who/what called it from where.
if it is hard to start the frontend inside gdb, you can start it normally,
and attach gdb later, using a "gdb frontend.exe pid" command.
K.O.
> I am not sure why
> this happened, as the end_of_run function returned successfully (at least the
> print statement right before "return SUCCESS;" appeared right away). So
> something else must have timed-out and caused it to be killed, I guess?
>
> Any suggestions on where to look to find out what causes this?
>
> Thanks in advance for your help! |
1601
|
08 Jul 2019 |
Vinzenz Bildstein | Bug Report | Frontend killed at stop of run | > run the frontend inside gdb and post the stack trace after the crash?
>
> if there is no crash (the program is stopped by exit()), you may need
> to set a breakpoint in exit() or _exit() (not sure what it's latest name is)
> then with luck your stack trace will show who/what called it from where.
>
If I remember correctly from the last time I tried that, it doesn't use the exit
function but gdb just reports that the program was terminated and no longer exists. I
can't set a breakpoint on SIGKILL as the point of SIGKILL is to kill the program and
gdb can't set a break at that point afaik. |
1602
|
08 Jul 2019 |
Konstantin Olchanski | Bug Report | Frontend killed at stop of run | > > run the frontend inside gdb and post the stack trace after the crash?
> >
> > if there is no crash (the program is stopped by exit()), you may need
> > to set a breakpoint in exit() or _exit() (not sure what it's latest name is)
> > then with luck your stack trace will show who/what called it from where.
> >
>
> If I remember correctly from the last time I tried that, it doesn't use the exit
> function but gdb just reports that the program was terminated and no longer exists. I
> can't set a breakpoint on SIGKILL as the point of SIGKILL is to kill the program and
> gdb can't set a break at that point afaik.
For SIGKILL, my gdb reports "Program terminated with signal SIGKILL, Killed." and there is no stack
trace. Is this what you see?
If your program stops "normally", not from receiving some signal, set breakpoints on "exit" and
"_exit".
The normal stop sequence is to call exit(), which runs all the atexit() functions (the midas atexit()
function prints the message about "cm_disconnect_experiment not called at end of program") and
calls _exit() to stop the program.
So if you see the midas message "cm_disconnect_experiment not called at end of program", it is a
good indication that somebody (not mfe.c) called exit() on you. A breakpoint on "exit" should catch
who does it.
Good luck,
K.O. |
1603
|
08 Jul 2019 |
Vinzenz Bildstein | Bug Report | Frontend killed at stop of run | > > > run the frontend inside gdb and post the stack trace after the crash?
> > >
> > > if there is no crash (the program is stopped by exit()), you may need
> > > to set a breakpoint in exit() or _exit() (not sure what it's latest name is)
> > > then with luck your stack trace will show who/what called it from where.
> > >
> >
> > If I remember correctly from the last time I tried that, it doesn't use the exit
> > function but gdb just reports that the program was terminated and no longer exists. I
> > can't set a breakpoint on SIGKILL as the point of SIGKILL is to kill the program and
> > gdb can't set a break at that point afaik.
>
> For SIGKILL, my gdb reports "Program terminated with signal SIGKILL, Killed." and there is no stack
> trace. Is this what you see?
Yes, that is exactly what I remember seeing.
>
> If your program stops "normally", not from receiving some signal, set breakpoints on "exit" and
> "_exit".
>
> The normal stop sequence is to call exit(), which runs all the atexit() functions (the midas atexit()
> function prints the message about "cm_disconnect_experiment not called at end of program") and
> calls _exit() to stop the program.
>
> So if you see the midas message "cm_disconnect_experiment not called at end of program", it is a
> good indication that somebody (not mfe.c) called exit() on you. A breakpoint on "exit" should catch
> who does it.
>
> Good luck,
> K.O.
So far I haven't seen the issue with the message "cm_disconnect_experiment not called at end of program"
again. Now I just have to restart the frontend after the run has (failed?) to stop. After restarting the
frontend everything seems to work again.
I haven't been writing data while doing these tests, so I can't say if there is any data missing or if the
runs were actually stopped properly (with a second dump of the ODB at the end). |
1604
|
08 Jul 2019 |
Konstantin Olchanski | Bug Report | Frontend killed at stop of run | > >
> > For SIGKILL, my gdb reports "Program terminated with signal SIGKILL, Killed." and there is no stack
> > trace. Is this what you see?
>
> Yes, that is exactly what I remember seeing.
>
Where would a SIGKILL come from?!?
Look in the syslog (/var/log/messages). If the program was killed by the linux kernel, it would be logged there,
the usual cause is the machine runs out of memory and programs are killed by the OOM killer, this is logged
into the syslog, always.
MIDAS also can issue a SIGKILL sometimes, again this is always logged in midas.log. see src/midas.c, search for SIGKILL to see
the exact messages printed before it is sent out.
K.O. |
1605
|
08 Jul 2019 |
Vinzenz Bildstein | Bug Report | Frontend killed at stop of run | > > >
> > > For SIGKILL, my gdb reports "Program terminated with signal SIGKILL, Killed." and there is no stack
> > > trace. Is this what you see?
> >
> > Yes, that is exactly what I remember seeing.
> >
>
> Where would a SIGKILL come from?!?
>
> Look in the syslog (/var/log/messages). If the program was killed by the linux kernel, it would be logged there,
> the usual cause is the machine runs out of memory and programs are killed by the OOM killer, this is logged
> into the syslog, always.
>
> MIDAS also can issue a SIGKILL sometimes, again this is always logged in midas.log. see src/midas.c, search for SIGKILL to see
> the exact messages printed before it is sent out.
>
> K.O.
I haven't been able to reproduce the error from the overnight run so far. I will try and leave this running in gdb overnight to see
if I can get that error again. |
1608
|
10 Jul 2019 |
Vinzenz Bildstein | Bug Report | Frontend killed at stop of run | > > > >
> > > > For SIGKILL, my gdb reports "Program terminated with signal SIGKILL, Killed." and there is no stack
> > > > trace. Is this what you see?
> > >
> > > Yes, that is exactly what I remember seeing.
> > >
> >
> > Where would a SIGKILL come from?!?
> >
> > Look in the syslog (/var/log/messages). If the program was killed by the linux kernel, it would be logged there,
> > the usual cause is the machine runs out of memory and programs are killed by the OOM killer, this is logged
> > into the syslog, always.
> >
> > MIDAS also can issue a SIGKILL sometimes, again this is always logged in midas.log. see src/midas.c, search for SIGKILL to see
> > the exact messages printed before it is sent out.
> >
> > K.O.
>
> I haven't been able to reproduce the error from the overnight run so far. I will try and leave this running in gdb overnight to see
> if I can get that error again.
I was able to reproduce the error after an overnight run. gdb reported that the program received a SIGKILL, but no sign of it in
/var/log/messages. I've tried finding a current midas.log file, but it seems we don't have one? The most recent one was last updated
on May 24th this year. |
|