ID |
Date |
Author |
Topic |
Subject |
2703
|
05 Feb 2024 |
Ben Smith | Bug Fix | string --> int64 conversion in the python interface ? | > The symptoms are consistent with a string --> int64 conversion not happening
> where it is needed.
Thanks for the report Pasha. Indeed I was missing a conversion in one place. Fixed now!
Ben |
2710
|
13 Feb 2024 |
Konstantin Olchanski | Bug Fix | string --> int64 conversion in the python interface ? | > > The symptoms are consistent with a string --> int64 conversion not happening
> > where it is needed.
>
> Thanks for the report Pasha. Indeed I was missing a conversion in one place. Fixed now!
>
Are we running these tests as part of the nightly build on bitbucket? They would be part of
the "make test" target. Correct python dependancies may need to be added to the bitbucket OS
image in bitbucket-pipelines.yml. (This is a PITA to get right).
K.O. |
515
|
23 Oct 2008 |
Konstantin Olchanski | Bug Report | strange output from "odbedit cleanup" | When I run odbedit remotely (odbedit -h ladd09), the "cleanup" command unexpectedly produces the
output of the "sor" command (sure enough, there is a call to db_get_open_records() there), but when I run
it locally, I do not get this output (but db_get_open_records() is still called). Strange. K.O. |
518
|
28 Oct 2008 |
Stefan Ritt | Bug Report | strange output from "odbedit cleanup" | > When I run odbedit remotely (odbedit -h ladd09), the "cleanup" command unexpectedly produces the
> output of the "sor" command (sure enough, there is a call to db_get_open_records() there), but when I run
> it locally, I do not get this output (but db_get_open_records() is still called). Strange. K.O.
The db_get_open_records() call was by mistake there, I removed it. What remains is that the notification
message if a client is removed from the ODB goes through the system messages. When running locally, odbedit
echoes it's own messages, but when running remotely, this is not the case. So the messages can be seen by
everybody else (plus it ends up in the message file), but not by the remote odbedit where the cleanup is
started. The quick fix for that is to say "old" in odbedit which shows the last few lines of the message
file, so one can see any successful cleanup. |
1259
|
13 Apr 2017 |
Andreas Suter | Bug Report | stop form odbedit broken | when I try to stop a run from odbedit I get a core dump.
[ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
midas commit 53af92a5d0...
-----
I checked what happens if I try to stop a run via the mhttpd web-page: this
works! So what is different?
-----
I placed a issue (# 47) on bitbucket as well.
What is the preferred channel to report potential bugs (elog / bitbucket issues)? |
1260
|
13 Apr 2017 |
Andreas Suter | Bug Report | stop form odbedit broken | > when I try to stop a run from odbedit I get a core dump.
>
> [ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
> Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
>
> midas commit 53af92a5d0...
>
> -----
>
> I checked what happens if I try to stop a run via the mhttpd web-page: this
> works! So what is different?
>
> -----
>
> I placed a issue (# 47) on bitbucket as well.
>
> What is the preferred channel to report potential bugs (elog / bitbucket issues)?
I think I found the problem. Some ODB String values which are **automatically**
generated:
CSS File = STRING : [1024] mhttpd.css
Sqlite dir = STRING : [1024]
History dir = STRING : [1024]
Sound = STRING : [1000] alarm.mp3
are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)
It looks as if this screws up quite a bit of the system! When deleting .ODB.SHM and
afterwards try to reload the ODB via a dump I previously made with odbedit, the
following is happening:
1) I get the error message that some strings are too long (exceeding
MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
variables this is.
2) After this reload, essentially nothing is working anymore. Any client I tried to
start just crashed.
Since it seems that the string length of MAX_STRING_LENGTH is very crucial I would
suggest that db_create_record (or whatever routine is dealing with it) checks for
STRING variables and ensures that they cannot exceed MAX_STRING_LENGTH.
When I shortened in my dump the above variables to MAX_STRING_LENGTH, regenerated the
ODB, also the 'stop' Problem in odbedit is gone. |
1265
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | stop form odbedit broken | > > when I try to stop a run from odbedit I get a core dump.
> >
> > [ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
> > Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
> >
I am quite puzzled by this situation. We have seen the above error before, tried to track it down, failed. I was
always thinking this is some kind of strange size mismatch between odb size and shared memory size and
shared memory save file odb.shm size.
Now with your information, it looks like it is memory corruption.
I always thought there is no length limit to odb strings, except for the odb api problem where you have to
know the maximum string length for db_get_value() & co otherwise long strings will be corrupted. Today
nobody uses fixed size buffers, either db_get_value() allocates the string of correct size (replacing buffer
overflow errors with memory leak errors), or return std::string.
I shall check on the use of MAX_STRING_SIZE at least in odb itself...
The default value 256 seems to be too small for today's use. (if you want to store json data, web page
fragments, etc).
K.O.
> > midas commit 53af92a5d0...
> >
> > -----
> >
> > I checked what happens if I try to stop a run via the mhttpd web-page: this
> > works! So what is different?
> >
> > -----
> >
> > I placed a issue (# 47) on bitbucket as well.
> >
> > What is the preferred channel to report potential bugs (elog / bitbucket issues)?
>
> I think I found the problem. Some ODB String values which are **automatically**
> generated:
>
> CSS File = STRING : [1024] mhttpd.css
> Sqlite dir = STRING : [1024]
> History dir = STRING : [1024]
> Sound = STRING : [1000] alarm.mp3
>
> are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)
>
> It looks as if this screws up quite a bit of the system! When deleting .ODB.SHM and
> afterwards try to reload the ODB via a dump I previously made with odbedit, the
> following is happening:
>
> 1) I get the error message that some strings are too long (exceeding
> MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
> variables this is.
>
> 2) After this reload, essentially nothing is working anymore. Any client I tried to
> start just crashed.
>
> Since it seems that the string length of MAX_STRING_LENGTH is very crucial I would
> suggest that db_create_record (or whatever routine is dealing with it) checks for
> STRING variables and ensures that they cannot exceed MAX_STRING_LENGTH.
>
> When I shortened in my dump the above variables to MAX_STRING_LENGTH, regenerated the
> ODB, also the 'stop' Problem in odbedit is gone. |
1270
|
15 Apr 2017 |
Konstantin Olchanski | Bug Report | stop form odbedit broken | > when I try to stop a run from odbedit I get a core dump.
> [ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
> Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
>
I am puzzled. The crash is at the very end of everything (save odb shared memory to odb.shm),
does the run actually stop, or the crash is before the run is fully stopped? (I guess if you want
to run more odbedit commands after stopping the run, so you care about not crashing).
K.O. |
1278
|
24 Apr 2017 |
Stefan Ritt | Bug Report | stop form odbedit broken | > CSS File = STRING : [1024] mhttpd.css
> Sqlite dir = STRING : [1024]
> History dir = STRING : [1024]
> Sound = STRING : [1000] alarm.mp3
After a quick discussion with Konstantin, I changed these strings to a length of 256 chars
(MAX_STRING_LENGTH). Actually all changes I had to made was on code introduced by KO, so I hope I
did everything correctly. He should carefully check my changes (actually I would have preferred if he
could change his code himself...).
I agree with KO that the preferred format for saving the ODB should be JSON, but there might be
experiments with have some old ODB dumps in other formats, so we should not remove the possibility to
read those formats back.
Stefan |
Draft
|
04 Jun 2020 |
Lukas Gerritzen | | stime() deprecated in glibc 2.31 | In glibc 2.31, the stime function was deprecated:
* The obsolete function stime is no longer available to newly linked
binaries, and its declaration has been removed from <time.h>.
Programs that set the system time should use clock_settime instead.
https://sourceware.org/legacy-ml/libc-announce/2020/msg00001.html
This creates a problem in src/system.cxx:3197:4 |
1410
|
22 Nov 2018 |
Konstantin Olchanski | Info | status of self-signed https certificates | I just happened to check the current situation with self-signed https certificates as implemented in mhttpd.
(To remember, the powers-that-be are pushing for universal use of https for all web access. The https
implementation in mhttpd at the moment can only generate self-signed certificates, so...)
plain unencrypted http:
- both google chrome and firefox say "connection not secure", but connect without any fuss.
- apple safari does not say anything
https with self-signed certificate:
- google chrome goes through an "are you sure?" page, "red not secure" status in toolbar
- firefox does the same thing, requires adding a security exception, but still shows "not secure" status in toolbar
- apple safari goes through a sequence of "are you sure?" pages, asks for the user password to add the self-signed certificate to
the macos key store, then marks the connection as "secure" (good)
So clearly powers-that-be do not want us to use self-signed certificates for https. (And frown on use of unencrypted
http even for localhost connections). Properly signed certificates can be obtained from letsencrypt almost
automatically, but of course mhttpd needs to know how to use them and how to do handle their automatic renewals.
I plan to update the mongoose web server library inside mhttpd and with luck I will straighten some of this certificate business at
the same time.
In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache
httpd, etc).
K.O. |
1411
|
30 Nov 2018 |
Stefan Ritt | Info | status of self-signed https certificates | > In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache
> httpd, etc).
I guess this is what moste people do anyhow these days. Do I understand correctly that this then rules out the usage of letsencrype certificates, since the
host needs to be accessed from outside, which is not possible if running behind a password protected firewall.
Stefan |
1412
|
03 Dec 2018 |
Konstantin Olchanski | Info | status of self-signed https certificates | > > In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache
> > httpd, etc).
>
> I guess this is what moste people do anyhow these days. Do I understand correctly that this then rules out the usage of letsencrype certificates, since the
> host needs to be accessed from outside, which is not possible if running behind a password protected firewall.
>
> Stefan
Careful, firewall != proxy, very different things.
A firewall prevents network communications, period. (Like fences and locked doors, there are good reasons to have them).
An https proxy is a way to have encrypted (protected) web communications with a machine behind a firewall.
Basically, we have 4 main cases, all with trouble.
1) mhttpd running on localhost, "just for testing", is in trouble. there is no simple way to get a "blessed" certificate, and self-signed certificates are now "almost forbidden". http is "okey
for now", but the writing is on the wall. There is no special exception for "local-only" connections.
2a) mhttpd running on an internet-connected machine, with apache httpd, our best case. To get this working one has to configure both apache httpd and the "blessed certificate"
certbot tool. With luck, both tools work smoothly on current OSes (they do NOT).
2b) same, but without apache httpd. One still has to run certbot, and the "glue" between mhttpd and certbot is currently missing: need a way to point mhttpd to the certbot certificate
files and a way to reload mhttpd when the certificate is auto-renewed.
3) mhttpd running on a machine behind a corporate firewall. worst case. if firewall Gods make an opening for ports 80 and 443, it becomes case (2a/b), otherwise, one must use some
kind of https proxy. (Plus there is no trivial way to setup an encrypted secure communication channel between mhttpd and this proxy, a double bad).
K.O.
P.S. I guess one can use nginx as the https proxy instead of apache httpd. I did not try yet. My impression is that everybody uses nginx, except for people who started with apache httpd
and are too lazy to try nginx.
K.O. |
1546
|
10 Jun 2019 |
Konstantin Olchanski | Info | status of self-signed https certificates | > > > In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache
> > > httpd, etc).
There we go. google-chrome 74 refuses to connect to mhttpd configured with a self-signed certificate generated per instructions printed by mhttpd.
Here is the full error text (there is no button to "let me connect to it anyway"):
Your connection is not private
Attackers might be trying to steal your information from musr03.triumf.ca (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_AUTHORITY_INVALID
Help improve Safe Browsing by sending some system information and page content to Google. Privacy policy
musr03.triumf.ca normally uses encryption to protect your information. When Google Chrome tried to connect to musr03.triumf.ca this time, the website sent back unusual and incorrect credentials. This may happen when an
attacker is trying to pretend to be musr03.triumf.ca, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Google Chrome stopped the connection before any data was exchanged.
You cannot visit musr03.triumf.ca right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later. |
1769
|
13 Jan 2020 |
Konstantin Olchanski | Info | status of self-signed https certificates | Now firefox returns the same error. version 72.0.1.
> daqlabpc.triumf.ca has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site.
> Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT
I think the problem is with HSTS. I enabled HSTS (in mhttpd and in apache httpd) because
SSLlabs encourage it and because my reading of it's description at
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security
makes it sound like a good idea without any downsides.
However, the actual HSTS RFC says something completely different:
https://tools.ietf.org/html/rfc6797
"The aim is to prevent click-through insecurity and address other potential threats".
To me this explains what I see. In contrast to the description at developer.mozilla.org,
firefox (and google chrome) disable "click-through" exceptions for "I do not like this https certificate",
and there is no way to connect to self-signed https.
Bottom line, either use certbot to get blessed https certificate or no https for you.
K.O.
> > > > In the mean time, we continue to recommend that mhttpd should be used behind a password protected https proxy (i.e. apache
> > > > httpd, etc).
>
> There we go. google-chrome 74 refuses to connect to mhttpd configured with a self-signed certificate generated per instructions printed by mhttpd.
>
> Here is the full error text (there is no button to "let me connect to it anyway"):
>
> Your connection is not private
> Attackers might be trying to steal your information from musr03.triumf.ca (for example, passwords, messages, or credit cards). Learn more
> NET::ERR_CERT_AUTHORITY_INVALID
>
> Help improve Safe Browsing by sending some system information and page content to Google. Privacy policy
> musr03.triumf.ca normally uses encryption to protect your information. When Google Chrome tried to connect to musr03.triumf.ca this time, the website sent back unusual and incorrect credentials. This may happen when an
> attacker is trying to pretend to be musr03.triumf.ca, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Google Chrome stopped the connection before any data was exchanged.
>
> You cannot visit musr03.triumf.ca right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later. |
841
|
12 Dec 2012 |
Shaun Mead | Bug Report | ss_thread_kill() kills entire program | Hi, I'm having some trouble getting ss_thread_kill() to work properly. It seems
to kill the entire program instead
of just the thread. Here is a test program to show the error:
_________________________________
#include <stdio.h>
#include <stdlib.h>
#include "midas.h"
#include "msystem.h"
INT f(void *param)
{
for (int x = 0; x < 100; x++)
sleep(1);
return 0;
}
int main()
{
printf("creating thread\n");
midas_thread_t thr = ss_thread_create(f, NULL);
sleep(2);
printf("killing thread\n");
ss_thread_kill(thr);
printf("success\n");
return 0;
}
_________________________________
Makefile:
_________________________________
FLAGS=-g -Wall -DLINUX -DOS_LINUX -I/home/deap/packages/midas/include
LIBS=-L/home/deap/packages/midas/linux-m64/lib -lmidas -lpthread -lrt -lutil
main.exe: main.cpp
g++ $(FLAGS) -o $@ $^ $(LIBS)
_________________________________
Output when run:
_________________________________
[deap@deap04 multithread]$ ./main.exe
creating thread
killing thread
Killed
[deap@deap04 multithread]$
_________________________________
The last "Killed" indicated the whole program got killed, when it should
actually just kill the thread and then
print "success".
I noticed the function in system.c uses pthread_kill(). Some google searches
show me that it may be better to use
pthread_cancel() (ie http://stackoverflow.com/questions/3438536/when-to-use-
pthread-cancel-and-not-pthread-kill ).
Shaun |
842
|
13 Dec 2012 |
Stefan Ritt | Bug Report | ss_thread_kill() kills entire program | The Linux thread functionality was introduced by Konstantin, so he might have a better idea about that.
What I usually do is a graceful thread shutdown just by a flag. Like
int stop_thread = 0;
INT f(void *param)
{
for (int x = 0; x < 100; x++) {
sleep(1);
if (stop_thread) {
// clean up things here...
return 0;
}
}
return 0;
}
int main()
{
printf("creating thread\n");
midas_thread_t thr = ss_thread_create(f, NULL);
sleep(2);
printf("killing thread\n");
stop_thread = 1;
sleep(2);
printf("success\n");
return 0;
}
This way I have a chance to clean up things in the thread, which otherwise I would not be able to. |
843
|
13 Dec 2012 |
Konstantin Olchanski | Bug Report | ss_thread_kill() kills entire program | > Hi, I'm having some trouble getting ss_thread_kill() to work properly. It seems
> to kill the entire program instead of just the thread.
You cannot kill a thread. It's not a well defined operation. Most OSes do have the
technical possibility to kill threads, but if you use them, you will not like the
results. For a taste of small trouble, if a thread is holding a lock and you kill
it, who's job is it to release the lock?
The best you can do is to ask the thread to gracefully shutdown itself. (I.e. by
using global variable flags).
P.S. I did not implement the ss_thread stuff, I do not know what ss_thread_kill()
does, but I recommend that you do not use it.
P.P.S. Programming using threads is complicated, I recommend that you read at least
some literature on the topic before using threads. At the least you must understand
the common pitfalls and mistakes. At the least, you must know about deadlocks,
livelocks, race conditions and semaphore priority inversions.
K.O. |
844
|
13 Dec 2012 |
Shaun Mead | Bug Report | ss_thread_kill() kills entire program | > > Hi, I'm having some trouble getting ss_thread_kill() to work properly. It seems
> > to kill the entire program instead of just the thread.
>
> You cannot kill a thread. It's not a well defined operation. Most OSes do have the
> technical possibility to kill threads, but if you use them, you will not like the
> results. For a taste of small trouble, if a thread is holding a lock and you kill
> it, who's job is it to release the lock?
>
> The best you can do is to ask the thread to gracefully shutdown itself. (I.e. by
> using global variable flags).
>
> P.S. I did not implement the ss_thread stuff, I do not know what ss_thread_kill()
> does, but I recommend that you do not use it.
>
> P.P.S. Programming using threads is complicated, I recommend that you read at least
> some literature on the topic before using threads. At the least you must understand
> the common pitfalls and mistakes. At the least, you must know about deadlocks,
> livelocks, race conditions and semaphore priority inversions.
>
> K.O.
Yes, but unfortunately what I was attempting to do was use a library function that I
can't alter. It sometimes gets stuck and I wanted a way to kill it. Anyway I ended up
not doing this at all in c++; I was able to do what I needed in python.
Shaun |
2267
|
31 Jul 2021 |
Peter Kunz | Bug Report | ss_shm_name: unsupported shared memory type, bye! | I ran into a problem trying to compile the latest MIDAS version on a Fedora
system.
mhttpd and odbedit return:
ss_shm_name: unsupported shared memory type, bye!
check_shm_type: preferred POSIXv4_SHM got SYSV_SHM
The check returns SYSV_SHM which doesn't seem to be supported in ss_shm_name.
Is there an easy solution for this?
Thanks. |
|