Thank you for the help. As it turns out, the problem was due to the fact that we were compiling MIDAS on our 64 bit backend machine, but one of the frontend machines is 32 bit. The problem was resolved by compiling a 32 bit version of MIDAS in
addition to the 64 bit version.

I am looking at the mlogger in the ALPHA anti-hydrogen experiment at CERN. It is 
mysteriously misbehaving during run start and stop.

The problem turns out to be with the select() system call.

The corresponding FD_SET(), FD_ISSET() & co operate on a an array of fixed size 
FD_SETSIZE, value 1024, in my case. But the socket number is 1409, so we overrun 
the FD_SET() array. Ouch.

I see that all uses of select() in midas have no protection against this.

(we should probably move away from select() to newer poll() or whatever it is)

Why does mlogger open so many file descriptors? The usual, scaling problems in the 
history. The old midas history does not reuse file descriptors, so opens the same 
3 history files (.hst, .idx, etc) for each history event. The new FILE history 
opens just one file per history event. But if the number of events is bigger than 
1024, we run into same trouble.

(BTW, the system limit on file descriptors is 4096 on the affected machine, 1024 
on some other machines, see "limit" or "ulimit -a").

K.O.

> I am looking at the mlogger in the ALPHA anti-hydrogen experiment at CERN. It is 
> mysteriously misbehaving during run start and stop.
> 
> The problem turns out to be with the select() system call.
> 
> The corresponding FD_SET(), FD_ISSET() & co operate on a an array of fixed size 
> FD_SETSIZE, value 1024, in my case. But the socket number is 1409, so we overrun 
> the FD_SET() array. Ouch.
> 
> I see that all uses of select() in midas have no protection against this.
> 
> (we should probably move away from select() to newer poll() or whatever it is)
> 
> Why does mlogger open so many file descriptors? The usual, scaling problems in the 
> history. The old midas history does not reuse file descriptors, so opens the same 
> 3 history files (.hst, .idx, etc) for each history event. The new FILE history 
> opens just one file per history event. But if the number of events is bigger than 
> 1024, we run into same trouble.
> 
> (BTW, the system limit on file descriptors is 4096 on the affected machine, 1024 
> on some other machines, see "limit" or "ulimit -a").
> 
> K.O.

I cannot imagine that you have more than 1024 different events in ALPHA. That wouldn't 
fit on your status page. 

I have some other suspicion: The logger opens a history file on access, then closes it 
again after writing to it. In the old days we had a case where we had a return from the 
write function BEFORE the file has been closed. This is kind of a memory leak, but with 
file descriptors. After some time of course you run out of file descriptors and crash. 
Now that bug has been fixed many years ago, but it sounds to me like there is another 
"fd leak" somewhere. You should add some debugging in the history code to print the 
file descriptors when you open a file and when you leave that routine. The leak could 
however also be somewhere else, like writing to the message file, ODB dump, ...

The right thing of course would be to rewrite everything with std::ofstream which 
closes automatically the file when the object gets out of scope.

Stefan

Hello,

When I  connect to analyzer on a x86_64 processor(with Roody),  
a analyzer break with segmentation violation in the root_server_thread  function.
Same code are working fine on a 32bit processor.
As I found the problem are in exchanging of pointers between analyzer and client.
Before to send a pointer, it is saved a pointer in int (size=4, instead of 8) at
this place:
Index: src/mana.c
===================================================================
--- src/mana.c  (revision 3498)
+++ src/mana.c  (working copy)
@@ -5386,7 +5386,7 @@

             //write pointer
             message->Reset(kMESS_ANY);
-            int p = (POINTER_T) obj;
+            POINTER_T p = (POINTER_T) obj;
             *message << p;
             sock->Send(*message);


Sincerely Yours,
Fedor Ignatov 

> Hello,
> 
> When I  connect to analyzer on a x86_64 processor(with Roody),  
> a analyzer break with segmentation violation in the root_server_thread  function.
> Same code are working fine on a 32bit processor.
> As I found the problem are in exchanging of pointers between analyzer and client.
> Before to send a pointer, it is saved a pointer in int (size=4, instead of 8) at
> this place:
> Index: src/mana.c
> ===================================================================
> --- src/mana.c  (revision 3498)
> +++ src/mana.c  (working copy)
> @@ -5386,7 +5386,7 @@
> 
>              //write pointer
>              message->Reset(kMESS_ANY);
> -            int p = (POINTER_T) obj;
> +            POINTER_T p = (POINTER_T) obj;
>              *message << p;
>              sock->Send(*message);
> 
> 
> Sincerely Yours,
> Fedor Ignatov 

Do I understand you right? With your patch it works even on 64 bit, right? Or do you
mean there is still a segmentation violation? Anyhow I committed your patch since the
"int" is clearly incorrect.

- Stefan

Yes right, Problem of a segmentation violation is solved with this patch. Now it works
fine on x86_64.

Fedor 

> Do I understand you right? With your patch it works even on 64 bit, right? Or do you
> mean there is still a segmentation violation? Anyhow I committed your patch since the
> "int" is clearly incorrect.
> 
> - Stefan

> Yes right, Problem of a segmentation violation is solved with this patch. Now it works
> fine on x86_64.

Right. I confirm this. I have this exact same fix in my stand-alone copy of the midas
histogram server, and should commit it to MIDAS CVS as well.

K.O.

Another small user experience request:

When making a link in the odb (web interface) a nice browser window pop's up. There is however not scrolling possible in the window. As a result, you can not reach a odb key if it is nested to deeply. 

Trying to type out the Link target in the field only allows for 32 characters

context: we are setting up a bunch of Links in the History

> When making a link in the odb (web interface) a nice browser window pop's up. There is however not scrolling possible in the window. As a result, you can not reach a odb key if it is nested to deeply. 
> 
> Trying to type out the Link target in the field only allows for 32 characters

Thanks for reporting the bug with the pop-up not being able to scroll, I fixed that and committed the change.

I do however not understand the issue with 32 characters. The link NAME should not be more than 32 chars (which applies to all ODB keys). 
But if I try I can write more than 32 chars in the link target.

Stefan

Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
script via Midas - which silently truncates command strings longer than 
256 characters.

I'd prefer that Midas place no limit on string length.  Failing that, it would be
helpful to have character limits called out in the documentation 
(https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).

As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
(Although key *names* are restricted to 256 characters.)  I've submitted one 
possible version of an arbitrary-length exec_script() as a pull request 
(https://bitbucket.org/tmidas/midas/pull-requests/).

Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
strings in the ODB as limited to 256 characters?

Thank you for reporting this problem:

a) ODB key *names* are restricted to 31 characters (32 bytes, last byte is a NUL), not 256 characters.
b) ODB string length is unlimited (32-bit length field)
c) ODB C API "db_get_value" & co require fixed length buffer and most users of this API provide a 256-byte fixed buffer for strings, some of them also do not 
check the status code, resulting in silent truncation. (I think the ODB functions themselves report truncation to midas.log, so not completely silent).

We try to fix this where we must - but it is cumbersome with the current ODB API - as in your fix on has to:
- get the ODB key, extract size
- allocate buffer
- call db_get_value() & co
- use the data
- remember to free the buffer on each and every return path

The first three steps could become one if we had an ODB "get_data" function that automatically allocated the data buffer.

But the main source of bugs will be the last step - remember to free the buffer, always.

P.S.

We are not alone in pondering how to do this best. If you want to see it "done right",
read the fresh-off-the-presses book "Go Programming Language" by Alan Donovan and Brian Kernighan,
http://www.gopl.io/

Brian Kernighan is the "K" in K&R "C programming language", still around and kicking, now at Google.
Sadly the "R" passed away in 2011 - http://www.nytimes.com/2011/10/14/technology/dennis-ritchie-programming-trailblazer-dies-at-70.html

K.O.

> Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
> script via Midas - which silently truncates command strings longer than 
> 256 characters.
> 
> I'd prefer that Midas place no limit on string length.  Failing that, it would be
> helpful to have character limits called out in the documentation 
> (https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
> https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).
> 
> As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
> (Although key *names* are restricted to 256 characters.)  I've submitted one 
> possible version of an arbitrary-length exec_script() as a pull request 
> (https://bitbucket.org/tmidas/midas/pull-requests/).
> 
> Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
> strings in the ODB as limited to 256 characters?

Using low-level memory allocation routines in higher-level programs like mhttpd makes me nervous.

We could use vector arrays to allow variable-sized allocation, and use the data() member function to access the char* needed for functions like strlcat,
db_get_data, and db_sprintf.

This conforms to the c++ standard, but doesn't require explicit freeing by the user - at least, not when you're allocating std::vector<char>.

Amy

> Thank you for reporting this problem:
> 
> a) ODB key *names* are restricted to 31 characters (32 bytes, last byte is a NUL), not 256 characters.
> b) ODB string length is unlimited (32-bit length field)
> c) ODB C API "db_get_value" & co require fixed length buffer and most users of this API provide a 256-byte fixed buffer for strings, some of them also do not 
> check the status code, resulting in silent truncation. (I think the ODB functions themselves report truncation to midas.log, so not completely silent).
> 
> We try to fix this where we must - but it is cumbersome with the current ODB API - as in your fix on has to:
> - get the ODB key, extract size
> - allocate buffer
> - call db_get_value() & co
> - use the data
> - remember to free the buffer on each and every return path
> 
> The first three steps could become one if we had an ODB "get_data" function that automatically allocated the data buffer.
> 
> But the main source of bugs will be the last step - remember to free the buffer, always.
> 
> P.S.
> 
> We are not alone in pondering how to do this best. If you want to see it "done right",
> read the fresh-off-the-presses book "Go Programming Language" by Alan Donovan and Brian Kernighan,
> http://www.gopl.io/
> 
> Brian Kernighan is the "K" in K&R "C programming language", still around and kicking, now at Google.
> Sadly the "R" passed away in 2011 - http://www.nytimes.com/2011/10/14/technology/dennis-ritchie-programming-trailblazer-dies-at-70.html
> 
> K.O.
> 
> > Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
> > script via Midas - which silently truncates command strings longer than 
> > 256 characters.
> > 
> > I'd prefer that Midas place no limit on string length.  Failing that, it would be
> > helpful to have character limits called out in the documentation 
> > (https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
> > https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).
> > 
> > As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
> > (Although key *names* are restricted to 256 characters.)  I've submitted one 
> > possible version of an arbitrary-length exec_script() as a pull request 
> > (https://bitbucket.org/tmidas/midas/pull-requests/).
> > 
> > Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
> > strings in the ODB as limited to 256 characters?

> Using low-level memory allocation routines in higher-level programs like mhttpd makes me nervous.

It should not, people have used malloc() for decades now without much injury to themselves. (Thomas corrects me: some people had big injury to their pride, me included).

> We could use vector arrays to allow variable-sized allocation, and use the data() member function to access the char* needed for functions like strlcat,
> db_get_data, and db_sprintf.

I thought auto_ptr was the correct tool to allocate "I just need a few bytes for a few minutes" arrays, but there is some discrepancy
between delete and delete[] (with brackets) and auto_ptr p(new char[i]) is verboten (even though it compiles just fine).

I ended up writing a custom replacement for auto_ptr called auto_string - now in mhttpd.cxx available for use in other places like this.

Still I think a db_get_data() that returns allocated memory is the correct solution. But this memory still needs to be released and lacking auto_ptr it opens the door for memory leaks.

> This conforms to the c++ standard, but doesn't require explicit freeing by the user - at least, not when you're allocating std::vector<char>

I do not think std::vector<char> can be cast into "char*" and used as replacement of "char str[100]" or "char* str = malloc(i);"

In other new, the limit on the command length is now removed.

K.O.

> 
> Amy
> 
> > Thank you for reporting this problem:
> > 
> > a) ODB key *names* are restricted to 31 characters (32 bytes, last byte is a NUL), not 256 characters.
> > b) ODB string length is unlimited (32-bit length field)
> > c) ODB C API "db_get_value" & co require fixed length buffer and most users of this API provide a 256-byte fixed buffer for strings, some of them also do not 
> > check the status code, resulting in silent truncation. (I think the ODB functions themselves report truncation to midas.log, so not completely silent).
> > 
> > We try to fix this where we must - but it is cumbersome with the current ODB API - as in your fix on has to:
> > - get the ODB key, extract size
> > - allocate buffer
> > - call db_get_value() & co
> > - use the data
> > - remember to free the buffer on each and every return path
> > 
> > The first three steps could become one if we had an ODB "get_data" function that automatically allocated the data buffer.
> > 
> > But the main source of bugs will be the last step - remember to free the buffer, always.
> > 
> > P.S.
> > 
> > We are not alone in pondering how to do this best. If you want to see it "done right",
> > read the fresh-off-the-presses book "Go Programming Language" by Alan Donovan and Brian Kernighan,
> > http://www.gopl.io/
> > 
> > Brian Kernighan is the "K" in K&R "C programming language", still around and kicking, now at Google.
> > Sadly the "R" passed away in 2011 - http://www.nytimes.com/2011/10/14/technology/dennis-ritchie-programming-trailblazer-dies-at-70.html
> > 
> > K.O.
> > 
> > > Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
> > > script via Midas - which silently truncates command strings longer than 
> > > 256 characters.
> > > 
> > > I'd prefer that Midas place no limit on string length.  Failing that, it would be
> > > helpful to have character limits called out in the documentation 
> > > (https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
> > > https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).
> > > 
> > > As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
> > > (Although key *names* are restricted to 256 characters.)  I've submitted one 
> > > possible version of an arbitrary-length exec_script() as a pull request 
> > > (https://bitbucket.org/tmidas/midas/pull-requests/).
> > > 
> > > Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
> > > strings in the ODB as limited to 256 characters?

mhttpd running as non-root cannot bind to standard https port 443. By default, mhttpd uses port 
8443 and it works just fine, but some applications such as the SSLlabs https tester insist on using 
port 443.

To connect mhttpd with port 443, I use the tcpproxy package from
git://git.spreadspace.org/tcpproxy.git

./tcpproxy -D -U -p 443 -r localhost4 -o 8443

(you can run this from rc.local)

(to remember, for best security one should run mhttpd behind an industry-standard https proxy)

K.O.

1. If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd. Wouldn't it be beneficial to have ODB entries where to get is from? This way people could change the look and feel more freely.
   
2. Especially the look and feel of the runlog is unsatisfactorily from my point of view. See . The old style was much more readable. I could recover the old style look and feel by slightly changing the mhttpd.cxx where I changed in show_rawfile(const char*) "dialogTable" to "runlogTable" in the table class. This way I could tinker around with the mhttpd.css by adding the following stuff there:
   
   • adding .runlogTable in line 289 :
     
   • adding some style information for the runlogTable :
     

> If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd.

mhttpd.css is served from $MIDASSYS/resources/mhttpd.css. The actual path is reported on the mhttpd 
"help" page.

(I think the internal mhttpd.css and mhttpd.js should be removed as no longer useful - nothing will work 
right if the real mhttpd.js and mhttpd.css cannot be served).

> Especially the look and feel of the runlog is unsatisfactorily from my point of view.

persons in charge of implementing the CSS stuff failed to convert quite a few pages, for example, the elog 
and the history editor pages were left completely broken. (mostly fixed now).

so thank you for reporting the runlog breakage, I hope Stefan & co can fix it quickly. (I cannot do - I have 
have no runlog pages on any of my test experiments).

> the old style was much more readable.

I think the new style is not too bad, except for a few visual artefacts here and there, the general comment 
that CSS is too complicated and hard to debug and the fact that over-subtle colouring yields inconsistent 
visuals between different monitors and ambient lighting conditions. (persons who select the colours always 
respond that "but to me, it looks just fine on my laptop", making it hard to resolve any issues).

> I could recover the old style look and feel by slightly changing the mhttpd.cxx

If you post the patches that fix it for you, I can commit them to midas. (git diff | mail olchansk@triumf.ca).

K.O.

Understand me right, I mostly like the new style, except the runlog as reported.
Attached you will find the diff's you were asking for. But as pointed out, I
haven't worked so far on CSS and hence this should be checked!!

I understand that the mhttpd.js needs to be the default one, however, mhttpd.css
might be left to the end-user to adopt to their specific needs. I shortly
checked in the mhttpd demon. It checks for the resources path in the ODB. If it
also would check for a CSS name, mhttpd.css could be changed/adopted by the
end-users without breaking things (at least it would then be their one business).

> > If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd.
> 
> mhttpd.css is served from $MIDASSYS/resources/mhttpd.css. The actual path is
reported on the mhttpd 
> "help" page.
> 
> (I think the internal mhttpd.css and mhttpd.js should be removed as no longer
useful - nothing will work 
> right if the real mhttpd.js and mhttpd.css cannot be served).
> 
> > Especially the look and feel of the runlog is unsatisfactorily from my point
of view.
> 
> persons in charge of implementing the CSS stuff failed to convert quite a few
pages, for example, the elog 
> and the history editor pages were left completely broken. (mostly fixed now).
> 
> so thank you for reporting the runlog breakage, I hope Stefan & co can fix it
quickly. (I cannot do - I have 
> have no runlog pages on any of my test experiments).
> 
> > the old style was much more readable.
> 
> I think the new style is not too bad, except for a few visual artefacts here
and there, the general comment 
> that CSS is too complicated and hard to debug and the fact that over-subtle
colouring yields inconsistent 
> visuals between different monitors and ambient lighting conditions. (persons
who select the colours always 
> respond that "but to me, it looks just fine on my laptop", making it hard to
resolve any issues).
> 
> > I could recover the old style look and feel by slightly changing the mhttpd.cxx
> 
> If you post the patches that fix it for you, I can commit them to midas. (git
diff | mail olchansk@triumf.ca).
> 
> K.O.

 > If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd.

I agree that this should be removed, Unfortunately I'm away right now, so I will fix it next week. Also will put in 
Andreas' diffs.

/Stefan

I put mhttpd.css and mhttpd.js into the ODB, so every experiment can change it. I put also Andreas' modifications of the CSS file for the runlog table and 
committed the changes.

/Stefan

I stumbled over an oddity which I would like to understand better. Here the
boundaries:
- Enable non-localhost RPC -> y
- Disable RPC hosts check  -> y

1) I am starting a run from ODBedit (start now -v):

07:13:11.272 2017/04/19 [ODBEdit,INFO] Run #26 started

07:13:25.516 2017/04/19 [Logger,LOG] File '/data/max/dlog/lem17_0026.root'
CRC32C checksum: 0x05ca4e7e, 1523383 bytes

On this little test experiment there is not much running, but it already shows
the effect I wanted to understand.

2) I am stopping the run from ODBedit (stop -v):

07:13:25.519 2017/04/19 [ODBEdit,INFO] Run #26 stopped

So, everything looks perfectly fine up to this point.

3) Now the 'strange' thing happens. To any point in time after this, I will stop
ODBEdit which results in the following messages:

07:13:32.335 2017/04/19 [ODBEdit,INFO] Program ODBEdit on host pc7962 stopped

07:13:32.335 2017/04/19 [Logger,ERROR] [midas.c:14079:rpc_server_receive,ERROR]
rpc check ok, abort canceled

This I do NOT understand! It looks as if the Logger (or any other client which
gets the run state transition) thinks that some Client (here ODBEdit) has a
broken connection. At least this is how I understand the comment in midas.c /
rpc_server_receive(). Is something broken in the de-registration from the RPC
server? By the way, all clients where running on the localhost, i.e. no remote
connection used here.

All this only happens if a run transition took place.

Unfortunately I do not understand the system well enough to suggest any fix to
this :-( and hence would appreciate any help.