[quote="Stefan Ritt"][quote="Thorsten Lux"]In addition now I cannot start
anymore the mlogger from the web interface but only manually. However, I can
stop it from the web interface.[/quote]

At least that one can be fixed easily. Each program has a certain command with
which one can start it. This has to be put into the ODB under
/Programs/<program>. In your case you probably need

/Programs/Logger/Start command = mlogger -D

to start the logger from the Web page. To debug your run stop problems, I would
recommend to start all programs in a terminal window and look which one crashes
on the run end.

/Stefan[/quote]


Hi Stefan,

under /Programs/Logger/Start command I have
/home/next/MIDAS/midas/linux/bin/mlogger -D . This command does not work if I
press the "Start Logger" button on the mhttpd webpage but when I copy and paste
this command to a terminal window, it does the job. 

Well, thanks to you both for the fast response. I wrote Konstantin an email with
the results of the tests he suggested me to do.

Ciao

> 
> > Hi Stefan,
> > 
> > under /Programs/Logger/Start command I have
> > /home/next/MIDAS/midas/linux/bin/mlogger -D . This command does not work if I
> > press the "Start Logger" button on the mhttpd webpage but when I copy and paste
> > this command to a terminal window, it does the job. 
> > 
> > Well, thanks to you both for the fast response. I wrote Konstantin an email with
> > the results of the tests he suggested me to do.
> > 
> > Ciao
> 
> Let me guess: mhttpd is started under root (to be able to connect to port 80), and for root the mlogger program 
> is not in the path. Try to put into the odb the full path:
> 
> /Programs/Logger Start command = /usr/local/bin/mlogger -D

Yes, mhttpd is started as sudo, but I have the full path in the start command. And every user has the right to
execute mlogger. But okay, I will concentrate first to get the rest working again and then I will fight this problem.

Thanks!

Hi,

it seems that I solved the problem in a quite brutal way.
I opened the database with odbedit and saved first the whole database as a ASCII
file and then I did the same for each section separately. Then I closed odbedit.
Afterwards I deleted all .*.SHM files including .ODB.SHM and rebooted the system.
After the restart I opened odbedit and started mhttpd. With this blank system
the problem had disappeared. Afterwards I loaded section by section from the
previous created ASCII files. After each section I tested if I can start and
stop runs and it worked without problems. At the end I also loaded the ASCII
file which contained the whole database. In this case I got the following error
message:
[odb.c:6038:db_paste,ERROR] found string exceeding MAX_STRING_LENGTH

However, after a reboot everything worked fine. I can start and stop runs, with
and without frontend, without any error message. Only the mlogger resisted to
work again. 

But also this problem we solved. It seems it was related to a missing library
path. It is strange since while in the mhttpd web page the command does not work
and is not giving any error message, copying the same command to a terminal and
to start it manually does the job. We solved it by putting the start of mlogger
in a simple shell script and to execute it then from the mhttpd web page.
Probably not an elegant solution but it does the job.

Well, with this I can enjoy my weekend to start over with data taking next week!

Thanks a lot!

Thorsten

Hello,

We observed a strange behavior, from our point of view:
After some issues with with a 100% full database and recovering from this by 
creating a new odb file from a previous copy, the Midas status page started to 
reload/refresh every second while on the other hand it solved the 100% full 
issue.
There are no error messages and the pages looks like normal but it is impossible 
to start a new run due to the permanent reloading.
It is possible to click for example on odb and check the settings. 

Any idea what could be the problem and what the solution? 

Thanks 

Thorsten 

I am sorry, yesterday evening I must have been a bit tired after a long day with a lot of 
problems and error messages, so that I did not realize that yes, the frontend was finally 
starting well again but by recovering the odb file from an old one, it was stuck in the 
transition "stopping run" and this caused the continuous reloading of the status page. 

A "obdedit -C stop" solved the problem. 

Sorry for this! 





 > We observed a strange behavior, from our point of view:
> > ... the Midas status page started to reload/refresh every second
> 
> What version of midas is this? Run the odbedit "ver" command please. Also which
> browser on what OS is this? (chrome->about google chrome, firefox->about firefox).
> 
> The current versions of midas do not reload the status page ever, and I think
> all the page-reload code has been removed and they cannot reload automatically.
> 
> Old versions of the midas status page were designed to reload every 60 seconds or so.
> The reload interval is adjustable, but I do not think it was stored in ODB. It was
> accessed from the status page "config" button and I think it stored the reload
> period is a browser cookie.
> 
> This reload value may have gotten confused, and in this case, to fix it,
> you can try to clear all the web cookies from the web page. Another test for this
> would be to try an alternate web browser, which would presumable not have the bad cookie
> and will not suffer from the reload problem.
> 
> You can also open the web page debugger (google chrome -> right click menu -> inspect ->> 
> console & etc) and see if anything shows up there. I think you can set a break point
> on the page reload function and catch the place that causes the reload.
> 
> K.O.

Hi, I would like to restructure our sequencer scripts and the paths. Until now many things are not generic at all. I would like to ask if it is possible to include files through a relative path for example something like 
INCLUDE ../chip/global_basic_functions
Maybe I just did not found how to do it.

I find that new mhttpd has strange behaviour for ODB strings.

- I create a new STRING variable in ODB through mhttpd.  It defaults to size 32.

- I then edit the STRING variable through mhttpd, writing a new string larger
than 32 characters.

- Initially everything looks fine; it seems as if the new string value has been
accepted.

- But if you reload the page, or navigate back to the page, you realize that
mhttpd has silently truncated the string back to 32 characters.

You can reproduce this problem on a test page here:

http://midptf01.triumf.ca:8081/AnnMessage

Older versions of mhttpd (I'm testing one from 2 years ago) don't have this
'feature'.  For older mhttpd the string variable would get resized when a larger
string was inputted.  That definitely seems like the right behavior to me.

I am using fresh copy of midas from bitbucket as of this morning. (How do I get
a particular tag/hash of the version of midas that I am using?)

MIDAS does not seem to be consistent (or at least convenient) with how it
handles booleans in AJAX functions.

When you request an ODB value that is a boolean with AJAX call like

http://neut14.triumf.ca:8081/?cmd=jcopy&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys

then you get

{ "Hidden/last_written" : 1437065425, "Hidden" : false }

This seems correct, since the JSON convention has booleans encoded as true/false.

But this convention does not work when trying to set the boolean value. For instance

http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=true

does not set the variable to true. To make this work you need to use the
characters y/n

http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y

I tested this with ajax/jset, but the same problem seems to occur when using the
javascript function ODBSet. The documentation doesn't say what sort of encoding
to use when using these functions, so I guess the idea is that these functions
use MIDAS encoding for booleans. But it seems to me that it would be more
convenient if jset/ODBSet allowed the option to use json/javascript encoding for
boolean values; or at least had that as a format option for jset/ODBSet.  That
way my javascript could look like

var mybool = true;
URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=" + mybool;

instead of

var mybool = true;
URI_command = ""
if(mybool){
  URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y";
else
  URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=n";


__________________________________________________________
Cross-posting from bitbucket issue tracker:

https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans

New version of MIDAS has a feature where it seems to automatically resize the
message page horizontally in order to fix each MIDAS message into one line. 
Some of my MIDAS messages (in particular error messages, where I need details)
are very long.  The result is that the MIDAS page automatically becomes very
wide and I have to scroll a lot left/right in order to read my messages.  This
is annoying.

I would vote to roll-back this new feature.

> 
> I recommend that you use "mhttpd --mg" as the alternative for running "mhttpd -p" behind an apache 
> proxy. Using "mhttpd -p" (no HTTPS/SSL) on an internet-connected machine is insecure and should not be 
> done. (private network such as 192.168.x.y addresses is okey for now, I guess).

Finally reading through your documentation in detail [1,2].  I find that I don't understand this recommendation to use secure mongoose 
instead of putting mhttpd behind an apache proxy.  I think that it is nice to have secure mhttpd with mongoose as an option, but your 
documentation seems to imply that mhttpd-mongoose is much better than mhttpd-behind-apache and that the latter solution is strongly 
deprecated.

Perhaps I am not understanding the benefits of the new system.  In reference [2] you say "If this is not possible, somewhat better security 
for HTTP is gained by using a password protected SSL (https) proxy."  This seems to imply that the security of mhttpd-mongoose is better 
than the security of mhttpd-behind-apache.  Is that correct?  I thought that they provided similar security (assuming you follow 
recommended configurations for APACHE).

Setting up apache is trivial and it seems that mhttpd-behind-apache has other advantages, like being able to put other web resources 
(ganglia, cameras, elog, etc) behind the same secure server.  Also you can start to build complicated custom pages that are served directly 
from apache and just use MIDAS AJAX calls.  I was imagining slowly moving away from using mhttpd at all and just having html/js/css 
resources served up by apache.

So, unless I'm missing something, at this point I would continue to recommend people use mhttpd-behind-apache and I'd suggest this be 
presented as an equally valid option in the documentation.

[1] https://midas.triumf.ca/MidasWiki/index.php/Mhttpd
[2] https://midas.triumf.ca/MidasWiki/index.php/Setup_MIDAS_experiment#Install_SSL_proxy

> > 
> > I find that I don't understand this recommendation to use secure mongoose 
> > instead of putting mhttpd behind an apache proxy.
> >
> 
> This is a very valid question.
> 
> I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.
> ...
> So, which one to use?
> 
> - for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
> - for light-weight cases, or when root access is not available use built-in https in mhttpd.
> 
> The midas wiki documentation should probably be updated to explain all of this.


Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.

My only small complaint is that since the default mhttpd comes with mongoose security turned on, you need to explicitly disable the mhttpd+mongoose security first before you can start setting up apache.  I guess that the motivation is 
that we should force people to disable security, rather than hoping that they will enable it.  That's a convincing argument; so all I really need is that this procedure be well documented.

On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:

[mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
openssl "modern cryptography" ECDH ciphers not available

mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
about the SSL (it complains about the certificate of course).  So maybe this error message is 
relatively harmless?

I don't get this error message with Scientific Linux 6.7.

Hi,

We have recently been changing the code in mhttpd that maps custom web pages and resources to 
particular files on the server file system.  Ie, changing the code that uses the ODB keys in /Custom to 
map a web address like 

http://myhost:8081/CS/MyCustomPage 

to some file like

/home/user/resource/mypage.html

This mapping gets complicated when you use the /Custom/Path key to specify a location for web 
resources like images.  We have tried to summarize how the current system works on the wiki

https://midas.triumf.ca/MidasWiki/index.php//Custom_ODB_tree

Please provide any suggestions on how either the documentation or the actual algorithm can be 
improved.

There were some complaints that the MIDAS sequencer was slow.  Specifically, the
complaint was that even lines in the sequence that didn't do any (like COMMENT
commands) tooks > 100ms to execute.  These slow sequencer steps could be a
little annoying if a script had to change a large number of ODB variables before
starting.

I tested this a little using a trivial sequence; note that I did all tests using
mhttpd with mongoose enabled on a newer macbook pro.  I found that with the
mongoose server each line in a sequencer script was taking ~100ms.  This is
consistent with the loop in the main thread, which is only doing a cm_yield and
a sleep:

   while (!_abort) {
      status = ss_mutex_wait_for(request_mutex, 0);
      status = cm_yield(0);
      if (status == RPC_SHUTDOWN)
         break;
      sequencer();
      status = ss_mutex_release(request_mutex);
      ss_sleep(100);
   }

I tested reducing the sleep to 20ms.  As expected, this made the sequencer more
zippy, able to execute ~50 commands per second.

I tried to think what would be downsides to making this change.  I think that
the main web communication should not be affected, because that communication is
all handled by the separate mongoose thread.

I checked how much extra CPU was used if the sleep was reduced from 100ms to
20ms.  I found that when a sequence was not running the CPU increased from 0% to
0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
4% with my change.  4% is a little high, though I'd say still reasonable.  I
found that most of the CPU usage was occuring because every call to
'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
guess that making that call 50 times causes the somewhat heavy CPU usage.

I would argue that it would still be worth making that change, so that the
sequencer can be more zippy.

> There were some complaints that the MIDAS sequencer was slow.  Specifically, the
> complaint was that even lines in the sequence that didn't do any (like COMMENT
> commands) tooks > 100ms to execute.  These slow sequencer steps could be a
> little annoying if a script had to change a large number of ODB variables before
> starting.
> ...
> I checked how much extra CPU was used if the sleep was reduced from 100ms to
> 20ms.  I found that when a sequence was not running the CPU increased from 0% to
> 0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
> 4% with my change.  4% is a little high, though I'd say still reasonable.  I
> found that most of the CPU usage was occuring because every call to
> 'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
> guess that making that call 50 times causes the somewhat heavy CPU usage.

One additional point: I think that it would be reasonably simple to reduce this CPU
usage even while a sequence was going on.  I would guess that for many sequences a
lot of time was spent in a 'WAIT SECONDS' command, since you would presumably want
to wait while data was being taken or conditions stabilizing.  I think that if you
are in a 'WAIT SECONDS' command that hasn't been satisfied then there probably isn't
any reason to do the db_set_record at the end of the sequencer() method.

> > I checked how much extra CPU was used if the sleep was reduced from 100ms to
> > 20ms.  I found that when a sequence was not running the CPU increased from 0% to
> > 0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
> > 4% with my change.  4% is a little high, though I'd say still reasonable.  I
> > found that most of the CPU usage was occuring because every call to
> > 'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
> > guess that making that call 50 times causes the somewhat heavy CPU usage.
> > 
> > I would argue that it would still be worth making that change, so that the
> > sequencer can be more zippy.
> 
> The minimal time slice on most systems is 10 ms, and nothing prevents us from switching to
> that. The original 100 ms was more for the fact that you can see the sequencer statements
> executed one after the other (with the color bar). But this is more a "debugging" feature which 
> we not really need. 

OK, I made this change; sleep is now 10ms on main thread.  Seems to work fine on SL6 and MacOS.

> To do it "right" the sequencer would have to _return_ a sleep time. Like if it is in a wait loop (as
> most of the time), the sleep time could be close to 1 second, to correctly update the wait
> progress bar. If the sequencer executes ODB set statements, the wait time could be zero, so
> thousands of statements can be executed in one second. The problem we will then have of course
> that the sequencer will block the "request_mutex" almost always, which would prevent the
> mongoose server from serving anything. So this should be carefully tested. It could be (on most OS)
> that releasing the mutex by the main loop immediately switches to the mongoose thread, which would
> make the web server still quite responsive, but I'm not sure about that. So as a first change making
> the sleep time 10ms should be fine.

Hmm, yeah, I'm not sure about how to handle reducing the wait time to zero after ODB set commands.

But it does seem like it would be straight-forward to increase the sleep time for waits; I'll look into
a clean way of doing that.

We have started a new elog for discussions of the ROOTANA MIDAS analyzer package
[1], which is used at TRIUMF and elsewhere for quick displays of MIDAS data. 
The forum is available here

https://midas.triumf.ca/elog/Rootana

I would note that we have recently finished implementing a system in rootana for
easy web displays of MIDAS data, using ROOT's THttpServer to post histograms. 
Details on this new scheme are here

https://midas.triumf.ca/elog/Rootana/1

and

https://midas.triumf.ca/MidasWiki/index.php/Rootana_javascript_displays

Please sign up for the forum if you are interested in getting ROOTANA-related
discussions.

Thomas

[1] https://midas.triumf.ca/MidasWiki/index.php/ROOTANA

I recently upgraded my macbook to MacOS 10.11.  The compilation of MIDAS failed after the upgrade, 
complaining about  

gcc  -c -g -O2 -Wall <snip> src/mongoose.c
src/mongoose.c:322:10: fatal error: 'openssl/ssl.h' file not found

It seems that MacOS has now fully removed openssl header files (they were deprecated for a while).  There 
seems to be some notes on that here

http://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html

Konstantin suggested installing open-source builds of openssl using MacPorts.  I did that and MIDAS 
compiled fine.  I documented the procedure here:

https://midas.triumf.ca/MidasWiki/index.php/Installation/Compilation_problems#MacOS_10.11_.28El_Capitan.2
9_openssl_errors