I have a possibly simple feature request for the MIDAS sequencer.  It would be
helpful to be able to save an ODB key's value to a variable, for later use, and
would be the analogue of the ODBSET command.  I had in mind an application where
a user wants to temporarily change some settings in the ODB, then restore the
ODB to its original values.  Maybe something like on ODBRead command:

<ODBRead path="/Path/ODBkey">varname</ODBRead>
<ODBSet path="/Path/ODBkey">0</ODBRead> 
<Wait for="events">3000</Wait>
<ODBSet path="/Path/ODBkey">$varname</ODBRead> 

(In which the key's value is saved to variable varname, then later written back
to the ODB.)

I'm open to other suggestions for simple ways to do this through the sequencer.

Thanks! 

> I have a possibly simple feature request for the MIDAS sequencer.  It would be
> helpful to be able to save an ODB key's value to a variable, for later use, and
> would be the analogue of the ODBSET command.  I had in mind an application where
> a user wants to temporarily change some settings in the ODB, then restore the
> ODB to its original values.  Maybe something like on ODBRead command:

I implemented your request, committed the changed to GIT and updated the documentation. Now you can run 
things like:

ODBSET /System/tmp/test 1234 
ODBGET /System/tmp/test v 
MESSAGE $v 

(first you must create the key in the ODB manually).

Best regards,
Stefan

Thanks, this seems very helpful, and we'll give it a try.

> > I have a possibly simple feature request for the MIDAS sequencer.  It would be
> > helpful to be able to save an ODB key's value to a variable, for later use, and
> > would be the analogue of the ODBSET command.  I had in mind an application where
> > a user wants to temporarily change some settings in the ODB, then restore the
> > ODB to its original values.  Maybe something like on ODBRead command:
> 
> I implemented your request, committed the changed to GIT and updated the documentation. Now you can run 
> things like:
> 
> ODBSET /System/tmp/test 1234 
> ODBGET /System/tmp/test v 
> MESSAGE $v 
> 
> (first you must create the key in the ODB manually).
> 
> Best regards,
> Stefan

Dear all
I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 
Thanks in advance,
Ivo

> I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 

Hi, Ivo, good to hear from a midas user in these difficult times.

We do not use FTP at TRIUMF, but Stefan asked us to keep FTP alive and working, so we should be able
to get you going. I will try to find the FTP instructions for you, I am pretty sure I have them somewhere.

In the mean time, I am very curious why you are using a FTP to record data, is it some kind
of data appliance where simplest input for data is FTP? Using NFS does not work or is too hard?

Also for example at CERN, we write data to Castor and EOS, for this mlogger writes data to local disk,
then the lazylogger runs a script to move the data to Castor and EOS. The example lazylogger
scripts for this are in the MIDAS "progs" directory. But maybe you do not have a local disk and this would
not work for you.

In other news, I hope to work on mlogger and lazylogger support for cloud storage (swift and s3 apis?),
would that be useful as replacement for FTP?

K.O.

> > I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 
> 
> Hi, Ivo, good to hear from a midas user in these difficult times.
> 
> We do not use FTP at TRIUMF, but Stefan asked us to keep FTP alive and working, so we should be able
> to get you going. I will try to find the FTP instructions for you, I am pretty sure I have them somewhere.
> 
> In the mean time, I am very curious why you are using a FTP to record data, is it some kind
> of data appliance where simplest input for data is FTP? Using NFS does not work or is too hard?
> 
> Also for example at CERN, we write data to Castor and EOS, for this mlogger writes data to local disk,
> then the lazylogger runs a script to move the data to Castor and EOS. The example lazylogger
> scripts for this are in the MIDAS "progs" directory. But maybe you do not have a local disk and this would
> not work for you.
> 
> In other news, I hope to work on mlogger and lazylogger support for cloud storage (swift and s3 apis?),
> would that be useful as replacement for FTP?
> 
> K.O.
>
Good Morning Konstantin

Thanks for the fast reply.  Yes, it is, Midas is one of the things we can at least improve from home. 

Our experiment is planned to measure (soon) at ILL. Now since we don't use the equipment/detector from the 
beamline but our own, all the data from Midas is saved on the local drive. This is fine in the first instance
but then we also need proper backup. Since our experiment is quite small, the easiest solution I came up with
is to copy all of our data to the ILL storage which has enough space and is properly backed up. The ILL data 
storage allows only SFTP connections, nothing else. Since Midas has the FTP feature, having a separate FTP 
logger channel seemed the easiest way to go. 

Thanks for your input, I will look into how to mount SFTP and then this would also be a solution. 

Since ILL only provides access via SFTP and everything else is not existent or blocked (not even ssh is possible),
this is the only thing we can work with by now. 

Best regards,
Ivo

Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain. 
What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the 
local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.

Best,
Stefan

> Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain. 
> What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the 
> local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.
> 
> Best,
> Stefan

Yes, see this now too. I will, therefore, try to set up the lazylogger properly. 

> 
> Since ILL only provides access via SFTP and everything else is not existent or blocked (not even ssh is possible),
> this is the only thing we can work with by now. 
> 

Oops. SFTP != FTP.

SFTP uses SSH for data transport, so we cannot do it directly from C++ code in MIDAS. (we could use libssh, etc, but...)

I suggest you use lazylogger with the lazy_dache script, replace "dccp" with "sftp", replace "nsls" with an sftp "ls" command.

If you get it working, please consider contributing your lazylogger script to midas. (and does not have to be written in perl, python should work equally well).

For setting up lazylogger with the script method, I am pretty sure I posted the instructions to the forum (ages ago),
let me know if you cannot find them.

Good luck.

K.O.

Hello,

I am quite new in both Linux and MIDAS.
I have install MIDAS on my desktop by going through this link: 
https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux 

in the last step when I send "mhttpd" command and try to open the link 
https://localhost:8443 (of course, changing the localhost with my host name), it 
failed to connect and shows this error: SSL_ERROR_NO_CYPHER_OVERLAP (please see 
attached file includes a screenshot of the error).

I have tried many ways to solve this problem: In Firefox: going to option/privacy 
and security/ security and uncheck the option "Block dangerous and deceptive 
content". but it does not help.

Looking forward your help
Thanks
Mehran

> I am quite new in both Linux and MIDAS.
> I have install MIDAS on my desktop by going through this link: 
> https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux 
> 
> in the last step when I send "mhttpd" command and try to open the link 
> https://localhost:8443 (of course, changing the localhost with my host name), it 
> failed to connect and shows this error: SSL_ERROR_NO_CYPHER_OVERLAP (please see 
> attached file includes a screenshot of the error).

What Linux? (on most linuxes, run "lsb_release -a")
What version of midas? (run odbedit "ver" command)
What version of firefox? (from the "about firefox" menu)

> I have tried many ways to solve this problem: In Firefox: going to option/privacy 
> and security/ security and uncheck the option "Block dangerous and deceptive 
> content". but it does not help.

No you cannot fix it from inside firefox. The issue is that the overlap of encryption methods
supported by your firefox and by your openssl library (used by mhttpd) is an empty set.
No common language, so to say, communication is impossible.

So either you have a very old openssl but very new firefox, or a very new openssl but very old 
firefox. Both very old or both very new can talk to each other, difficulties start with greater  
difference in age, as new (better) encryption methods are added and old (no-longer-secure) 
methods are banished.

BTW, for good security we recommend using apache httpd as the https proxy (instead of built-in 
https support in mhttpd). (I am not sure what it says in the current documentation). (But apache 
httpd will use the same openssl library, so this may not solve your problem. Let's see what 
versions of software you are using, per questions above, first).

K.O.

Hi Konstantin,
Thanks for your reply, 

> What Linux? (on most linuxes, run "lsb_release -a")
> What version of midas? (run odbedit "ver" command)
I am using CentOS 8

> What version of firefox? (from the "about firefox" menu)
Firefox 71.0

Thanks 
Mehran

> No you cannot fix it from inside firefox. The issue is that the overlap of encryption methods
> supported by your firefox and by your openssl library (used by mhttpd) is an empty set.
> No common language, so to say, communication is impossible.
> 
> So either you have a very old openssl but very new firefox, or a very new openssl but very old 
> firefox. Both very old or both very new can talk to each other, difficulties start with greater  
> difference in age, as new (better) encryption methods are added and old (no-longer-secure) 
> methods are banished.
> 
> BTW, for good security we recommend using apache httpd as the https proxy (instead of built-in 
> https support in mhttpd). (I am not sure what it says in the current documentation). (But apache 
> httpd will use the same openssl library, so this may not solve your problem. Let's see what 
> versions of software you are using, per questions above, first).
> 
> K.O.

Hi, I have not run midas on Centos-8 yet. Maybe there is a problem with the openssl library there. The Centos-7 
instructions for setting up apache httpd proxy are here, with luck they work on centos-8:
https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29

K.O.


> Hi Konstantin,
> Thanks for your reply, 
> 
> > What Linux? (on most linuxes, run "lsb_release -a")
> > What version of midas? (run odbedit "ver" command)
> I am using CentOS 8
> 
> > What version of firefox? (from the "about firefox" menu)
> Firefox 71.0
> 
> Thanks 
> Mehran
> 
> > No you cannot fix it from inside firefox. The issue is that the overlap of encryption methods
> > supported by your firefox and by your openssl library (used by mhttpd) is an empty set.
> > No common language, so to say, communication is impossible.
> > 
> > So either you have a very old openssl but very new firefox, or a very new openssl but very old 
> > firefox. Both very old or both very new can talk to each other, difficulties start with greater  
> > difference in age, as new (better) encryption methods are added and old (no-longer-secure) 
> > methods are banished.
> > 
> > BTW, for good security we recommend using apache httpd as the https proxy (instead of built-in 
> > https support in mhttpd). (I am not sure what it says in the current documentation). (But apache 
> > httpd will use the same openssl library, so this may not solve your problem. Let's see what 
> > versions of software you are using, per questions above, first).
> > 
> > K.O.

Hi,
As, the link suggests, I perform "yum install -y mod_ssl certwatch crypto-utils" but it complains as:
No match for argument: certwatch
No match for argument: crypto-utils

You may have a look on this link: https://blog.cloudware.bg/en/whats-new-in-centos-linux-8/
What’s gone?
In with the new, out with the old. CentOS 8 also says goodbye to some features. The OS removes several security functionalities. Among them is the Clevis HTTP pin, Coolkey and crypto-utils.
Cent OS 8 comes with securetty disabled by default. The configuration file is no longer included. You can add it back, but you will have to do it yourself. Another change is that shadow-utils no longer allow all-numeric user and group names.

Thanks
Mehran

> Hi, I have not run midas on Centos-8 yet. Maybe there is a problem with the openssl library there. The Centos-7 
> instructions for setting up apache httpd proxy are here, with luck they work on centos-8:
> https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
> 
> K.O.
> 

> I am using CentOS 8 [and]
> Firefox 71.0

I now have a centos-8 machine, I successfully built midas and I confirm that there is a problem.

But I get different errors from you:

- google chrome - does not connect at all (without any useful error message: "This site can’t be reached. The 
connection was reset.")
- firefox complains about the self-signed certificate, but connects ok, I see the midas status page and it works. "page 
info" reports connection is TLS 1.3, encryption TLS_AES_128_GCM_SHA256. However, the function "view certificate" 
does not work (without any useful error message).

I tried to run the SSLlabs tool to get some more information from mhttpd, but it does not want to run against mhttpd on 
port 8443... I do have a port redirect program somewhere... need to find it...

K.O.

> > The Centos-7 instructions for setting up apache httpd proxy are here, with luck they work on centos-8:
> > https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29

I now have a centos-8 computer, I followed my instructions and they generally worked.

There is a number of problems with the certbot package that prevent me from writing coherent production quality instructions for centos-8.

But at the end I was successful, httpd runs, gets "A+" rating from SSLlabs, forwards the requests to mhttpd, I can access the midas status page etc.

With luck the certbot packages for centos-8 will be sorted out soon (the apache plugin seems to be missing, this causes the automatic
certificate renewal to not work) and I will update my instructions to include centos-8.

Until then, I recommend that people continue to use centos-7 or the current Ubuntu LTS release.

K.O.

Documentation for writing midas history data to SQL (mysql) is now documented in midas doxygen files 
(make dox; firefox doxfiles/html/index.html). The corresponding logger and mhttpd code has been 
committed for some time now and it is used in production environment by the t2k/nd280 slow controls 
daq system at TRIUMF.

svn rev 4487
K.O.

> Documentation for writing midas history data to SQL (mysql) is now documented in midas doxygen files 
> (make dox; firefox doxfiles/html/index.html). The corresponding logger and mhttpd code has been 
> committed for some time now and it is used in production environment by the t2k/nd280 slow controls 
> daq system at TRIUMF.
> svn rev 4487


An updated version of the SQL history code is now committed to midas svn.

The new code is in history_sql.cxx. It implements a C++ interface to the MIDAS history (history.h),
and improves on the old code history_odbc.cxx by adding:
- an index table for remembering MIDAS names of SQL tables and columns (our midas users like to use funny characters in history 
names that are not permitted in SQL table and column names),
- caching of database schema (event names, etc) with a noticeable speedup of mhttpd (there is a new button on the history panel editor 
"clear history cache" to make mhttpd reload the database schema.

The updated documentation for using SQL history is committed to midas svn doxfiles/internal.dox (svn up; make dox; firefox 
doxfiles/html/index.html), or see my copy on the web at
http://ladd00.triumf.ca/~olchansk/midas/Internal.html#History_sql_internal

svn rev 4595
K.O.

   /* append argument "-b" for batch mode without graphics */
   rargv[rargc] = (char *) malloc(3);
   rargv[rargc++] = "-b";

   TApplication theApp("mlogger", &rargc, rargv);

   /* free argument memory */
   free(rargv[0]);
   free(rargv[1]);
   free(rargv);

   /* append argument "-b" for batch mode without graphics */
   rargv[rargc] = (char *) malloc(3);
   rargv[rargc++] = "-b";

   TApplication theApp("mlogger", &rargc, rargv);

   /* free argument memory */
   free(rargv[0]);
   /*free(rargv[1]);*/
   free(rargv);

#include <stdio.h>
#include <malloc.h>

int main(int argc, char** argv)
{
        char* pp;
        pp = (char *)malloc(sizeof(char)*3);
        /* pp = "-b"; */
        strcpy(pp,"-b");
        printf("PP=%s\n",pp);
        free(pp);

        return 0;
}

   /* append argument "-b" for batch mode without graphics */
   rargv[rargc] = (char *) malloc(3);
   rargv[rargc++] = "-b";

   TApplication theApp("mlogger", &rargc, rargv);

   /* free argument memory */
   free(rargv[0]);
   free(rargv[1]);
   free(rargv);

   /* append argument "-b" for batch mode without graphics */
   rargv[rargc] = (char *) malloc(3);
   rargv[rargc++] = "-b";

   TApplication theApp("mlogger", &rargc, rargv);

   /* free argument memory */
   free(rargv[0]);
   /*free(rargv[1]);*/
   free(rargv);

rargv[rargc] = (char *) malloc(3);