> I am using the Midas sequencer to run subsequent measurements in a loop, without 
> knowing how many iterations in advance. Therefore, I am using the "infinity" 
> option. Since I have other commands after the loop, it would be nice to have the 
> possibility to break the loop, but let the sequencer then finish the rest of the 
> commands. 
> Cheers,
> Ivo

You can do that with the "GOTO" statement, jumping to the first line after the loop.

Here is a working example:


LOOP runs, 5
     WAIT Seconds 3
     IF $runs > 2
         GOTO 7
     ENDIF
ENDLOOP
MESSAGE "Finished", 1

Best,
Stefan

I am using the Midas sequencer to run subsequent measurements in a loop, without 
knowing how many iterations in advance. Therefore, I am using the "infinity" 
option. Since I have other commands after the loop, it would be nice to have the 
possibility to break the loop, but let the sequencer then finish the rest of the 
commands. 
Cheers,
Ivo

> Dear all,
>        I'm trying to compile MIDAS on MacOS 10.10 and I get this error:
> 
> /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier 
> 'CLOCK_REALTIME'
>    clock_settime(CLOCK_REALTIME, &ltm);
> 
> Is it related to my (old) version of MacOS? Can I fix it somehow?
> 
> Thank you,
>       Francesco

If I see this correctly, you need at least MacOSX 10.12. If you can't upgrade, you can just remove line 3187 
from system.cxx. This function is only used in an online environment, where you would run a frontend on your 
Mac, which you probably don't do. So removing it does not hurt you.

Stefan

Dear all,
       I'm trying to compile MIDAS on MacOS 10.10 and I get this error:

/Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier 
'CLOCK_REALTIME'
   clock_settime(CLOCK_REALTIME, &ltm);

Is it related to my (old) version of MacOS? Can I fix it somehow?

Thank you,
      Francesco

We have to request of a 64-bit integer data type to be included in MIDAS banks.
Since 64-bit integers are on some systems "long" and on other systems "long long",
I decided to create the two new data types

TID_INT64
TID_UINT64

which follows more the standard C++ tradition:

https://en.cppreference.com/w/cpp/types/integer

To be consistent, I renamed the old types:

TID_BYTE       -> TID_UINT8
TID_SBYTE      -> TID_INT8
TID_WORD       -> TID_UINT16
TID_SHORT      -> TID_INT16
TID_DWORD      -> TID_UINT32
TID_INT        -> TID_INT32

I left the old definitions in midas.h, so old code will still compile fine and be binary
compatible. But if you write new code, the new types are recommended.

If you save the ODB in ASCII format, the new types are used as stings as well, like

[/Experiment]
ODB timeout = INT32 : 10000

but the old types are still understood when you load an old ODB file.

I hope I didn't break anything, please report if you have any issue.

Stefan

I had some quick look at the new mongoose code and didn't find anything I dislike. Did a quick test of the proxy which worked and is nice to have. 
Agree with all KO said about authentication.

So if there are no complaints, I would suggest that we move the summary of this thread into the official documentation.

Stefan

Ok, changed.

Stefan

Hi Stefan,

I think this only partially resolves the issue, in log_write:

#ifdef HAVE_ROOT
   } else if (log_chn->format == FORMAT_ROOT) {
      status = root_write(log_chn, pevent, pevent->data_size + sizeof(EVENT_HEADER));
#endif
   }

   actual_time = ss_millitime();
   if ((int) actual_time - (int) start_time > 3000)
      cm_msg(MINFO, "log_write", "Write operation on \'%s\' took %d ms", log_chn->path.c_str(), actual_time - start_time);

   if (status != SS_SUCCESS && !stop_requested) {
      cm_msg(MTALK, "log_write", "Error writing output file, stopping run");
      cm_msg(MERROR, "log_write", "Cannot write \'%s\', error %d, stopping run", log_chn->path.c_str(), status);
      stop_the_run(0);

      return status;
   }

In your solution root_write returns quietly but status == SS_INVALID_FORMAT (not SS_SUCCESS) and hence I get another misleading error message "Error writing output file, stopping run".

In order to prevent this you also would need to change the return value to SS_SUCCESS.

Dear simplest solution seems to me to just remove the error message generation and silently ignore the BOE EOR events. 

Committed that change.

Stefan

> [Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8000

Hi, Andreas, please open a bug report for this problem on bitbucket, there is now at least 2 bugs against
the ROOT writer (some events are written in duplicate sometimes), and I hope to fix this next time i review
the mlogger (RSN!). Biggest problem is that I do not use the ROOT output myself, so I have no way
to know if ROOT files produced by mlogger are correct or make sense. (without setting up some kind
of test environment with a ROOT file reader.

Thank you for reporting this problem here, so more people know about it.

If somebody has a patch to fix this, please send it in!

K.O.

Dear All,

At our experiment we write ROOT files. When starting/stopping runs we get the following error messages:

[Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8000

[Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8001

Looking into the source code I found that log_write (line 4248) sends these Midas System Events (BOR,EOR) to root_write without filtering them. root_write() checks in a first step if it gets such Midas System Events and if yes, moans.

Wouldn't it be better just to filter these events in log_write, before calling root_write, avoiding unnecessary error messages?

Is there something I miss?

Thanks,
  Andreas

> 
> Since ILL only provides access via SFTP and everything else is not existent or blocked (not even ssh is possible),
> this is the only thing we can work with by now. 
> 

Oops. SFTP != FTP.

SFTP uses SSH for data transport, so we cannot do it directly from C++ code in MIDAS. (we could use libssh, etc, but...)

I suggest you use lazylogger with the lazy_dache script, replace "dccp" with "sftp", replace "nsls" with an sftp "ls" command.

If you get it working, please consider contributing your lazylogger script to midas. (and does not have to be written in perl, python should work equally well).

For setting up lazylogger with the script method, I am pretty sure I posted the instructions to the forum (ages ago),
let me know if you cannot find them.

Good luck.

K.O.

> Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain. 
> What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the 
> local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.
> 
> Best,
> Stefan

Yes, see this now too. I will, therefore, try to set up the lazylogger properly. 

Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain. 
What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the 
local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.

Best,
Stefan

> > I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 
> 
> Hi, Ivo, good to hear from a midas user in these difficult times.
> 
> We do not use FTP at TRIUMF, but Stefan asked us to keep FTP alive and working, so we should be able
> to get you going. I will try to find the FTP instructions for you, I am pretty sure I have them somewhere.
> 
> In the mean time, I am very curious why you are using a FTP to record data, is it some kind
> of data appliance where simplest input for data is FTP? Using NFS does not work or is too hard?
> 
> Also for example at CERN, we write data to Castor and EOS, for this mlogger writes data to local disk,
> then the lazylogger runs a script to move the data to Castor and EOS. The example lazylogger
> scripts for this are in the MIDAS "progs" directory. But maybe you do not have a local disk and this would
> not work for you.
> 
> In other news, I hope to work on mlogger and lazylogger support for cloud storage (swift and s3 apis?),
> would that be useful as replacement for FTP?
> 
> K.O.
>
Good Morning Konstantin

Thanks for the fast reply.  Yes, it is, Midas is one of the things we can at least improve from home. 

Our experiment is planned to measure (soon) at ILL. Now since we don't use the equipment/detector from the 
beamline but our own, all the data from Midas is saved on the local drive. This is fine in the first instance
but then we also need proper backup. Since our experiment is quite small, the easiest solution I came up with
is to copy all of our data to the ILL storage which has enough space and is properly backed up. The ILL data 
storage allows only SFTP connections, nothing else. Since Midas has the FTP feature, having a separate FTP 
logger channel seemed the easiest way to go. 

Thanks for your input, I will look into how to mount SFTP and then this would also be a solution. 

Since ILL only provides access via SFTP and everything else is not existent or blocked (not even ssh is possible),
this is the only thing we can work with by now. 

Best regards,
Ivo

> I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 

Hi, Ivo, good to hear from a midas user in these difficult times.

We do not use FTP at TRIUMF, but Stefan asked us to keep FTP alive and working, so we should be able
to get you going. I will try to find the FTP instructions for you, I am pretty sure I have them somewhere.

In the mean time, I am very curious why you are using a FTP to record data, is it some kind
of data appliance where simplest input for data is FTP? Using NFS does not work or is too hard?

Also for example at CERN, we write data to Castor and EOS, for this mlogger writes data to local disk,
then the lazylogger runs a script to move the data to Castor and EOS. The example lazylogger
scripts for this are in the MIDAS "progs" directory. But maybe you do not have a local disk and this would
not work for you.

In other news, I hope to work on mlogger and lazylogger support for cloud storage (swift and s3 apis?),
would that be useful as replacement for FTP?

K.O.

Dear all
I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all. 
Thanks in advance,
Ivo

> > > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.

current code looks for the mbedtls library in ../mbedtls (next to midas)

if cmake misdetects it, turn it off by setting NO_MBEDTLS (same as NO_ROOT & co)

if you do want to build mhttpd with mbedtls, do this:

cd .../midas
cd ../
git clone https://github.com/ARMmbed/mbedtls.git
cd mbedtls
git submodule update --init ### this will populate the "crypto" directory
make ### if "python2" is missing, building of test suite programs will fail, but the libraries needed for midas will be built

cd ../midas
make cmake...

K.O.

> > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.

Configuration is done by ODB /WebServer:

---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
mime.types                      DIR
Enable localhost port           BOOL    1     4     2h   0   RWD  y
localhost port                  INT     1     4     2h   0   RWD  8080
localhost port passwords        BOOL    1     4     2h   0   RWD  n
Enable insecure port            BOOL    1     4     12h  0   RWD  n
insecure port                   INT     1     4     2h   0   RWD  8081
insecure port passwords         BOOL    1     4     2h   0   RWD  y
insecure port host list         BOOL    1     4     2h   0   RWD  y
Enable https port               BOOL    1     4     12h  0   RWD  n
https port                      INT     1     4     2h   0   RWD  8443
https port passwords            BOOL    1     4     2h   0   RWD  y
https port host list            BOOL    1     4     2h   0   RWD  y
Host list                       STRING  10    32    2h   0   RWD
                                        [0]             localhost
                                        [1]
                                        [2]
                                        [3]
                                        [4]
                                        [5]
                                        [6]
                                        [7]
                                        [8]
                                        [9]
Enable IPv6                     BOOL    1     4     2h   0   RWD  y
Proxy                           DIR
---------------------------------------------------------------------------

Most entries are self-obvious, but note:

- mime.types contains the mapping of file extensions of file content-type telling browser what to do:

---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l mime.types/
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
.HTML                           STRING  1     10    2h   0   RWD  text/html
.HTM                            STRING  1     10    2h   0   RWD  text/html
.CSS                            STRING  1     9     2h   0   RWD  text/css
---------------------------------------------------------------------------

- Proxy directory configures the http proxy (as implemented by mongoose, I am
not sure if I understand all limitations):

---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l Proxy/
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
example                         STRING  1     27    17h  0   RWD  #http://localhost:8080
---------------------------------------------------------------------------

("#" means - commented-out)

http://localhost:8080/proxy/example/foo/bar/baz proxies to http://localhost:8080/foo/bar/baz

- "Enable IPv6" tells mhttpd to also listen on the IPv6 ports. The best I can tell IPv6 works on the Mac,
and with luck will get some testing at CERN where IPv6 is in use.

Documentation on the midas wiki still needs to be updated for this.
K.O.

> the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.

The new code implements 3 http ports:

- localhost port 8080 - enabled by default - suitable for "I want to test midas on my laptop" and for connecting from the apache httpd
https password protected gateway.

- insecure http port 8081 - disabled by default - with optional password protection (HTTP Digest auth), and optional hostlist access
control - for the case when the https gateway is running on a different computer (i.e. ALPHA at CERN).

(My reading of "internet opinions" about HTTP Digest authentication over unencrypted HTTP is
that while considered very obsolete, there are no specific security problems and exploits
against it - other than the usual - man-in-the-middle and "steal the password file" attacks.
So while I do not recommend using it, I do not feel justified to remove/disable it on security grounds.
It provides an alternative password protection when use of SSL/HTTPS is too difficult).

- https port 8443 - disabled by default - also with optional password protection (HTTP Digest auth), and optional hostlist access
control. HTTP Digest password protection over HTTPS is deemed as secure at "HTTP Basic" password protection over HTTPS and
that is what is used by apache httpd password protection.

(The main problem with mhttpd support of HTTPS is obtaining an https certificate. Right now mhttpd
instructs the user to generate a self-signed certificate. But there is 2 problems: modern browsers dislike self-signed
certificates (even when explicitely marked "trust it!") and there is no check for certificate expiration.
I guess one could try to integrate mhttpd with certbot and the let's-encrypt system, but there
is problems, i.e. the certificate files live in readable-only-by-root directories, etc. I would rather
wait until mongoose implement certbot integration in their code).

More later...
K.O.