The HTTPS SSL certificate on ladd00.triumf.ca has been updated. Same as the old
certificate, the new one is self-signed and your web browser may complain about
that and ask you to "save a security exception".

When you save the new certificate, you can verify that you are connected to the
real ladd00.triumf.ca by comparing the "SHA1 fingerprint" reported by your web
browser to the one given below (as reported by "svn update"):

Certificate information:
 - Hostname: ladd00.triumf.ca
 - Valid: from Mon, 10 Jun 2013 18:43:19 GMT until Tue, 10 Jun 2014 18:43:19 GMT
 - Issuer: DAQ, TRIUMF, Vancouver, BC, CA
 - Fingerprint: 3f:15:d0:1d:68:ae:f0:73:10:78:84:66:f3:af:c7:67:5c:70:00:62

At the same time, elogd was updated to latest version from Stefan (elog-2.9.2-1.i386).

Also as part of local computer updates, elogd is temporarily running on ladd03.triumf.ca. This should be 
transparent to all users but please let me know if there are strange artefacts, etc. (ladd03 will become the 
new ladd00 "soon").

K.O.

Dear MIDAS programmers,

I have been using your software in my lab (APC, Paris)
to run our data acqusition system. It is very robust and flexible.s

I would like to give you the large amount of credit which you are due.
How should I cite both MIDAS and ROODY? I have not been able to find any
information in the usual places.

Cheers, and thanks for the great program!
-Carl

> 
> I have been using your software in my lab (APC, Paris)
> to run our data acqusition system. It is very robust and flexible.s
> 
> I would like to give you the large amount of credit which you are due.
> How should I cite both MIDAS and ROODY? I have not been able to find any
> information in the usual places.
> 

Good to hear from a happy user.

I think the best way to give us credit is to recommend MIDAS to 10 of your friends.

For MIDAS citations, I think Pierre and Stefan have a standard one somewhere, we should have it linked from 
midas.triumf.ca.

For ROODY citations, I am not sure we have one. The idea behind it was to make a ROOT version of PAW++. 
Main authors are Greg King (UBC COOP student, where is he now?), Joe Chuma (also author of PHYSICA, 
R.I.P.), Pierre Amaudruz, myself and a few others.

K.O.

> Dear MIDAS programmers,
> 
> I have been using your software in my lab (APC, Paris)
> to run our data acqusition system. It is very robust and flexible.s
> 
> I would like to give you the large amount of credit which you are due.
> How should I cite both MIDAS and ROODY? I have not been able to find any
> information in the usual places.
> 
> Cheers, and thanks for the great program!
> -Carl

The standard citation for midas is a link to

http://midas.psi.ch

At the moment this points automatically to http://midas.triumf.ca, so both institutes are credited.

/Stefan

Hi,

I run into problems while trying to install Midas on Slackware 14.0. In the past
I have easily installed Midas on many other versions of Slackware. I have a new
computer set up with Slackware 14.0 and I just got the Midas latest version from 
https://bitbucket.org/tmidas/midas

Apparently there is a problem with a shared library which should be on the
system, I think make checks for /usr/include/mysql and then supposes that
libodbc.so should be on disk. I don't know why on my system it is not.

But I was wondering if I have some other problems (configuration problem?)
because I get a very large number of warnings. My last installation of Midas is
like from two years ago but I don't remember getting many warnings. Do I do
something obviously wrong? Here is uname -a output and I attached a file with
the output from make in midas folder (GNU Make 3.82 Built for
x86_64-slackware-linux-gnu). Thanks a lot!

Linux lheppc83 3.2.29 #2 SMP Mon Sep 17 14:19:22 CDT 2012 x86_64 Intel(R)
Xeon(R) CPU           E5520  @ 2.27GHz GenuineIntel GNU/Linux

Seems to me a problem with the ODBC library, so maybe Konstantin can comment.

/Stefan

> I run into problems while trying to install Midas on Slackware 14.0.

Thank you for reporting this. We do not have any slackware computers so we cannot see these message usually.

We use SL/RHEL 5/6 and MacOS for most development, plus we now have an Ubuntu test machine, where I see a 
whole different spew of compiler messages.

Most of the messages are:

a) useless compiler whining:

src/midas.c: In function 'cm_transition2':
src/midas.c:3769:74: warning: variable 'error' set but not used [-Wunused-but-set-variable]

b) an actual error in fal.c:

src/fal.c:131:0: warning: "EQUIPMENT_COMMON_STR" redefined [enabled by default]

c) actual error in fal.c: assignment into string constant is not permitted: char*x="aaa"; x[0]='c'; // core dump

src/fal.c:383:1: warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]

these are fixed by making sure all such pointers are "const char*" and the corresponding midas functions are 
also "const char*".

d) maybe an error (gcc sometimes gets this one wrong)

./mscb/mscb.c: In function 'int mscb_info(int, short unsigned int, MSCB_INFO*)':
./mscb/mscb.c:1682:8: warning: 'size' may be used uninitialized in this function [-Wuninitialized]
 
> Apparently there is a problem with a shared library which should be on the
> system, I think make checks for /usr/include/mysql and then supposes that
> libodbc.so should be on disk. I don't know why on my system it is not.

g++  -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -Idrivers -I../mxml -I./mscb -Llinux/lib -
DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_MYSQL -I/usr/include/mysql -DHAVE_ODBC -DHAVE_SQLITE 
-DHAVE_ROOT -pthread -m64 -I/home/exodaq/root_5.35.10/include -DHAVE_ZLIB -DHAVE_MSCB -DOS_LINUX 
-fPIC -Wno-unused-function -o linux/bin/mhttpd linux/lib/mhttpd.o linux/lib/mgd.o linux/lib/mscb.o 
linux/lib/sequencer.o linux/lib/libmidas.a linux/lib/libmidas.a -lodbc -lsqlite3 -lutil -lpthread -lrt -lz -lm
/usr/lib64/gcc/x86_64-slackware-linux/4.7.1/../../../../x86_64-slackware-linux/bin/ld: cannot find -lodbc

The ODBC library is not found (shared .so or static .a).

The Makefile check is for /usr/include/sql.h (usually part of the ODBC package). On the command line above, 
HAVE_ODBC is set, and the rest of MIDAS compiled okey, so the ODBC header files at least are present. But why 
the library is not found?

I do not know how slackware packages this stuff the way they do and I do not have a slackware system to check 
how it should look like, so I cannot suggest anything other than commenting out "HAVE_ODBC := ..." in the 
Makefile.


> But I was wondering if I have some other problems (configuration problem?)
> because I get a very large number of warnings. My last installation of Midas is
> like from two years ago but I don't remember getting many warnings.

There are no "many warnings". Mostly it's just one same warning repeated many times that complains about 
perfectly valid code:

src/midas.c: In function 'cm_transition':
src/midas.c:4388:19: warning: variable 'tr_main' set but not used [-Wunused-but-set-variable]

They complain about code:
 { int i=foo(); ... } // yes, "i" is not used, yes, if you have to keep it if you want to be able to see the return value 
of foo() in gdb.



> Do I do something obviously wrong?

No you. GCC people turned on one more noisy junk warning.

> Thanks a lot!

No idea about your missing ODBC library, I do not even know how to get a package listing on slackware (and 
proud of it).

But if you do know how to get a package listing for your odbc package, please send it here. On RHEL/SL, I would 
do:

rpm -qf /usr/include/sql.h ### find out the name of the package that owns this file
rpm -ql xxx ### list all files in this package

K.O.

> > I run into problems while trying to install Midas on Slackware 14.0.
> 
> b) an actual error in fal.c:
> 
> src/fal.c:131:0: warning: "EQUIPMENT_COMMON_STR" redefined [enabled by default]
> 
> c) actual error in fal.c: assignment into string constant is not permitted: char*x="aaa"; x[0]='c'; // core dump
> 
> src/fal.c:383:1: warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]
> 
> these are fixed by making sure all such pointers are "const char*" and the corresponding midas functions are 

the warnings in fal.c are now fixed.

K.O.

The warnings with the set but unused variables are real. While John O'Donnell proposed:

==========

somewhere I long the way I found an include file to help remove this kind of message.  try something like:

#include "use.h"
int foo () { return 3; }
int main () {
 { USED int i=foo(); }
 return 0;
}

with -Wall, and you will see the unused messages are gone.

==========

I would rather go and remove the unused variables to clean up the code a bit. Unfortunately my gcc version does 
not yet bark on that. So once I get a new version and I got plenty of spare time (....) I will consider removing all 
these variables.

/Stefan

> > I run into problems while trying to install Midas on Slackware 14.0.
> 
> Thank you for reporting this. We do not have any slackware computers so we cannot see these message usually.
> 
> 
> src/midas.c: In function 'cm_transition2':
> src/midas.c:3769:74: warning: variable 'error' set but not used [-Wunused-but-set-variable]
> 

got around to look at compile messages on ubuntu: in addition to "variable 'error' set but not used" we have these:

warning: ignoring return value of 'ssize_t write(int, const void*, size_t)'
warning: ignoring return value of 'ssize_t read(int, void*, size_t)'
warning: ignoring return value of 'int setuid(__uid_t)'
and a few more of similar

K.O.

> got around to look at compile messages on ubuntu: in addition to "variable 'error' set but not used" we have these:
> 
> warning: ignoring return value of 'ssize_t write(int, const void*, size_t)'
> warning: ignoring return value of 'ssize_t read(int, void*, size_t)'
> warning: ignoring return value of 'int setuid(__uid_t)'
> and a few more of similar

Arghh, now it is getting even more picky. I can understand the "variable xyz set but not used" and I'm willing to remove all the variables. But checking the 
return value from every function? Well, if the disk gets full, our code will silently ignore this for write(), so maybe it's not a bad idea to add a few checks. Also 
for the read(), there could be some problem, where an explicit cm_msg() in case of an error would help.

Hi, Thanks a lot for the response! Yes to search packages and list their content in Slackware it is pretty similar to your illustration. Slackware seems to use iODBC in which case it would link with -liodbc I guess.

root@lheppc83:~# slackpkg file-search sql.h

Looking for sql.h in package list. Please wait... DONE

The list below shows the packages that contains "sql\.h" file.

[ installed ] - libiodbc-3.52.7-x86_64-2

You can search specific packages using "slackpkg search package".

root@lheppc83:~# cat /var/log/packages/libiodbc-3.52.7-x86_64-2
PACKAGE NAME:     libiodbc-3.52.7-x86_64-2
COMPRESSED PACKAGE SIZE:     255.0K
UNCOMPRESSED PACKAGE SIZE:     1.0M
PACKAGE LOCATION: /var/log/mount/slackware64/l/libiodbc-3.52.7-x86_64-2.txz
PACKAGE DESCRIPTION:
libiodbc: libiodbc (Independent Open DataBase Connectivity)
libiodbc:
libiodbc: iODBC is the acronym for Independent Open DataBase Connectivity,
libiodbc: an Open Source platform independent implementation of both the ODBC
libiodbc: and X/Open specifications.  It allows for developing solutions
libiodbc: that are language, platform and database independent.
libiodbc:
libiodbc:
libiodbc:
libiodbc: Homepage: http://iodbc.org/
libiodbc:
FILE LIST:
./
usr/
usr/share/
usr/share/libiodbc/
usr/share/libiodbc/samples/
usr/share/libiodbc/samples/iodbctest.c
usr/share/libiodbc/samples/Makefile
usr/man/
usr/man/man1/
usr/man/man1/iodbc-config.1.gz
usr/man/man1/iodbctestw.1.gz
usr/man/man1/iodbctest.1.gz
usr/man/man1/iodbcadm-gtk.1.gz
usr/bin/
usr/bin/iodbctest
usr/bin/iodbcadm-gtk
usr/bin/iodbctestw
usr/bin/iodbc-config
usr/include/
usr/include/iodbcinst.h
usr/include/sqlext.h
usr/include/iodbcunix.h
usr/include/isqltypes.h
usr/include/sql.h
usr/include/iodbcext.h
usr/include/isql.h
usr/include/odbcinst.h
usr/include/isqlext.h
usr/include/sqlucode.h
usr/include/sqltypes.h
usr/lib64/
usr/lib64/libiodbc.la
usr/lib64/libdrvproxy.so.2.1.19
usr/lib64/libiodbcinst.la
usr/lib64/libiodbcadm.so.2.1.19
usr/lib64/libiodbcinst.so.2.1.19
usr/lib64/libiodbcadm.la
usr/lib64/pkgconfig/
usr/lib64/pkgconfig/libiodbc.pc
usr/lib64/libiodbc.so.2.1.19
usr/lib64/libdrvproxy.la
usr/doc/
usr/doc/libiodbc-3.52.7/
usr/doc/libiodbc-3.52.7/ChangeLog
usr/doc/libiodbc-3.52.7/README
usr/doc/libiodbc-3.52.7/COPYING
usr/doc/libiodbc-3.52.7/AUTHORS
usr/doc/libiodbc-3.52.7/INSTALL
install/
install/doinst.sh
install/slack-desc

# slackpkg file-search sql.h
[ installed ] - libiodbc-3.52.7-x86_64-2
...
# slackpkg search package
...
# cat /var/log/packages/libiodbc-3.52.7-x86_64-2
            usr/include/sql.h
...
            usr/lib64/libiodbc.so.2.1.19
...

Thanks, I am saving the slackpkg commands for future reference. Looks like the immediate problem is 
with the library name: libiodbc instead of libodbc. But the header file sql.h is the same.

I am not sure if it is worth making a generic solution for this: on MacOS, all ODBC functions are now 
obsoleted, to be removed, and since we are stanardized on MySQL anyway, so I think I will rewrite the SQL 
history driver to use the MySQL interface directly. Then all this ODBC extra layering will go away.

K.O.

> #include "use.h"
>  { USED int i=foo(); }

Sounds nifty, but google does not find use.h.

As for unused variables, some can be removed, others not so much, there is some code in there:

int i = blah...
#if 0
if (i=42) printf("wow, we got a 42!\n");
#endif
and
if (0) printf("debug: i=%d\n", i);

(difference is if you remove "i" or otherwise break the disabled debug code, "#if 0" will complain the next time you need that debugging code, "if (0)" will 
complain right away).

Some of this disabled debug code I would rather not remove - so much debug scaffolding I have added, removed, added again, removed again, all in the same 
places that I cannot be bothered with removing it anymore. I "#if 0" it and it stays there until I need it next time. But of course now gcc complains about it.

K.O.

I'm looking into using MIDAS for an experiment that creates one large event
(20MB or more) every second.

Q1: It looks like I should use EQ_FRAGMENTED. Has this feature been in use
recently? Is it known to work/not work?

More specifically, the computer should initiate a 1 second data taking, start to
such the data out of the electronics (which may take a while), change some
experimental parameters, and start over. 

Q2: What's the best way to do this? EQ_PERIODIC? 
I cannot guarantee that the time required to read the hardware has an upper bound.
In a standalone-prog I would simply use a big loop and let the machine execute
it as fast as it can: 1.1s, 1.5s, 1.1s, 1.3s, 2.5s, ..... depending on the HW
deadtimes.
Will this work with EQ_PERIODIC?

(Sorry for these maybe stupid questions, but I have so far only used MIDAS for
externally generated events, with <32kB event size).


Thanks a lot,

Randolf

> I'm looking into using MIDAS for an experiment that creates one large event
> (20MB or more) every second.
> 
> Q1: It looks like I should use EQ_FRAGMENTED. Has this feature been in use
> recently? Is it known to work/not work?
> 
> More specifically, the computer should initiate a 1 second data taking, start to
> such the data out of the electronics (which may take a while), change some
> experimental parameters, and start over. 
> 
> Q2: What's the best way to do this? EQ_PERIODIC? 
> I cannot guarantee that the time required to read the hardware has an upper bound.
> In a standalone-prog I would simply use a big loop and let the machine execute
> it as fast as it can: 1.1s, 1.5s, 1.1s, 1.3s, 2.5s, ..... depending on the HW
> deadtimes.
> Will this work with EQ_PERIODIC?
> 
> (Sorry for these maybe stupid questions, but I have so far only used MIDAS for
> externally generated events, with <32kB event size).
> 
> 
> Thanks a lot,
> 
> Randolf

Hi Randolf,

EQ_FRAGMENTED is kind of historically, when computers had a few MB of memory and you have to play special tricks to get large data buffers through. Today I 
would just use EQ_PERIODIC and increase the midas maximal event size to your needs. For details look here:

https://midas.triumf.ca/MidasWiki/index.php/Event_Buffer

The front-end scheduler is asynchronous, which means that your readout is called when the given period (1 second) is elapsed. If the readout takes longer 
than 1s, the schedule will (hopefully) call your readout immediately after the event has been sent. So you get automatically your maximal data rate. At MEG, we 
use 2 MB events with 10 Hz, so a 20 MB/sec data rate should not be a problem on decent computers.

Best,
Stefan

> I'm looking into using MIDAS for an experiment that creates one large event
> (20MB or more) every second.

Hi, there - 20 Mbyte event at 1/sec is not so large these days. (Well, depending on your hardware).

Using typical 1-2 year old PC hardware, 20 M/sec to local disk should work right away. Sending data from a 
remote front end (through the mserver), or writing to a remote disk (NFS, etc) - will of course requre a GigE 
network connection.

By default, MIDAS is configured for using about 1-2 Mbyte events, so for your case, you will need to:

- increase the event size limits in your frontend,
- increase /Experiment/MAX_EVENT_SIZE in ODB
- increase the size of the SYSTEM event buffer (/Experiment/Buffer sizes/SYSTEM in ODB)

I generally recommend sizing the SYSTEM event buffer to hold a few seconds worth of data (ot 
accommodate any delays in writing to local disk - competing  reads, internal delays of the disks, etc).

So for 20 M/s, the SYSTEM buffer size should be about 40-60 Mbytes.

For your case, you also want to buffer 3-5-10 events, so the SYSTEM buffer size would be between 100 and 
200 MBytes.

Assuming you have between 8-16-32 GBytes of RAM, this should not be a problem.

One the other hand, if you are running on a low-power ("green") ARM system with 1 Gbyte of RAM and a 
1GHz CPU, you should be able to handle the data rate of 20 Mbytes/sec, as long as your network and 
storage can handle it - I see GigE ethernet running at about 30-40 Mbytes/sec, so you should be okey,
but local storage to SD flash is only about 10 Mbytes/sec - too slow. You can try USB-attached HDD or SSD, 
this should run at up to 30-40 Mbytes/sec. I would expect no problems with this rate from MIDAS, as long 
as you can fit into your 1 GByte of RAM - obviously your SYSTEM buffer will have to be a little smaller than 
on a full-featured PC.

More information on MIDAS event size limits is here (as already reported by Stefan)
https://midas.triumf.ca/MidasWiki/index.php/Event_Buffer

Let us know how it works out for you.

K.O.

Hi,

I have been trying to use db_check_record() in order to verify that a subtree in the ODB has the correct 
variables, variable order, and overall size. I'm going off the documentation 
(https://midas.psi.ch/htmldoc/group__odbfunctionc.html) and use a string to compare against the ODB 
structure.  Since the string format is not specified for db_check_record(), I'm formatting my string 
according to the db_create_record() example.

Instead of db_check_record() checking the entire ODB subtree against all the variables represented in the 
string, I'm finding that only the first variable is checked.  The later variables in the string can be 
misspelled, out of order, or inexistent, and db_check_record() will still return 1.

Am I using db_check_record incorrectly?  

Thank you for any help with this issue.


I also believe that some of the documentation for db_check_record is outdated.  For example, init_string 
is referenced in the documentation but isn't part of the function definition.

Works, and here is how I did it. The attached example is based on the standard MIDAS
example in "src/midas/examples/experiment". 

My somewhat unsorted notes, haven't really tweaked the numbers. But it WORKS.

(1) mlogger writes "last.xml" (hard-coded!) which takes an awful amount of time
    as it writes the complete ODB containing the 10MB bank!
    just outcomment 
       // odb_save("last.xml");
    in mlogger.cxx, function 
    INT tr_start(INT run_number, char *error)
    (line ~3870 in mlogger rev. 5377, .cxx-file included)

(2) frontend.c:
     * the most important declarations are

/* BIG_DATA_BYTES is the data in 1 bank
   BIG_EVENT_SIZE is the event size. It's a bit larger than the bank size
                  because MIDAS needs to add some header bytes, I think
 */

#define BIG_DATA_BYTES  (10*1024*1024)   // 10 MB
#define BIG_EVENT_SIZE  (BIG_DATA_BYTES + 100)

/* maximum event size produced by this frontend */
INT max_event_size = BIG_EVENT_SIZE;

/* maximum event size for fragmented events (EQ_FRAGMENTED) */
INT max_event_size_frag = 5 * BIG_EVENT_SIZE;

/* buffer size to hold 10 events */
INT event_buffer_size = 10 * BIG_EVENT_SIZE;


     * bk_init() can hold at most 32kByte size events! Use bk_init32() instead.

     * complete frontend.c is attached

(3) in an xterm do
    # . setup.sh
    # odbedit -s 41943040
          (first invocation of odbedit must create large enough odb,
           otherwise you'll get "odb full" errors)
(4) odbedit> load big.odb      
    (attached). Essentials are:

    /Experiment/MAX_EVENT_SIZE = 20971520
    /Experiment/Buffer sizes/SYSTEM = 41943040   <- at least 2 events!

    To avoid excessive latecies when starting/stopping a run, do
    /Logger/ODB Dump = no 
    /Logger/Channels/0/Settings/ODB Dump = no 
     
    and create an Equipment Tree to make the mlogger happy

(5) a few more xterms, always ". setup.sh":
    # mlogger_patched (see (1))
    # ./frontend  (attched)

(6) in your odbedit (4) say "start". You should fill your disk rather quickly.

> Works, and here is how I did it. The attached example is based on the standard MIDAS
> example in "src/midas/examples/experiment". 

If you have such huge events, it does not make sense to put them into the ODB. The size needs to be increased (as 
you realized correctly) and the run stop takes long if you write last.xml. So just remove the RO_ODB flag in the 
frontend program and you won't have these problems.

/Stefan