> The custom and alias links in the standard midas status page were shown as HTML 
> links so far. If there are many links with names having spaces in their names, 
> it's a bit hard to distinguish between them. Therefore, they are packed now into 
> individual buttons (see attachment) starting from SVN revision 4633 on. This makes 
> also the look more homogeneous. If there is any problem with that, please report.

Would you consider using a different colour for the alias buttons (or background
colour)? At present it's hard to know whether a button is an alias link, a custom page
link or a user-button especially if you are not familiar with the button layout. 

> > The custom and alias links in the standard midas status page were shown as HTML 
> > links so far. If there are many links with names having spaces in their names, 
> > it's a bit hard to distinguish between them. Therefore, they are packed now into 
> > individual buttons (see attachment) starting from SVN revision 4633 on. This makes 
> > also the look more homogeneous. If there is any problem with that, please report.
> 
> Would you consider using a different colour for the alias buttons (or background
> colour)? At present it's hard to know whether a button is an alias link, a custom page
> link or a user-button especially if you are not familiar with the button layout. 

Ok, I changed the background colors for the button rows. There are now four different 
colors: Main menu buttons, Scripts, Manually triggered events, Alias & Custom pages. Hope 
this is ok. Of course one could have each button in a different color, but then it gets 
complicated... In that case I would recommend to make a dedicated custom page with all these 
buttons, which you can then tailor exactly to your needs.

1. Untar daq-midas_deb.tar.gz somewhere, assumming ~/Temp.
2. cd ~/Temp/daq-midas
3. svn co -r 5065 svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas
4. svn co -r 68 svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
5. cp -rvp debian/autoconf/* ./
6. ./configure --help
7. ./configure <--options>
8. make && make install

1. untar daq-midas_deb.tar.gz somewhere, assuming ~/Temp.
2. cd ~/Temp/daq-midas
3. svn co -r 5065 svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas
4. svn co -r 68 svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
5. dpkg-buildpackage -b -us -uc

• daq-midas-doc -- The documents and references
  
• daq-midas-root -- the midas runtime library and utilities built with root
  
• daq-midas-noroot -- the midas runtime library and utilities built without root
  
• daq-midas-dev-root -- the midas devel files (headers, objects, drivers, examples) built with root
  
• daq-midas-dev-noroot -- the midas devel files (headers, objects, drivers, examples) built without root
  

• executalbes -- /usr/lib/daq-midas/bin
  
• library and objs -- /usr/lib/daq-midas/lib
  
• headers -- /usr/lib/daq-midas/include
  
• sources and drivers -- /usr/share/daq-midas/
  
• docs and examples -- /usr/share/doc/daq-midas
  
• mdaq-config -- /usr/bin/mdaq-config
  

gcc `mdaq-config --cflags` -c -o myfe.o myfe.c
gcc `mdaq-config --libs` -o myfe myfe.o `mdaq-config --libdir`/mfe.o

• each midas experiment will be configured in a file named "mdaq.yaml"
  
• mdaq.py reads the configure file and prepare the daq environment, just like "examples/experiment/start_daq.sh"
  
• mdaq.py will handle "start/stop/restart/info" about the daq codes.
  

> I deployed several Debian Linux boxes as the DAQ systems in our lab. But I
feel it's boring to build and install midas and its related softwares (such as
root) on each box.


Our solution at TRIUMF is to install such packages on a shared NFS filesystem
visible to all client computers. This works well for ROOT and but MIDAS we found
it nearly impossible to keep MIDAS versions in sync between different projects
and expiments, so each experiment uses it's own copy of MIDAS, usually located
in the experiment home directory ($HOME/packages/midas). Because we often need
to make local modifications to MIDAS sources (Makefile, etc), we do not
"install" MIDAS into non-user-writable /usr/local & etc.


> I use autoconf/automake


The promise (premise) of autoconf/automake is to "hide" system dependencies. The
scripts are supposed to automatically probe the build environment and construct
an appropriate Makefile.

In practice, the autotool scripts always have bugs and incorrect assumptions
about the build environment and only work well for a few standardized systems
(RHEL and Debian derivatives) where the differences are so trivial that
autotools is an overkill and a normal Makefile is adequate for the job.

In my experience, as soon as I try to build an autotool-ized package on anything
that does not look like RHEL or Debian, autotool scripts explode and have to be
debugged and kludged by hand. Anybody who has ever done that would agree with me
that one would rather hack the ugliest Makefile than any of the  autotool
generated gibberish.

And of course autotools have never handled cross-compilation in any reasonable
way. Since we do cross-compile MIDAS (for VxWorks and embedded Linux, see "make
crosscompile") a Makefile is required and it so happens that the same Makefile
also works for normal Linux and MacOS, thank you very much.



> Here are the installation:
> [*] executalbes -- /usr/lib/daq-midas/bin
> [*] library and objs -- /usr/lib/daq-midas/lib


Is this in violation of the LSB (or LFS)? I though they mandate that files
controlled by package manager should be /usr/bin/odbedit, /usr/lib64/libmidas.a,
etc (/usr/bin/midas/odbedit no permitted).


> gcc `mdaq-config --cflags` -c -o myfe.o myfe.c


Please check if your config scripts correctly handle the "-m32" and "-m64" flags
- we frequently cross-compile 32-bit MIDAS executables on 64-bit machines.


K.O.

I've been trying to set up midas to create an automatic elog entry at the end of
each run and I've run into a problem. I've setup an elog on our server which
uses SSL and it seems that the melog provided by midas to create logbook entries
doesn't know any SSL.

My solution to this was to copy the crypt.c from the elog package to the
computer running midas and changed melog.c and the makefile to use SSL if a flag
-s is used. Does this seem like a sensible solution or did I oversee the obvious
and/or right way to do this?

> I've been trying to set up midas to create an automatic elog entry at the end of
> each run and I've run into a problem. I've setup an elog on our server which
> uses SSL and it seems that the melog provided by midas to create logbook entries
> doesn't know any SSL.
> 
> My solution to this was to copy the crypt.c from the elog package to the
> computer running midas and changed melog.c and the makefile to use SSL if a flag
> -s is used. Does this seem like a sensible solution or did I oversee the obvious
> and/or right way to do this?

Indeed melog.c is an old version of the elog.c utility in the elog package, which has not been maintained since a 
long time. Can't you just use the recent elog.c utility from the elog package?

> > I've been trying to set up midas to create an automatic elog entry at the end of
> > each run and I've run into a problem. I've setup an elog on our server which
> > uses SSL and it seems that the melog provided by midas to create logbook entries
> > doesn't know any SSL.
> > 
> > My solution to this was to copy the crypt.c from the elog package to the
> > computer running midas and changed melog.c and the makefile to use SSL if a flag
> > -s is used. Does this seem like a sensible solution or did I oversee the obvious
> > and/or right way to do this?
> 
> Indeed melog.c is an old version of the elog.c utility in the elog package, which has not been maintained since a 
> long time. Can't you just use the recent elog.c utility from the elog package?

Well, that's essentially what I did, I just didn't want to install the whole elog package on the midas server. Whether
the utility is called elog or melog doesn't really matter. I just wanted to make sure that this is the right way to do
it. 

Thanks!

Please, tell me how to choose a value of a "key" like DCM, pulser period,
presamples, upper thresholds to run a experiment? where can I find the related
informations? 

1. If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd. Wouldn't it be beneficial to have ODB entries where to get is from? This way people could change the look and feel more freely.
   
2. Especially the look and feel of the runlog is unsatisfactorily from my point of view. See . The old style was much more readable. I could recover the old style look and feel by slightly changing the mhttpd.cxx where I changed in show_rawfile(const char*) "dialogTable" to "runlogTable" in the table class. This way I could tinker around with the mhttpd.css by adding the following stuff there:
   
   • adding .runlogTable in line 289 :
     
   • adding some style information for the runlogTable :
     

> If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd.

mhttpd.css is served from $MIDASSYS/resources/mhttpd.css. The actual path is reported on the mhttpd 
"help" page.

(I think the internal mhttpd.css and mhttpd.js should be removed as no longer useful - nothing will work 
right if the real mhttpd.js and mhttpd.css cannot be served).

> Especially the look and feel of the runlog is unsatisfactorily from my point of view.

persons in charge of implementing the CSS stuff failed to convert quite a few pages, for example, the elog 
and the history editor pages were left completely broken. (mostly fixed now).

so thank you for reporting the runlog breakage, I hope Stefan & co can fix it quickly. (I cannot do - I have 
have no runlog pages on any of my test experiments).

> the old style was much more readable.

I think the new style is not too bad, except for a few visual artefacts here and there, the general comment 
that CSS is too complicated and hard to debug and the fact that over-subtle colouring yields inconsistent 
visuals between different monitors and ambient lighting conditions. (persons who select the colours always 
respond that "but to me, it looks just fine on my laptop", making it hard to resolve any issues).

> I could recover the old style look and feel by slightly changing the mhttpd.cxx

If you post the patches that fix it for you, I can commit them to midas. (git diff | mail olchansk@triumf.ca).

K.O.

Understand me right, I mostly like the new style, except the runlog as reported.
Attached you will find the diff's you were asking for. But as pointed out, I
haven't worked so far on CSS and hence this should be checked!!

I understand that the mhttpd.js needs to be the default one, however, mhttpd.css
might be left to the end-user to adopt to their specific needs. I shortly
checked in the mhttpd demon. It checks for the resources path in the ODB. If it
also would check for a CSS name, mhttpd.css could be changed/adopted by the
end-users without breaking things (at least it would then be their one business).

> > If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd.
> 
> mhttpd.css is served from $MIDASSYS/resources/mhttpd.css. The actual path is
reported on the mhttpd 
> "help" page.
> 
> (I think the internal mhttpd.css and mhttpd.js should be removed as no longer
useful - nothing will work 
> right if the real mhttpd.js and mhttpd.css cannot be served).
> 
> > Especially the look and feel of the runlog is unsatisfactorily from my point
of view.
> 
> persons in charge of implementing the CSS stuff failed to convert quite a few
pages, for example, the elog 
> and the history editor pages were left completely broken. (mostly fixed now).
> 
> so thank you for reporting the runlog breakage, I hope Stefan & co can fix it
quickly. (I cannot do - I have 
> have no runlog pages on any of my test experiments).
> 
> > the old style was much more readable.
> 
> I think the new style is not too bad, except for a few visual artefacts here
and there, the general comment 
> that CSS is too complicated and hard to debug and the fact that over-subtle
colouring yields inconsistent 
> visuals between different monitors and ambient lighting conditions. (persons
who select the colours always 
> respond that "but to me, it looks just fine on my laptop", making it hard to
resolve any issues).
> 
> > I could recover the old style look and feel by slightly changing the mhttpd.cxx
> 
> If you post the patches that fix it for you, I can commit them to midas. (git
diff | mail olchansk@triumf.ca).
> 
> K.O.

 > If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd.

I agree that this should be removed, Unfortunately I'm away right now, so I will fix it next week. Also will put in 
Andreas' diffs.

/Stefan

I put mhttpd.css and mhttpd.js into the ODB, so every experiment can change it. I put also Andreas' modifications of the CSS file for the runlog table and 
committed the changes.

/Stefan

We use the "Edit-on-start" and it's great.  But sometimes, something breaks
during the run, or you didn't realize you forgot to plug in a cable, or
whatever.  It'd be nice to have an "Edit-on-end" where you could prompt the user
to answer simple questions (like "Was this a good run?  [y/n]" or "Was the data
polarized?  [y/n]") and/or add a quick summary of what happened that run.


Thanks in advance,

Dan

We have similar demands, and we solve it in the following:

We use a run database. In the simplest case, this can be a text file which gets written at the end of the file. The 
mlogger has a built in SQL interface, so one can keep that table even inside a SQL interface. The per-run-
information then contains the run number, start/stop time, number of events, some run parameters and a "junk" 
flag. So if a run has a problem, one can set the junk flag by accessing the database (or text file) and setting this 
flag. In many cases you see that a run had a problem not at the end of the run, but a bit later. You mayby realize 
that the last two or three runs had the problem. With the run database approach, you can flag any run as "junk" 
later, which we need often, An edit-on-end would not make this possible.

So technically putting and edit-on-end is not a problem, but your life might be much easier if you use a run 
database as outlined above.

Best regards,
Stefan

I have a possibly simple feature request for the MIDAS sequencer.  It would be
helpful to be able to save an ODB key's value to a variable, for later use, and
would be the analogue of the ODBSET command.  I had in mind an application where
a user wants to temporarily change some settings in the ODB, then restore the
ODB to its original values.  Maybe something like on ODBRead command:

<ODBRead path="/Path/ODBkey">varname</ODBRead>
<ODBSet path="/Path/ODBkey">0</ODBRead> 
<Wait for="events">3000</Wait>
<ODBSet path="/Path/ODBkey">$varname</ODBRead> 

(In which the key's value is saved to variable varname, then later written back
to the ODB.)

I'm open to other suggestions for simple ways to do this through the sequencer.

Thanks! 

> I have a possibly simple feature request for the MIDAS sequencer.  It would be
> helpful to be able to save an ODB key's value to a variable, for later use, and
> would be the analogue of the ODBSET command.  I had in mind an application where
> a user wants to temporarily change some settings in the ODB, then restore the
> ODB to its original values.  Maybe something like on ODBRead command:

I implemented your request, committed the changed to GIT and updated the documentation. Now you can run 
things like:

ODBSET /System/tmp/test 1234 
ODBGET /System/tmp/test v 
MESSAGE $v 

(first you must create the key in the ODB manually).

Best regards,
Stefan

Thanks, this seems very helpful, and we'll give it a try.

> > I have a possibly simple feature request for the MIDAS sequencer.  It would be
> > helpful to be able to save an ODB key's value to a variable, for later use, and
> > would be the analogue of the ODBSET command.  I had in mind an application where
> > a user wants to temporarily change some settings in the ODB, then restore the
> > ODB to its original values.  Maybe something like on ODBRead command:
> 
> I implemented your request, committed the changed to GIT and updated the documentation. Now you can run 
> things like:
> 
> ODBSET /System/tmp/test 1234 
> ODBGET /System/tmp/test v 
> MESSAGE $v 
> 
> (first you must create the key in the ODB manually).
> 
> Best regards,
> Stefan

I am adding LZ4 and LZO compression the mlogger and as part of this work, I would like to add 
computation of checksums for the midas files.

On one side, such checksums help me confirm that uncompressed data contents is the same as original 
data (compression/decompression is okey).

On the other side, such checksums can confirm to the end user that today's contents of the midas file is 
the same as originally written by mlogger (maybe years ago) - there was no bit rot, no file corruption, no 
accidental or intentional modification of contents.

There are several choices of checksums available:
crc32 - as implemented by zlib (already written inside mid.gz files)
crc32c - improved and hardware accelerated version of CRC32 (http://tools.ietf.org/html/rfc3309)
md5 - cryptographically strong checksum, but obsolete
sha1 - same, also obsolete
sha256 - currently considered to be cryptographically strong

Of these checksums, only sha256 (sha512, etc) are presently considered to be cryptographically strong,
meaning that they can detect intentional file modifications. As opposed to (for example) crc32 where
it is easy to construct 2 files with different contents but the same checksum. Both md5 and sha1 are 
presently considered to be similarly cryptographically broken. But all of them are still usable
as checksums - as they will detect non-intentional data modifications (bit rot, etc) with
very high probability.

(Of course the strongest checksum is also the most expensive to compute).

I will probably implement crc32 (already in zlib), crc32c (easy to find hardware-accelerated
implementations) and sha256 (cryptographically strong).

I can write the computed checksums into midas.log, or into runNNN.crc32, runNNN.sha256, etc files. (or 
both).

Any thoughts on this?

K.O.

> Any thoughts on this?

We use binary midas files now for ~20 years and never felt the necessity to put any checksums or even encryption on these files. The reason for that is the following: Data on 
modern hard disks is already protected by CRC code or even ERC on the lower level, so it's very unlikely that single bytes change. If something happens, then it's a 
corruption of the file system, so a few sectors of a file are missing or wrong. In that case a CRC won't help you much, just tells you that the files are corrupt. But you see that 
also in the midas event structure. Each event has a header with the size of the event, so you can follow the file event by event. If something is missing, the next event header 
is no event header but something in the middle of the date, and you recognise this immediately since the header does not make any send (date is off by many years, event ID 
is arbitrary, event size is very different). So this redundancy in the midas event structure helps you to identify any corrupt files as good in my opinion as a CRC code will. I 
would not want to waste a single CPU cycle on lengthy CRC or even SHA algorithms, unless I see single bytes change inside events. But in this case this can even happen at 
the network level between frontend and backends. So we should add the CRC/SHA code at the frontend level. This could increase the dead time of the experiment which is 
bad. And what about VME transfer? While hard disks and Ethernet networks have already built-in CRC checks, VME transfer doesn't. So how can you be sure that no bits 
get corrupt between your ADC and your frontend computer?

If people insist of having CRC or SHA protection/encryption for some reason I do not understand yet, we should make this optional, so that I can turn it off, since I don't 
need it.

/Stefan