Back Midas Rome Roody Rootana
  Midas DAQ System, Page 39 of 120  Not logged in ELOG logo
ID Dateup Author Topic Subject
  765   20 Jun 2011 Jimmy NgaiForumCannot open input file (file too large?)
Dear All,

Thanks Konstantin Olchanski for providing me a hint. The file can be opened now after I 
changed the line: 

file->gzfile = gzopen(file_name, "rb");

in function ma_open() in mana.c to the followings: 

INT fd = open(file_name, O_RDONLY | O_LARGEFILE);
if (fd <= 0)
   return NULL;

file->gzfile = gzdopen(fd, "rb");

No modifications to the Makefile is needed in this case.

Best Regards,
Jimmy


> Dear All,
> 
> I got a "Cannot open input file" error when I tried to analyze a .mid.gz file with 
> size over 5 GB on a 32-bit Linux. The error traced back to gzopen() in mana.c 
> where it returned NULL when opening the file. I understand that 32-bit Linux may 
> not be able to handle files with size over 2 GB. I tried to add -
> D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 to CFLAGS in the Makefile of MIDAS and 
> the analyzer, but I still got the same error. Is there any workarounds that enable 
> me to analyze large files on 32-bit systems?
> 
> p.s. The data file was also produced on a 32-bit Linux.
> 
> Thanks & Best Regards,
> 
> Jimmy
  766   20 Jun 2011 Stefan RittInfoJavascript ODB interface revised
The Javascript interface to the ODB has been revised. This extends the capabilities of custom web pages requesting data from the ODB. By grouping several request together, the number of round-trips is minimized and the response time is reduced. Following functions are new or extended:

  • ODBGet(path[, format]): This functions works now also with subdirectories in the ODB. The command ODBGet('/Runinfo') returns for example:
    1
    1
    1024
    0
    0
    0
    Mon Jun 20 09:40:14 2011
    1308588014
    Mon Jun 20 09:40:46 2011
    1308588046
    

  • ODBGetRecord(path), ODBExtractRecord(key): While ODBGet can be used for subdirectories, an easier way is to use ODBGetRecord and ODBExtractRecord. The first function retrieves the subtree (record), while the second one can be used to extract individual items. Here is an example:
    result = ODBGetRecord('/Runinfo');
    run_number = ODBExtractRecord(result, 'Run number');
    start_time = ODBExtractRecord(result, 'Start time');
    

  • ODBMGet(paths[, callback, formats]): This function ("Multi-Get") can be used to obtain ODB values from different paths in one call. The ODB paths have to be supplied in an array, the result is again an array. An optional callback routine might be supplied for asynchronous operation. Optional formats might be supplied if the resulting number should be formatted in a specific way. Here is an example:
       var req = new Array();
       req[0] = "/Runinfo/Run number";
       req[1] = "/Equipment/Trigger/Statistics/Events sent";
       var result = ODBMGet(req);
       run_number = result[0];
       events_sent = result[1];
    

The new functions are implemented in mhttpd revision 5075.
  767   21 Jun 2011 Stefan RittInfoNew MIDAS sequencer
A new sequencer for starting and stopping runs has been implemented. Although it is till kind of in a preliminary phase, it is usable, so I would like to share the syntax with you.

The sequencer runs inside mhttpd, and creates a new ODB subdirectory "/Sequencer". There is a new button on the status page called "Sequencer". In can run scripts in XML format, which reside on the server (where mhttpd is running). The sequencer is stateless, that means even if mhttpd is stopped and restarted, it resumes operation from where it has been stopped. Following statements are implemented:

  • <Comment>comment</Comment>
    a comment for this XML file, for information only

  • <ODBSet path="path">value</ODBSet>
    to set a value in the ODB

  • <ODBInc path="path">delta</ODBInc>
    to increment a value in the ODB

  • <RunDescription>Description</RunDescription>
    a run description which is stored under /Experiment/Run Parameters/Run Description.

  • <Transition>Start | Stop</Transition>
    to start or stop a run

  • <Loop n="n"> ... </Loop>
    to execute a loop n times. For infinite loops, "infinit" can be specified as n

  • <Wait for="events | ODBvalue | seconds" [path="ODB path"]>x</Wait>
    wait until a number of events is acquired (testing /Equipment/Trigger/Statistics/Events sent), or until a value in the ODB exceeds x, or wait for x seconds.

  • <Script [loop_counter="1"]>Script</Script>
    to call a script on the server side. Optionally, the loop counter(s) are passed to the script

Attached is a simple script which can be used as a starting point.
Attachment 1: seqtest.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<RunSequence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="">
<Comment>This is the XML test file</Comment>

<ODBSet path="/Experiment/Run Parameters/Comment">Test comment</ODBSet>
<RunDescription>Test Run</RunDescription>

<!-- do 10 runs -->
<Loop n="10">
   <!-- increment high voltage and wait for some time to settle -->
   <ODBInc path="/Equipment/HV/Variables/Demand[0]">10</ODBInc>
   <Wait for="seconds">10</Wait>

   <!-- start a run, acquire 3000 events -->
   <Transition>Start</Transition>
   <Wait for="events">3000</Wait>
   <Transition>Stop</Transition> -->
</Loop>
</RunSequence>
  768   24 Jun 2011 Exaos LeeSuggestionBuild MIDAS debian packages using autoconf/automake.
Here is my story. I deployed several Debian Linux boxes as the DAQ systems in our lab. But I feel it's boring to build and install midas and its related softwares (such as root) on each box. So I need a local debian software repository and put midas and its related packages in it. First of all, I need a midas debian package. After a week's study and searching, I finally finished the job. Hope you feel it useful.

All the work is attached as "daq-midas_deb.tar.gz". The detail is followed. I also created several debian packages. But it's too large to be uploaded. I havn't my own site accessible from internet. So, if you need the debian packages, please give me an accessible ftp or other similar service, then I can upload them to you.

First, I use autoconf/automake to rewrite the building system of MIDAS. You can check it this way:
1. Untar daq-midas_deb.tar.gz somewhere, assumming ~/Temp.
2. cd ~/Temp/daq-midas
3. svn co -r 5065 svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas
4. svn co -r 68 svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
5. cp -rvp debian/autoconf/* ./
6. ./configure --help
7. ./configure <--options>
8. make && make install

Then, I created the debian packages based on the new building files. You need to install root-system package from http://lcg-heppkg.web.cern.ch/lcg-heppkg/debian/. You can build debs this way:
1. untar daq-midas_deb.tar.gz somewhere, assuming ~/Temp.
2. cd ~/Temp/daq-midas
3. svn co -r 5065 svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas
4. svn co -r 68 svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
5. dpkg-buildpackage -b -us -uc

I split the package into serverals parts:
  • daq-midas-doc -- The documents and references
  • daq-midas-root -- the midas runtime library and utilities built with root
  • daq-midas-noroot -- the midas runtime library and utilities built without root
  • daq-midas-dev-root -- the midas devel files (headers, objects, drivers, examples) built with root
  • daq-midas-dev-noroot -- the midas devel files (headers, objects, drivers, examples) built without root

Here are the installation:
  • executalbes -- /usr/lib/daq-midas/bin
  • library and objs -- /usr/lib/daq-midas/lib
  • headers -- /usr/lib/daq-midas/include
  • sources and drivers -- /usr/share/daq-midas/
  • docs and examples -- /usr/share/doc/daq-midas
  • mdaq-config -- /usr/bin/mdaq-config

I add an auto-generated shell script -- mdaq-config. It behaves just like "root-config". You can get midas build flags and link flags this way:
gcc `mdaq-config --cflags` -c -o myfe.o myfe.c
gcc `mdaq-config --libs` -o myfe myfe.o `mdaq-config --libdir`/mfe.o

Bugs and suggestions are welcomed.

P.S. Based on debian packages, I am planing to write another script, "mdaq.py":
  • each midas experiment will be configured in a file named "mdaq.yaml"
  • mdaq.py reads the configure file and prepare the daq environment, just like "examples/experiment/start_daq.sh"
  • mdaq.py will handle "start/stop/restart/info" about the daq codes.
The attached "mdaq.py" is the old one.
Attachment 1: daq-midas_deb.tar.gz
Attachment 2: mdaq.py
#!/usr/bin/env python
'''
   Python script to handle daq sessions using MIDAS/PSI.
   Auther: Exaos Lee <Exaos DOT Lee AT gmail DOT com>
   Date:   2009-07-15
'''

import os, os.path
import sys, signal, time
from commands     import getoutput
from ConfigParser import ConfigParser
from pprint       import pprint

class mdaq:
    _codes_ = { 'MSERVER':"MIDAS Server", 'MHTTPD':"MIDAS HTTPD Server",
                'ODBEDIT':"ODB Editor",   'LOGGER':"Data Logger",
                'PROG_FE':"Frontend",   'PROG_ANA':"Analyzer" }
    _env_ = dict()

    def __init__(self,**kws):
        env = dict()
        if kws.has_key('fname') and kws['fname'] != "":
            env = self.parse_conf(kws['fname'])
        self.apply_env(env)

    def parse_conf(self, fname):
        if not os.path.isfile(fname):
            print("**WARNING** File \"%s\" does not exist!"%fname)
            return None

        conf = ConfigParser()
        conf.read(fname)
        env={'CONF_FILE':fname }
        fcpath = os.path.dirname(fname)

        sec = 'EXP_INFO' ## Parsing Experiment Information
        env['EXP_INFO'] = [ self.safe_parse(conf,sec,'Name'),
                            self.safe_parse(conf,sec,'Info'), ]
        msg = self.safe_parse(conf,sec,'DAQ_DIR')
        if msg != "" and os.path.isdir(msg):
            dpath = msg
            if not os.path.isabs(msg):
                dpath = os.path.join(os.path.abspath(fcpath),msg)
            env['DAQ_DIR'] = os.path.normpath(dpath)

        ## Parsing MIDAS configuration
        env['MIDASSYS'] = ["MIDAS System Path",
                           self.safe_parse(conf,"MIDAS","MIDASSYS")]
        if env['MIDASSYS'][1]=="": env['MIDASSYS'][1] = self.get_default_msys()

        ## Parsing Info of programs
        for key in self._codes_.keys():
            code = self.parse_prog_opt(conf, key, fpath=fcpath)
            if code:
                env[key] = [self._codes_[key], ]
                env[key].extend(code)

        port = self.safe_parse(conf, "MHTTPD","PORT")
        if port != "":
            if not env.has_key('MHTTPD'):
                env['MHTTPD'] = ["MIDAS HTTP Server",
                                 os.path.join(env['MIDASSYS'][1],"bin/mhttpd"),
                                 "-h localhost -D"]
            if "-p %s"%port not in env['MHTTPD'][2]:
                env['MHTTPD'][2] = env['MHTTPD'][2] + " -p %s"%port

        return env

    def parse_prog_opt(self, cnf, sec, fpath="./"):
        name = self.safe_parse(cnf, sec, "NAME")
        path = self.safe_parse(cnf, sec, "PATH")
        fn = os.path.join(path,name)
        if name == "": fn = ""
        elif not os.path.isabs(fn):
            fn = os.path.normpath(os.path.join(os.path.abspath(fpath),fn))
        else:
            fn = os.path.normpath(fn)
        opts = self.safe_parse(cnf, sec, "Option")
        if opts != "" or name != "":
            return [fn, opts]

        return None
 
    def safe_parse(self, cnf, sec, opt):
        try:    return cnf.get(sec,opt)
        except: return ""

    def get_default_msys(self):
        if os.environ.has_key('MIDASSYS'): return os.environ['MIDASSYS']
        return "/opt/MIDAS.PSI/Version/Current"

    def apply_env(self, daq_env=dict()):
        env = daq_env
        if not env:
            print("** WARNING** No environment defined! Using the default values. ****")
            env = {'MIDASSYS':['MIDAS System Path', ""], 'CONF_FILE':"", }
        if not env.has_key('MIDASSYS'): env['MIDASSYS'] = ['MIDAS System Path',""]
        if env['MIDASSYS'][1]=="": env['MIDASSYS'][1] = self.get_default_msys()
        def_env = {
            'MSERVER' :["MIDAS Server",
                        os.path.join(env['MIDASSYS'][1],"bin/mserver"),
                        "-D", True],
            'MHTTPD'  :["MIDAS HTTP Server",
                        os.path.join(env['MIDASSYS'][1],"bin/mhttpd"),
                        "-h localhost -D -p 8080", True],
            'ODBEDIT' :["ODB Editor",
                        os.path.join(env['MIDASSYS'][1],"bin/odbedit"),
                        "-c clean", False],
            'LOGGER'  :["Data Logger",
                        os.path.join(env['MIDASSYS'][1],"bin/mlogger"),
                        "", True],
            'PROG_FE' :["Frontend","", "", True],
            'PROG_ANA':["Analyzer","", "", True],
            'EXP_INFO':["test","The test experiment"],
            'DAQ_DIR':os.path.join(os.environ['HOME'],"online/test") }

        for key in def_env.keys():
            if not env.has_key(key): env[key] = def_env[key]

        for key in self._codes_.keys():
            if env[key][1] == "": env[key][1] = def_env[key][1]
            if env[key][2] == "": env[key][2] = def_env[key][2]

        if not os.path.isdir(env['DAQ_DIR']):
            try: os.makedirs(env['DAQ_DIR'])
            except: print("ERROR to make file")

        ## Asign default program options
        exp_opt = "-e %s"%env['EXP_INFO'][0]
        for key in ['LOGGER', 'PROG_FE', 'PROG_ANA', "MHTTPD"]:
            if exp_opt not in env[key][2]:
                env[key][2] = exp_opt + ' ' + env[key][2]
            if "-D" not in env[key][2]:
                env[key][2] = env[key][2] + ' -D'
        if exp_opt not in env['ODBEDIT'][2]:
            env['ODBEDIT'][2] = env['ODBEDIT'][2] + ' ' + exp_opt

        self._env_ = env

    def print_conf(self):
        print "\n================= DAQ Config ====================="
        for key in self._env_.keys():
            print key,": ",
            pprint(self._env_[key])

    def info(self): self.print_conf()

    ## running programs
    def getpids(self, pname):
        return [int(k) for k in getoutput("pidof %s"%pname).strip().split()]

    def kill_prog(self,pname):
        for i in self.getpids(pname): os.kill(i,signal.SIGTERM)

    def run_prog(self, prog):
        if len(prog)>=4 and not prog[3]: return

        if len(prog)>=3 and prog[1]:
            print("Starting %s: %s"%(prog[0],prog[1]))
            os.system("%s %s"%(prog[1],prog[2]))
        else:
            print("%s: no executable exist!"%prog[0])

    def get_status(self,pfile):
        msg = "Not running!"
        pids = self.getpids( os.path.basename(pfile) )
        if pids: return 'PIDs = ' + str(pids)
        return msg

    def echo_status(self,pname,pfile):
        print("%s:\n   %s\n   "%(pname,pfile)
              + self.get_status(pfile))

    ## Session manage: start, stop, status, restart

    def status(self):
        print( "\n===== Session status for experiment \"%s\" =====\n" %
               self._env_['EXP_INFO'][0] )
        for code in ['MSERVER','MHTTPD','LOGGER','PROG_FE','PROG_ANA']:
            self.echo_status(self._env_[code][0], self._env_[code][1])

    def start(self):
        ## Running mserver & mhttpd
        ms_pids = self.getpids(os.path.basename(self._env_['MSERVER'][1]))
        if not ms_pids:
            self.run_prog(self._env_['MSERVER'])
            print("Killing %s ...\n"%self._env_['MHTTPD'][0])
            self.kill_prog(self._env_['MHTTPD'][1])
        if not self.getpids(os.path.basename(self._env_['MHTTPD'][1])):
            self.run_prog(self._env_['MHTTPD'])

        ## Running FE and Analyzer
        old_path = os.getcwd()
        os.chdir( self._env_['DAQ_DIR'] )

        # Init ODB
        print("Initializing ODB for experiment %s ..." % self._env_['EXP_INFO'][0])
        self.run_prog(self._env_['ODBEDIT'])
        time.sleep(1)

        # First stop, then start ...
        self.stop()
        time.sleep(1)

        # Start FE & ANA
        self.run_prog(self._env_['PROG_FE'])
        self.run_prog(self._env_['PROG_ANA'])
        time.sleep(1)

        # Start Logger
        self.run_prog(self._env_['LOGGER'])

        os.chdir(old_path)

    def stop(self):
        for key in ["LOGGER", "PROG_FE", "PROG_ANA"]:
            if self._env_[key][1]:
                print("Killing threads:\n  %s"%self._env_[key][1])
                self.kill_prog(self._env_[key][1])

    def stop_all(self):
        self.stop()
        for key in ['MHTTPD', 'MSERVER']:
            if self._env_[key][1]:
                print("Killing threads:\n  %s"%self._env_[key][1])
                self.kill_prog(self._env_[key][1])

    def restart(self): self.start()


if __name__=='__main__':
    from optparse import OptionParser

    usage = "usage: %prog [options] <command>"
    parser = OptionParser(usage=usage, version="%prog 0.1")
    parser.add_option("-c","--config", metavar="filename",
                      help="Read config from FILENAME")
    cmds = ["start","stop","stop_all","restart","status","info"]

    (opts,args) = parser.parse_args( sys.argv[1:] )
    if not args or len(args)>2:
         parser.print_help()
         print "\nCommands:\n\t",
         pprint(cmds)
         exit()

    fcnf = "daq.conf"
    if opts.config: fcnf = opts.config

    if args[0] in cmds:
        daq = mdaq(fname=fcnf)
        exec( "daq.%s()"%args[0] )
    else:
        parser.print_help()
        print "\nCommands:\n\t",
        pprint(cmds)
  769   27 Jun 2011 Konstantin OlchanskiSuggestionBuild MIDAS debian packages using autoconf/automake.
> I deployed several Debian Linux boxes as the DAQ systems in our lab. But I
feel it's boring to build and install midas and its related softwares (such as
root) on each box.


Our solution at TRIUMF is to install such packages on a shared NFS filesystem
visible to all client computers. This works well for ROOT and but MIDAS we found
it nearly impossible to keep MIDAS versions in sync between different projects
and expiments, so each experiment uses it's own copy of MIDAS, usually located
in the experiment home directory ($HOME/packages/midas). Because we often need
to make local modifications to MIDAS sources (Makefile, etc), we do not
"install" MIDAS into non-user-writable /usr/local & etc.


> I use autoconf/automake


The promise (premise) of autoconf/automake is to "hide" system dependencies. The
scripts are supposed to automatically probe the build environment and construct
an appropriate Makefile.

In practice, the autotool scripts always have bugs and incorrect assumptions
about the build environment and only work well for a few standardized systems
(RHEL and Debian derivatives) where the differences are so trivial that
autotools is an overkill and a normal Makefile is adequate for the job.

In my experience, as soon as I try to build an autotool-ized package on anything
that does not look like RHEL or Debian, autotool scripts explode and have to be
debugged and kludged by hand. Anybody who has ever done that would agree with me
that one would rather hack the ugliest Makefile than any of the  autotool
generated gibberish.

And of course autotools have never handled cross-compilation in any reasonable
way. Since we do cross-compile MIDAS (for VxWorks and embedded Linux, see "make
crosscompile") a Makefile is required and it so happens that the same Makefile
also works for normal Linux and MacOS, thank you very much.



> Here are the installation:
> [*] executalbes -- /usr/lib/daq-midas/bin
> [*] library and objs -- /usr/lib/daq-midas/lib


Is this in violation of the LSB (or LFS)? I though they mandate that files
controlled by package manager should be /usr/bin/odbedit, /usr/lib64/libmidas.a,
etc (/usr/bin/midas/odbedit no permitted).


> gcc `mdaq-config --cflags` -c -o myfe.o myfe.c


Please check if your config scripts correctly handle the "-m32" and "-m64" flags
- we frequently cross-compile 32-bit MIDAS executables on 64-bit machines.


K.O.
  770   27 Jun 2011 Konstantin OlchanskiInfoupdated mhttpd history "export" function
The mhttpd history "export" function has been converted to the new midas history
interface and should now work for SQL-based history systems. In the process,
improvements by Eoin Butler (CERN AD-5/ALPHA) were merged - adding a UNIX
timestamp and a better text timestamp. Also now "export" outputs the actual
values from the history file - the scaling values from the definition of the
history plot panel are no longer applied.

Here is an example of the new file format:

Time, Timestamp, Run, Run State, SLOW
2011.06.21 15:45:21, 1308696321, 13292, 3, -89.1007

svn rev 5104
K.O.
  771   27 Jun 2011 Konstantin OlchanskiInfomidas shared memory changes
A number of changes were made to the midas shared memory implementation for
Linux and MacOS:

1) SysV or POSIX shared memory compile-type choice is removed. Both shared
memory types are compiled-in and are selected at run time.
2) the shared memory type used by an experiment is recorded in the file
.SHM_TYPE.TXT. Currently implemented are "POSIXv2_SHM" (the new default for new
experiments), "POSIX_SHM", "MMAP_SHM" and "SYSV_SHM". (see system.c) (MMAP_SHM
is fully functional but is not recommended). The POSIXv2_SHM uses an improved
filename scheme (on Linux, see "ls -l /dev/shm") and permits multiple
experiments to coexist on a MacOS computer (where there is a severe limit on
shared memory filename length).
3) following a number of mishaps where "odbedit" has been run on the wrong
computer (causing havoc with ODB and .xxx.SHM files), for each experiment, the
hostname of the computer where the ODB shared memory is meant to reside is now
recorded in the file .SHM_HOST.TXT. Typically, this is the machine running
mserver, mhttpd and mlogger. If some client is accidentally started on the wrong
machine or if MIDAS_SERVER_HOST is accidentally left undefined, MIDAS will now
print a stern message reporting the hostname mismatch, tell the user to use the
mserver and refuse to run. The user has the choice of starting the client on the
correct computer (as reported in the error message), using the mserver (start
client with -H flag) or edit/delete the .SHM_HOST.TXT file (full pathname is
reported by the error message).

With this update, MIDAS on MacOS becomes fully functional (before, only one
experiment could be used at a time).

svn rev 5105
K.O.
  772   27 Jun 2011 Konstantin OlchanskiInfomlogger lock for runNNN.mid.gz files
By popular request, Stefan R. implemented a locking scheme for mlogger output files.

To use this function, set the mlogger ODB /Logger/Channels/NNN/Settings/Filename
to ".run%05dsub%05d.mid.gz" (note the leading dot).

In this mode, active output files will have a filename with a leading dot
(.run00001sub00001.mid.gz) while the file is being written to. After the file is
closed, it is renamed and the leading dot is removed.

To use this function with the lazylogger, please set ODB
"/Lazy/Foo/Settings/Filename format" to "run*.mid.gz,run*.xml" (note the leading
text "run"). Set "stay behind" to 0.

svn rev 5080 (or so, checking by Stefan R.)
K.O.
  773   05 Jul 2011 Konstantin OlchanskiInfomidas shared memory changes
> 2) the shared memory type used by an experiment is recorded in the file .SHM_TYPE.TXT.

An error with creating the file .SHM_TYPE.TXT was corrected in system.c svn rev 5125 - if file did not exist, it is 
created correctly, but MIDAS reports "cannot connect to ODB". Second try works correctly because the file exists 
now.

> 3) the hostname of the computer where the ODB shared memory is meant to reside is now
> recorded in the file .SHM_HOST.TXT.

This is causing problems on mobile computers where "hostname" changes all the time (i.e. set according to 
DHCP on whatever network happens to be connected).

If you run into this problem, keep deleting .SHM_HOST.TXT or use this workaround: disable the hostname check 
by making the file .SHM_HOST.TXT empty (zero length).

K.O.
  774   05 Jul 2011 Konstantin OlchanskiBug ReportMacOS network socket timeouts non-functional
It turns out that because of differences between select() syscall implementation between UNIX (MacOS, 
maybe BSD) and Linux,  network socket timeouts do not work.

This affects timeouts during run transitions (transition calls to dead clients do not timeout), maybe other 
places.

I am looking into fixing this. The main difficulty is with UNIX select() not updating the timeout parameter 
when it is interrupted by the MIDAS watchdog alarm signal. Linux select() subtracts the elapsed time from 
the timeout value and this code from system.c works correctly: while (1) { status = select(..., &timeout); if 
(status==0) break; } (value of timeout becomes smaller each time), while on MacOS it loops forever (value 
of timeout does not change).
K.O.
  775   10 Jul 2011 Konstantin OlchanskiBug Fixmidas shared memory changes
> > 2) the shared memory type used by an experiment is recorded in the file .SHM_TYPE.TXT.
> > 3) the hostname of the computer where the ODB shared memory is meant to reside is now
> > recorded in the file .SHM_HOST.TXT.

Due to a typo in src/system.c svn rev 5125, ss_shm_delete() did not work at all. This broke "odbedit -R", "odbedit -s 5000000" (to change ODB size), etc. 
Fixed in src/system.c svn rev 5134. (It is safe to update just tis one file to fix this problem).

Sorry for the inconvenience,
K.O.
  776   11 Jul 2011 Konstantin OlchanskiBug Fixmidas shared memory changes
> > > 2) the shared memory type used by an experiment is recorded in the file .SHM_TYPE.TXT.
> > > 3) the hostname of the computer where the ODB shared memory is meant to reside is now
> > > recorded in the file .SHM_HOST.TXT.


Because the mserver did not setup correct experiment name and path, POSIX shared memory did not work at all when used with the mserver. Fixed in mserver.c rev 5135


Sorry for the inconvenience,
K.O.
  777   11 Jul 2011 Konstantin OlchanskiInfoMake "STOP" run transition always succeed
Over the years, there was some back-and-forth changes in what happens to run transitions when some 
of the participants misbehave (do not respond to RPC calls, timeout, crash, etc).

The very original behaviour was to ignore all errors. This resulted in user confusion when some clients 
would start, some would not, data from frontends that missed the transition did not arrive, etc.

So it was changed to fail the transition if any client misbehaves.

This left mlogger (who is usually the first one to see the TR_START transition) in a funny state - output 
file is open, etc, but there is no run active. This was fixed by adding a TR_STARTABORT transition to tell 
mlogger, event builder & co that the just started run did not start after all.

Also at some point code was added to forcefully kill clients that do not respond to run transitions (do 
not respond to RPC, timeout, etc).

Recently, it was observed how during unattended overnight operation of a MIDAS DAQ system, with the 
logger set to "auto restart", some unnecessary clients misbehave during the run stop transition, and 
prevent the run from stopping and restarting. The user comes in the morning and is unhappy that data 
taking stopped some time during the night.

midas.c svn rev 5136 changes the TR_STOP transition to always succeed, even if some clients had 
transition errors. If these clients are unnecessary for normal operation of the DAQ, the following run 
"auto restart" will continue taking data. If those were important clients, data taking will continue the 
best it can - it *is* unattended operation - nobody is looking - but users can always setup alarms for 
checking that important clients are always running during data taking. (For very important clients, one 
can setup alarms to send email, send SMS messages, etc).

K.O.
  778   25 Aug 2011 Francesco PrelzForum64-bit integer support in MIDAS
Hi,

I've been doing some preliminary work to use at least the MIDAS
SQL history component for a new CERN experiment (Aegis). I wonder
whether there is any plan to support 64-bit signed/unsigned integer data types
in MIDAS. time_t on 64-bit architectures is actually signed 64-bit
(the 'easy' way to work around the 2038 crisis), and this may be enough to
cause problems.

Thanks.
Francesco Prelz
INFN Milano
  779   05 Sep 2011 John McMillanForumkhyt1331 under scientific linux 5.5?
Hello,
      I'm trying to build khyt1331 under scientific linux 5.5, kernel
2.6.18-238.9.1el5.  Has anyone succeeded with this.  So far, I've
managed to compile by hacking all the references to man9 pages out
of the makefile.  I've then hand installed the kernel driver with 
insmod.  cat /proc/khyt1331 produces 
Hytec 5331 card found at address 0xE800, using interrupt 10
Device not in use
CAMAC crate 0: responding
CAMAC crate 1: not responding
CAMAC crate 2: not responding
CAMAC crate 3: not responding 
and the "addr" LED blinks - so progress of some sort.  
There's no sign of /dev/camac.

Next up I'm going to compile stuff like camactest.c - though the 
makefiles in the drivers folder don't mention these, so I'll have to
work through what is needed by hand.   

At some point I'll have to rewrite a bit so that it all load automatically.

Any hints or tips greatfully received.

John McMillan
  780   12 Dec 2011 Michael MurrayBug Reportbk_delete uses memcpy instead of memmove
In midas.c, the bk_delete function removes a bank by decrementing the total
event size and then copying the remaining banks into the location of the first
using memcpy from string.h.

memcpy is not specified to handle overlapping memory regions (such as MIDAS
banks), though it seems most common implementations do.

memmove should be used instead, which is specified to behave as if copying
through an intermediate buffer.

I noticed the misbehavior using glibc with gcc version 4.4.4 and scientific
linux 6.0. Other gcc versions changed nothing, as this originates from the
implementation of memcpy in libc.

libc version:
GNU C Library stable release version 2.12, by Roland McGrath et al.
Compiled by GNU CC version 4.4.5 20110214 (Red Hat 4.4.5-6).
Compiled on a Linux 2.6.32 system on 2011-12-06.
  781   16 Dec 2011 Konstantin OlchanskiBug Reportbk_delete uses memcpy instead of memmove
> In midas.c, the bk_delete function removes a bank by decrementing the total
> event size and then copying the remaining banks into the location of the first
> using memcpy from string.h.


I confirm the documented difference between memcpy() and memmove() and I confirm the 
questionable use of memcpy() in bk_delete(). I think it should be memmove(). I made it so in my copy 
of midas, so this change will not be lost.

But I am not sure how to test it - I do not think I ever used bk_delete(). I will probably ponder upon 
this and do a blind commit.


K.O.
  782   19 Dec 2011 Stefan RittBug Reportbk_delete uses memcpy instead of memmove
> > In midas.c, the bk_delete function removes a bank by decrementing the total
> > event size and then copying the remaining banks into the location of the first
> > using memcpy from string.h.
> 
> 
> I confirm the documented difference between memcpy() and memmove() and I confirm the 
> questionable use of memcpy() in bk_delete(). I think it should be memmove(). I made it so in my copy 
> of midas, so this change will not be lost.
> 
> But I am not sure how to test it - I do not think I ever used bk_delete(). I will probably ponder upon 
> this and do a blind commit.
> 
> 
> K.O.

It cannot hurt to use memmove(), so please go ahead to commit the changes.

- Stefan
  783   30 Jan 2012 Stefan RittInfoIEEE Real Time 2012 Call for Abstracts
Hello,

I'm co-organizing the upcoming Real Time Conference, which covers also the field of data acquisition, so it might be interesting for people working 
with MIDAS. If you have something to report, you could also consider to send an abstract to this conference.  It will be nicely located in Berkeley, 
California. We plan excursions to San Francisco and to Napa Valley.

Best regards,
Stefan Ritt

---------------------------

18th Real Time Conference
June 11  15, 2012
Berkeley, CA

We invite you to the Hotel Shattuck Plaza in downtown Berkeley, California for
the 2012 Real-Time Conference (RT2012).   It will take place Monday, June 11
through Friday, June 15, 2012, with optional pre-conference tutorials Saturday
and Sunday, June 9-10.

Like the previous editions, RT2012 will be a multidisciplinary conference
devoted to the latest developments on realtime techniques in the fields of
plasma and nuclear fusion, particle physics, nuclear physics and astrophysics,
space science, accelerators, medical physics, nuclear power instrumentation and
other radiation instrumentation.

Abstract submission is open as of 18 January (deadline 2 March). Please visit

http://www.npss-confs.org/rtc/welcome.asp?flag=44675.77&Retry=1 to submit an

abstract.

Call for Abstracts 

RT 2012 is an interdisciplinary conference on realtime data acquisition and
computing applications in the physical sciences. These applications include:

* High energy physics 
* Nuclear physics 
* Astrophysics and astroparticle physics 
* Nuclear fusion 
* Medical physics 
* Space instrumentation 
* Nuclear power instrumentation 
* Realtime security and safety 
* General Radiation Instrumentation 

Specific topics include (but are certainly not limited to) the list shown below.
We welcome correspondence to see how your research fits our venue.   

Key Dates

* Abstract submission opened:  January 18, 2012 
* Abstract deadline:  March 2, 2012 
* Program available: April 2 

Suggested Topics

* Realtime system architectures 
* Intelligent signal processing 
* Programmable devices 
* Fast data transfer links and networks 
* Trigger systems 
* Data acquisition 
* Processing farms 
* Control, monitoring, and test systems 
* Upgrades 
* Emerging realtime technologies 
* New standards 
* Realtime safety and security 
* Feedback on experiences 

Contact Information

If you have a question or wish to opt in for occasional e-mail updates about
RT2012, send us a message at RT2012@lbl.gov. To view full conference
information, visit http://rt2012.lbl.gov/index.html
  784   29 Feb 2012 Konstantin OlchanskiBug ReportProblem with semaphores
Hi there! In the T2K/ND280 experiment in Japan, we keep having problems with MIDAS locking (probably 
of ODB). The symptoms are: some program reports a timeout waiting for the ODB lock, then all programs 
eventually die with this same error. Complete system meltdown. This does not look like the deadlock 
between locks for ODB, cm_msg and the data buffers that I looked into last year. It looks more like 
somebody locks ODB, dies and the Linux kernel fails to unlock the lock (via the SYSV "sem undo" 
function). But it is hard to confirm, hence this message:

The implementation of semaphores in MIDAS (used for locking ODB and the shared memory data buffers) 
uses the straight SYSV semaphore API - which lacks basic debugging features - there is no tracking of 
who locked what when, so if anything at all goes wrong at all, i.e. we are confronted with a timeout 
waiting for the ODB lock, the only corrective action possible is to kill all MIDAS clients and tell the user to 
start from scratch. There is no additional information available from the SYSV semaphore API to identify 
which MIDAS program caused the fault.

The POSIX semaphore API is even worse - no debugging features are available, *and* if a program dies 
while holding a lock, the lock stays locked forever (everybody else will wait forever or see a semaphore 
timeout, and then what?).

So I am looking for an "advanced semaphore library" to use in MIDAS. In addition to the boring functions 
of reliable locking and unlocking, it should support:
- wait with timeout
- remember who is holding the lock
- detect that the process holding the lock is dead and take corrective action (automatic unlock as done by 
SYSV semaphores, call back to user code where we can cleanup and unlock ourselves, etc)
- maybe permit recursive locking (not really required as ODB locks are already made recursive "by hand")
- maybe remember some of the locking history (so we can dump it into a log file when we detect a 
deadlock or other lock malfunction).

Quick google search only find sundry wrappers for SYSV and POSIX semaphores. How they deal with the 
problem of processes locking the semaphore and dying remains a mystery to me (other than telling users 
to remove the Ctrl-C button from their keyboard). BTW, we have seen this problem with several 
commercial applications that use SYSV semaphores but forget to enable the SEM_UNDO function).

Anyhow, if anybody can suggest such an advanced locking library it would be great. Will save me the 
effort of writing one.

K.O.
ELOG V3.1.4-2e1708b5