ELOG Midas

Back Midas Rome Roody Rootana

Midas DAQ System, Page 139 of 145

Not logged in

Find | Login | Help

Full | Summary | Threaded | Hide attachments

2896 Entries

Goto page Previous 1, 2, 3 ... 138, 139, 140 ... 143, 144, 145 Next

Build error with mlogger: invalid conversion from ‘void*’ to ‘gzFile’

KO wrote:

BTW, I read the midas elog via email and if you post html or elcode messages, I receive complete
gibberish. For prompt service, please select message type "plain". (yes, you cannot use fancy colours and
blinking text, but better than me not reading your stuff at all).

BTW2, for easier reading, please include error messages as plain text in your message. As opposed to
compressed attachements.

K.O.

BTW3, if you use a real email program you don't get glibberish. I know some people prefer good-old-text-only pine, but I'm sure you do not use the ascii-only browser lynx to browse the internet, right? So if you browse the web in graphics, why not read your email in graphics as well. Better change yourself than the whole rest of the world Wink

Build error with mana.c while using CERNLIB, svn 3366

If you use CERNLIB to build hmana.o, you may encounter the following error:

src/mana.c: In function ‘write_event_hbook’:
src/mana.c:2881: error: invalid assignment

or somthing like this:

src/mana.c: In function ‘write_event_hbook’:
src/mana.c:2881: warning: target of assignment not really an lvalue; this will be a hard error in the future

So I checked the mana.c and found these lines

2880            /* shift data pointer to next item */
2881            (char *) pdata += key.item_size * key.num_values;

should be changed to

2880            /* shift data pointer to next item */
2881            pdata += key.item_size * key.num_values * sizeof(char) ;

Build error with mana.c while using CERNLIB, svn 3366

Committed, thanks.

Build error due to missing header

I encountered a build error as "sort undefined...". It is caused by missing C++ header <algorithm> in which "sort" is defined. It can be fixed as the attachment.

Environment:
G++: 4.3.4
Platform: Debian Linux testing

> I committed an updated lazylogger with updated documentation. The new version supports subruns and
> can save to external storage arbitrary files (i.e. odb dump files). It also moves most book keeping out of
> odb to permit handling more files on bigger storage disks.
>
> Example lazylogger scripts for castor (CERN) and dcache (TRIUMF) are in the directory "utils".
>
> The lazylogger documentation was updated to remove obsolete information and to describe the new
> functions. As usual "make dox; cd doxfiles/html; firefox index.html" or see my copy at:
>
> http://ladd00.triumf.ca/~olchansk/midas/Utilities.html#lazylogger_task
>
> svn rev 4615, 4616.
> K.O.

Attachment 1: lazylogger.diff

diff --git a/src/lazylogger.c b/src/lazylogger.c
index 34be17a..a7ba4a3 100644
--- a/src/lazylogger.c
+++ b/src/lazylogger.c
@@ -17,6 +17,7 @@ $Id$
 
 #include <vector>
 #include <string>
+#include <algorithm>
 
 #define NOTHING_TODO  0
 #define FORCE_EXIT    1

Build MIDAS debian packages using autoconf/automake.

Here is my story. I deployed several Debian Linux boxes as the DAQ systems in our lab. But I feel it's boring to build and install midas and its related softwares (such as root) on each box. So I need a local debian software repository and put midas and its related packages in it. First of all, I need a midas debian package. After a week's study and searching, I finally finished the job. Hope you feel it useful.

All the work is attached as "daq-midas_deb.tar.gz". The detail is followed. I also created several debian packages. But it's too large to be uploaded. I havn't my own site accessible from internet. So, if you need the debian packages, please give me an accessible ftp or other similar service, then I can upload them to you.

First, I use autoconf/automake to rewrite the building system of MIDAS. You can check it this way:

1. Untar daq-midas_deb.tar.gz somewhere, assumming ~/Temp.
2. cd ~/Temp/daq-midas
3. svn co -r 5065 svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas
4. svn co -r 68 svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
5. cp -rvp debian/autoconf/* ./
6. ./configure --help
7. ./configure <--options>
8. make && make install

Then, I created the debian packages based on the new building files. You need to install root-system package from http://lcg-heppkg.web.cern.ch/lcg-heppkg/debian/. You can build debs this way:

1. untar daq-midas_deb.tar.gz somewhere, assuming ~/Temp.
2. cd ~/Temp/daq-midas
3. svn co -r 5065 svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas
4. svn co -r 68 svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
5. dpkg-buildpackage -b -us -uc

I split the package into serverals parts:

daq-midas-doc -- The documents and references
daq-midas-root -- the midas runtime library and utilities built with root
daq-midas-noroot -- the midas runtime library and utilities built without root
daq-midas-dev-root -- the midas devel files (headers, objects, drivers, examples) built with root
daq-midas-dev-noroot -- the midas devel files (headers, objects, drivers, examples) built without root

Here are the installation:

executalbes -- /usr/lib/daq-midas/bin
library and objs -- /usr/lib/daq-midas/lib
headers -- /usr/lib/daq-midas/include
sources and drivers -- /usr/share/daq-midas/
docs and examples -- /usr/share/doc/daq-midas
mdaq-config -- /usr/bin/mdaq-config

I add an auto-generated shell script -- mdaq-config. It behaves just like "root-config". You can get midas build flags and link flags this way:

gcc `mdaq-config --cflags` -c -o myfe.o myfe.c
gcc `mdaq-config --libs` -o myfe myfe.o `mdaq-config --libdir`/mfe.o

Bugs and suggestions are welcomed.

P.S. Based on debian packages, I am planing to write another script, "mdaq.py":

each midas experiment will be configured in a file named "mdaq.yaml"
mdaq.py reads the configure file and prepare the daq environment, just like "examples/experiment/start_daq.sh"
mdaq.py will handle "start/stop/restart/info" about the daq codes.

The attached "mdaq.py" is the old one.

Attachment 1: daq-midas_deb.tar.gz

Attachment 2: mdaq.py

#!/usr/bin/env python
'''
   Python script to handle daq sessions using MIDAS/PSI.
   Auther: Exaos Lee <Exaos DOT Lee AT gmail DOT com>
   Date:   2009-07-15
'''

import os, os.path
import sys, signal, time
from commands     import getoutput
from ConfigParser import ConfigParser
from pprint       import pprint

class mdaq:
    _codes_ = { 'MSERVER':"MIDAS Server", 'MHTTPD':"MIDAS HTTPD Server",
                'ODBEDIT':"ODB Editor",   'LOGGER':"Data Logger",
                'PROG_FE':"Frontend",   'PROG_ANA':"Analyzer" }
    _env_ = dict()

    def __init__(self,**kws):
        env = dict()
        if kws.has_key('fname') and kws['fname'] != "":
            env = self.parse_conf(kws['fname'])
        self.apply_env(env)

    def parse_conf(self, fname):
        if not os.path.isfile(fname):
            print("**WARNING** File \"%s\" does not exist!"%fname)
            return None

        conf = ConfigParser()
        conf.read(fname)
        env={'CONF_FILE':fname }
        fcpath = os.path.dirname(fname)

        sec = 'EXP_INFO' ## Parsing Experiment Information
        env['EXP_INFO'] = [ self.safe_parse(conf,sec,'Name'),
                            self.safe_parse(conf,sec,'Info'), ]
        msg = self.safe_parse(conf,sec,'DAQ_DIR')
        if msg != "" and os.path.isdir(msg):
            dpath = msg
            if not os.path.isabs(msg):
                dpath = os.path.join(os.path.abspath(fcpath),msg)
            env['DAQ_DIR'] = os.path.normpath(dpath)

        ## Parsing MIDAS configuration
        env['MIDASSYS'] = ["MIDAS System Path",
                           self.safe_parse(conf,"MIDAS","MIDASSYS")]
        if env['MIDASSYS'][1]=="": env['MIDASSYS'][1] = self.get_default_msys()

        ## Parsing Info of programs
        for key in self._codes_.keys():
            code = self.parse_prog_opt(conf, key, fpath=fcpath)
            if code:
                env[key] = [self._codes_[key], ]
                env[key].extend(code)

        port = self.safe_parse(conf, "MHTTPD","PORT")
        if port != "":
            if not env.has_key('MHTTPD'):
                env['MHTTPD'] = ["MIDAS HTTP Server",
                                 os.path.join(env['MIDASSYS'][1],"bin/mhttpd"),
                                 "-h localhost -D"]
            if "-p %s"%port not in env['MHTTPD'][2]:
                env['MHTTPD'][2] = env['MHTTPD'][2] + " -p %s"%port

        return env

    def parse_prog_opt(self, cnf, sec, fpath="./"):
        name = self.safe_parse(cnf, sec, "NAME")
        path = self.safe_parse(cnf, sec, "PATH")
        fn = os.path.join(path,name)
        if name == "": fn = ""
        elif not os.path.isabs(fn):
            fn = os.path.normpath(os.path.join(os.path.abspath(fpath),fn))
        else:
            fn = os.path.normpath(fn)
        opts = self.safe_parse(cnf, sec, "Option")
        if opts != "" or name != "":
            return [fn, opts]

        return None
 
    def safe_parse(self, cnf, sec, opt):
        try:    return cnf.get(sec,opt)
        except: return ""

    def get_default_msys(self):
        if os.environ.has_key('MIDASSYS'): return os.environ['MIDASSYS']
        return "/opt/MIDAS.PSI/Version/Current"

    def apply_env(self, daq_env=dict()):
        env = daq_env
        if not env:
            print("** WARNING** No environment defined! Using the default values. ****")
            env = {'MIDASSYS':['MIDAS System Path', ""], 'CONF_FILE':"", }
        if not env.has_key('MIDASSYS'): env['MIDASSYS'] = ['MIDAS System Path',""]
        if env['MIDASSYS'][1]=="": env['MIDASSYS'][1] = self.get_default_msys()
        def_env = {
            'MSERVER' :["MIDAS Server",
                        os.path.join(env['MIDASSYS'][1],"bin/mserver"),
                        "-D", True],
            'MHTTPD'  :["MIDAS HTTP Server",
                        os.path.join(env['MIDASSYS'][1],"bin/mhttpd"),
                        "-h localhost -D -p 8080", True],
            'ODBEDIT' :["ODB Editor",
                        os.path.join(env['MIDASSYS'][1],"bin/odbedit"),
                        "-c clean", False],
            'LOGGER'  :["Data Logger",
                        os.path.join(env['MIDASSYS'][1],"bin/mlogger"),
                        "", True],
            'PROG_FE' :["Frontend","", "", True],
            'PROG_ANA':["Analyzer","", "", True],
            'EXP_INFO':["test","The test experiment"],
            'DAQ_DIR':os.path.join(os.environ['HOME'],"online/test") }

        for key in def_env.keys():
            if not env.has_key(key): env[key] = def_env[key]

        for key in self._codes_.keys():
            if env[key][1] == "": env[key][1] = def_env[key][1]
            if env[key][2] == "": env[key][2] = def_env[key][2]

        if not os.path.isdir(env['DAQ_DIR']):
            try: os.makedirs(env['DAQ_DIR'])
            except: print("ERROR to make file")

        ## Asign default program options
        exp_opt = "-e %s"%env['EXP_INFO'][0]
        for key in ['LOGGER', 'PROG_FE', 'PROG_ANA', "MHTTPD"]:
            if exp_opt not in env[key][2]:
                env[key][2] = exp_opt + ' ' + env[key][2]
            if "-D" not in env[key][2]:
                env[key][2] = env[key][2] + ' -D'
        if exp_opt not in env['ODBEDIT'][2]:
            env['ODBEDIT'][2] = env['ODBEDIT'][2] + ' ' + exp_opt

        self._env_ = env

    def print_conf(self):
        print "\n================= DAQ Config ====================="
        for key in self._env_.keys():
            print key,": ",
            pprint(self._env_[key])

    def info(self): self.print_conf()

    ## running programs
    def getpids(self, pname):
        return [int(k) for k in getoutput("pidof %s"%pname).strip().split()]

    def kill_prog(self,pname):
        for i in self.getpids(pname): os.kill(i,signal.SIGTERM)

    def run_prog(self, prog):
        if len(prog)>=4 and not prog[3]: return

        if len(prog)>=3 and prog[1]:
            print("Starting %s: %s"%(prog[0],prog[1]))
            os.system("%s %s"%(prog[1],prog[2]))
        else:
            print("%s: no executable exist!"%prog[0])

    def get_status(self,pfile):
        msg = "Not running!"
        pids = self.getpids( os.path.basename(pfile) )
        if pids: return 'PIDs = ' + str(pids)
        return msg

    def echo_status(self,pname,pfile):
        print("%s:\n   %s\n   "%(pname,pfile)
              + self.get_status(pfile))

    ## Session manage: start, stop, status, restart

    def status(self):
        print( "\n===== Session status for experiment \"%s\" =====\n" %
               self._env_['EXP_INFO'][0] )
        for code in ['MSERVER','MHTTPD','LOGGER','PROG_FE','PROG_ANA']:
            self.echo_status(self._env_[code][0], self._env_[code][1])

    def start(self):
        ## Running mserver & mhttpd
        ms_pids = self.getpids(os.path.basename(self._env_['MSERVER'][1]))
        if not ms_pids:
            self.run_prog(self._env_['MSERVER'])
            print("Killing %s ...\n"%self._env_['MHTTPD'][0])
            self.kill_prog(self._env_['MHTTPD'][1])
        if not self.getpids(os.path.basename(self._env_['MHTTPD'][1])):
            self.run_prog(self._env_['MHTTPD'])

        ## Running FE and Analyzer
        old_path = os.getcwd()
        os.chdir( self._env_['DAQ_DIR'] )

        # Init ODB
        print("Initializing ODB for experiment %s ..." % self._env_['EXP_INFO'][0])
        self.run_prog(self._env_['ODBEDIT'])
        time.sleep(1)

        # First stop, then start ...
        self.stop()
        time.sleep(1)

        # Start FE & ANA
        self.run_prog(self._env_['PROG_FE'])
        self.run_prog(self._env_['PROG_ANA'])
        time.sleep(1)

        # Start Logger
        self.run_prog(self._env_['LOGGER'])

        os.chdir(old_path)

    def stop(self):
        for key in ["LOGGER", "PROG_FE", "PROG_ANA"]:
            if self._env_[key][1]:
                print("Killing threads:\n  %s"%self._env_[key][1])
                self.kill_prog(self._env_[key][1])

    def stop_all(self):
        self.stop()
        for key in ['MHTTPD', 'MSERVER']:
            if self._env_[key][1]:
                print("Killing threads:\n  %s"%self._env_[key][1])
                self.kill_prog(self._env_[key][1])

    def restart(self): self.start()


if __name__=='__main__':
    from optparse import OptionParser

    usage = "usage: %prog [options] <command>"
    parser = OptionParser(usage=usage, version="%prog 0.1")
    parser.add_option("-c","--config", metavar="filename",
                      help="Read config from FILENAME")
    cmds = ["start","stop","stop_all","restart","status","info"]

    (opts,args) = parser.parse_args( sys.argv[1:] )
    if not args or len(args)>2:
         parser.print_help()
         print "\nCommands:\n\t",
         pprint(cmds)
         exit()

    fcnf = "daq.conf"
    if opts.config: fcnf = opts.config

    if args[0] in cmds:
        daq = mdaq(fname=fcnf)
        exec( "daq.%s()"%args[0] )
    else:
        parser.print_help()
        print "\nCommands:\n\t",
        pprint(cmds)

Build MIDAS debian packages using autoconf/automake.

> I deployed several Debian Linux boxes as the DAQ systems in our lab. But I
feel it's boring to build and install midas and its related softwares (such as
root) on each box.


Our solution at TRIUMF is to install such packages on a shared NFS filesystem
visible to all client computers. This works well for ROOT and but MIDAS we found
it nearly impossible to keep MIDAS versions in sync between different projects
and expiments, so each experiment uses it's own copy of MIDAS, usually located
in the experiment home directory ($HOME/packages/midas). Because we often need
to make local modifications to MIDAS sources (Makefile, etc), we do not
"install" MIDAS into non-user-writable /usr/local & etc.


> I use autoconf/automake


The promise (premise) of autoconf/automake is to "hide" system dependencies. The
scripts are supposed to automatically probe the build environment and construct
an appropriate Makefile.

In practice, the autotool scripts always have bugs and incorrect assumptions
about the build environment and only work well for a few standardized systems
(RHEL and Debian derivatives) where the differences are so trivial that
autotools is an overkill and a normal Makefile is adequate for the job.

In my experience, as soon as I try to build an autotool-ized package on anything
that does not look like RHEL or Debian, autotool scripts explode and have to be
debugged and kludged by hand. Anybody who has ever done that would agree with me
that one would rather hack the ugliest Makefile than any of the  autotool
generated gibberish.

And of course autotools have never handled cross-compilation in any reasonable
way. Since we do cross-compile MIDAS (for VxWorks and embedded Linux, see "make
crosscompile") a Makefile is required and it so happens that the same Makefile
also works for normal Linux and MacOS, thank you very much.



> Here are the installation:
> [*] executalbes -- /usr/lib/daq-midas/bin
> [*] library and objs -- /usr/lib/daq-midas/lib


Is this in violation of the LSB (or LFS)? I though they mandate that files
controlled by package manager should be /usr/bin/odbedit, /usr/lib64/libmidas.a,
etc (/usr/bin/midas/odbedit no permitted).


> gcc `mdaq-config --cflags` -c -o myfe.o myfe.c


Please check if your config scripts correctly handle the "-m32" and "-m64" flags
- we frequently cross-compile 32-bit MIDAS executables on 64-bit machines.


K.O.

Hi,

I am running a simple FE executable that is supposed to define a PRAW DWORD bank.
The issue is that, right after the start of the run, the logger crashes without messages.
Then the FE reports this error, which is rather confusing.
```
12:59:29.140 2021/05/24 [feTestDatastruct,ERROR] [odb.cxx:6986:db_set_data1,ERROR] "/Equipment/Trigger/Variables/PRAW" is of type UINT32, not UINT32
```

> Hi,
> 
> I am running a simple FE executable that is supposed to define a PRAW DWORD bank.
> The issue is that, right after the start of the run, the logger crashes without messages.
> Then the FE reports this error, which is rather confusing.
> ```
> 12:59:29.140 2021/05/24 [feTestDatastruct,ERROR] [odb.cxx:6986:db_set_data1,ERROR] "/Equipment/Trigger/Variables/PRAW" is of type UINT32, not UINT32
> ```

I think this is fixed in latest midas. There was a typo in this message, the same tid was printed twice,
with result you report "mismatch UINT32 and UINT32", instead of "mismatch of UINT32 vs what is actually there".

This fixes the message, after that you have to manually fix the mismatch in the data type in ODB (delete old one, I guess).

K.O.

Best MIDAS branch/version for "production"

Hello!
My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
University (Japan). I also happen to be a T2K collaborator.
I am currently developing the DAQ software for a T2K near detector called WAGASCI 
(different from ND280) and we recently decided to adopt MIDAS as a user 
interface.
Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
the remote repository every now and then with my local copy and that is fine ... 
but on the 25th of April our experiment is officially starting and I was 
wondering which version of MIDAS should I use for "production".
So my question is:
For a running experiment that needs software stability what branch of the MIDAS 
repository is better suited? The master branch or the develop branch? Moreover, 
what point in time do you think is more stable?
Best regards
Giorgio

Best MIDAS branch/version for "production"

Hi, Giorgio - you are asking excellent questions. I will try to answer them, but as ever, there are no 
easy answers.

In general, the top of the midas "develop" branch is "the best midas there is".

So for a new experiment, it is a reasonable place to start. Of course you can see
that there is quite a bit of commit activity going on, however, most substantial
changes are done on separate branches, where we try the new code, debug
it and test it. Only when the new code is "ready", we commit it to the "develop"
branch. Then, most often, we find some more last minute post-merge bugs,
and fix them right then and there on the head of the "develop" branch. Eventually
the dust settles and we have stable code that stays stable for a long time.

For example, right now we are waiting for the dust to settle on the change
of the MIDAS URL scheme, which was necessitated by needs of several experiments
that have more-complex-then-usual https proxy configurations. Unusual today,
but more common as we move forward, I think.

So if the head of the "develop" branch does not work for you, we encourage
that you file a bug report (here on this forum or on the bitbucket issue tracker).

While we try to sort out your problem, you can fall back to a previous version
of midas:

a) go back to one of the older "midas-YYYY-MM-X" tags
b) go back to one of the release candidate branches "feature/midas-YYYY-MM".

But if you have an already running system and you already have a working
instance of MIDAS, you do not have to update it to the latest version
unless you need some newest feature or you suffer from a bug
that has been fixed in a newer version.

In general, I find that it is fairly safe to update a working instance of MIDAS to
the latest code. But do keep your old working copy, if there is trouble, you can
always go back to something that works.

Now to your questions:

> For a running experiment that needs software stability what branch of the MIDAS 
> repository is better suited?

easy to answer. the most stable is the version you are using right now (doh!). each time
you upgrade, there is a risk that something will go wrong.

if you start from scratch, use the head of the "develop" branch (the latest and greatest),
if you run into trouble, report the trouble and update to a newer version with your
trouble fixed, or go back in time to previous tags and release candidate branches (as I described 
above).

> The master branch or the develop branch?

the "develop" branch.

> Moreover, what point in time do you think is more stable?

I find that it is impossible to have a stable-stable-stable version of midas
because the rest of the world flows forward in time. Old versions of midas
stop compiling because of OS and compiler changes; they stop running
because of OS or web browser or hardware changes. Then somebody
always asks for new features and new or old bugs surface constantly.

But we try to mark some "good" spots using git tags and release branches,
however the more far you go back in time, the more likely the code will
not even compile anymore.


K.O.



> Hello!
> My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
> University (Japan). I also happen to be a T2K collaborator.
> I am currently developing the DAQ software for a T2K near detector called WAGASCI 
> (different from ND280) and we recently decided to adopt MIDAS as a user 
> interface.
> Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
> the remote repository every now and then with my local copy and that is fine ... 
> but on the 25th of April our experiment is officially starting and I was 
> wondering which version of MIDAS should I use for "production".
> So my question is:
> For a running experiment that needs software stability what branch of the MIDAS 
> repository is better suited? The master branch or the develop branch? Moreover, 
> what point in time do you think is more stable?
> Best regards
> Giorgio

Best MIDAS branch/version for "production"

Hi Konstantin,
thank you for the very in-depth answer. Right now I am using the latest version of MIDAS (the head of the 
develop branch) and I have not noticed any bugs or problems in the MIDAS code, except the ones that I 
myself put in the part of the code that I am developing of course.
So I will do as you suggest and continue to use the head of the develop branch.
Thank you again for taking the time to answer all my questions!

PS other than the code peculiar to our experiment, I have made a little modification to the MIDAS install 
Makefile (I noticed that there is an "install" target but not an "uninstall" target so I wrote it).
If you are interested, I could make a merge request on BitBucket, just let me know.

Bests
Giorgio

> Hi, Giorgio - you are asking excellent questions. I will try to answer them, but as ever, there are no 
> easy answers.
> 
> In general, the top of the midas "develop" branch is "the best midas there is".
> 
> So for a new experiment, it is a reasonable place to start. Of course you can see
> that there is quite a bit of commit activity going on, however, most substantial
> changes are done on separate branches, where we try the new code, debug
> it and test it. Only when the new code is "ready", we commit it to the "develop"
> branch. Then, most often, we find some more last minute post-merge bugs,
> and fix them right then and there on the head of the "develop" branch. Eventually
> the dust settles and we have stable code that stays stable for a long time.
> 
> For example, right now we are waiting for the dust to settle on the change
> of the MIDAS URL scheme, which was necessitated by needs of several experiments
> that have more-complex-then-usual https proxy configurations. Unusual today,
> but more common as we move forward, I think.
> 
> So if the head of the "develop" branch does not work for you, we encourage
> that you file a bug report (here on this forum or on the bitbucket issue tracker).
> 
> While we try to sort out your problem, you can fall back to a previous version
> of midas:
> 
> a) go back to one of the older "midas-YYYY-MM-X" tags
> b) go back to one of the release candidate branches "feature/midas-YYYY-MM".
> 
> But if you have an already running system and you already have a working
> instance of MIDAS, you do not have to update it to the latest version
> unless you need some newest feature or you suffer from a bug
> that has been fixed in a newer version.
> 
> In general, I find that it is fairly safe to update a working instance of MIDAS to
> the latest code. But do keep your old working copy, if there is trouble, you can
> always go back to something that works.
> 
> Now to your questions:
> 
> > For a running experiment that needs software stability what branch of the MIDAS 
> > repository is better suited?
> 
> easy to answer. the most stable is the version you are using right now (doh!). each time
> you upgrade, there is a risk that something will go wrong.
> 
> if you start from scratch, use the head of the "develop" branch (the latest and greatest),
> if you run into trouble, report the trouble and update to a newer version with your
> trouble fixed, or go back in time to previous tags and release candidate branches (as I described 
> above).
> 
> > The master branch or the develop branch?
> 
> the "develop" branch.
> 
> > Moreover, what point in time do you think is more stable?
> 
> I find that it is impossible to have a stable-stable-stable version of midas
> because the rest of the world flows forward in time. Old versions of midas
> stop compiling because of OS and compiler changes; they stop running
> because of OS or web browser or hardware changes. Then somebody
> always asks for new features and new or old bugs surface constantly.
> 
> But we try to mark some "good" spots using git tags and release branches,
> however the more far you go back in time, the more likely the code will
> not even compile anymore.
> 
> 
> K.O.
> 
> 
> 
> > Hello!
> > My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
> > University (Japan). I also happen to be a T2K collaborator.
> > I am currently developing the DAQ software for a T2K near detector called WAGASCI 
> > (different from ND280) and we recently decided to adopt MIDAS as a user 
> > interface.
> > Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
> > the remote repository every now and then with my local copy and that is fine ... 
> > but on the 25th of April our experiment is officially starting and I was 
> > wondering which version of MIDAS should I use for "production".
> > So my question is:
> > For a running experiment that needs software stability what branch of the MIDAS 
> > repository is better suited? The master branch or the develop branch? Moreover, 
> > what point in time do you think is more stable?
> > Best regards
> > Giorgio

Best MIDAS branch/version for "production"

> 
> PS other than the code peculiar to our experiment, I have made a little modification to the MIDAS install 
> Makefile (I noticed that there is an "install" target but not an "uninstall" target so I wrote it).
> If you are interested, I could make a merge request on BitBucket, just let me know.
> 

Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.

K.O.



> Bests
> Giorgio
> 
> > Hi, Giorgio - you are asking excellent questions. I will try to answer them, but as ever, there are no 
> > easy answers.
> > 
> > In general, the top of the midas "develop" branch is "the best midas there is".
> > 
> > So for a new experiment, it is a reasonable place to start. Of course you can see
> > that there is quite a bit of commit activity going on, however, most substantial
> > changes are done on separate branches, where we try the new code, debug
> > it and test it. Only when the new code is "ready", we commit it to the "develop"
> > branch. Then, most often, we find some more last minute post-merge bugs,
> > and fix them right then and there on the head of the "develop" branch. Eventually
> > the dust settles and we have stable code that stays stable for a long time.
> > 
> > For example, right now we are waiting for the dust to settle on the change
> > of the MIDAS URL scheme, which was necessitated by needs of several experiments
> > that have more-complex-then-usual https proxy configurations. Unusual today,
> > but more common as we move forward, I think.
> > 
> > So if the head of the "develop" branch does not work for you, we encourage
> > that you file a bug report (here on this forum or on the bitbucket issue tracker).
> > 
> > While we try to sort out your problem, you can fall back to a previous version
> > of midas:
> > 
> > a) go back to one of the older "midas-YYYY-MM-X" tags
> > b) go back to one of the release candidate branches "feature/midas-YYYY-MM".
> > 
> > But if you have an already running system and you already have a working
> > instance of MIDAS, you do not have to update it to the latest version
> > unless you need some newest feature or you suffer from a bug
> > that has been fixed in a newer version.
> > 
> > In general, I find that it is fairly safe to update a working instance of MIDAS to
> > the latest code. But do keep your old working copy, if there is trouble, you can
> > always go back to something that works.
> > 
> > Now to your questions:
> > 
> > > For a running experiment that needs software stability what branch of the MIDAS 
> > > repository is better suited?
> > 
> > easy to answer. the most stable is the version you are using right now (doh!). each time
> > you upgrade, there is a risk that something will go wrong.
> > 
> > if you start from scratch, use the head of the "develop" branch (the latest and greatest),
> > if you run into trouble, report the trouble and update to a newer version with your
> > trouble fixed, or go back in time to previous tags and release candidate branches (as I described 
> > above).
> > 
> > > The master branch or the develop branch?
> > 
> > the "develop" branch.
> > 
> > > Moreover, what point in time do you think is more stable?
> > 
> > I find that it is impossible to have a stable-stable-stable version of midas
> > because the rest of the world flows forward in time. Old versions of midas
> > stop compiling because of OS and compiler changes; they stop running
> > because of OS or web browser or hardware changes. Then somebody
> > always asks for new features and new or old bugs surface constantly.
> > 
> > But we try to mark some "good" spots using git tags and release branches,
> > however the more far you go back in time, the more likely the code will
> > not even compile anymore.
> > 
> > 
> > K.O.
> > 
> > 
> > 
> > > Hello!
> > > My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
> > > University (Japan). I also happen to be a T2K collaborator.
> > > I am currently developing the DAQ software for a T2K near detector called WAGASCI 
> > > (different from ND280) and we recently decided to adopt MIDAS as a user 
> > > interface.
> > > Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
> > > the remote repository every now and then with my local copy and that is fine ... 
> > > but on the 25th of April our experiment is officially starting and I was 
> > > wondering which version of MIDAS should I use for "production".
> > > So my question is:
> > > For a running experiment that needs software stability what branch of the MIDAS 
> > > repository is better suited? The master branch or the develop branch? Moreover, 
> > > what point in time do you think is more stable?
> > > Best regards
> > > Giorgio

Best MIDAS branch/version for "production"

> Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.


... and change the documentation accordingly (Suzannah!?). Installing midas these days does not really make sense, since normally only one 
users uses it on a given machine.

Stefan

Best MIDAS branch/version for "production"

> > Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.
> 
> 
> ... and change the documentation accordingly (Suzannah!?). Installing midas these days does not really make sense, since normally only one 
> users uses it on a given machine.
> 
> Stefan

I understand. Anyway, I preferred to install MIDAS in the /opt/midas folder to be consistent with the other programs (Pyrame and Calicoes from LLR) that we are using for our experiment (they are all installed in the /opt folder). Moreover, I am using Linux systemd to enable mhttpd on startup (and other handy features like auto-restart after a crash). And quite unfortunately CentOS doesn't support to start systemd units as a non-root user.
So in my particular case, perhaps it made some sense to install MIDAS in a folder other than the source code folder.

Giorgio

Best MIDAS branch/version for "production"

> > Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.
> 
> 
> ... and change the documentation accordingly (Suzannah!?). Installing midas these days does not really make sense, since normally only one 
> users uses it on a given machine.
> 
> Stefan

I understand. Anyway, I preferred to install MIDAS in the /opt/midas folder to remain consistent with the other programs that we are using for 
our experiment (Pyrame and Calicoes from LLR). I am also using Linux systemd to enable mhttpd on startup (and other handy features like auto-
restart after a crash) and unfortunally CentOS doesn't support to enable systemd units as a non-root user.
So in my particular case, perhaps it made some sense to install MIDAS in a folder other than the source code folder
Giorgio

Best MIDAS branch/version for "production"

> > > Hmm... for most experiments, we do not "install" midas. I should probably remove the "install" target from the Makefile.
>
> install MIDAS in the /opt/midas folder to remain consistent with the other programs that we are using for 
> our experiment (Pyrame and Calicoes from LLR).

I see. Would this work as well? Instead of "make install" do this:
su - root
cd /opt
git pull midas
cd midas
make
add /opt/midas/linux/bin to your PATH. (is it time to get rid of the "linux" part from the default build path?!?)

> I am also using Linux systemd to enable mhttpd on startup

We use cron @reboot to start mhttpd.

But. CentOS7 systemd cron unit file has a bug and they run @reboot cron jobs before NIS and autofs is ready, see
http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_crontab_.40reboot_for_MIDAS_.28CentOS7.29

Can you post your systemd unit file to this elog, others may find it useful.

K.O.

Best MIDAS branch/version for "production"

> I see. Would this work as well? Instead of "make install" do this:
> su - root
> cd /opt
> git pull midas
> cd midas
> make
> add /opt/midas/linux/bin to your PATH. (is it time to get rid of the "linux" part from the default build path?!?)

Got it. I will do that in the future.

> Can you post your systemd unit file to this elog, others may find it useful.

[Unit]
Description=MIDAS data acquisition system
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=3
User=neo
ExecStart=/opt/midas/bin/mhttpd -e WAGASCI --http 8081 --https 8444
Environment="MIDASSYS=/opt/midas" "MIDAS_EXPTAB=/home/neo/Code/WAGASCI/MIDAS/online/exptab" "MIDAS_EXPT_NAME=WAGASCI" 
"SVN_EDITOR=emacs -nw" "GIT_EDITOR=emacs -nw"
PassEnvironment=MIDASSYS MIDAS_EXPTAB MIDAS_EXPT_NAME SVN_EDITOR GIT_EDITOR

[Install]
WantedBy=multi-user.target

Bash Script for handling an experiment code

I rearanged the files in "examples/experiment" as the attached "mtest_exp.zip". I re-write the start/stop script as the attached "daq.sh". The script "daq.sh" can be re-used for many experiments. The user only needs to provide an script "daq_env.sh" as the following containing the settings for the experiment environment.

#!/bin/sh

[ ! "$MIDASSYS" ] && MIDASSYS=/opt/MIDAS.PSI/Version/Current
[ ! "$HTTPPORT" ] && HTTPPORT=8080
[ ! "$SRVHOST" ]  && SRVHOST=localhost
LOGGER=${MIDASSYS}/bin/mlogger

EXPPATH=/home/das/online/test
CODEPATH=${EXPPATH}/code
LOGGER=${MIDASSYS}/bin/mlogger

PROG_FE=${CODEPATH}/frontend
PROG_ANA=${CODEPATH}/analyzer

if [ ! "$MIDAS_EXPTAB" ]; then
	MIDAS_DIR=${EXPPATH}
else
	MIDAS_EXPT_NAME="test"
fi

I hope this can be helpful. Smile

There seem to be some problems such as:
1. When several experiments are defined, the $LOGGER may be not the one used for this exp.
2. The "pidof" may be not in some platforms, so this script is limited.

Hope anybody can help me to improve it for general purpose. All my best!

Attachment 1: daq.sh

#!/bin/sh

DAQ_SETENV=./daq_env.sh

[ -f "${DAQ_SETENV}" ] && source ${DAQ_SETENV}

### Start ....
start() 
{
    #### Running mserver
    pidmsrv=`pidof mserver`
    if [ ! "$pidmsrv" ]; then
	echo Starting mserver ...
	$MIDASSYS/bin/mserver -D -m
	pidmhttpd=`pidof mhttpd`
	[ "$pidmhttpd" ] && kill -9 $pidmhttpd
	echo Starting mhttpd ...
	$MIDASSYS/bin/mhttpd -h localhost -D -p $HTTPPORT
    fi

    #### Change work directory
    OLD_DIR=`pwd`
    cd ${EXPPATH}
    
    #### Running FE and Analyzer ...

    # Options for multi experiments running ...
    [ "$MIDAS_EXPTAB" ] && OPTIONS="-e $MIDAS_EXPT_NAME"

    $MIDASSYS/bin/odbedit ${OPTIONS} -c clean

    sleep 2

    echo Starting frontend session: ${PROG_FE} ...
    ${PROG_FE}  ${OPTIONS} -D

    echo Starting analyzer session: ${PROG_ANA} ...
    ${PROG_ANA} ${OPTIONS} -D

    sleep 1

    echo Starting logger utility: mlogger ...
    ${LOGGER} ${OPTIONS} -D

    cat <<EOF
Please point your web browser to http://localhost:$HTTPPORT
Or run: firefox http://localhost:$HTTPPORT
To look at live histograms, run: roody -Hlocalhost

EOF

    #### Resume old directory
    cd ${OLD_DIR}
}

### Stop ....
stop() 
{
    #### Kill Running sessions
    pidlog=`pidof $(basename ${LOGGER})`
    pidfe=` pidof $(basename ${PROG_FE})`
    pidana=`pidof $(basename ${PROG_ANA})`
    [ "$pidlog" ] && kill $pidlog
    [ "$pidfe"  ] && kill $pidfe
    [ "$pidana" ] && kill $pidana
}

### Status ....
status()
{
    echo "MIDAS Server (mserver): "
    pid=`pidof mserver`
    echo_status $pid

    echo "MIDAS HTTPD Server (mhttpd): "
    pid=`pidof mhttpd`
    echo_status $pid

    echo "Logger (${LOGGER}): "
    pid=`pidof $(basename ${LOGGER})`
    echo_status $pid

    echo "Frontend Code ($PROG_FE): "
    pid=`pidof $(basename ${PROG_FE})`
    echo_status $pid

    echo "Analyzer Code ($PROG_ANA): "
    pid=`pidof $(basename ${PROG_ANA})`
    echo_status $pid
}

echo_status()
{
    if [ "$1" ]; then
	echo -e "\t Running as $1."
    else
	echo -e "\t Not Running!"
    fi
}

### Usage ....
usage()
{
    echo "Usage: $0 [<start>|<stop>|<restart>|<status>]"
}

###################### Main Procedure ###########################
case "${1:-''}" in 
    "start")
	start
	;;
    "stop")
	stop
	;;
    "restart")
	stop
	start
	;;
    "status")
	status
	;;
    *)
	usage
	;;
esac

#########################  The END  #############################

Attachment 2: mtest_exp.zip

Hello everyone,

It seems that if a frontend is started automatically by using Program->Auto start then the status page does not show it as started. This is since the FE name has a number after the name. If I stop and start manually then the status page shows the correct state of the FE. Am I doing something wrong or is this a bug somewhere?

thanks,
Zaher

> It seems that if a frontend is started automatically by using Program->Auto start then the status page does not show it as started. This is since the FE name has a number after the name. If I stop and start manually then the status page shows the correct state of the FE. Am I doing something wrong or is this a bug somewhere?

Zaher, please read https://daq00.triumf.ca/elog-midas/Midas/919

K.O.

Goto page Previous 1, 2, 3 ... 138, 139, 140 ... 143, 144, 145 Next

ELOG V3.1.4-2e1708b5