Hi,

I had to apply the attached patch to convince SuSE Linux 12.2 to compile mana.c
gcc version is "(SUSE Linux) 4.6.2"

Problem is that gz{write,close, etc.} expect a 1st argument of type gzFile (see
zlib.h), whereas out_file is FILE*. In fact, out_file is a cast to FILE*, even
in the case when we work on a gzfile (HAVE_ZLIB).

Could you please confirm that the patch is correct, and possibly apply it to trunk?

I haven't checked if mana works as advertised now.

Cheers,


Randolf

The simplest thing is probably to delete all files .[A-Z]*.SHM in the odb directory (the
one you specified in /etc/exptab).
This wipes the ODB, shared memory and all the other obscure stuff, giving you a clean,
fresh start.

Of course it wipes all the valuable stuff, too. That's why it's handy to sometimes open
odbedit and "save odb_<yyyymmdd>.odb". You can reload the thing after such a fatal 
"rm .[A-Z]*.SHM" 

And my 2ct:

Go for git!

I've been using git since 2007 or so, after cvs and svn. Git has some killer features which I can't miss any more:

* No central repo. Have all the history with you on the train.
* Branching and merging, with stable branches and feature branches.
  Happy hacking while my students do analysis on a stable version.
  Or multiple development branches for several features.
  And merging really works, including fixing up merge conflicts.
* "git bisect" for finding which commit introduced a (reproducible) bug.
* "gitk --all"

I use git for everything: Software, tex, even (Ooffice) Word documents.

Go for git. :-)

Randolf

Hi Konstantin,

> > * No central repo. Have all the history with you on the train.
> > * Branching and merging, with stable branches and feature branches.
> >   Happy hacking while my students do analysis on a stable version.
> >   Or multiple development branches for several features.
> 
> This is the part that worries me the most. Without a "central" "authoritative" repository,
> in just a few quick days, everybody will have their own incompatible version of midas.

No! This is probably one of the biggest misunderstandings of the git workflow.

You can of course _define_ one central repo: This is the one that you and Stefan decide to be "the source" (as
Linus does for the kernel). It's like the central svn repo: Only Stefan and you can push to it, and everybody
else will pull from it. Why should I pull MIDAS from some obscure source, when your "public" repo is available.

Look at the Linux Kernel: Linus' version is authoritative, even though everybody and his best friend has his
own kernel repo.

So, the main workflow does not change a lot: You collect patches, commit them, and "push" them to the central
repo. All users "pull" from this central repo. This is very much what svn offers.

> 
> I guess I am okey with your private midas diverging from mainstream, but when *I* end up
> with 10 different incompatible versions just in *my* repository, can that be good?

See above: _You_ define what the central repo is.

But: I _bet_ you will very soon have 10 versions in your personal repo, because _you choose_ to do so. It's
just SO much easier. The non-linear history with many branches is a _feature_. I can't live without it any more:


Looking at my MIDAS analyzer:

I have a "public" repo in /pub/git/lamb.git. This is where I publish my analyzer versions. All my collaborators
pull from this.

Then I have my personal repo in ~/src/lamb. 
This is where I develop. When I think something is ready for the public, I merge this branch into the public repo. 

Whenever I start to work on a new feature, I create a branch in my _local_ repo (~/src/lamb).  I can fiddle and
play, not affecting anybody else, because it never sees the public repo.
OK, collaborator A finds a bug. I switch to my local copy of the public version, fix the bug, and push the fix
to the publix repo. Then I go back to my (local) feature branch, merge the bug fix, and continue hacking.
Only when the feature is ready, I push it to the public repo.

Things get moe interesting as you work on several features simultaneously. You have e.g. 3 topic branches:
(a) is nearly ready, and you want a bunch of people to test it.
    push branch "feature (a)" to the public repo and tell the people which branch to pull.
(b) is WIP, you hack on it without affecting (a).
(c) is bug fixes which may or may not affect (a) or (b).
And so on.

You will soon discover the beauty of several parallel branches.

Plus, git merges are SO simple that you never think about "how to merge"

> 
> >   And merging really works, including fixing up merge conflicts.
> 
> But somebody still has to do it. With a central repository, the problem takes care of
> itself - each developer has to do their own merging - with svn, you cannot commit
> to the head without merging the head into your code first. But with git, I can just throw
> my changes int some branch out there hoping that somebody else would do the merging.
> But guess what, there aint anybody home but us chickens. We do not have a mad finn here
> to enforce discipline and keep us in shape...

See above: You will have the exact same workflow in git, if you like.




> As an example, look at the HADOOP/HDFS code development, they have at least 3 "mainstream"
> branches going, neither has all the features combined together and each branch has bugs with
> the fixes in a different branch. What a way to run a railroad.

I haven't look at this. All I can say: Branches are one of the best features.

> 
> > * "git bisect" for finding which commit introduced a (reproducible) bug.
> > * "gitk --all"
> >
> > Go for git. :-)
> 
> Absolutely. For me, as soon as I can wrap my head around this business of "who does all the merging".

Easy: YOU do it.

Keep going as in svn: Collect patches, and send them out.

And then, try "git checkout -b my_first_branch", hack, hack, hack,
"git merge master".

Best,

Randolf


> 
> K.O.

> > * "git bisect" for finding which commit introduced a (reproducible) bug.
> 
> I did not know this command, so I read about it. This IS WONDERFUL! I had once (actually with MSCB) the case that a bug was introduced i the last 100 
> revisions, but I did not know in which. So I checked out -1, -2, -3 revisions, then thought a bit, then tried -99, -98, then had the bright idea to try -50, then 
> slowly converged. Later I realised that I should have done a binary search, like -50, if ok try -25, if bad try -37, and so on to iteratively find the offending 
> commit. Finding that there is a command it git which does this automatically is great news.

even more so considering the nonlinear history (due to branching) in a regular git repo.

I'm looking into using MIDAS for an experiment that creates one large event
(20MB or more) every second.

Q1: It looks like I should use EQ_FRAGMENTED. Has this feature been in use
recently? Is it known to work/not work?

More specifically, the computer should initiate a 1 second data taking, start to
such the data out of the electronics (which may take a while), change some
experimental parameters, and start over. 

Q2: What's the best way to do this? EQ_PERIODIC? 
I cannot guarantee that the time required to read the hardware has an upper bound.
In a standalone-prog I would simply use a big loop and let the machine execute
it as fast as it can: 1.1s, 1.5s, 1.1s, 1.3s, 2.5s, ..... depending on the HW
deadtimes.
Will this work with EQ_PERIODIC?

(Sorry for these maybe stupid questions, but I have so far only used MIDAS for
externally generated events, with <32kB event size).


Thanks a lot,

Randolf

Works, and here is how I did it. The attached example is based on the standard MIDAS
example in "src/midas/examples/experiment". 

My somewhat unsorted notes, haven't really tweaked the numbers. But it WORKS.

(1) mlogger writes "last.xml" (hard-coded!) which takes an awful amount of time
    as it writes the complete ODB containing the 10MB bank!
    just outcomment 
       // odb_save("last.xml");
    in mlogger.cxx, function 
    INT tr_start(INT run_number, char *error)
    (line ~3870 in mlogger rev. 5377, .cxx-file included)

(2) frontend.c:
     * the most important declarations are

/* BIG_DATA_BYTES is the data in 1 bank
   BIG_EVENT_SIZE is the event size. It's a bit larger than the bank size
                  because MIDAS needs to add some header bytes, I think
 */

#define BIG_DATA_BYTES  (10*1024*1024)   // 10 MB
#define BIG_EVENT_SIZE  (BIG_DATA_BYTES + 100)

/* maximum event size produced by this frontend */
INT max_event_size = BIG_EVENT_SIZE;

/* maximum event size for fragmented events (EQ_FRAGMENTED) */
INT max_event_size_frag = 5 * BIG_EVENT_SIZE;

/* buffer size to hold 10 events */
INT event_buffer_size = 10 * BIG_EVENT_SIZE;


     * bk_init() can hold at most 32kByte size events! Use bk_init32() instead.

     * complete frontend.c is attached

(3) in an xterm do
    # . setup.sh
    # odbedit -s 41943040
          (first invocation of odbedit must create large enough odb,
           otherwise you'll get "odb full" errors)
(4) odbedit> load big.odb      
    (attached). Essentials are:

    /Experiment/MAX_EVENT_SIZE = 20971520
    /Experiment/Buffer sizes/SYSTEM = 41943040   <- at least 2 events!

    To avoid excessive latecies when starting/stopping a run, do
    /Logger/ODB Dump = no 
    /Logger/Channels/0/Settings/ODB Dump = no 
     
    and create an Equipment Tree to make the mlogger happy

(5) a few more xterms, always ". setup.sh":
    # mlogger_patched (see (1))
    # ./frontend  (attched)

(6) in your odbedit (4) say "start". You should fill your disk rather quickly.

> > When I compile current cvs-head midas, I get errors about undefined function
> > min(). I do not think min() is in the list of standard C functions, so
> > something else should be used instead, like a MIN(a,b) macro. To make life
> > more interesting, in a few places, there is also a variable called "min".
> > Here is the error:
> > 
> > src/mana.c: In function `INT write_event_ascii(FILE*, EVENT_HEADER*, 
> >    ANALYZE_REQUEST*)':
> > src/mana.c:2571: `min' undeclared (first use this function)
> > src/mana.c:2571: (Each undeclared identifier is reported only once for each 
> >    function it appears in.)
> > make: *** [linux/lib/rmana.o] Error 1
> 
> This is really a miracle to me. The min/max macros are defined both in midas.h
> and msystem.h and worked the last ten years or so. However, I agree that macros
> should follow the standard and use capital letters, so I changed that.

The problem is that /usr/include/c++/3.*/bits/stl_algobase.h contains 
#undef min
#undef max

and in C++ with STL one should really use something like this
    std::min<INT>(a,b)


Cheers
  Piotr

Is the midas CVS server set-up so that I can pull the newest 
version off the CVS server?

What would be my CVSROOT?
pserver:anoncvs@midas.psi.ch:/cvs/midas *this did not work* :)

Piotr

 

Hi,
 I am really puzzled: I am running the very same as far as sources
are concerned (Dec 12, 2003 snapsot) midas frontend (miniexp + camacnul)
on two different machines (and the same trusted private network):

1) one is an ancient Pentium/100 MHz laptop with RedHat Linux 7.3 and 
2) another one is event more ancient MVME167 25MHz running VxWorks 5.4.2

The front end on my Linux PC works just fine, whereas on the MVME167
I get intermittent crashes (most often at the end of the run).
[Correction: the crashes happen, I think, when the frontend wants 
to update the ODB]

The crashes happen in db_set_record routines

Any ideas what might be wrong? 
Except that MVME167 is a piece of ...#@!% 

Piotr

A followup: I traced back the problem to version 1.9.2.

Version 1.9.1 does not have this problem but 1.9.2 does. 
For now I stick with 1.9.1

Piotr
 

> Starting from midas version 1.9.4 on, the environment variable 'MIDASSYS'
> should be defined and point to the installation directory of midas. The
> purpose of that is that add-on packages (like the upcoming ROME system) can
> find the midas libraries and include files. It is excatly the same as for
> ROOT which defines ROOTSYS and should therefore be straight forward. The
> libraries should then reside in $MIDASSYS/lib (or %MIDASSYS%\lib under windows).
> 
> To remind users about this new variable, a test has been added to odbedit,
> which shows a warning when starting odbedit and MIDASSYS is not defined.

1. Finally! It's about time to do that! 

2. What will the entire structure tree look like?

Here's my suggestion
MIDASSYS=/opt/midas-1.9.4 (for example)   


so the Linux binaries would go to 
MIDASHOST=i386-pc-linux-gnu
$MIDASSYS/$MIDASHOST/bin
$MIDASSYS/$MIDASHOST/lib

the VxWorks binaries
MIDASHOST=m68k-wrs-vxworks
$MIDASSYS/$MIDASHOST/bin
$MIDASSYS/$MIDASHOST/lib

and the shared stuff would go to 
$MIDASSYS/include
$MIDASSYS/share/drivers
$MIDASSYS/share/examples

The Makefile would need to be adjusted (for make install) but that is not
too complicated

What do you think?

Regards
  Piotr

> I guess we should follow the "standard" as much as possible. MIDASSYS was inspired by
> ROOTSYS. Now where do people usually install ROOT? Is it /opt/root-x.x.x or something
> else. Some years ago (when I did the last time some linux administration) optional
> packages were put into /usr/local by default. I guess you have more experience with
> today's tradition, so do whatever you thing is standard.
I agree that we should follow the standard. 
I used /opt as an example. 
There are several "schools" as to where put things my philosophy is
/usr/{bin,lib,include}       - std OS packages (RPMS, .deb or whatever your flavor likes)
/usr/local/{bin,lib,include} - make/make install packages
/opt/..                      - additional packages (RPMS, ...) 

But it should be up to the user what $MIDASSYS she/he likes.

> 
> > so the Linux binaries would go to 
> > MIDASHOST=i386-pc-linux-gnu
> > $MIDASSYS/$MIDASHOST/bin
> > $MIDASSYS/$MIDASHOST/lib
> 
> Does that mean that the path has to be modified to include $MIDASSYS/$MIDASHOST/bin?
> If we put a link to /usr/local/bin, the path does not have to be modified. What about
> shared libraries? Does ldconfig know about /usr/local/lib, or $MIDASYS/$MIDASHOST/lib?
The path could/should be modified in users .bashrc/.tcshrc or we could provide a simple
system-wide script(s) that would do the job.
For years, I've been using such a scenario on my Linux PCs with regards to various
add-on packages (e.g. cern). 

Here's an example of my cern.sh that goes into /etc/profile.d on my RedHat Linux PC
#===================================
. /etc/profile.d/.functions
export CERN=/cern
export CERN_LEVEL=pro
addpath $CERN/$CERN_LEVEL/bin
#===================================

As for library path: there are several ways (as with exec path)
a) nice way: modify /etc/ld.so.conf by adding $MIDASYS/$MIDASHOST/lib
b) modifying LD_LIBRARY_PATH (there's some security issues with it)
c) symlinking to /usr/local/lib


> 
> What about /usr/share? Is that a common place for documentatino etc?
Yes. Check any recent Linux distribution /usr/share is full of docs, icons, etc.

This is my bias. 

I (obviously) prefer packing things into rpm which makes install/updates 
very easy - especially if you are managing several machines.

Cheers
    Piotr

Hi,
  I think that rather than spending too much time on where to 
put files and how to define the environment - I am guilty of that myself.
We should be perhaps have some discussion on the future of MIDAS.

Are we ready for 2.0? 
Stefan - do you have any ideas/enhancements?

1) For one I would like to explore memory mapping (mmap()) on Linux
- I've used it once upon a time on DEC OSF/1 and I found it really
nice compared to shared memory. 
From a user standpoint it behaves as a shared memory but is mapped 
to a real file that can be easily "removed" when neccessary. 
One really annoying thing in MIDAS is when it goes ballistic
the cleanup which is somewhat tricky. 
The question if there is any performance penalty associated

2) Expanding hardware support: 
   a) custom microcontrolers?
   b) more hardware
   c) how about a "standard" Linux device /dev/midas
   for various PCI cards (PCI<->CAMAC) (PCI<->VME) 

3) I have never really seen a midas deployment that uses interrupts. 
I do understand the ease of polling and the fact that these days
CPU's are cheap but sometimes it is important to use interrupts.
Any examples/experience?

?)

Piotr

Sorry the previous message got mangled:

Hi, 
I think that rather than spending too much time on where to put files 
and how to define the environment - I am guilty of that myself -  we should 
perhaps have some discussion on the future of MIDAS. 

Are we ready for 2.0? 

Stefan - do you have any ideas/enhancements? 

1) For one I would like to explore memory mapping (mmap()) on Linux.
I've used it once upon a time on DEC OSF/1 and I found it really nice 
compared to shared memory. From a user standpoint it behaves as a shared
memory but is mapped to a real file that can be easily "removed" when 
neccessary. One really annoying thing in MIDAS is when it goes ballistic 
the cleanup is somewhat tricky. 
The question if there is any performance penalty associated 

2) Expanding hardware support: 
  a) custom microcontrolers? 
  b) other hardware
  c) how about a "standard" Linux device /dev/midas for various 
  PCI cards (PCI<->CAMAC) (PCI<->VME) 

3) I have never really seen a midas deployment that uses interrupts. 
I do understand the ease of polling and the fact that these days CPU's 
are cheap but sometimes it is important to use interrupts. 

Any examples/experience? 

?) 

Piotr

Hi,
 has anyone tried runnning midas frontend on a Linux running 
on a Motorola MVME167 motorola embedded CPU?
I have seen people running Linux on a MV167 
(http://www.sleepie.demon.co.uk/linuxvme/)
so in principle this can be done.

The reason I am asking is that we have a lot of them in house 
and we would like to avoid paying for VxWorks
(I have succesfully run Midas on a mvme167/VxWorks node)

Or maybe one has come up with a much better solution 
[short of dumping mv167 into a sewer :)]

Piotr

Hi there:
I have a question if there's anybody out there running MIDAS with event builder
that assembles events from more that just a few front ends (say on the order of
0x10 or more)?
Any experiences with scalability?

Cheers
 Piotr

Hi all,
thank you for all responses. 

It seems that there's no problem running MIDAS with event builder assembling
data from ~10 front-ends. How about ~100? One possible solution is to have a
multi-tiered architecture. 

The reason I am asking is that we are in the process of designing an Ethernet
based DAQ system with front-ends running on embedded computers (Linux/ARM
CPU/Xilinix FPGA) and MIDAS is one of my options as a DAQ framework.
I am open for advice/suggestions.

Thanks again
  Piotr

Hello!
My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
University (Japan). I also happen to be a T2K collaborator.
I am currently developing the DAQ software for a T2K near detector called WAGASCI 
(different from ND280) and we recently decided to adopt MIDAS as a user 
interface.
Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
the remote repository every now and then with my local copy and that is fine ... 
but on the 25th of April our experiment is officially starting and I was 
wondering which version of MIDAS should I use for "production".
So my question is:
For a running experiment that needs software stability what branch of the MIDAS 
repository is better suited? The master branch or the develop branch? Moreover, 
what point in time do you think is more stable?
Best regards
Giorgio

Hi Konstantin,
thank you for the very in-depth answer. Right now I am using the latest version of MIDAS (the head of the 
develop branch) and I have not noticed any bugs or problems in the MIDAS code, except the ones that I 
myself put in the part of the code that I am developing of course.
So I will do as you suggest and continue to use the head of the develop branch.
Thank you again for taking the time to answer all my questions!

PS other than the code peculiar to our experiment, I have made a little modification to the MIDAS install 
Makefile (I noticed that there is an "install" target but not an "uninstall" target so I wrote it).
If you are interested, I could make a merge request on BitBucket, just let me know.

Bests
Giorgio

> Hi, Giorgio - you are asking excellent questions. I will try to answer them, but as ever, there are no 
> easy answers.
> 
> In general, the top of the midas "develop" branch is "the best midas there is".
> 
> So for a new experiment, it is a reasonable place to start. Of course you can see
> that there is quite a bit of commit activity going on, however, most substantial
> changes are done on separate branches, where we try the new code, debug
> it and test it. Only when the new code is "ready", we commit it to the "develop"
> branch. Then, most often, we find some more last minute post-merge bugs,
> and fix them right then and there on the head of the "develop" branch. Eventually
> the dust settles and we have stable code that stays stable for a long time.
> 
> For example, right now we are waiting for the dust to settle on the change
> of the MIDAS URL scheme, which was necessitated by needs of several experiments
> that have more-complex-then-usual https proxy configurations. Unusual today,
> but more common as we move forward, I think.
> 
> So if the head of the "develop" branch does not work for you, we encourage
> that you file a bug report (here on this forum or on the bitbucket issue tracker).
> 
> While we try to sort out your problem, you can fall back to a previous version
> of midas:
> 
> a) go back to one of the older "midas-YYYY-MM-X" tags
> b) go back to one of the release candidate branches "feature/midas-YYYY-MM".
> 
> But if you have an already running system and you already have a working
> instance of MIDAS, you do not have to update it to the latest version
> unless you need some newest feature or you suffer from a bug
> that has been fixed in a newer version.
> 
> In general, I find that it is fairly safe to update a working instance of MIDAS to
> the latest code. But do keep your old working copy, if there is trouble, you can
> always go back to something that works.
> 
> Now to your questions:
> 
> > For a running experiment that needs software stability what branch of the MIDAS 
> > repository is better suited?
> 
> easy to answer. the most stable is the version you are using right now (doh!). each time
> you upgrade, there is a risk that something will go wrong.
> 
> if you start from scratch, use the head of the "develop" branch (the latest and greatest),
> if you run into trouble, report the trouble and update to a newer version with your
> trouble fixed, or go back in time to previous tags and release candidate branches (as I described 
> above).
> 
> > The master branch or the develop branch?
> 
> the "develop" branch.
> 
> > Moreover, what point in time do you think is more stable?
> 
> I find that it is impossible to have a stable-stable-stable version of midas
> because the rest of the world flows forward in time. Old versions of midas
> stop compiling because of OS and compiler changes; they stop running
> because of OS or web browser or hardware changes. Then somebody
> always asks for new features and new or old bugs surface constantly.
> 
> But we try to mark some "good" spots using git tags and release branches,
> however the more far you go back in time, the more likely the code will
> not even compile anymore.
> 
> 
> K.O.
> 
> 
> 
> > Hello!
> > My name is Giorgio Pintaudi and I am a Ph.D. student at Yokohama National 
> > University (Japan). I also happen to be a T2K collaborator.
> > I am currently developing the DAQ software for a T2K near detector called WAGASCI 
> > (different from ND280) and we recently decided to adopt MIDAS as a user 
> > interface.
> > Now I am using the "develop" branch of the MIDAS BitBucket repository: I merge 
> > the remote repository every now and then with my local copy and that is fine ... 
> > but on the 25th of April our experiment is officially starting and I was 
> > wondering which version of MIDAS should I use for "production".
> > So my question is:
> > For a running experiment that needs software stability what branch of the MIDAS 
> > repository is better suited? The master branch or the develop branch? Moreover, 
> > what point in time do you think is more stable?
> > Best regards
> > Giorgio