Back Midas Rome Roody Rootana
  Midas DAQ System, Page 1 of 45  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
Entry  06 Jun 2003, Pierre-André Amaudruz, , Welcome 
Dear Midas users,

As you certainly aware, ELOG (Electronic Logbook) has been written
by Stefan Ritt and its functionality is part of the Midas package too.
This web site using Elog is replacing the W-Agora Forum previously setup.

You will need to register to this forum in order to gain Write access and 
possible Email notification.

We would like to encourage you to post your questions or comments at
this Midas Elog site instead of using private Email to the authors as your 
remarks are surely of interest to the other users too.

 
 
Entry  27 Aug 2003, Pierre-André Amaudruz, , Operation under 1.9.3 with the analyzer rmidas-mlogger_root_file.jpg
1) Prior upgrading midas to 1.9.3, make sure you've saved your ODB in ASCII
   format using "odbedit> save my_odb.odb", as the internal structure is
   incompatible with previous version. You will be able to restore it once
   the new odb is up using "odbedit> load my_odb.odb".

2) since version 1.9.2, the analyzer supports ROOT and PAW packages.
   The general Midas makefile build the analyzer core system mana.c
   differently depending on presence of the environment variable $ROOTSYS.

   In the case $ROOTSYS is not defined, the Makefile will create:
   ~/os/lib/mana.o, build for NO HBOOK calls.
   ~/os/lib/hmana.o, build with HBOOK calls for PAW analyzer
    (requires /cern/pro/lib to be present).

   In the case $ROOTSYS is defined and pointing to a valid root directory:
   ~/os/lib/mana.o, build for NO HBOOK calls.
   ~/os/lib/rmana.o, build for ROOT analyzer.

3) Since 1.9.2, the ~/examples/experiment contains the ROOT
   analyzer example instead of HBOOK. The local Makefile uses the source
   examples and the ~/os/lib/rmana.o for building the final user
   application.
   
   The previous HBOOK(PAW) analyzer has been moved into ~examples/hbookexpt
   directory. The analyzer is build using the ~/os/lib/hmana.o 

4) A new application "rmidas" is available when the system is build with
   ROOT support. This application is an initial "pure" ROOT GUI implementing
   TSocket for remote ROOT histogram display. 
   Once a ONLINE ROOT analyzer is up and running, by invoking "rmidas"
   you will be prompt for a host name. Enter the node name hosting the
   analyzer. You will be presented with a list of histogram which can
   be display in a ROOT frame environment (see attachment). 

5) The support of ROOT is also available for the logger by changing  
   the data format and the destination file name in the ODB structure.
   This option will save on file the Midas banks converted into ROOT Tree.
   This file can be opened with ROOT (see attachment).

------- ODB structure of /Logger/Channels/0/Settings
   [local:midas:R]Settings>ls
    Active                          y
    Type                            Disk
    Filename                        run%05d.root    <<<<<<<<< new extension
    Format                          ROOT            <<<<<<<<< new format
    Compression                     0
    ODB dump                        y
    Log messages                    0
    Buffer                          SYSTEM
    Event ID                        -1
    Trigger mask                    -1
    Event limit                     0
    Byte limit                      0
    Tape capacity                   0
    Subdir format                   
    Current filename                run00211.root
-------   

.
Entry  11 Aug 2003, Konstantin Olchanski, , Alarm on no ping? 
I want midas alarms to go off when I cannot ping arbitrary remote hosts. Is
there is easy/preferred way to do this? K.O.
    Reply  18 Dec 2003, Stefan Ritt, , Alarm on no ping? 
> I want midas alarms to go off when I cannot ping arbitrary remote hosts. Is
> there is easy/preferred way to do this? K.O.

There are "internal alarms" with type AT_EVALUATED. Just find a program 
where you can put some code which gets periodically executed (like the idle 
loop in the frontend), and so something like:

DWORD last = 0;

  if (ss_time() > last+60)
    {
    last = ss_time();

    /* do a ping via socket(), bind() and connect() */
    ...

    if (status != CM_SUCCESS)
      al_trigger_alarm("XYZ Ping", str, "Warning", 
                       "Host is dead", AT_INTERNAL);
    }

Pierre does the same thing in lazylogger.c, just have a look. I don't know 
how to do a ping correctly in C, I guess you have to send an UDP packet 
somewhere, but I never did it. If you find it out, please post it.


 
Entry  14 Jan 2004, Razvan Stefan Gornea, , Access to hardware in the MIDAS framework 
I am just starting to explore MIDAS, i.e. reading the manual and trying 
some examples. For the moment I would like to make a simple frontend that 
access a portable multimeter through RS-232 port. I think this could help 
me understand how to access hardare inside MIDAS framework. Initially I've 
started from the MiniFE.c example and tried to initialize the serial port 
on run start transition and build a readout loop in the main function. I 
know that this is not a full frontend but I was just interested in getting 
some experience with the drivers available in the distribution, in this 
case RS-232. The portable multimeter is very simple in principle, one just 
has to configure the port settings and then send character 'R' and read 14 
ASCII characters from the device. Unfortunately I could not understand how 
to invoke the driver services so I changed and started again with the 
slowcont/frontend.c example. From this example and after reading the "Slow 
Control System" section in the MIDAS manual I think that all I need to do 
is to define my own equipment structure based on the multi.c class driver 
with a single input channel (and replace the null driver with the RS-232).

Here I got stuck. I see from the code source that there is a relationship 
between drivers at all levels (even bus) and the ODB but I don't yet fully 
understand how they work. Actually for a couple of days now I am in a loop 
going from class to device to bus and then back again to class drivers 
trying to see how to create my own device driver and especially how to call 
the bus driver. It could be that the framework is invoking the drivers and 
the user just has to configure things ... up to now I didn't dare to look 
at the mfe.c.

Is there a more detailed documentation about slow control and drivers then 
the MIDAS manual? What is the data flow through the three layers system for 
drivers? What is the role of the framework and what is left to the user 
choice?

Thanks
    Reply  14 Jan 2004, Stefan Ritt, , Access to hardware in the MIDAS framework 
There is some information at

http://midas.triumf.ca/doc/html/Internal.html#Slow_Control_system

and at

http://midas/download/course/course_rt03.zip , file "part1.ppt", expecially 
page 59 and page 62 "writing your own device driver".

So what you are missing for your application is a "device driver" for your 
multimeter. The only function it has to implement is the function CMD_INIT 
where you initialize the RS232 port, and the funciton CMD_GET, which sends 
a "R" and reads the value. Now you have two options:

1) You implement RS232 calls directly in your device driver

You link against rs232.c and directly call rs232_init() at the inizialization, 
then call rs232_write() and rs232_read() where you read your 14 ASCII 
characters.

2) You call a "bus driver" in your device driver

This method makes the device driver independent of the underlying transport 
interface. So if your next multimeter accepts the same "R" command over 
Ethernet, you can just replace the RS232 bus driver by the TCPIP bus driver 
without having to change your device driver. But I guess that method 2) is not 
worth for such a simple device like your multimeter.

So take nulldev.c or dastemp.c as your starting point, put some RS232 
initialization into the init routine and the communication via "R" into 
the "get" routine. The slow control frontend, driven by mfe.c, should then 
regularly read your multimeter and the value should appear in the ODB. Take 
the examples/slowcont/frontend.c as an example, and adjust the multi_driver[] 
list to use your new device driver (instead of the nulldev).

I would like to mention that the usage of midas only makes sense for some 
experiemnts which require event based readout, using VME or CAMAC crates. If 
your only task is to read out some devices which are called "slow control 
equipment" in the midas language, then you might be better of with labview or 
something.
       Reply  16 Jan 2004, Razvan Stefan Gornea, , Access to hardware in the MIDAS framework frontend.cmeterdev.c
The multimeter device is indeed to simple to use MIDAS but I am just trying 
it as a learning experience. The DAQ system to develop involves VME crates 
and general purpose I/O boards. The slow control part, especially accessing 
the I/O boards seem to me more complex then the VME access. I want to 
understand very well the "correct" way of using the MIDAS slow control 
framework before starting the project.

I chose the second method and created a meterdev.c driver (essentially a 
copy of the nulldev.c) where I changed the init. function and the get 
function. I am not sending a "INIT ..." string because for this device it 
is useless. In the get function I send a "D" and read my string. I changed 
the frontend of the example to have a new driver list (in the first try I 
eliminated the Output device but the ODB got corrupted, I guess the class 
multi needs to have defined output channels). The output channel is linked 
with nulldev and null (I guess this is like if they would not be present).

The result is strange because the get function is called all the time very 
fast (much faster then the 9 seconds as set in the equipment) and even 
before starting the run (I just put the flag RO_RUNNING).

Thanks for any help
          Reply  17 Jan 2004, Stefan Ritt, , Access to hardware in the MIDAS framework 
> The result is strange because the get function is called all the time very 
> fast (much faster then the 9 seconds as set in the equipment) and even 
> before starting the run (I just put the flag RO_RUNNING).

This is on purpose. When the frontend is idle, it loops over the slow control 
equipment as fast as possible. This way, you see changes in your hardware very 
quickly. I see no reason to waste CPU cycles in the frontend when there are 
better things to do like reading slow control equipment. Presume you have the 
alarm system running, which turns off some equipment in case of an over 
current. You better do this as quickly as possible, not wasting up to 9 
seconds each time.

The 9 seconds you mention are for reading *EVENTS*. You have double 
functionality: First, reading the slow control system, writing updated values 
to the ODB, where someone else can display or evaluate them (in the alarm 
system for example). Second, assemble events and sending them with the other 
data to disk or tape. Only the second one gets controlled by RO_RUNNING and 
the 9 seconds. You can see this by the updating event statists on your 
frontend display, which increments only when running and then every 9 seconds.
Entry  10 Mar 2004, Jan Wouters, , Creation of secondary Midas output file. dance193.tar
Dear Midas Team,

I have run into a problem with Midas and was wondering if you could explain what I 
am doing wrong.  I have included a simple demo to illustrate what I am doing and 
can send a small input data file if needed.

WHAT I AM TRYING TO DO:
Every midas event for the DANCE experiment consists of many physics events.  I am 
trying to create a secondary mid file where the event boundaries are now the 
physics events rather than the midas events.  This secondary mid file will be 
analyzed using a second stage midas analyzer.

For the demo, I use the data from EV02 (one of our 15 frontends), which consists of a 
variable number of fixed length structures where each structure contains the data for 
one crystal from the DANCE detector. 
 I treat each crystal as a separate physics event and write it out in the TREK bank, 
which is a demo calculated output bank, as a separate event.   

(The only difference between this demo and our real system is that we would include 
all the crystals from the other frontends that have approximately the same time stamp 
in the output bank.  Thus the output bank would consist of a varing number of 
crystals in one event rather than the fixed one crystal per event used in this demo.)

THE CHANGES TO analyzer.c AND adccalib.c
I loop through the EV02 bank examining each crystal structure in turn.  I calculate 
"calibrated" parameters and put them into an output bank called TREK.  The unusual 
part of this example is that the TREK bank is no longer part of the main list of input 
banks, ana_trigger_bank_list[].   Instead it is now part of a new bank list called 
ana_physics_bank_list[].  See the analyzer.c file for this definition.

In adccalib.c I  create the space for this new bank as follows. 

	EVENT_HEADER 	gPhysicsEventHeaders[ MAX_EVENT_SIZE / sizeof( 
EVENT_HEADER ) ];  
	WORD* 		gPhysicsEventData = ( WORD * )( gPhysicsEventHeaders + 1 );		

In the adc_calib routine I create the bank header as follows.  Note that the serial 
numbers will restart at 0 at the beginning of each midas event.  Should I let the serial 
number increment monotonically until the end of the run?:

	gPhysicsEventHeaders->serial_number = (DWORD) - 1;
	gPhysicsEventHeaders->event_id = 2;
	gPhysicsEventHeaders->trigger_mask = 0;
	gPhysicsEventHeaders->time_stamp = pheader->time_stamp;

In a loop that loops through all the crystals contained in EV02,  I extract each crystal, 
calibrate it, and store it in a TREK structure.  In creating the TREK bank I assume that 
each one will be a separate physics event thus I update the event serial number and 
use bk_init32 to initialize the memory.   

   	for ( short i = 0; i < nItems; i++ )
  	{	++(gPhysicsEventHeaders->serial_number);  	// Update serial number.
  		bk_init32( gPhysicsEventData );		// Initialize storage.
  		bk_create( gPhysicsEventData, "TREK", TID_STRUCT, &trek );
  	
  		trek->one = (double) pev->areahg * 1.0;
  		trek->two = (float) pev->timelo * 1.0;

  		bk_close( gPhysicsEventData, trek+1 );
  		
  		pev++; 					// Loop to next crystal's data. 
	}	

The output bank should consist of multiple events for each individual EV02 midas 
input event. 

 As far as I can tell the code compiles and runs fine, but I get no data in the .mid 
output file except for the ODB. I have a print statement at the beginning of each 
midas event stating how many crystals were found in the EV02 bank.  I also print out 
the calibrated value for each crystal as it is being placed in its own TREK output 
bank.  The data appears correct.

 I cannot place TREK in the input bank the way it normally is done in the examples 
because there is not a one-to-one correspondence between a midas event and a 
true physics event.  Instead one midas event has many physics events.  Thus the 
output bank needs to be in a new memory area so that I can create a custom header 
and increment the serial number properly for each event.  Our follow-on analysis 
using a second Midas analyzer only needs to analyze one physics event at a time 
rather than one Midas event at a time, which is why we are going to all the trouble to 
get this paradigm working.

I include all the code for this very simple example. 

RUNNING THE CODE:
To run the example just use the run01220.mid file I will send:

./analyzer -i run01220.mid.gz -o run01220out.mid -c settings.odb_cfg -n 50

The only thing done by the settings.odb_cfg file is to turn on the TREK output bank.  I 
have verified that the bank is on.

SUMMARY:
I believe that I must not be creating the new TREK output bank correctly so that 
midas understands that the event-by-event calculated physics data should be written 
out event-by-event.  I have pointed out several places in the above discussion where 
I might be making a mistake.

I would like to get both this example running and a similar which create Root trees, 
though the Root trees are of secondary importance.  With this example I can finish 
writing the second stage analyzer and get the DANCE collaboration moving forward 
with their analysis.  Currently, we cannot use this paradigm because I cannot create 
a secondary mid file in our stage one analysis.  I would be very grateful if you could 
take a look at this example and tell me what I am doing incorrectly.

Jan
    Reply  10 Mar 2004, Stefan Ritt, , Creation of secondary Midas output file. adccalib.c
Dear Jan,

I had a look at your code. You create a gPhysicsEventHeader array, fill it, and expect the 
framework to write it to disk. But how can the framework "guess" that you want your private 
global array being written? Unfortunately it cannot do magic!

Do do what you want, you have to write a "secondary" midas file yourself. I modified your 
code to do that. First, I define the event storage like

BYTE           gSecEvent[ MAX_EVENT_SIZE ];
EVENT_HEADER   *gPhysicsEventHeader = (EVENT_HEADER *) gSecEvent;
WORD* 	       gPhysicsEventData = ( WORD * )( gPhysicsEventHeader + 1 );		

I use gSecEvent as a BYTE array, since it only contains one avent at a time, so this is more 
appropriate. Then, in the BOR routine, I open a file:

  sprintf(str, "sec%05d.mid", run_number);
  sec_fh = open(str, O_CREAT | O_RDWR | O_BINARY, 0644);

and close it in the EOR routine

  close(sec_fh);

The event routine now manually fills events into the secondary file:

      /* write event to secondary .mid file */
      gPhysicsEventHeader->data_size = bk_size(gPhysicsEventData);
      write(sec_fh, gPhysicsEventHeader, sizeof(EVENT_HEADER)+bk_size(gPhysicsEventData));

Note that this code is placed *inside* the for() loop over nItems, so for each detector you 
create and event and write it.

That's all you need, the full file adccalib.c is attached. I tried to produce a sec01220.mid 
file and was able to read it back with the mdump utility.

Best regards,

  Stefan
    Reply  11 Mar 2004, Renee Poutissou, , Creation of secondary Midas output file. 
Jan , 

Do you need to log this stage 1 output?  If not, you would use the 
eventbuilder mechanism to create your stage 2 events.  
I use the eventbuilder mechanism with success for my TWIST experiment.

Renee
Entry  21 Jun 2004, Piotr Zolnierczuk, , FAQ: anonymous cvs access? 
Is the midas CVS server set-up so that I can pull the newest 
version off the CVS server?

What would be my CVSROOT?
pserver:anoncvs@midas.psi.ch:/cvs/midas *this did not work* :)

Piotr

 
    Reply  21 Jun 2004, Pierre-André Amaudruz, , FAQ: anonymous cvs access? 
> Is the midas CVS server set-up so that I can pull the newest 
> version off the CVS server?
> 
> What would be my CVSROOT?
> pserver:anoncvs@midas.psi.ch:/cvs/midas *this did not work* :)
> 
> Piotr
> 
>  

In the Midas doc under "Quick Start"
http://midas.triumf.ca/doc/html/quickstart.html
you will find the proper cvs command for accessing the latest cvs Midas
version. The public pwd is cvs. You will only be able to checkout/update the
package.
       Reply  22 Jun 2004, Exaos Lee, , FAQ: anonymous cvs access? 
> In the Midas doc under "Quick Start"
> http://midas.triumf.ca/doc/html/quickstart.html
> you will find the proper cvs command for accessing the latest cvs Midas
> version. The public pwd is cvs. You will only be able to checkout/update the
> package.

I cannot checkout module:
------------
01:52:16: pc2075.psi.ch: Operation timed out
01:52:16: cvs [checkout aborted]: end of file from server (consult above
messages if any)
------------

Could anybody add download tar package in the WWW interface of CVS repository.
I know the original CGI script has such a feature. Thanks.
    Reply  29 Jun 2004, Konstantin Olchanski, , FAQ: anonymous cvs access? 
> Is the midas CVS server set-up so that I can pull the newest 
> version off the CVS server?
> 
> What would be my CVSROOT?
> pserver:anoncvs@midas.psi.ch:/cvs/midas *this did not work* :)

I use:

setenv CVS_RSH ssh
cvs -d cvs@midas.psi.ch:/usr/local/cvsroot checkout midas

It works most of the time. Sometimes I get locking and "permission denied"
errors on some subdirectories, presumably when CVS is being reorganized or
when there are stale locks. Does not happen too often.

K.O.
Entry  09 Jul 2004, Stefan Ritt, , Version 1.9.4 released today 
Version 1.9.4 of midas has been released today. It is mainly a maintenance
update, for all the little things which have been fixed since the last
release, and does not contain major new functionality.
Entry  15 Jul 2004, Stefan Ritt, , Severe bug in 1.9.4 
Hello midas'ers,

Today I discovered a severe bug in the routine bm_check_buffers(), which
causes the logger to crash when it stops a run due to a reached event limit.
The funny thing is that this bug was there since the beginning, but only
recent versions of gcc and libc reveal it.

Since I consider this severe, I fixed it and updated 1.9.4 just now. I did
not go with 1.9.4-1, but maybe in future we should consider patch levels.

So please everybody who uses 1.9.4 and has problems with crashing loggers,
please update to 1.9.4 from today (July 15th, 2004).

- Stefan
Entry  14 Jul 2004, Piotr Zolnierczuk, , future direction discussion? 
Hi,
  I think that rather than spending too much time on where to 
put files and how to define the environment - I am guilty of that myself.
We should be perhaps have some discussion on the future of MIDAS.

Are we ready for 2.0? 
Stefan - do you have any ideas/enhancements?

1) For one I would like to explore memory mapping (mmap()) on Linux
- I've used it once upon a time on DEC OSF/1 and I found it really
nice compared to shared memory. 
From a user standpoint it behaves as a shared memory but is mapped 
to a real file that can be easily "removed" when neccessary. 
One really annoying thing in MIDAS is when it goes ballistic
the cleanup which is somewhat tricky. 
The question if there is any performance penalty associated

2) Expanding hardware support: 
   a) custom microcontrolers?
   b) more hardware
   c) how about a "standard" Linux device /dev/midas
   for various PCI cards (PCI<->CAMAC) (PCI<->VME) 

3) I have never really seen a midas deployment that uses interrupts. 
I do understand the ease of polling and the fact that these days
CPU's are cheap but sometimes it is important to use interrupts.
Any examples/experience?

?)

Piotr
    Reply  14 Jul 2004, Piotr Zolnierczuk, , future directions discussion? 
Sorry the previous message got mangled:

Hi, 
I think that rather than spending too much time on where to put files 
and how to define the environment - I am guilty of that myself -  we should 
perhaps have some discussion on the future of MIDAS. 

Are we ready for 2.0? 

Stefan - do you have any ideas/enhancements? 

1) For one I would like to explore memory mapping (mmap()) on Linux.
I've used it once upon a time on DEC OSF/1 and I found it really nice 
compared to shared memory. From a user standpoint it behaves as a shared
memory but is mapped to a real file that can be easily "removed" when 
neccessary. One really annoying thing in MIDAS is when it goes ballistic 
the cleanup is somewhat tricky. 
The question if there is any performance penalty associated 

2) Expanding hardware support: 
  a) custom microcontrolers? 
  b) other hardware
  c) how about a "standard" Linux device /dev/midas for various 
  PCI cards (PCI<->CAMAC) (PCI<->VME) 

3) I have never really seen a midas deployment that uses interrupts. 
I do understand the ease of polling and the fact that these days CPU's 
are cheap but sometimes it is important to use interrupts. 

Any examples/experience? 

?) 

Piotr
    Reply  14 Jul 2004, Stefan Ritt, , future direction discussion? 
Have changed your entry as Non-HTML (easier to reply to...)

Here are some "initial" comments, by no means complete...

> Are we ready for 2.0? 
> Stefan - do you have any ideas/enhancements?

A big thing along the horizon I see is the ROME environment
(http://midas.psi.ch/rome/). So we will move away from PAW to ROOT. Although
the DAQ part will stay untouched, the whole analysis back-end changes,
including some XML configuration and MySQL support. I guess that would justify
a 2.0. I will discuss this at TRIUMF when I come in September, see how useful
ROME is for other users...

> 1) For one I would like to explore memory mapping (mmap()) on Linux
> - I've used it once upon a time on DEC OSF/1 and I found it really
> nice compared to shared memory. 
> From a user standpoint it behaves as a shared memory but is mapped 
> to a real file that can be easily "removed" when neccessary. 
> One really annoying thing in MIDAS is when it goes ballistic
> the cleanup which is somewhat tricky. 
> The question if there is any performance penalty associated

I guess there are no performance penalties, since under the hood both
techniques are handled similarly. The problem is that besides share memories
you need also semaphores to controll exclusive access to the memory, either
shm() funcitons or mmap(), so this would only fix half of the problem. I seem
to remember that mmap() was not available on some Ultix systems or so, but I
guess that's obsolete by now...

> 2) Expanding hardware support: 
>    a) custom microcontrolers?
>    b) more hardware
>    c) how about a "standard" Linux device /dev/midas
>    for various PCI cards (PCI<->CAMAC) (PCI<->VME) 

Well, you cannot develop hardware support for hardware you don't have, so the
policy up to now was that everyone developing some special drivers or hardware
support contributed it to the package. About c), we have already a CAMAC
driver standard, but at the user space level, so I don't see the benefit of
having kernel-mode standardized drivers. The only difference will be that the
debugging will be harder. VME standard is there in a kind of poor start right
now, but I expect to finish it this fall.

As for a), there is the MSCB system (http://midas.psi.ch/mscb) which has midas
support on the device driver and bus driver level, but I learned that
distributing hardware (or PCB designs if you like) is much harder than sharing
software. 

> 3) I have never really seen a midas deployment that uses interrupts. 
> I do understand the ease of polling and the fact that these days
> CPU's are cheap but sometimes it is important to use interrupts.
> Any examples/experience?

It's not only the "ease" of polling, but also that it's faster (in almost all
cases) and less troublesome. But hey, interrupt support is included in mfe.c,
so if you are fanatic about interrupts, please feel free to use them.
       Reply  15 Jul 2004, Konstantin Olchanski, , future direction discussion? 
> > Are we ready for 2.0? 

I disapprove of version number inflation. Why not go straight for midas version
3000-Pro-Z?

> A big thing along the horizon I see is the ROME environment
> (http://midas.psi.ch/rome/). So we will move away from PAW to ROOT. Although
> the DAQ part will stay untouched, the whole analysis back-end changes,
> including some XML configuration and MySQL support.

I looked at the ROME slides from Pierre, and it seems to suffer badly from the
second-system syndrome (read The Mythical Man-Month).

For us, it is important to get the data into a form where we can process it with
ROOT and I would prefer if we could concentrate in improving the (embryonic) ROOT
online analysis capabilities.

> > 1) For one I would like to explore memory mapping (mmap())

It is trivial to replace System-V shared memory with mmap(). I am surprised that
it has not been done yet. System-V semaphores are a little bit harder to get rid of.

> > 2) Expanding hardware support: 
> >    a) custom microcontrolers?
> >    b) more hardware
> >    c) how about a "standard" Linux device /dev/midas
> >    for various PCI cards (PCI<->CAMAC) (PCI<->VME) 

We cannot expect MIDAS Authors to provide drivers for all possible hardware.

At best, we can mount an effort to collect exisiting drivers from all MIDAS users
"out there" and to integrate them into MIDAS.

Even that is highly non-trivial- many drivers use non-portable native hardware
access interfaces (direct bit-banging on PPCs, VMIC library on VMIC Linux
machines, etc). We have already failed to create an efficient generic portable VME
access library.

> As for a), there is the MSCB system (http://midas.psi.ch/mscb)

I never saw the point of having tons of MIDAS code for MSCB hardware that nobody
has and nobody will ever have.

> > 3) I have never really seen a midas deployment that uses interrupts. 
>
> It's not only the "ease" of polling, but also that it's faster (in almost all
> cases) and less troublesome. But hey, interrupt support is included in mfe.c,
> so if you are fanatic about interrupts...

Interrupts are important when one cannot afford chewing up CPU cycles, memory
cycles, PCI cycles and VME cycles on polling.

As I understand, common CAMAC hardware does not generate interrupts- this explains
lack of examples and lack of interrupt use. At TRIUMF, our new VME hardware
supports interrupts and I have a VMIC-based setup where I can (and intend to) test
MIDAS support of interrupts.

K.O.
Entry  09 Jul 2004, Stefan Ritt, , Introduction of environment variable MIDASSYS 
Starting from midas version 1.9.4 on, the environment variable 'MIDASSYS'
should be defined and point to the installation directory of midas. The
purpose of that is that add-on packages (like the upcoming ROME system) can
find the midas libraries and include files. It is excatly the same as for
ROOT which defines ROOTSYS and should therefore be straight forward. The
libraries should then reside in $MIDASSYS/lib (or %MIDASSYS%\lib under windows).

To remind users about this new variable, a test has been added to odbedit,
which shows a warning when starting odbedit and MIDASSYS is not defined.
    Reply  09 Jul 2004, Piotr Zolnierczuk, , Introduction of environment variable MIDASSYS 
> Starting from midas version 1.9.4 on, the environment variable 'MIDASSYS'
> should be defined and point to the installation directory of midas. The
> purpose of that is that add-on packages (like the upcoming ROME system) can
> find the midas libraries and include files. It is excatly the same as for
> ROOT which defines ROOTSYS and should therefore be straight forward. The
> libraries should then reside in $MIDASSYS/lib (or %MIDASSYS%\lib under windows).
> 
> To remind users about this new variable, a test has been added to odbedit,
> which shows a warning when starting odbedit and MIDASSYS is not defined.

1. Finally! It's about time to do that! 

2. What will the entire structure tree look like?

Here's my suggestion
MIDASSYS=/opt/midas-1.9.4 (for example)   


so the Linux binaries would go to 
MIDASHOST=i386-pc-linux-gnu
$MIDASSYS/$MIDASHOST/bin
$MIDASSYS/$MIDASHOST/lib

the VxWorks binaries
MIDASHOST=m68k-wrs-vxworks
$MIDASSYS/$MIDASHOST/bin
$MIDASSYS/$MIDASHOST/lib

and the shared stuff would go to 
$MIDASSYS/include
$MIDASSYS/share/drivers
$MIDASSYS/share/examples

The Makefile would need to be adjusted (for make install) but that is not
too complicated

What do you think?

Regards
  Piotr
       Reply  09 Jul 2004, Stefan Ritt, , Introduction of environment variable MIDASSYS 
> Here's my suggestion
> MIDASSYS=/opt/midas-1.9.4 (for example)   

I guess we should follow the "standard" as much as possible. MIDASSYS was inspired by
ROOTSYS. Now where do people usually install ROOT? Is it /opt/root-x.x.x or something
else. Some years ago (when I did the last time some linux administration) optional
packages were put into /usr/local by default. I guess you have more experience with
today's tradition, so do whatever you thing is standard.

> so the Linux binaries would go to 
> MIDASHOST=i386-pc-linux-gnu
> $MIDASSYS/$MIDASHOST/bin
> $MIDASSYS/$MIDASHOST/lib

Does that mean that the path has to be modified to include $MIDASSYS/$MIDASHOST/bin?
If we put a link to /usr/local/bin, the path does not have to be modified. What about
shared libraries? Does ldconfig know about /usr/local/lib, or $MIDASYS/$MIDASHOST/lib?

> and the shared stuff would go to 
> $MIDASSYS/include
> $MIDASSYS/share/drivers
> $MIDASSYS/share/examples

What about /usr/share? Is that a common place for documentatino etc?

Thanks for your advice.

- Stefan
          Reply  09 Jul 2004, Piotr Zolnierczuk, , Introduction of environment variable MIDASSYS 
> I guess we should follow the "standard" as much as possible. MIDASSYS was inspired by
> ROOTSYS. Now where do people usually install ROOT? Is it /opt/root-x.x.x or something
> else. Some years ago (when I did the last time some linux administration) optional
> packages were put into /usr/local by default. I guess you have more experience with
> today's tradition, so do whatever you thing is standard.
I agree that we should follow the standard. 
I used /opt as an example. 
There are several "schools" as to where put things my philosophy is
/usr/{bin,lib,include}       - std OS packages (RPMS, .deb or whatever your flavor likes)
/usr/local/{bin,lib,include} - make/make install packages
/opt/..                      - additional packages (RPMS, ...) 

But it should be up to the user what $MIDASSYS she/he likes.

> 
> > so the Linux binaries would go to 
> > MIDASHOST=i386-pc-linux-gnu
> > $MIDASSYS/$MIDASHOST/bin
> > $MIDASSYS/$MIDASHOST/lib
> 
> Does that mean that the path has to be modified to include $MIDASSYS/$MIDASHOST/bin?
> If we put a link to /usr/local/bin, the path does not have to be modified. What about
> shared libraries? Does ldconfig know about /usr/local/lib, or $MIDASYS/$MIDASHOST/lib?
The path could/should be modified in users .bashrc/.tcshrc or we could provide a simple
system-wide script(s) that would do the job.
For years, I've been using such a scenario on my Linux PCs with regards to various
add-on packages (e.g. cern). 

Here's an example of my cern.sh that goes into /etc/profile.d on my RedHat Linux PC
#===================================
. /etc/profile.d/.functions
export CERN=/cern
export CERN_LEVEL=pro
addpath $CERN/$CERN_LEVEL/bin
#===================================

As for library path: there are several ways (as with exec path)
a) nice way: modify /etc/ld.so.conf by adding $MIDASYS/$MIDASHOST/lib
b) modifying LD_LIBRARY_PATH (there's some security issues with it)
c) symlinking to /usr/local/lib


> 
> What about /usr/share? Is that a common place for documentatino etc?
Yes. Check any recent Linux distribution /usr/share is full of docs, icons, etc.

This is my bias. 

I (obviously) prefer packing things into rpm which makes install/updates 
very easy - especially if you are managing several machines.

Cheers
    Piotr
          Reply  09 Jul 2004, John M O'Donnell, , Introduction of environment variable MIDASSYS 
For a long time the "de facto" standard was to spread a package around in many
directories under /usr/local.  This proved to be a bad idea, as removing the
package
became very difficult.

With POSIX there is a written standard, which says that each pacakge goes in
it's own
directory under /opt. eg. /opt/midas.  Each package gets to define it's own
structure
within that directory.  One could imagine several versions installed at the
same time
/opt/midas/v1.9.2 and /opt/midas/v1.9.4 each with a bin, lib include etc. 
Following the
ROOT example, you could make a link from /opt/midas/pro to
/opt/midas/v1.9.4, so that
system files and login files are easy to maintain etc.  The basic idea is

MIDASSYS=/opt/midas/pro
PATH=$PATH:$MIDASSYS/bin

though a more sophisticated approach is

MIDASSYS=/opt/midas/pro
echo $PATH | grep -q $MIDASSYS || PATH=$PATH:$MIDASSYS/bin

where the assignment line (Bourne shell, and BASH shell) ensures
that multiple entries are not added on the PATH even if the script is more
than once.

POSIX also goes on to say that links from /opt/bin can be made if desired. 
I find this
usefull if a package has only one or two executables, and I don't to make
multiple
versions available.

I hope that the POSIX ideas are usefull,

John.

> > Here's my suggestion
> > MIDASSYS=/opt/midas-1.9.4 (for example)   
> 
> I guess we should follow the "standard" as much as possible. MIDASSYS was
inspired by
> ROOTSYS. Now where do people usually install ROOT? Is it /opt/root-x.x.x
or something
> else. Some years ago (when I did the last time some linux administration)
optional
> packages were put into /usr/local by default. I guess you have more
experience with
> today's tradition, so do whatever you thing is standard.
> 
> > so the Linux binaries would go to 
> > MIDASHOST=i386-pc-linux-gnu
> > $MIDASSYS/$MIDASHOST/bin
> > $MIDASSYS/$MIDASHOST/lib
> 
> Does that mean that the path has to be modified to include
$MIDASSYS/$MIDASHOST/bin?
> If we put a link to /usr/local/bin, the path does not have to be modified.
What about
> shared libraries? Does ldconfig know about /usr/local/lib, or
$MIDASYS/$MIDASHOST/lib?
> 
> > and the shared stuff would go to 
> > $MIDASSYS/include
> > $MIDASSYS/share/drivers
> > $MIDASSYS/share/examples
> 
> What about /usr/share? Is that a common place for documentatino etc?
> 
> Thanks for your advice.
> 
> - Stefan
             Reply  12 Jul 2004, Stefan Ritt, , Introduction of environment variable MIDASSYS 
> With POSIX there is a written standard, which says that each pacakge goes in
> it's own
> directory under /opt. eg. /opt/midas.  Each package gets to define it's own
> structure
> within that directory.  One could imagine several versions installed at the
> same time
> /opt/midas/v1.9.2 and /opt/midas/v1.9.4 each with a bin, lib include etc. 
> Following the
> ROOT example, you could make a link from /opt/midas/pro to
> /opt/midas/v1.9.4, so that
> system files and login files are easy to maintain etc.  The basic idea is
> 
> MIDASSYS=/opt/midas/pro
> PATH=$PATH:$MIDASSYS/bin
> 
> though a more sophisticated approach is
> 
> MIDASSYS=/opt/midas/pro
> echo $PATH | grep -q $MIDASSYS || PATH=$PATH:$MIDASSYS/bin
> 
> where the assignment line (Bourne shell, and BASH shell) ensures
> that multiple entries are not added on the PATH even if the script is more
> than once.

That sounds all very good to me. So can you please sit together (at least John,
Piotr, and Pierre-Andre), discuss a common scheme and and propose it officially in
this forum for comments. After a week or so, it should be implemented into the
Makefile and installation scripts. I also would like to have Paul Knowles giving
it a look, since he voluteered to make the midas RPMs, which also heavily depends
on the chosen directory structure.
       Reply  20 Jul 2004, Konstantin Olchanski, , Introduction of environment variable MIDASSYS 
> > Starting from midas version 1.9.4 on, the environment variable 'MIDASSYS' ...
> 2. What will the entire structure tree look like?
> 
> Here's my suggestion
> MIDASSYS=/opt/midas-1.9.4 (for example)   

Where should MIDAS be installed?

After looking at the LSB and at the FHS, it appears that the standards permit all of:
1) /opt/midas...
2) /usr/{bin,lib,...}
3) /usr/local/{bin,lib,...}

Some handy references:
http://www.pathname.com/fhs/pub/fhs-2.3.html
http://www.linuxbase.org/spec/

The "example LSB-compliant packages" appear to install into /opt/lsb, but I do not see
 any guidance as to where "my" packages should go.

Then, after some googling, I see that IBM "recommends" /opt (see
http://www-106.ibm.com/developerworks/linux/library/l-lsb.html):

begin-quote---
To avoid name space collisions when installing LSB-conforming applications, the
applications belonging to the base operating system or the distribution are to be
installed in /sbin/, /bin/, or /usr/. System administrators can build packages from
source and install them into the /usr/local/ directory. However, third-party packages
of add-on software must be installed in /opt/<package>/, where <package> is the name
that describes a software suite.
end-quote---


K.O.
          Reply  21 Jul 2004, Stefan Ritt, , Introduction of environment variable MIDASSYS 
> Where should MIDAS be installed?

I personally don't have any preference, as long as it's in accordance with "the standard"
(whatever this is). Maybe one should add a flag to the makefile to specify the
installation directory, either /opt or /usr/local, so people then have the choice. I have
seen that in other packages. As for the RPM, I leave the final proposal to the person
writing the spec file (Paul? Piotr? Konstantin?). We should then commonly agree on the
location based on that proposal. The person supplying the RPM will "officially" become the
RPM maintainer and be responsible for maintaining it.

> installed in /sbin/, /bin/, or /usr/. System administrators can build packages from
> source and install them into the /usr/local/ directory. However, third-party packages
> of add-on software must be installed in /opt/<package>/, where <package> is the name
> that describes a software suite.

Well, midas is kind of in the middle. On one hand it's a third-party package (-> /opt),
but it requires some compilation to allow meaningful work (frontend, analyzer). So maybe
the RPM should go to /opt, and if compiled from the TAR ball it should go to /usr/local?
But that means if someone has to maintain a large basis of midas machines, he/she has to
always search two locations. On the other hand one can alway do a "cd $MIDASSYS" ...

- Stefan
Entry  05 Dec 2003, Konstantin Olchanski, , HOWTO setup MIDAS ROOT tree analysis 
> root -l
root> TFile *f = new TFile("run00064.root")
root> TTree *t = f->Get("Trigger")
root> t->StartViewer() // look at the ROOT TTree
root> t->MakeSelector() // generates Trigger.h, Trigger.C

edit run.C, the main program:
{
gROOT->Reset();
TFile f("data/run00064.root");
TTree *t = f.Get("Trigger");

TH1D* adc8 = new TH1D("adc8","ADC8",1500,0,1500-1);
TH1D* tdc2 = new TH1D("tdc2","TDC2",1500,0,1500-1);
TH2D* h12 = new TH2D("h2","ADC8 vs TDC2",100,0,1500,100,0,1500);
TH2D* h12cut = new TH2D("h2cut","ADC8 vs TDC2",50,0,1000-1,50,0,1500);

TSelector *s = TSelector::GetSelector("Trigger.C");
t->Process(s);

adc8->Draw();
tdc2->Draw();
h12->Draw();
h12cut->Draw();
}

edit Trigger.C:

Bool_t Trigger::ProcessCut(Int_t entry)
{
  fChain->GetTree()->GetEntry(entry);
  if (entry%100 == 0) printf("entry %d\r",entry);
  return kTRUE;
}

void Trigger::ProcessFill(Int_t entry)
{
  adc8->Fill(ADCS_ADCS[8]);
  tdc2->Fill(TDCS_TDCS[2]);
  h12->Fill(TDCS_TDCS[2],ADCS_ADCS[8]);
  if (ADCS_ADCS[8] > 100)
    h12cut->Fill(TDCS_TDCS[2],ADCS_ADCS[8]);
}

Run the analysis:

root -l
root> .x run.C

K.O.
    Reply  20 Jul 2004, Konstantin Olchanski, , HOWTO setup MIDAS ROOT tree analysis 
Updating the instructions to ROOT version 3.10.2. Example is from TRIUMF-KOPIO
tree analysis.

shell> root -l
root> TFile *f = new TFile("run00064.root")
root> Trigger->MakeSelector("TriggerSelector")  // "Trigger" is the tree name
inside the root file. Generates TriggerSelector.h and TriggerSelector.cpp

= edit run.C, the main program:

{
gROOT->Reset();
TSelector *s = TSelector::GetSelector("TriggerSelector.C");

TChain chain("Trigger"); // "Trigger" is the tree name inside the root files
chain.Add("run03016.root"); // can chain multiple files

TH1D* tdc2 = new TH1D("tdc2","TDC2",1500,0,1500-1);

chain.Process(s,"",500); // process 500 events
//chain.Process(s); // or process all events

tdc2->Draw();
}

= edit TriggerSelector.h:

in the TriggerSelector class members, i.e. "UInt_t TDC1_TDC1[47];" edit the
array size to be bigger than the maximum possible bank size

= edit TriggerSelector.C:

Bool_t TriggerSelector::Process(Int_t entry)
{
  fChain->GetTree()->GetEntry(entry);

  if (entry%100 == 0)
    printf("process %d, nTDC %3d, 0x%08x\n",entry,TDC1_nTDC1,TDC1_TDC1[1]);

  tdc2->Fill(TDC1_nTDC1);
  return kTRUE;
}

= Run the analysis:

shell> root -l
root> .x run.C
 
K.O.
Entry  05 Dec 2003, Konstantin Olchanski, , HOWTO setup MIDAS ROOT tree analysis 
> root -l
root> TFile *f = new TFile("run00064.root")
root> TTree *t = f->Get("Trigger")
root> t->StartViewer() // look at the ROOT TTree
root> t->MakeSelector() // generates Trigger.h, Trigger.C

edit run.C, the main program:
{
gROOT->Reset();
TFile f("data/run00064.root");
TTree *t = f.Get("Trigger");

TH1D* adc8 = new TH1D("adc8","ADC8",1500,0,1500-1);
TH1D* tdc2 = new TH1D("tdc2","TDC2",1500,0,1500-1);
TH2D* h12 = new TH2D("h2","ADC8 vs TDC2",100,0,1500,100,0,1500);
TH2D* h12cut = new TH2D("h2cut","ADC8 vs TDC2",50,0,1000-1,50,0,1500);

TSelector *s = TSelector::GetSelector("Trigger.C");
t->Process(s);

adc8->Draw();
tdc2->Draw();
h12->Draw();
h12cut->Draw();
}

edit Trigger.C:

Bool_t Trigger::ProcessCut(Int_t entry)
{
  fChain->GetTree()->GetEntry(entry);
  if (entry%100 == 0) printf("entry %d\r",entry);
  return kTRUE;
}

void Trigger::ProcessFill(Int_t entry)
{
  adc8->Fill(ADCS_ADCS[8]);
  tdc2->Fill(TDCS_TDCS[2]);
  h12->Fill(TDCS_TDCS[2],ADCS_ADCS[8]);
  if (ADCS_ADCS[8] > 100)
    h12cut->Fill(TDCS_TDCS[2],ADCS_ADCS[8]);
}

Run the analysis:

root -l
root> .x run.C

K.O.
    Reply  20 Jul 2004, Konstantin Olchanski, , HOWTO setup MIDAS ROOT tree analysis 
Updating the instructions to ROOT version 3.10.2. Example is from TRIUMF-KOPIO
tree analysis.

shell> root -l
root> TFile *f = new TFile("run00064.root")
root> Trigger->MakeSelector("TriggerSelector")  // "Trigger" is the tree name
inside the root file. Generates TriggerSelector.h and TriggerSelector.cpp

= edit run.C, the main program:

{
gROOT->Reset();
TSelector *s = TSelector::GetSelector("TriggerSelector.C");

TChain chain("Trigger"); // "Trigger" is the tree name inside the root files
chain.Add("run03016.root"); // can chain multiple files

TH1D* tdc2 = new TH1D("tdc2","TDC2",1500,0,1500-1);

chain.Process(s,"",500); // process 500 events
//chain.Process(s); // or process all events

tdc2->Draw();
}

= edit TriggerSelector.h:

in the TriggerSelector class members, i.e. "UInt_t TDC1_TDC1[47];" edit the
array size to be bigger than the maximum possible bank size

= edit TriggerSelector.C:

Bool_t TriggerSelector::Process(Int_t entry)
{
  fChain->GetTree()->GetEntry(entry);

  if (entry%100 == 0)
    printf("process %d, nTDC %3d, 0x%08x\n",entry,TDC1_nTDC1,TDC1_TDC1[1]);

  tdc2->Fill(TDC1_nTDC1);
  return kTRUE;
}

= Run the analysis:

shell> root -l
root> .x run.C
 
K.O.
Entry  05 Dec 2003, Konstantin Olchanski, , HOWTO setup MIDAS ROOT tree analysis 
> root -l
root> TFile *f = new TFile("run00064.root")
root> TTree *t = f->Get("Trigger")
root> t->StartViewer() // look at the ROOT TTree
root> t->MakeSelector() // generates Trigger.h, Trigger.C

edit run.C, the main program:
{
gROOT->Reset();
TFile f("data/run00064.root");
TTree *t = f.Get("Trigger");

TH1D* adc8 = new TH1D("adc8","ADC8",1500,0,1500-1);
TH1D* tdc2 = new TH1D("tdc2","TDC2",1500,0,1500-1);
TH2D* h12 = new TH2D("h2","ADC8 vs TDC2",100,0,1500,100,0,1500);
TH2D* h12cut = new TH2D("h2cut","ADC8 vs TDC2",50,0,1000-1,50,0,1500);

TSelector *s = TSelector::GetSelector("Trigger.C");
t->Process(s);

adc8->Draw();
tdc2->Draw();
h12->Draw();
h12cut->Draw();
}

edit Trigger.C:

Bool_t Trigger::ProcessCut(Int_t entry)
{
  fChain->GetTree()->GetEntry(entry);
  if (entry%100 == 0) printf("entry %d\r",entry);
  return kTRUE;
}

void Trigger::ProcessFill(Int_t entry)
{
  adc8->Fill(ADCS_ADCS[8]);
  tdc2->Fill(TDCS_TDCS[2]);
  h12->Fill(TDCS_TDCS[2],ADCS_ADCS[8]);
  if (ADCS_ADCS[8] > 100)
    h12cut->Fill(TDCS_TDCS[2],ADCS_ADCS[8]);
}

Run the analysis:

root -l
root> .x run.C

K.O.
    Reply  20 Jul 2004, Konstantin Olchanski, , HOWTO setup MIDAS ROOT tree analysis 
Updating the instructions to ROOT version 3.10.2. Example is from TRIUMF-KOPIO
tree analysis.

shell> root -l
root> TFile *f = new TFile("run00064.root")
root> Trigger->MakeSelector("TriggerSelector")  // "Trigger" is the tree name
inside the root file. Generates TriggerSelector.h and TriggerSelector.cpp

= edit run.C, the main program:

{
gROOT->Reset();
TSelector *s = TSelector::GetSelector("TriggerSelector.C");

TChain chain("Trigger"); // "Trigger" is the tree name inside the root files
chain.Add("run03016.root"); // can chain multiple files

TH1D* tdc2 = new TH1D("tdc2","TDC2",1500,0,1500-1);

chain.Process(s,"",500); // process 500 events
//chain.Process(s); // or process all events

tdc2->Draw();
}

= edit TriggerSelector.h:

in the TriggerSelector class members, i.e. "UInt_t TDC1_TDC1[47];" edit the
array size to be bigger than the maximum possible bank size

= edit TriggerSelector.C:

Bool_t TriggerSelector::Process(Int_t entry)
{
  fChain->GetTree()->GetEntry(entry);

  if (entry%100 == 0)
    printf("process %d, nTDC %3d, 0x%08x\n",entry,TDC1_nTDC1,TDC1_TDC1[1]);

  tdc2->Fill(TDC1_nTDC1);
  return kTRUE;
}

= Run the analysis:

shell> root -l
root> .x run.C
 
K.O.
Entry  15 Dec 2003, Pierre-André Amaudruz, , ROOT GUI at Triumf 
The current Triumf DAQ standard (Midas) since the second quarter of this
year (2003) has the capability to deal with ROOT histograms. The internal
midas logger can save data files in ROOT format and the analyzer can book
and fill ROOT histograms. These features triggered a new project started
during summer 2003 for building a Triumf GUI ROOT/Midas display utility.

The initial requirements for this utility are:
1) Solely based on ROOT (VirtualX, no Qt)
2) Similar overall functionality than PAW.
   - Open concurrent ROOT files.
   - Open connection to a single Midas Online experiment (requires analyzer
                                                          as server)
   - Optional Auto-update in ONLINE mode.
   - Zoning, Zooming option display.
   - Simple Historgram gaphic manipulation. (based on current ROOT
                                             implementation)
   - Tree manipulation ( use of TBrowser())
   - Simple user script invocation.
   - Optional experiment specific customization.
3) Session configuration save/restore option.

An initial version has been developed and currently is under evaluation.
Improvement and further development will based on the local experimenters
responses. 

This utility will be available for external use around the second quarter of
the 2004 at the latest.

.
Entry  02 Jul 2003, Pierre-André Amaudruz, , Midas/ROOT Analyser situation midas-root.jpg
The current and future situation of the Midas analyzer is summarized in the
attachment below.

Box explanation:
================
Front end:
---------
Midas code for accessing/gathering the hardware information into the Midas
format.

Midas SHM:
---------
Midas back end shared memory where the front end data are sent to.

mlogger:
-------
Data logger collecting the midas events and storing them on a physical
logging device (Disk, Tape)

Midas Analyzer:
--------------
Midas client for event-by-event analysis. Incoming data can be either online
or offline.

mserver:
-------
Subprocess interfacing external (remote) midas client to the centralized
data collection and database system.

PAW:
---
Standalone physics data analyzer (CERN).

ROOT:
----
Standalone Physics data analyser (CERN).


This diagram represents the data path from the Frontend to the analyzer in
online and offline mode. Each data path is annoted with a circled number
discussed below. In all cases, the data will flow from the front end
application to the midas back end data buffers which reside in a specific
share memory for a given experiment.

Path:
(1): From the shared memory, the midas analyzer can request events directly
and process them for output to divers destination.

(2): The data logger is a specific application which stores all the data to
 a storage media such as a disk or tape. This path is specific to the
creation of file.mid file format. The actual storage file in this .mid
format can be readout later on by the midas analyzer.

(3): The Midas analyzer has been developed originally for interfacing to the
PAW analyzer which uses its own shared memory segment for online display.
The analyzer can also save the data into a specific data format consistent
with PAW (HBOOK and Ntuples, extension .rz).

(4): Presently the data logger support a creation of the ROOT file format.
This file contains in the form of a Tree the midas event-by-event data. This
file is fully compatible with ROOT and therefore can be read out by the
standard ROOT application.

(5): Equivalent to the data logger, the analyzer receiving from the data
buffer or reading from a .mid file data can apply an event-by-event analysis
and on request produce a compliant ROOT file for further analysis. This
.root file can be composed of Trees as well as histograms.

(6): The possibility of ONLINE ROOT analysis has been implemented in a first
stage through the TMapFile (ROOT shared memory). While this configuration is
still in use an experiment, the intention is to deprecate it and replace it
with the data path (7).

(7): This path uses the network socket channel to transfer data out of the
analyzer to the ROOT environment. The current analyzer has a limited support
for ROOT analysis by only publishing on request the Midas analysis built in
histograms. No mean is yet implemented for Tree passing mechanism.

(8): The pass has not been yet investigated, but ROOT does provide
accessibility to external function calls which makes this option possible.
The ROOT framework will then perform dedicated event call to the main midas
data buffer using the standard midas communication scheme. The data format
translation from Midas banks to ROOT format will have to be taken care at
the user level in the ROOT environment.


Discussion:
==========
Presently the Socket communication between Midas and ROOT (7) is under
revision by Stefan Ritt and René Brun. This revision will simplify the
remote access of an object such as an histogram. For the Tree itself, the
requirement would be to implement a "ring buffer" mechanism for remote tree
request. This is currently under discussion.

The path (8) has been suggested by Triumf to address small experiment setup
where only a single analyzer is required. This path minimize the DAQ
requirements by moving all the data analysis handling to the user.
The same ROOT analysis code would be applicable to a ONLINE as well as
OFFLINE analysis.

Cons:
- Necessity of publishing raw data through the network for every instance of
the remote analyzer.
- Result sharing of the analysis cannot be done yet in real time.

Pros:
- No need of extra task for data translation (midas/root).
- Unique data unpacking code part of the user code.
- Less CPU requirement.

Other issues:
============
- The current necessity of the Midas shared memory for the midas analyzer to
run is a concern in particular for offline analysis where a priori no midas
is available. 

- The handling of the run/analyzer parameters. Possible parameter extraction
from file.odb.
Entry  21 Sep 2004, Konstantin Olchanski, , ODB-EPICS gateway 
At TRIUMF, we use several different versions of code to interface MIDAS and
EPICS (http://www.aps.anl.gov/epics). Now that we more or less understand
our needs, I propose this design for a simplified "EPICS" MIDAS frontend. I
would like to keep this new front end in the MIDAS CVS repository, possibly
replacing the existing EPICS frontend in examples/epics.

The basic idea is to provide an ODB-driven bi-directional gateway between
EPICS and ODB with this functionality: periodically read EPICS data and save
it in ODB, optionally generate MIDAS events with EPICS data; for writing
data to EPICS, use hotlinks- if the user changes "write" variables in ODB,
the changes are sent to EPICS.

1) ODB structure
   /equipment/epicsgw/
     common/...
     statistics/...
     variables/
       epics[...] <--- EPICS->ODB data (double[])
       write[...] <--- ODB->EPICS data (double[])
     settings/
       num epics  <--- number of epics variables (int)
       num write  <--- number of write variables (int)
       names epics <--- human-readable names for EPICS variables (string[])
       names write <--- human-readable names for EPICS variables (string[])
       chans epics <--- EPICS channels for epics-read data (string[])
       chans write <--- EPICS channels for epics-write data (string[])
       period      <--- EPICS read period in milliseconds (int[])
       enable epics <-- enable (y/n) epics-read (bool[])
       enable write <-- enable (y/n) epics-write (bool[])
       enable events <- enable event generation (bool)

2) EPICS to ODB data path: periodically read each enabled "epics" variable
and write the data values to ODB. Other front ends can hotlink the "epics"
variables to receive updated epics data.

3) ODB to EPICS data path: monitor the hotlink to ".../variables/write". If
data changes, send the changes to EPICS. At startup, write all "write"
variables to EPICS.

4) event generation: TBD.

5) error handling: TBD.

K.O.
    Reply  21 Sep 2004, Stefan Ritt, , ODB-EPICS gateway 
The easiest way to achieve this is to write a new class driver, probably derived
from the multi.c class driver. One has just to rename all "output" with "write"
(or better "ODB2EPICS") and all "input" with "EPICS2ODB". The multi class driver
handles already a factor/offset for each channel (which could be 1/0 of course),
a threshold to update the ODB/EPICS only when a value changes significantly, to
retrieve labes from the bus driver (EPICS labes -> ODB settings), automatic
event generation and error handling. So it would be a good starting point.

What one gets from the class driver in the ODB is:

  /equipment/<name>/
     variables/
        Input[]     <--- read from the bus driver (float)
        Output[]    <--- witten to the bus driver (float)
     settings/
        Names Input[]        <--- human readable names
        Names Output[]       <--- human readable names
        Update Threshold[]
        Input Offset[]
        Input Factor[]
        Output Offset[]
        Output Factor[]
        Devices/
           Input/
              DD/   <--- parameters for Device Driver
                 ... Epics addresses, flags etc.
              BD/   <--- parameters for Bus Driver
           Output/
        
So if one uses the standard mfe.c code together with the multi.c class driver
and epics_ca.c device driver all what is left is the following:

- replace cd_gen.c by multi.c in the examples/epics directory
- break down the already existing flags into enable epics/write/events
- maybe add th EPICS read period

The last two things should be done in the epics_ca.c device driver, so one can
use the multi.c class driver without any change. Event generation and error
handling then comes for free.
Entry  31 Aug 2004, Konstantin Olchanski, , midas odb locking 
One of our experiments is suffering from periodic ODB corruption and I
suspected that there might be a problem with ODB locking. In the last few
days, I finally had time to read the ODB locking code, to write a little
test program and to play with ODB. This is what I found:

1) ODB locking appears to be sound, my test program failed to find any
locking flaws, except for a big problem, described below. Please read on.

2) ODB locking is "unfair". A program "while (1) { lock(); do_stuff();
unlock(); /* no sleep here */ }" would lock out other users of ODB
(including odbedit) for seconds and minutes at a time. I see this as a flaw
in the semop() implementation in the Linux kernel and I cannot think of an
easy way to fix it in our code. (I tested only on RHL9 2.4.20-31.9smp on a
dual CPU machine. 2.6 kernels may work better).

3) presently, we use an infinite timeout waiting for the ODB lock. I suggest
we set the timeout to, say, 5 minutes, to protect against dead (or live)
locks that we saw a few times here at TRIUMF- every ODB client would hang
without any error messages or explanations forever waiting for the ODB lock
that is held by some rogue ODB client stuck in an infinite loop in corrupted
ODB.

4) while reading the locking code in db_{lock,unlock}_database(), I thought
that there is a race condition against the "lock_cnt" variable, until I
realized that this variable is local and there is no race condition. I would
like to comment this in the code?

5) I found a failure mode where db_close_database() erroneously deletes the
lock semaphore. Once the semaphore is deleted, ODB locking silently fails
(in db_lock_database() we do not check for success status of
mutex_wait_for()) and remaining ODB clients operate without locking protection.

This failure happens after ODB undercounts active clients after losing track
of clients removed by "idle timeouts" (and by other checks?). At some point,
db_close_database() decides that there are no more clients left, attempts to
delete the shared memory (this fails because there are still active clients
attached) and deletes the lock semaphore. Afterwards, the remaining "lost"
active clients operate without lock protection. This would tend to happen
while shutting down all clients, a time when they all rush-in to delete
themselves from "/system/clients", unsuring ODB corruption.

A quick solution I just coded would not work for mmap()-based shared memory
(I destroy the lock semaphore after the ODB shared memory is destroyed) as
this relies on "shm_nattch" counting feature of System-V shared memories,
absent in the mmap() based shared memories. Since the Windows implementation
uses mmap(), my "solution" is an obvious no-go. Alternatives would be to add
a second semaphore, just for counting active ODB clients (kluncky); or never
delete the semaphore in the first place (dirty, and how does one clear it if
it gets stuck in the locked state?).

For now, I would like to add a check to ss_mutex_waitfor() call in
db_lock_database() and crash if we can't get the mutex. Returning an error
code would be cleaner, but would not work because nobody checks the return
status of db_lock_database(). If can't get the mutex for (say) 5 minutes, I
think we should crash, too- something is very wrong and it is pointless to
continue waiting.

K.O.
    Reply  15 Sep 2004, Konstantin Olchanski, , midas odb locking 
After some discussion with Stefan-

> 1) ODB locking appears to be sound...
> 2) ODB locking is "unfair"

Stefan reminded me that "priority boosting" is the standard solution for this
problem. Since Linux does not appear to implement this, we may try doing it inside
midas, time permitting. "Fairness" behaviour of Win32, BSD and MacOSX may be worth
investigating.

> 3) presently, we use an infinite timeout waiting for the ODB lock.

I will add a timeout of 10 minutes, then shutdown the ODB client with an error message.

> 4) in db_{lock,unlock}_database(), [there is no] race condition against the
"lock_cnt" variable [because it is local].

I will document this.

> 5) I found a failure mode where db_close_database() erroneously deletes the
> lock semaphore. Once the semaphore is deleted, ODB locking silently fails
> (in db_lock_database() we do not check for success status of
> mutex_wait_for()) and remaining ODB clients operate without locking protection.

I will add a check and shutdown the ODB client with an error message if the lock
cannot be obtained (the mutex was deleted, the "lock" system call returns an error,
etc).

> [how to decide when the last ODB client disconnected from the shared memory and
when to delete the lock semaphore?]

We considered using a counting semaphore to count active ODB clients, if counting
semaphores do the right things on all supported systems (Linux, Win32, MacOSX).

K.O.
       Reply  16 Sep 2004, Stefan Ritt, , midas odb locking 
> I will add a timeout of 10 minutes, then shutdown the ODB client with an error message.

I added a timeout handling to db_lock_database. It was already present in
ss_mutex_wait_for, so it was just a matter of passing the status up the calling stack.
ODBEdit stops if it cannot obtain a lock after 5 minutes.
Entry  31 Aug 2004, Konstantin Olchanski, , mlogger crash if using mserver. 
Our users keep making a simple mistake- they set MIDAS_SERVER_HOST in their
environement. Most midas programs do not mind this- they go through the
mserver, inefficient but benign- except for the mlogger, which dumps core
about 10 seconds after starting. This mightily confuses the users-
everything works perfectly, except for the mlogger, (for most users) the
most obscure and magical part of midas. Obviously they can't take data
without the mlogger and they fail to correlate this crash with editing their
.cshrc file, so we get panic calls at midnight or whenever. And every time,
while debugging midas malfunctions, changes to .cshrc is absolutely the last
place we look for. Ouch!

As it turns out, mlogger does crash if it uses the mserver-
log_system_history() calls db_lock_database(), with a prompt crash because
the mlogger is not directly connected to any ODB (it's mserver is).

Obviously, running the mlogger via the mserver makes no sense, but we should
warn about this rather than dump core.

I propose this patch to src/mlogger.c::log_system_history():

-   db_lock_database(hDB);
-   db_notify_clients(hDB, hist_log[index].hKeyVar, FALSE);
-   db_unlock_database(hDB);
-
+   if (!rpc_is_remote())
+     {
+       db_lock_database(hDB);
+       db_notify_clients(hDB, hist_log[index].hKeyVar, FALSE);
+       db_unlock_database(hDB);
+     }
+   else
+     {
+       cm_msg(MERROR, "log_system_history", "Warning: mlogger is running
remotely via the mserver. This is an unsupported configuration. Please unset
MIDAS_SERVER_HOST and restart the mlogger");
+     }

K.O.
    Reply  07 Sep 2004, Stefan Ritt, , mlogger crash if using mserver. 
I trapped myself into that problem recently so it's the right time to fix it (;-).

We have two options: 

a) Make the logger work remotely, even if it's suboptimal and 
b) Make the logger refuse to run remotely. 

I have no case where I need to run the logger remotely, so I would opt for b).
This would mean removing the "-h" command line switch and the evaluation of
MIDAS_SERVER_HOST, or just supplying an empty host string to
cm_connect_experiment().

Let me know if you agree, I can then remove the "-h" option. The patch you
suggested I would apply in addition.

- Stefan
       Reply  15 Sep 2004, Konstantin Olchanski, , mlogger crash if using mserver. 
> I trapped myself into that problem recently so it's the right time to fix it (;-).
> We have two options: 
> a) Make the logger work remotely, even if it's suboptimal and 
> b) Make the logger refuse to run remotely.

After some discussion between Stefan, Pierre and myself, it was decided to disallow
running mlogger remotely via the mserver.

K.O.
Entry  14 Jul 2004, Exaos Lee, , install problem of Makefile on MacOS X (Darwin 7.4.0, gcc 3.3) 
I have compiled the sources on Darwin 7.4.0 with gcc 3.3. After the compilation of source codes, I 
try to execute "gmake install". I got the following message:
-------
Nothing to be done for "install".
-------

The install target could not be executed. Then I add the following line to the Makefile:
------
.PHONY: install
------

The install target can be executed. But when it is tring to copy "dio" to the proper directory, it 
cannot find the file. Then I found that the "utils" target isn't built. 
I try to build the target: darwin/bin/dio, I got the following error:
-------
cc -g -O2 -Wall -Iinclude -Idrivers -Ldarwin/lib -DINCLUDE_FTPLIB   -DOS_LINUX -DOS_DARWIN 
-DHAVE_STRLCPY -fPIC -Wno-unused-function -o darwin/bin/dio utils/dio.c
utils/dio.c:39:20: sys/io.h: No such file or directory
utils/dio.c: In function `main':
utils/dio.c:46: warning: implicit declaration of function `iopl'
gmake: *** [darwin/bin/dio] Error 1
--------
So, the include file "sys/io.h" may be changed under Darwin. I don't know how. I will try later. I 
hope somebody can notice this. 
Best regards.
    Reply  14 Jul 2004, Exaos Lee, , install problem of Makefile on MacOS X (Darwin 7.4.0, gcc 3.3) 
There are not such a file "io.h" inside my MacOS X. In fact, I didn't find any file containing function iopl().
So what is the equivalent function of iopl() under MacOS X? The utility dio should be modified in order to be compiled under Darwin-
gcc platform. 
       Reply  14 Jul 2004, Konstantin Olchanski, , install problem of Makefile on MacOS X (Darwin 7.4.0, gcc 3.3) 
> There are not such a file "io.h" inside my MacOS X. In fact, I didn't find any file containing function iopl().
> So what is the equivalent function of iopl() under MacOS X? The utility dio should be modified in order to be compiled under Darwin-
> gcc platform. 

"dio" is not supported under MacOSX. It is used to grant user programs access to PCI and ISA cards (usually CAMAC interfaces). We have
no MacOSX hardware with PCI or ISA slots so we cannot test and support this functionality.

The MacOSX Makefile should not try to build "dio". I will accept a patch to fix this Makefile bug.

K.O.
Entry  30 Jun 2004, Piotr Zolnierczuk, , mvme167 problems 
Hi,
 I am really puzzled: I am running the very same as far as sources
are concerned (Dec 12, 2003 snapsot) midas frontend (miniexp + camacnul)
on two different machines (and the same trusted private network):

1) one is an ancient Pentium/100 MHz laptop with RedHat Linux 7.3 and 
2) another one is event more ancient MVME167 25MHz running VxWorks 5.4.2

The front end on my Linux PC works just fine, whereas on the MVME167
I get intermittent crashes (most often at the end of the run).
[Correction: the crashes happen, I think, when the frontend wants 
to update the ODB]

The crashes happen in db_set_record routines

Any ideas what might be wrong? 
Except that MVME167 is a piece of ...#@!% 

Piotr
    Reply  30 Jun 2004, Piotr Zolnierczuk, , mvme167 problems 
A followup: I traced back the problem to version 1.9.2.

Version 1.9.1 does not have this problem but 1.9.2 does. 
For now I stick with 1.9.1

Piotr
 
ELOG V3.1.4-2e1708b5