> > - system.c: do not chdir("/") in ss_daemon_init()- it prevents us from ever
> >   getting core dumps from midas daemons.
> 
> The chdir("/") is from one of the unix text books. They say you HAVE to do it. If you start a
> daemon on an NFS file system, you cannot unmount that file system as long as the daemon is
> running.

Right, I remember this NFS problem from a while back.

This problem does not exist in the current crop of Linux systems (since Red Hat 7.3 at least) - they
either kill off all user programs or use "umount -f" and "umount -l".

"umount -l" works in any case to unmount a "busy" filesystem.

For systems where the NFS problem does still exist, one should do this: "mlogger -D" becomes "(cd /; mlogger -D)".

So I suspect that the "cd /" advice from the unix programming book is no longer as necessary
as it used to be. (Perhaps a better advice would have been to "cd /tmp", so we could still get
core dumps from non-root daemons).

K.O.

With recent changes to mana.c, creation of user ROOT objects in the MIDAS
analyser has changed. Here is the new example code for creating ROOT objects
that are visible in ROODY and are saved into the histogram file.

1) in the "global" context (outside of any function)

#include <TH1D.h>
#include <TProfile.h>

static TH1D* gMyHist1 = 0;
static TProfile* gMyHist2 = 0;

2) In the analyzer "init" or "begin run" method, create the histogram:

//extern TFolder *gManaHistosFolder; // from midas.h
gMyHist1 = new TH1D("gMyHist1",...);
gMyHist2 = new TProfile("gMyHist2",...);
gManaHistosFolder->Add(gMyHist1);
gManaHistosFolder->Add(gMyHist2);

(note: this will produce an warning about "possible memory leak")

3) In the per-event method, fill the histograms

gMyHist1->Fill(x);
gMyHist2->Fill(x,y);

4) In the Makefile, where you compile the frontend, add "-DUSE_ROOT" right after
"-I$(ROOTSYS)/include"

K.O.

The current h1_book() macros (and the previous example analyzer code) have an
odd persistency problem: for example, the user wants to change some histogram
limits, edits the h1_book() calls, rebuilds and restarts the analyzer, starts a
new run, and observes that all histograms are filled using the old limits, his
changes "did not take". The user panics, I get paged during the Holy Lunch Hour,
everybody is unhappy.

This is what I think happens:

1) analyzer starts
2) LoadRootHistgrams() loads old histograms from file
3) user code calls h1_book()
4) h1_book template in midas.h does this (roughly):
      hist = (TH1X *) gManaHistosFolder->FindObjectAny(name);
      if (hist == NULL) {
         hist = new TH1X(name, title, bins, min, max);
5) since the histogram already exists (loaded from the file, with the old
limits), the TH1X constructor is not called at all, new histogram limits are
utterly ignored.

A possible solution is to unconditionally create the ROOT objects, like I do in
the example code posted at http://dasdevpc.triumf.ca:9080/Midas/191. That code
produces an annoying warning from ROOT about possible memory leaks. This could
be fixed by adding a two liner to "find and delete" the object before it is
created, trippling the number of user code lines per histogram (find & delete,
then create). Highly ugly.

midas.h macros (h1_book & co) can be fixed by adding checks for histogram limits
and such, but I would much prefer a generic solution/convention that would work
for arbitrary ROOT objects without MIDAS-specific wrappers (think TProfile,
TGraph, etc...).

Any suggestions?

K.O.

We stumbled upon and fixed a "double free" bug in src/ybos.c causing crashes in
lazylogger writing .mid files in the FORMAT_MIDAS format (why does it use
ybos.c? Pierre says- for generic file i/o). Why this code had ever worked before
remains a mystery. K.O.

> > All the XML functionality is implemented in the new mxml.c/h library
> 
> mxml.c/h ... I separated it's CVS tree.
>
> The midas Makefile has been adjusted accordingly.

Looks like the midas mxml Makefile bits did not make it to CVS. Current Makefile
revision 1.67 does not have them and building midas from cvs sources fails because it
does not find mxml.h and mxml.c

K.O.

> I'm getting segfaults in yb_any_file_rclose (closing a file opened with
> yb_any_file_ropen with type MIDAS).
> 
> I think there are bugs with freeing from uninitialized pointers my.pmagta,
> my.pyh, and my.pylrl (which are only set when opening a YBOS file).  These
> should be set to NULL in yb_any_file_ropen (case MIDAS).  Likewise, the MIDAS
> format pointers my.pmp and my.pmrd should be NULLed for YBOS opens. 
> 
> It might be wise to also initialize the pointers in the "my" structure to null.

Do you see this crash even after my fix to (another?) double free?

K.O.

Current MIDAS versions nag me about setting the env.variable MIDASSYS to the
"midas installation directory", but I do not have one, so what should I set
MIDASSYS to? I checkout MIDAS from cvs into /home/olchansk/daq/midas, build it
there, run it from there. I never do "make install" (I am not "root" on every
machine; I am not the only MIDAS user on every machine). What should I set
MIDASSYS to? K.O.

I fixed some minor bit rot in the example experiment: a few minor Makefile
problems, make the analyzer use the current histogram creation macros, etc. I
also added startup and shutdown scripts. These will be documented as we work
through them with our Summer student. K.O.

I have been plagued by ODB corruption when I run the analyzer for the first time
after setting up the new experiment. Some time ago, I traced this to
mana.c::book_ttree() and now I found and fixed the bug, fix now commited to
midas cvs. In book_ttree(), db_find("/Analyzer/Bank switches") was returning an
error and setting hkey to zero. Then we called db_open_record() with hkey==0,
which cased ODB corruption later on. The normal db_validate_hkey() did not catch
this because it considers hkey==0 to be valid (when most likely it is not). K.O.

Minor Makefile changes:
- add "-m32" gcc flag to force 32-bit compilation on 64-bit Linux.
- do not link ybos.o into lazylogger and mdump.
K.O.

Some time ago, the "remote CAMAC" functionality in mfe.c was made conditional on
HAVE_CAMAC. This flag is not set by default so remote camac calls silently do
not work, unless midas is compiled in a special way. I am too lazy to compile
midas differently depending on what hardware I use, so I split
register_cnaf_callback() into a separate file and made it easy to call directly
from the user front end.

I left the HAVE_CAMAC bits in mfe.c so people who use that would see no change.

Affected files:
Makefile (add cnaf_callback.o)
midas.h (add void register_cnaf_callback(int debug);
mfe.c (move the rpc code to cnaf_callback.c, call register_cnaf_callback())
cnaf_callback.c (new file)

K.O.

Minor changes to run transitions code:
- improve debug messages
- fail transition if cannot connect to one of the clients
K.O.

It turns out that the new priority sequencing of run state transitions had a
flaw: the frontends, the analyzer and the logger all registered at priority 500
and were invoked in essentially a random order. For example the frontend could
get a begin-run transition before the logger and so start sending data before
the logger opened the output file. Same for the analyzer and same for the end of
run. Also the sequencing for pause/resume run and begin/end run was different
when the two pairs ought to have identical sequencing.

I now commited changes to mana.c and mlogger.c changing their transition sequencing:

start and resume:
200 - logger (mlogger.c, no change)
300 - analyzer (mana.c, was 500)
500 - frontends (mfe.c, no change)

stop and pause:
500 - frontends (mfe.c, no change)
700 - analyzer (mana.c, was 500)
800 - mlogger (mlogger.c, was 500)

P.S. However, even after this change, the TRIUMF ISAC/Dragon experiment still
see an anomaly in the analyzer, where it receives data events after the
end-of-run transition.

K.O.

For some time now, we have been thinking of updating the programming interface
for the VME bus interface drivers- mvmestd.h.

Until recently, we only had one type of vme interface- the PowerPC and
Universe-II based Motorolla MVME230x single board computers running VxWorks, and
that is the only VME interface supported by the present mvmestd.h & co in the
midas cvs.

Now we also have the Intel-PC and Universe-II based VMIC-VME single board
computers running Linux (RHL9 and RHEL4). They come with their own VME drivers
and interface library (from VMIC), and we (Pierre and myself) wrote a simplified
MIDAS-style library for using it with our ADC and TDC drivers.

After working with the VMIC-VME based systems this Summer, I am about to commit
our VME ADC and TDC drivers to MIDAS CVS. Since they use our VMIC-VME library, I
was inspired to integrate our library with the existing MIDAS VME API.

Both VME interfaces we use, MVME230x and VMIC-VME, use the same Universe-II
PCI-to-VME bridge. This brodge (+ OS drivers) provides memory mapped access to
VME directly from user memory space. Other VME interfaces require more
complicated interfacing and I tried to accomodate them in my design.

Note that this design is incomplete, it only has the VME features that we
currently use. I expect that the missing features (interrupts, DMA) will be
added to the "MIDAS VME API" as we start using them. Alternatively, they may be
implemented as interface-dependant "extensions".

So here goes:

void* mvme_getHandleA16(int crate,mvme_addr_t vmeA16addr,int numbytes,int vmeamod);
void* mvme_getHandleA24(int crate,mvme_addr_t vmeA24addr,int numbytes,int vmeamod);
void* mvme_getHandleA32(int crate,mvme_addr_t vmeA32addr,int numbytes,int vmeamod);

void mvme_writeD8(void* handle,int offset,int data);
void mvme_writeD16(void* handle,int offset,int data);
void mvme_writeD32(void* handle,int offset,int data);

int  mvme_readD8(void* handle,int offset);
int  mvme_readD16(void* handle,int offset);
int  mvme_readD32(void* handle,int offset);

The "getHandle" methods return a handle for accessing the required VME address
space. For Universe-II based drivers with direct memory mapping, the handle is a
pointer to the vme-mapped memory and can be directly dereferenced (after casting
from void*). For other drivers, it may be a pointer to an internal data
structure or whatever.

The "readDnn" and "writeDnn" methods implement the single-word vme transfers. It
is intended that directly mapped interfaces (Universe-II) can implement them as
"extern inline" (RTFM C docs) for maximum efficiency.

I am still struggling with a specification for vme block transfers. How does one
specify chained transfers? (mimic "man readv" using "struct iovec"?) How to
specify when the transfers stop (on word count, on BERR, etc). How to specify
FIFO modes (where the vme address is not incremented, all data is read from the
same address. The Universe-II bridge does not have this mode, others do). How to
decode whether to use DMA or not? (The VMIC-VME DMA driver has high startup
overhead, short transfers are faster using PIO more).

Anyhow, I do not need those advanced features immediately, so I omit them.

An implementation of this new interface will be commited to
midas/drivers/bus/vmicvme.{c,h} (and eventually I will modify vxVME.c to
conform). Drivers for sundry CAEN VME modules that use the new interface will be
commited to midas/drivers/divers (where I see drivers for other VME stuff).

Feedback is most welcome. I will try to get the stuff commited within the next
few days, plus a few days to shake down any bugs introduced during midasification.

K.O.

> Good that you brought up the MIDAS VME API again, since this is still not complete, but
> has to be completed soon.

Right, but I can only complete the parts that I thought of and for which I already have
code. This leaves out support for DMA (read: any block transfers) and interrupts.

> Let me summarize the goals:
> - have a single set of functions which can be used with all VME CPUs/Interfaces at our
> institutes. Using this technique, one can change the interface or CPU and still keep
> the same frontend source code. This was already successfully done with the MIDAS CAMAC
> standard (as defined in mcstd.h)

Well, all interfaces are different and no amount of software will make them look all the
same. I am now facing this problem with the Wiener CCUSB CAMAC-USB2 interface. I can
implement all of mcstd.h, but the interface is intended to be used by downloading it with a
CAMAC readout program and mcstd.h knows nothing about that.

> - base any ADC/TDC driver we write on that API, so these modules can be used with any
> CPU/Interface without changing the driver

Right. Most useful.

> - have a simple and easy to understand set of functions

Right.

> - "cover" any specialities from the drivers, like memory mapping.

Exactly. We are facing a tricky task of inventing one API for two completely different
modes of operation- purely memory mapped access on UniverseII based hardware and message
passing access for the SIS3100 and VMUSB (Wiener VME-USB2).

> So one should only use explicity vme_read/vme_write functions.

Rightey-ho. The fly in the ointement is that all VME ADC and TDC drivers in TRIUMF are
written assuming memory mapped access, and I will not convert them to vme_read/vme_write
overnight (think of testing).

> So I would propose the following:
> 
>   mvme_set_crate()   each funciton has a _get_ partner, like mvme_get_crate()
>   mvme_set_address_mode()
>   mvme_set_amod()
>   mvme_set_blocktransfer()
>   mvme_set_fifomode()             // speciality of the SIS3100 interface, write a
>                                   // block of data to the same address
>   ...
> 
>   mvme_read(vme_address, dest_addr, num_bytes);
>   mvme_write(src_addr, vme_address, num_bytes);

This is compatible with what we do now and I will look into implementing this for
VMIC/Linux and MVME/VxWorks interfaces.

> Now I would like to hear anybody's comments. If we agree on this method, we have to
> define a complete set of functions mvme_set_xxx.

We currently require only single-word transfers so we can concentrate on mvme_set_xxx for
block-transfers later.

> If we get a new interface in the
> future which has new functionality (like 2eVME block transfers), we have to change the
> API each time (while with the ioctl() we only would have to add one parameter).

This amounts to the same thing: add a new function or add a new ioctl() call.

> maybe we can make a more generic mvme_set_vme_mode(mode), where mode could be fifomode,
> 2eVME mode, chained block transfer mode and so on.

This is a can of worms and I would rather postpone discussion of block transfers. To give
you a taste: UniverseII does not have a "fifo mode"- it *always* increments the vme address
 (silly). A fifo mode can be emulated using chained transfers (read 256 bytes from
addresses A through A+256, then read 256 more from address A, etc.), but the present VMIC
VME library does not support chained transfers. On VxWorks, we do not even have a driver
for the DMA engine, so not block transfers there at all.

I will now think about and post an updated proposal for mvmestd.h

K.O.

P.S. There is a proposal for musbstd.h heading your way, too.

Commited to CVS is the preliminary driver for the Wiener CC-USB CAMAC interface.
The driver implements all the mcstd.h camac access functions, except for those
not supported by hardware (8-bit operations, interrupts) and a few esoteric
functions not implemented in any other camac driver. The driver uses the
musbstd.h library to access USB, also commited in preliminary form.

Affected files:
midas/Makefile (added musbstd.c to libmidas.{a,so})
include/musbstd.h, src/musbstd.c (preliminary USB access library)
drivers/bus/ccusb.{c,h}

Most of the CAMAC access functions have been tested (see comments in ccusb.c).
If you find errors and problems, please email me (olchansk@triumf.ca) or write
an elog reply to this elog message.

Missing is the documentation and finalization of USB access library.
Missing is conformity to some MIDAS coding conventions.

Enjoy,
K.O.

I would like to document a few problems I ran into while setting up a new
experiment (two USB interfaces to Alice TPC electronics, plus maybe a USB
interface to CAMAC). I am using a midas cvs checkout from last October, so I am
not sure if these problems exist in the very latest code. I have fixes for all
of them and I will commit them after some more testing and after I figure out
how to commit into this new svn thingy.

- mxml: writing xml into an in-memory buffer probably produces invalid xml
because one of the mxml functions always writes "/>" into writer->fh, which is 0
for in-memory writers, so the "/>" tag goes to the console instead of the xml
data stream.

- hs_write_event() closes fd 0 (standard input), which confuses ss_getch(),
which makes mlogger not work (at least on my machine). I traced this down to the
history file file descriptors being initialized to zero and hs_write_event()
closing files without checking that it ever opened them.

- mevb: event builder did not work with a single frontend (a two-liner fix, once
Pierre showed me where to look. Why? My second TPC-USB interface did not yet
arrive and I wanted to test my frontend code. Yes, it had enough bugs to prevent
the event builder from working).

- mevb: consumes 100% CPU. Fix: add a delay in the main busy-loop.

- mlogger ROOT tree output does not work for data banks coming through the event
builder: mlogger looks for the bank definition under the event_id of mevb, in 
/equipment/evb/variables, which is empty, as the data banks are under
/equipment/frontendNN/variables. This may be hard to fix: bank "TPCA" may be
under "fe01", "TPCB" under "fe02" and mlogger knows nothing about any of this.
Fix: go back to .mid files.

K.O.

My TPC events are fairly large: 18 FEC cards * 128 channels per card * 2 Kbytes
per channel = about 4 Mbytes. In my
frontend, when I request this event size, MIDAS complaints (in mfe.c) that it is
bigger than MAX_EVENT_SIZE, which
is set to 0.5 Mbytes in midas.h. What is the best way to deal with this? Should
we increase MAX_EVENT_SIZE to
something bigger? Remove the MAX_EVENT_SIZE limitation altogether?
  
For now, I increased the value MAX_EVENT_SIZE & co to (10*1024*1024) and it
seems to work (I also had to bump the
sanity check in bm_open_buffer() from 10E6 to 100E6). With 1/4 of the FEC cards,
the event size is 1 Mbyte at ~6
ev/sec the machine is almost idle, with the biggest CPU user being the event
builder at 10% CPU utilization.

K.O.

It turns out the the standard event builder fragment matching algorithm cannot
be used in my TPC application. I have two TPC-USB interfaces, which lack any
"busy" or synchronization logic. I send the hardware trigger into both
interfaces, and if one of them misses it, the data is out of sync forever. Consider:

Hardware
trigger    trig1     trig2    trig3    trig4
TPC01      serial1   serial2  serial3  serial4
TPC02      serial1  (missing) serial2  serial3

With the event builder matching only the event serial numbers, the first event
will be okey, but the second event will have trig2 data from TPC01 and trig3
data from TPC02, etc.

The problem exists even if the TPC-USB interfaces do not miss any triggers:
during begin and end of run, the interfaces are enabled one at a time, so if a
trigger arrives after the first interface was enabled, but before the second is
enabled, the data starts being out of sync (and if the same happens during the
end-of-run, the event counts from both frontends will match, but all data would
*still* be out of sync).

Obviously additional data is needed to match the fragments.

So in each frontend, I have a high-precision timestamp (gettimeofday(), usec
resolution) and I would like to have the event builder match the timestamps
instead of event serial numbers. What is the best way to do this? The mevb.c
code does not have any user callbacks for checking "do these fragments belong to
the same event?".

P.S. The event rate will be about 1/sec from cosmic ray tests and at most
10-50/sec in the M11 beam line at TRIUMF, at these low rates, the gettimeofday()
timestamps should be adequate.

K.O.

> Minor changes to run transitions code:
> - fail transition if cannot connect to one of the clients

This change introduced a problem:
1) a run is happily taking data
2) a frontend crashes
3) the web interface cannot stop the run (cannot contact the crashed frontend)
until  it is removed by the timeout (10-60 seconds?).

I am now considering allowing the run to end even if some clients cannot be
contacted. The begin, pause and resume transitions would continue to fail if
clients cannot be contacted.

K.O.