I would like to document a few problems I ran into while setting up a new
experiment (two USB interfaces to Alice TPC electronics, plus maybe a USB
interface to CAMAC). I am using a midas cvs checkout from last October, so I am
not sure if these problems exist in the very latest code. I have fixes for all
of them and I will commit them after some more testing and after I figure out
how to commit into this new svn thingy.

- mxml: writing xml into an in-memory buffer probably produces invalid xml
because one of the mxml functions always writes "/>" into writer->fh, which is 0
for in-memory writers, so the "/>" tag goes to the console instead of the xml
data stream.

- hs_write_event() closes fd 0 (standard input), which confuses ss_getch(),
which makes mlogger not work (at least on my machine). I traced this down to the
history file file descriptors being initialized to zero and hs_write_event()
closing files without checking that it ever opened them.

- mevb: event builder did not work with a single frontend (a two-liner fix, once
Pierre showed me where to look. Why? My second TPC-USB interface did not yet
arrive and I wanted to test my frontend code. Yes, it had enough bugs to prevent
the event builder from working).

- mevb: consumes 100% CPU. Fix: add a delay in the main busy-loop.

- mlogger ROOT tree output does not work for data banks coming through the event
builder: mlogger looks for the bank definition under the event_id of mevb, in 
/equipment/evb/variables, which is empty, as the data banks are under
/equipment/frontendNN/variables. This may be hard to fix: bank "TPCA" may be
under "fe01", "TPCB" under "fe02" and mlogger knows nothing about any of this.
Fix: go back to .mid files.

K.O.

  /* swap event header if in wrong format */
  if (pevent->serial_number > 0x1000000) {
     WORD_SWAP(&pevent->event_id);
     WORD_SWAP(&pevent->trigger_mask);
     DWORD_SWAP(&pevent->serial_number);
     DWORD_SWAP(&pevent->time_stamp);
     DWORD_SWAP(&pevent->data_size);
  }

  /* swap event header if in wrong format */
#ifdef SWAP_EVENTS
  WORD_SWAP(&pevent->event_id);
  WORD_SWAP(&pevent->trigger_mask);
  DWORD_SWAP(&pevent->serial_number);
  DWORD_SWAP(&pevent->time_stamp);
  DWORD_SWAP(&pevent->data_size);
#endif

> I manage to access some vme modules with the older vmicvme interface and seemed
> confused with the
> new interface as the sample code provided does not have a specific test sample.
> The test code
> provided in the earlier version for accessing  V792 32ch. QDC was quite handy,
> how can I apply it
> for the new interface?  

Hello Ian,

I'm in the process of updating the V1190B, V792 and other to the new mvmestd.
These drivers will soon be committed to the repository.

Cheers, Pierre-André 

I manage to access some vme modules with the older vmicvme interface and seemed
confused with the
new interface as the sample code provided does not have a specific test sample.
The test code
provided in the earlier version for accessing  V792 32ch. QDC was quite handy,
how can I apply it
for the new interface?  

...
install: darwin/bin/dio: No such file or directory
make: *** [install] Error 71

404,405c404,405
< $(BIN_DIR)/mcnaf: $(UTL_DIR)/mcnaf.c $(DRV_DIR)/camac/camacrpc.c
<       $(CC) $(CFLAGS) $(OSFLAGS) -o $@ $(UTL_DIR)/mcnaf.c $(DRV_DIR)/camac/camacrpc.c $(LIB) $(LIBS)
---
> $(BIN_DIR)/mcnaf: $(UTL_DIR)/mcnaf.c $(DRV_DIR)/bus/camacrpc.c
>       $(CC) $(CFLAGS) $(OSFLAGS) -o $@ $(UTL_DIR)/mcnaf.c $(DRV_DIR)/bus/camacrpc.c $(LIB) $(LIBS)
438c438,439
<       @for i in mserver mhttpd odbedit mlogger ; \
---
> 
>       @for i in mserver mhttpd odbedit mlogger dio ; \
444,447d444
<       chmod +s $(SYSBIN_DIR)/mhttpd
< 
< ifeq ($(OSTYPE),linux)
<       install -v -m 755 $(BIN_DIR)/dio $(SYSBIN_DIR)
449c446
< endif
---
>       chmod +s $(SYSBIN_DIR)/mhttpd

diff Makefile Makefile.modify
-------------------------------------
404,405c404,405
< $(BIN_DIR)/mcnaf: $(UTL_DIR)/mcnaf.c $(DRV_DIR)/bus/camacrpc.c
<       $(CC) $(CFLAGS) $(OSFLAGS) -o $@ $(UTL_DIR)/mcnaf.c $(DRV_DIR)/bus/camacrpc.c $(LIB) $(LIBS)
---
> $(BIN_DIR)/mcnaf: $(UTL_DIR)/mcnaf.c $(DRV_DIR)/camac/camacrpc.c
>       $(CC) $(CFLAGS) $(OSFLAGS) -o $@ $(UTL_DIR)/mcnaf.c $(DRV_DIR)/camac/camacrpc.c $(LIB) $(LIBS)

• Anonymous checkout can be done now with
  svn co svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/midas/trunk midas
  svn co svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/mxml/trunk mxml
  Use password svn (you might have to enter it several times). The mxml package is now outside from midas, so you have to check it out separately.
  
• Non-anonymous access (for commits!) is only possible if you have an account at PSI. While it is possible via
  svn co svn+ssh://<your_name>@savannah.psi.ch/afs/psi.ch/project/meg/svn/midas/trunk midas
  it is more convenient if you access the repository via AFS, since then you only have to obtain a valid AFS token once a day and do not have to supply passwords on each SVN access
  
• Before you do a checkout, delete (or rename) your old CVS working directory
  
• Subversion does not use file revisions, but a global revision number for the whole repository, which is now at 2752. To get some idea about subversion, read this very good book
  
• The Web access to the repository is at http://savannah.psi.ch/viewcvs/trunk/?root=midas
  
• The ViewCVS web interface allows on-the-fly generation of TAR balls from the current repository. Just click on the link Download tarball
  
• The old CVS repository has been switched to read-only and will be completely closed in a few weeks
  
• The machine midas.psi.ch will in the near future not be available any more for any repository
  
• All the $Log: tags in the midas files have been replaced by $Id: tags, since the former ones are not supported by SVN (for good reasons actually). To view the change log, do a svn log <filename>.
  

• Do not use the Cygwin subversion package, but the binaries from here if you plan to access the SVN repository through AFS at PSI (or other places where AFS is available). If you map the AFS repository for example to "Y:", then the binaries access this under file:///Y:/svn/meg/... whicl the cygwin ones access this under file:///cygwin/y/svn/meg/... While this is ok in principle, it gives a conflict with the TortoiseSVN which expects the first path. So if you want to use command line utilities together with TortoiseSVN, the Cygwin package won't work.
  
• Use the TortoiseSVN package. It's really great! It has a very nice "diff" viewer/merger, it's integrated into the Windows explorer, has a spell checker for composing comments for commits, etc.
  
• For the SVN binaries under Windows, you have to set the environment variable LANG=en_US, otherwise svn will talk in German to you on a standard PSI Windows PC.
  

Dear MIDAS users and developers,

The "Midas VME Standard API" has been revised. We tried to incorporate all
comments and ideas we got so far. The mvme_ioctl() function was abandoned in
favor of several mvme_get/set_xxx functions. Furthermore, two additional
functions for read and write have been implemented to simplify writing/reading
single values to VME. The current API looks like this:

int mvme_open(MVME_INTERFACE **vme, int index);
int mvme_close(MVME_INTERFACE *vme);
int mvme_sysreset(MVME_INTERFACE *vme);
int mvme_read(MVME_INTERFACE *vme, void *dst, mvme_addr_t vme_addr,
              mvme_size_t n_bytes);
DWORD mvme_read_value(MVME_INTERFACE *vme, mvme_addr_t vme_addr);
int mvme_write(MVME_INTERFACE *vme, mvme_addr_t vme_addr, void *src,
               mvme_size_t n_bytes);
int mvme_write_value(MVME_INTERFACE *vme, mvme_addr_t vme_addr, DWORD value);
int mvme_set_am(MVME_INTERFACE *vme, int am);
int mvme_get_am(MVME_INTERFACE *vme, int *am);
int mvme_set_dmode(MVME_INTERFACE *vme, int dmode);
int mvme_get_dmode(MVME_INTERFACE *vme, int *dmode);
int mvme_set_blt(MVME_INTERFACE *vme, int mode);
int mvme_get_blt(MVME_INTERFACE *vme, int *mode);

The MVME_INTERFACE structure holds all internal data, similar to the FILE
structure in stdio.h. If several VME interfaces (of the same type) are present
in a PC, the function mvme_open can be called once for each crate, specifying
the index. The block transfer modes passed to mvme_set_blt control the usage of
DMA, MBLT64 and so on. Not all interfaces might support all modes, in which case
mvme_set_blt should return MVME_UNSUPPORTED. Then it's up to the user code to
ignore this error or choose a different mode.

So far we have implemented drivers for the SIS3100, SBS617/SBS618 and VMIC
interfaces using this standard. It should be noted that the VMIC uses solely
memory mapped VME I/O, which is completely hidden in the VMIC MVMESTD driver.

We would like to encourage people to switch to the revised MVMESTD API wherever
possible. If new drivers for ADCs and TDCs for example are written using this
standard, groups with different VME interfaces can use them without modification.

Although the standard works now for three different interfaces, it might be that
new interfaces need slight additions. They should be identified as soon as
possible, in order to adapt the MVMESTD quickly and freeze the API soon.

Interrupts are not (yet) implemented in the MVMESTD, because most experiments
use polling anyhow. If there is a need for interrupts by someone, he should come
up quickly with this and make a proposal for implementation.

Commited to CVS is the preliminary driver for the Wiener CC-USB CAMAC interface.
The driver implements all the mcstd.h camac access functions, except for those
not supported by hardware (8-bit operations, interrupts) and a few esoteric
functions not implemented in any other camac driver. The driver uses the
musbstd.h library to access USB, also commited in preliminary form.

Affected files:
midas/Makefile (added musbstd.c to libmidas.{a,so})
include/musbstd.h, src/musbstd.c (preliminary USB access library)
drivers/bus/ccusb.{c,h}

Most of the CAMAC access functions have been tested (see comments in ccusb.c).
If you find errors and problems, please email me (olchansk@triumf.ca) or write
an elog reply to this elog message.

Missing is the documentation and finalization of USB access library.
Missing is conformity to some MIDAS coding conventions.

Enjoy,
K.O.

> Right, but I can only complete the parts that I thought of and for which I already have
> code. This leaves out support for DMA (read: any block transfers) and interrupts.

DMA should be simple. We have a dma_flag in the MVME_INTERFACE structure, which only needs to
be set with mvme_set_dma_mode(...). The mvme_read/write subroutine then checks this flag and
calls the appropriate routine from the native API. About interrupts I haven't thought so much.
Does TRIUMF use interrupts anywhere? Or are all midas frontends in polled mode?

> Well, all interfaces are different and no amount of software will make them look all the
> same. 
>
> > - "cover" any specialities from the drivers, like memory mapping.
> 
> Exactly. We are facing a tricky task of inventing one API for two completely different
> modes of operation- purely memory mapped access on UniverseII based hardware and message
> passing access for the SIS3100 and VMUSB (Wiener VME-USB2).

Not all the same, but some common denominator. The memory mapped architecture can probably be
hidden in an API. So if one calls mvme_read/write, the routine checks if that region is already
mapped, and maps it if necessary. Then all you need is a proper offset and a memcpy(). Checking
about mapping causes some overhead. You have to check a hash table or a linked list which takes
time. But I think (see previous message) that this overhead should be small compared with the
IO operation. 

> I am now facing this problem with the Wiener CCUSB CAMAC-USB2 interface. I can
> implement all of mcstd.h, but the interface is intended to be used by downloading it with a
> CAMAC readout program and mcstd.h knows nothing about that.

Downloading a program you probably cannot cover with a common API, you are right. The problem
with USB is that you can only make ~1000 transfers per second, even with 2.0. So if you want
more, you need the old list concept.
> > So one should only use explicity vme_read/vme_write functions.
> 
> Rightey-ho. The fly in the ointement is that all VME ADC and TDC drivers in TRIUMF are
> written assuming memory mapped access, and I will not convert them to vme_read/vme_write
> overnight (think of testing).

You don't have to. This question only comes up if you (have to) use a non-memory mapped
interface. You can then either write then two separate drivers, or one driver and two MVME APIs.

> > Now I would like to hear anybody's comments. If we agree on this method, we have to
> > define a complete set of functions mvme_set_xxx.
> 
> We currently require only single-word transfers so we can concentrate on mvme_set_xxx for
> block-transfers later.

I need block transfers end of this month, so we should it include it in our current discussion.
The problem is that I use our (own) DRS2 waveform digitizing board, where each board produces
70kB of data per event. In non-DMA mode, the transfer would take forever.

> > maybe we can make a more generic mvme_set_vme_mode(mode), where mode could be fifomode,
> > 2eVME mode, chained block transfer mode and so on.
> 
> This is a can of worms and I would rather postpone discussion of block transfers. To give
> you a taste: UniverseII does not have a "fifo mode"- it *always* increments the vme address
>  (silly). A fifo mode can be emulated using chained transfers (read 256 bytes from
> addresses A through A+256, then read 256 more from address A, etc.), but the present VMIC
> VME library does not support chained transfers. On VxWorks, we do not even have a driver
> for the DMA engine, so not block transfers there at all.

If a native API does not support block transfer, the MVME driver should just ignore the DMA
setting. A ADC driver might then run slower, but still run. 

> I will now think about and post an updated proposal for mvmestd.h

Please also consider elog:221, I guess this is a cleaner and more flexible way of implementing
any MXXX standard.

- Stefan

> Good that you brought up the MIDAS VME API again, since this is still not complete, but
> has to be completed soon.

Right, but I can only complete the parts that I thought of and for which I already have
code. This leaves out support for DMA (read: any block transfers) and interrupts.

> Let me summarize the goals:
> - have a single set of functions which can be used with all VME CPUs/Interfaces at our
> institutes. Using this technique, one can change the interface or CPU and still keep
> the same frontend source code. This was already successfully done with the MIDAS CAMAC
> standard (as defined in mcstd.h)

Well, all interfaces are different and no amount of software will make them look all the
same. I am now facing this problem with the Wiener CCUSB CAMAC-USB2 interface. I can
implement all of mcstd.h, but the interface is intended to be used by downloading it with a
CAMAC readout program and mcstd.h knows nothing about that.

> - base any ADC/TDC driver we write on that API, so these modules can be used with any
> CPU/Interface without changing the driver

Right. Most useful.

> - have a simple and easy to understand set of functions

Right.

> - "cover" any specialities from the drivers, like memory mapping.

Exactly. We are facing a tricky task of inventing one API for two completely different
modes of operation- purely memory mapped access on UniverseII based hardware and message
passing access for the SIS3100 and VMUSB (Wiener VME-USB2).

> So one should only use explicity vme_read/vme_write functions.

Rightey-ho. The fly in the ointement is that all VME ADC and TDC drivers in TRIUMF are
written assuming memory mapped access, and I will not convert them to vme_read/vme_write
overnight (think of testing).

> So I would propose the following:
> 
>   mvme_set_crate()   each funciton has a _get_ partner, like mvme_get_crate()
>   mvme_set_address_mode()
>   mvme_set_amod()
>   mvme_set_blocktransfer()
>   mvme_set_fifomode()             // speciality of the SIS3100 interface, write a
>                                   // block of data to the same address
>   ...
> 
>   mvme_read(vme_address, dest_addr, num_bytes);
>   mvme_write(src_addr, vme_address, num_bytes);

This is compatible with what we do now and I will look into implementing this for
VMIC/Linux and MVME/VxWorks interfaces.

> Now I would like to hear anybody's comments. If we agree on this method, we have to
> define a complete set of functions mvme_set_xxx.

We currently require only single-word transfers so we can concentrate on mvme_set_xxx for
block-transfers later.

> If we get a new interface in the
> future which has new functionality (like 2eVME block transfers), we have to change the
> API each time (while with the ioctl() we only would have to add one parameter).

This amounts to the same thing: add a new function or add a new ioctl() call.

> maybe we can make a more generic mvme_set_vme_mode(mode), where mode could be fifomode,
> 2eVME mode, chained block transfer mode and so on.

This is a can of worms and I would rather postpone discussion of block transfers. To give
you a taste: UniverseII does not have a "fifo mode"- it *always* increments the vme address
 (silly). A fifo mode can be emulated using chained transfers (read 256 bytes from
addresses A through A+256, then read 256 more from address A, etc.), but the present VMIC
VME library does not support chained transfers. On VxWorks, we do not even have a driver
for the DMA engine, so not block transfers there at all.

I will now think about and post an updated proposal for mvmestd.h

K.O.

P.S. There is a proposal for musbstd.h heading your way, too.

Anothe idea which comes to my mind, we could make it kind of object oriented, like

typedef struct {
  int handle;
  int crate;
  int amod;
  int fifo_mode;
  ...
} MVME_INTERFACE;

main()
{
  MVME_INERFACE *vme;

  vme = mvme_init(); // allocated and fills MVME_INTERFACE structure

  mvme_set_crate(vme, crate_no);
  mvme_set_address_mode(vme, A24);
  ...

  mvme_read(vme, vme_address, dest_addr, num_bytes);
  mvme_exit(vme);    // frees memory allocated in mvme_init()
}

------------------------------------------

This way we would only have one structure containing all required parameters, and get/set
functions for it, like the OO textbooks propose it. This would actually make it thread
save. The "vme" pointer from above still has to be passed around to subroutines, but a
single pointer is better than lots of handles.

Good that you brought up the MIDAS VME API again, since this is still not complete, but
has to be completed soon.

Let me summarize the goals:

- have a single set of functions which can be used with all VME CPUs/Interfaces at our
institutes. Using this technique, one can change the interface or CPU and still keep
the same frontend source code. This was already successfully done with the MIDAS CAMAC
standard (as defined in mcstd.h)

- base any ADC/TDC driver we write on that API, so these modules can be used with any
CPU/Interface without changing the driver

- have a simple and easy to understand set of functions

- "cover" any specialities from the drivers, like memory mapping. 

Especially this point is very delicate. If one explicitely uses memory mapping in the
API, one cannot use interfaces which do not support this (like the Struck SIS3100). So
one should only use explicity vme_read/vme_write functions. Now people might argue that
going for each single access through a function is an overhead as compared to a memory
mapped operation. This might be true (even with inline functions of modern C
compilers), but it should be small on fast computers. Typically a single VME operation
take ~1us, while a function call takes much less.

Regarding the API implementation, I see now three "philosophies":

1) Handle oriented. One obtains a handle for each VME crate for each addressing mode,
then uses this handle for subsequent operation. This is the way the proposal from K.O.
is written.

2) Parameter oriented. There is no handle visible to the user code. All parameters are
passed in each call, like

mvme_read(crate, address_mode, vme_amod, source_addr, destination_addr, num_bytes);

3) ioctl() based. Same as 2), but the parameters like the address mode only get changed
via ioctl() when needed, like

vme_ioctl(request, parameter)   such as
 
   vme_ioctl(SET_CRATE, 1);
   vme_ioctl(MVME_AMOD, A24);
   mvme_read(source_addr, destination_addr, num_bytes);

This is how the current mvmestd.h is defined and how the
midas\drivers\bus\sis3100\sis3100.c is implemented.

Now the question is: should we implement 1), 2) or 3) ?

I had already lots of discussions with Pierre, and he convinced me that the ioctl() way
is not very nice. The advantage is that there is only one function to change
everything, so the complete API would be only 5 functions (init, exit, read, write,
ioctl), but of course there are many parameters to the ioctl() function. 

On the other hand I do not like the option 1). If you have five crates on a single PC
(and that's what we will have in our MEG experiment), you need 5x3 handles. If you use
many nested subroutines in your event readout, you have to pass lots of handles around.
I do not like option 2) as well, beacause each VME call contains many parmeters, which
make it hard to read.

So I would propose the following: We implement something like 3), but with explicit
routines:

  mvme_set_crate()   each funciton has a _get_ partner, like mvme_get_crate()
  mvme_set_address_mode()
  mvme_set_amod()
  mvme_set_blocktransfer()
  mvme_set_fifomode()             // speciality of the SIS3100 interface, write a
                                  // block of data to the same address
  ...

  mvme_read(vme_address, dest_addr, num_bytes);
  mvme_write(src_addr, vme_address, num_bytes);

It might look unfamiliar to have to set the address mode explicitely, but in practice
one typically has a few configuration calls in A16 mode, then the data readout in A32
mode. So omitting the address mode in the vme_read/write calls saves typing effort.

Since one does not use explicit handles, they have to stored internally in the driver.
I did this in the sis3100.c, and found that this overhead is negligible. The
implementation if of course not thread save, but does anybody use threads in the
experiment? I guess not.

Now I would like to hear anybody's comments. If we agree on this method, we have to
define a complete set of functions mvme_set_xxx. If we get a new interface in the
future which has new functionality (like 2eVME block transfers), we have to change the
API each time (while with the ioctl() we only would have to add one parameter). Or
maybe we can make a more generic mvme_set_vme_mode(mode), where mode could be fifomode,
2eVME mode, chained block transfer mode and so on.

Now there might be experiments which require the last bit of performance at the
frontend. They can decide to use the MIDAS API with some performance overhead, or they
can call directly the native driver API, but then be locked to the API. So everybody
has to decide himself.

I meet with Pierre end of September, and would like to finalize the API at that time.
So please give it a thought and let me know.

Best regards,

  Stefan

> It turns out that the new priority sequencing of run state transitions had a
> flaw: the frontends, the analyzer and the logger all registered at priority 500
> and were invoked in essentially a random order. For example the frontend could
> get a begin-run transition before the logger and so start sending data before
> the logger opened the output file. Same for the analyzer and same for the end of
> run. Also the sequencing for pause/resume run and begin/end run was different
> when the two pairs ought to have identical sequencing.
> 
> I now commited changes to mana.c and mlogger.c changing their transition sequencing:
> 
> start and resume:
> 200 - logger (mlogger.c, no change)
> 300 - analyzer (mana.c, was 500)
> 500 - frontends (mfe.c, no change)
> 
> stop and pause:
> 500 - frontends (mfe.c, no change)
> 700 - analyzer (mana.c, was 500)
> 800 - mlogger (mlogger.c, was 500)
> 
> P.S. However, even after this change, the TRIUMF ISAC/Dragon experiment still
> see an anomaly in the analyzer, where it receives data events after the
> end-of-run transition.
> 
> K.O.

Thanks for fixing that bug. It happend because during the implementatoin of the priority
sequencing we have up the pre/post tansition, which took care of the proper sequence
between the logger, frontend and analyzer. The way you modified the sequence is
absolutely correct. It is important to have >10 numbers "around" the frontends (like
450...550) in case one has an experiment with >10 frontends which need to make a
transition in a certain sequence (like the DANCE experiment in Los Alamos).

> Some time ago, the "remote CAMAC" functionality in mfe.c was made conditional on
> HAVE_CAMAC. This flag is not set by default so remote camac calls silently do
> not work, unless midas is compiled in a special way. I am too lazy to compile
> midas differently depending on what hardware I use, so I split
> register_cnaf_callback() into a separate file and made it easy to call directly
> from the user front end.
> 
> I left the HAVE_CAMAC bits in mfe.c so people who use that would see no change.
> 
> Affected files:
> Makefile (add cnaf_callback.o)
> midas.h (add void register_cnaf_callback(int debug);
> mfe.c (move the rpc code to cnaf_callback.c, call register_cnaf_callback())
> cnaf_callback.c (new file)
> 
> K.O.

That's a good idea. The frontend framework should be independent of the used
hardware (CAMAC or VME or whatever). I event went further and removed the HAVE_CAMAC
completely. This means that people have to add the call to register_cnaf_callback()
explicitly into the frontend user init routine. I think this inconvenience is not a
big deal because even before that people had to add the cnaf_callback.c file
explicitely into their Makefile. So they have to be aware of that change, and then
it's not a big deal to modify the init routine as well. But this way we have mfe.c
completely independen of the DAQ hardware which is how it should be.

To make things a bit easier, I modified the midas\examples\experiment\fronted.c to
contain this call, so people should be guided by that. I also added cnaf_callback.c
to the Makefile of the example frontend.