> We have seen on several daq systems this problem: we start a run and observe that the number of 
> events written by mlogger to the output file is double the number of events actually collected. Upon 
> inspection of the output file, we see that every event is written twice. Restarting the run usually fixes 
> this problem.

mlogger.c fixed svn rev 4497. (from tr_start(), call tr_stop() if somehow it was not called already by end-run transition).

K.O.

> When using remote midas clients with mserver, you may have noticed the zero-size .RPC.SHM files 
> these clients create in the directory where you run them. These files are associated with the semaphore 
> created by the midas rpc layer (rpc_call) to synchronize rpc calls between multiple threads. This 
> semaphore is always created, even for single-threaded midas applications. Also normally midas 
> semaphore files are created in the midas experiment directory specified in exptab (same place as 
> .ODB.SHM), but for remote clients, we do not know that location until we start making rpc calls, so the 
> semaphore file is created in the current directory (and it is on a remote machine anyway, so this 
> location may not be visible locally).
> 
> There are 2 problems with these semaphores:

A 3rd problem surfaced - on SL5 Linux, the global limit is 128 or so semaphores and on at least one heavily used machine that hosts multiple 
experiments we simply run out of semaphores.

For "normal" semaphores, their number is fixed to about 5 per experiment (one for each shared memory buffer), but the number of RPC 
semaphores is not bounded by the number of experiments or even by the number of user accounts - they are created (and never deleted) for 
each experiment, for each user that connects to each experiment, for each subdirectory where the each user happened to try to start a 
program that connects to the each experiment. (to reuse the old children's rhyme).

Right now, MIDAS does not have an abstraction for "local multi-thread mutex" (i.e. pthread_mutex & co) and mostly uses global semaphores 
for this task (with interesting coding results, i.e. for multithreaded locking of ODB). Perhaps such an abstraction should be introduced?

K.O.

The odb hot-link function db_open_record() did not return an error when the system limit for hotlinks is 
exceeded and no more hot links could be added (silent failure). This is now fixed.
odb.c svn rev 4500
K.O.

> I guess you have to do some debugging there. Note that "detached" transitions have 
> been implemented recently by Konstantin, so maybe your problem is related to that. 
> In this case Konstantin should check what's wrong.

Yes, I think there is a problem - cm_transition() starts the mtransition helper without the "-h expt" switch, so 
mtransition can only connect to the "default" experiment. Will fix. K.O.

> > We have seen on several daq systems this problem: we start a run and observe that the number of 
> > events written by mlogger to the output file is double the number of events actually collected. Upon 
> > inspection of the output file, we see that every event is written twice. Restarting the run usually fixes 
> > this problem.
> 
> mlogger.c fixed svn rev 4497. (from tr_start(), call tr_stop() if somehow it was not called already by end-run transition).

There is a new problem: after an unsuccessful run start, the next run start bombs with the error "output file runNNN.mid already exists". One way around this is to 
manually remove the useless data file, another is to bump up the run number. Better solution is to automatically erase the output file created by unsuccessful run 
starts.

K.O.

> > I guess you have to do some debugging there. Note that "detached" transitions have 
> > been implemented recently by Konstantin, so maybe your problem is related to that. 
> > In this case Konstantin should check what's wrong.
> 
> Yes, I think there is a problem - cm_transition() starts the mtransition helper without the "-h expt" switch, so 
> mtransition can only connect to the "default" experiment. Will fix. K.O.

Fixed midas.c svn rev 4506: in cm_transition(), always pass "-e expt" to mtransition, if connected remotely, pass the
"-h host:port".

svn rev 4506
K.O.

> > > We have seen on several daq systems this problem: we start a run and observe that the number of 
> > > events written by mlogger to the output file is double the number of events actually collected. Upon 
> > > inspection of the output file, we see that every event is written twice. Restarting the run usually fixes 
> > > this problem.
> > 
> > mlogger.c fixed svn rev 4497. (from tr_start(), call tr_stop() if somehow it was not called already by end-run transition).
> 
> There is a new problem: after an unsuccessful run start, the next run start bombs with the error "output file runNNN.mid already exists". One way around this is to 
> manually remove the useless data file, another is to bump up the run number. Better solution is to automatically erase the output file created by unsuccessful run 
> starts.

Stefan suggested implementing a new transition, TR_STARTABORT, issued if TR_START fails. mlogger can use it to cleanup open files, etc, similar to TR_STOP.

This is now implemented. In mlogger, TR_STARTABORT is similar to TR_STOP, but deletes open output files and does not save end-of-run information into databases, etc. mfe.c does not handle this trnasition yet, but I 
plan to add it - to fix the observed situations where the run failed to start, but some equipment does not know about it and continues to generate events and send data.

svn rev 4514
K.O.

> > Stefan suggested implementing a new transition, TR_STARTABORT, issued if TR_START fails. mlogger can use it to cleanup open files, etc, similar to TR_STOP.
> > 
> > This is now implemented. In mlogger, TR_STARTABORT is similar to TR_STOP, but deletes open output files and does not save end-of-run information into databases, etc. mfe.c does not handle this trnasition yet, but I 
> > plan to add it - to fix the observed situations where the run failed to start, but some equipment does not know about it and continues to generate events and send data.
> > 
> > svn rev 4514
> > K.O.
> 
> There is one problem with the TR_STARTABORT: If you combine old and new clients they will crash, since the old clients don't know anything about TR_STARTABORT. The way to prevent this is to increase the Midas version from 
> 2.0.0 to 2.1.0. Then you will get a warning if you mix clients. Please test this and commit the change if it works.

Are you sure? Only clients that register themselves to receive the TR_STARTABORT transition (via cm_register_transition()) will receive this transition.

As of now, the only client that registers and receives this transition is mlogger.

I also confirm that old clients that know nothing about TR_STARTABORT are *not* sent this transition. (this is tested).

K.O.

> > > mhttpd function for starting and stopping runs now uses cm_transition(DETACH) which spawns an 
> > > external helper program called mtransition to handle the transition sequencing.

Problem reported by Stefan - user presses the "stop the run" button, and the web page comes back saying "running" as if the button did not work. This is 
confusing. It happens because mtransition did not start yet - we have a race condition against it.

To improve this situation, mhttpd now remembers that a start/stop button was pushed and displays a message "Run start/stop requested" until it detects 
that mtransition started and set "runinfo/transition in progress" (or the run state changed).

svn rev 4520
K.O.

> > Could you give more info on your setup:
> > - CAMAC controller model
> Jorway 73A, we have three in hand and the problem doesn't depend on which controller
> we were using.

Dawei sent me a copy of his jorway73a.c scsi-camac driver. It is quite different from the
file in the MIDAS distribution. Dawei tells me that the file from the MIDAS distribution
does not compile. Stack traces from Dawei indicate a hang in this modified jorway73a.c
scsi-camac driver.

K.O.

> uint32_t mvme_read_value(MVME_INTERFACE * vme, mvme_addr_t vme_addr);
> Question: How to distinguish the status and value returned?

On VME interfaces using the Universe and tsi148 PCI-VME bridges, your question has no meaning.

The VME address space is directly mapped into the PCI address space, then mmap()ed into your 
program address space. Internally mvme_read_value() is "return *(uint32_t*)(mmap_base + vme_addr);" 
and there is no such thing as "status".

Physically on the VME bus, for single-word VME cycles (mvme_read_value), there are only 2 error 
conditions, an AS or a DS timeout, and these bridges return the bit pattern 0xFFFFFFFF for either error, 
the same as the traditional VME bus always worked (i.e. before PCI, before ISA, back when the VME bus 
*was* the main CPU-memory-IO bus).

So the answer to your question is "yes". If mvme_read_value() returned 0xFFFFFFFF, there was a VME 
bus timeout because the board you are trying to address a) is not installed, b) was unplugged, c) does 
not decode the address you tried to access (maybe you used the wrong AM code or the wrong data 
width).

With Universe and tsi148 PCI-VME bridges, the mvme_read() call runs the DMA interface that can issue 
block transfer cycles on the VME bus. These DMA interfaces have interesting error handling, but 
basically, they only tell you the estimated VME address at which the AS or DS timeout or BERR has 
occurred. For sane VME boards, DMA errors mean very basic breakage of the VME crate and VME board.

With non-directly attached VME interfaces, i.e. the SIS3100, you can also have communication errors. I 
do not know how those are reported by the SIS3100 Linux drivers, and I do not know how the MIDAS 
driver reports them. But I do know that if you see those errors, your interface is very broken and VME 
bus errors are the least of your worries.

P.S. There also exist PCI bus errors, they also return the bit pattern 0xFFFFFFFF and mean basic 
breakage inside your computer. PCI-PCI and PCI-host bridges have special registers you can read to 
find out the exact cause of the error ("your computer is broken").

K.O.

FYI - committed the last changes from TRIUMF DAQ systems for PIENU and T2K/ND280 FGD and TPC 
tests:

- mhttpd: add <odb xxx format="%d">xxx</odb>, similar to AJAX ODBget() method
- alarm.c: if alarm stops the run, log a message (sometimes it is hard to tell "why did this run stop?!?") 
use DETACH transition (was ASYNC - does not follow requested transition sequencing, now calls 
mtransition helper). Also verified that alarm handler always runs on the main computer - for remote 
clients, alarms are processed inside the corresponding mserver process.
- midas.c: event buffer fixes:
-- mserver 100% cpu busy loop if event buffer is full
-- consolidate event buffer cleanup into one routine. do things similar to odb cleanup - check for client 
pid, etc.
-- do not kill clients that have the watchdog timeout set to zero.

svn rev 4541
K.O.

> FYI - committed the last changes from TRIUMF DAQ systems for PIENU and T2K/ND280 FGD and TPC 
> tests:
> svn rev 4541

Also:

- add traps to event buffer code to catch event buffer (shared memory) corruption observed in PIENU
- dynamically allocate some RPC network data buffers to permit better communication between MIDAS clients built with different values of 
MAX_EVENT_SIZE (in T2K/ND280 the default 4 Mbytes is too small for some users, while other users use the default size - this change permits all 
these programs to talk to each other).

K.O.

> If you compile each source into an object, you may encounter a link error ...

Also camac rpc did not work at all last time we tried to use it at triumf, maybe 4 month ago in the Dragon 
experiment (upgrade from older version of midas). Never got around to trace down why. YMMV.

K.O.

> I encountered many warning while building MIDAS (svn r4556). Please see the
> attached log file. Most of them are caused by type conversion from string to
> "char*".
> Though I can ignore all the warning without any problem, I still hate to see
> them. :-)

There is no "type conversions". The compiler is whining about code like this:

/* data type names */
static char *tid_name[] = {
   "NULL",
   "BYTE",
   ...

I guess we should keep the compiler happy and make them "static const char*".

BTW, my compiler is SL5.2 gcc-4.1.2 and it does not complain. What's your compiler?

K.O.

> $ odbedit -e expcvadc
> odbedit: /opt/DAQ/repos/bot/midas/src/odb.c:651: db_validate_sizes: Assertion 
`sizeof(EQUIPMENT_INFO) == 400' failed.

Yes, this is now fixed, svn rev 4571 should be okey. Sorry about causing this problem - Stefan added 
some useful additional data to EQUIPMENT_INFO and my check for binary compatibility caught it and 
complained. Unfortunately on Saturday Stefan had to abruptly go back to PSI and things have been a little 
bit chaotic because we did not complete the testing of all the new changes and additions.

K.O.

> Linking CXX executable bin/mh2sql
> CMakeFiles/mh2sql.dir/utils/mh2sql.cxx.o: In function `main':
> /opt/DAQ/repos/bot/midas/utils/mh2sql.cxx:150: undefined reference to `MakeMidasHistoryODBC()'

Yes, I am in the process of changing the midas history interface and accidentally committed a version of 
mh2sql (utility for converting MIDAS history .hst files to SQL database) that uses the new interface.

This is now fixed in svn rev 4571.

The new C++ interface to the MIDAS history is in include/history.h and implementations for data storage 
using both midas .hst files and SQL (ODBC/MySQL) database are also committed (history_midas.cxx
and history_sql.cxx). The file history_odbc.cxx will be removed after some more testing of the new 
interface.

(All the new code is not activated yet, pending more testing).

K.O.

> --- Makefile
> < 	ld -shared -o $@ $^ $(LIBS) -lc
> ---
> > 	$(CXX) -shared -o $@ $^ $(LIBS) -lc

Will do. We also have a long standing request to change shared library name from lidmidas.so to libmidas-shared.so.

Different name for the .so file will permit us to build the shared library by default, but still link all MIDAS executables
with the static libmidas.a.

(there is no benefit from linking MIDAS executables - mlogger, mhttpd, etc - with the shared library,
and there is a significant cost in confusion from version skew between the executables and shared
libraries - I have had enough midnight calls "why did odbedit stop working? Oh, who changed LD_LIBRARY_PATH
and why is it now binding against this obsolete libmidas.so left over from 2 years ago?").

For user applications you can do whatever, but for MIDAS core applications I strongly suggest that they
be linked to the midas static library.

K.O.

> Linking CXX shared library lib/libmidas.so
/usr/bin/c++  ... -o lib/libmidas.so ... CMakeFiles/midas-shared.dir/src/history_odbc.cxx.o 
CMakeFiles/midas-shared.dir/src/history_sql.cxx.o 
CMakeFiles/midas-shared.dir/src/history_sql.cxx.o: In function `SqlODBC':
/opt/DAQ/repos/bot/midas/src/history_sql.cxx:200: multiple definition of `SqlODBC::SqlODBC()'
CMakeFiles/midas-
shared.dir/src/history_odbc.cxx.o:/opt/DAQ/repos/bot/midas/src/history_odbc.cxx:315: first defined 
here
> Why is the class "SqlODBC" duplicated?

This is interesting. I do not think my C++ book spells it out that I cannot have class A in foo.cxx
and a class A in bar.cxx. In fact I already discovered that the two classes A will collide (duplicate default 
constructor A::A, even if all other member functions are different) if I link an executable that uses both 
foo.o and bar.o.

Is there a way around this problem? In C, I can declare functions and variables "static" to "hide" them
from the linker.

What is the C++ equivalent for classes? (Any C++ experts here?)

In this specific case, the file history_odbc.cxx will disappear with the next commit of mhttpd, hopefully 
today. mlogger and mh2sql already do not use it.

And I will commit the Makefile change renaming libmidas.so to libmidas-shared.so, so we can build it 
by default but still link midas core executables to the normal midas library.

This will catch such a problem if it happens again.

K.O.

>  The "g++" is whining while compiling history_midas.cxx. Please see the attached log file.

Fixed. svn 4594. K.O.