Commited to CVS is the preliminary driver for the Wiener CC-USB CAMAC interface.
The driver implements all the mcstd.h camac access functions, except for those
not supported by hardware (8-bit operations, interrupts) and a few esoteric
functions not implemented in any other camac driver. The driver uses the
musbstd.h library to access USB, also commited in preliminary form.

Affected files:
midas/Makefile (added musbstd.c to libmidas.{a,so})
include/musbstd.h, src/musbstd.c (preliminary USB access library)
drivers/bus/ccusb.{c,h}

Most of the CAMAC access functions have been tested (see comments in ccusb.c).
If you find errors and problems, please email me (olchansk@triumf.ca) or write
an elog reply to this elog message.

Missing is the documentation and finalization of USB access library.
Missing is conformity to some MIDAS coding conventions.

Enjoy,
K.O.

It turns out that the new priority sequencing of run state transitions had a
flaw: the frontends, the analyzer and the logger all registered at priority 500
and were invoked in essentially a random order. For example the frontend could
get a begin-run transition before the logger and so start sending data before
the logger opened the output file. Same for the analyzer and same for the end of
run. Also the sequencing for pause/resume run and begin/end run was different
when the two pairs ought to have identical sequencing.

I now commited changes to mana.c and mlogger.c changing their transition sequencing:

start and resume:
200 - logger (mlogger.c, no change)
300 - analyzer (mana.c, was 500)
500 - frontends (mfe.c, no change)

stop and pause:
500 - frontends (mfe.c, no change)
700 - analyzer (mana.c, was 500)
800 - mlogger (mlogger.c, was 500)

P.S. However, even after this change, the TRIUMF ISAC/Dragon experiment still
see an anomaly in the analyzer, where it receives data events after the
end-of-run transition.

K.O.

> It turns out that the new priority sequencing of run state transitions had a
> flaw: the frontends, the analyzer and the logger all registered at priority 500
> and were invoked in essentially a random order. For example the frontend could
> get a begin-run transition before the logger and so start sending data before
> the logger opened the output file. Same for the analyzer and same for the end of
> run. Also the sequencing for pause/resume run and begin/end run was different
> when the two pairs ought to have identical sequencing.
> 
> I now commited changes to mana.c and mlogger.c changing their transition sequencing:
> 
> start and resume:
> 200 - logger (mlogger.c, no change)
> 300 - analyzer (mana.c, was 500)
> 500 - frontends (mfe.c, no change)
> 
> stop and pause:
> 500 - frontends (mfe.c, no change)
> 700 - analyzer (mana.c, was 500)
> 800 - mlogger (mlogger.c, was 500)
> 
> P.S. However, even after this change, the TRIUMF ISAC/Dragon experiment still
> see an anomaly in the analyzer, where it receives data events after the
> end-of-run transition.
> 
> K.O.

Thanks for fixing that bug. It happend because during the implementatoin of the priority
sequencing we have up the pre/post tansition, which took care of the proper sequence
between the logger, frontend and analyzer. The way you modified the sequence is
absolutely correct. It is important to have >10 numbers "around" the frontends (like
450...550) in case one has an experiment with >10 frontends which need to make a
transition in a certain sequence (like the DANCE experiment in Los Alamos).

Minor Makefile changes:
- add "-m32" gcc flag to force 32-bit compilation on 64-bit Linux.
- do not link ybos.o into lazylogger and mdump.
K.O.

I have been plagued by ODB corruption when I run the analyzer for the first time
after setting up the new experiment. Some time ago, I traced this to
mana.c::book_ttree() and now I found and fixed the bug, fix now commited to
midas cvs. In book_ttree(), db_find("/Analyzer/Bank switches") was returning an
error and setting hkey to zero. Then we called db_open_record() with hkey==0,
which cased ODB corruption later on. The normal db_validate_hkey() did not catch
this because it considers hkey==0 to be valid (when most likely it is not). K.O.

I fixed some minor bit rot in the example experiment: a few minor Makefile
problems, make the analyzer use the current histogram creation macros, etc. I
also added startup and shutdown scripts. These will be documented as we work
through them with our Summer student. K.O.

I had to move strlcpy & strlcat into a separate file "strlcpy.c". A header file
"strlcpy.h" was added as well. This way one can omit the old HAVE_STRLCPY which
made life hard. The windows and linux makefiles were adjusted accordingly, but
for Max OS X there might be some fixes necessary which I could not test.

Current MIDAS versions nag me about setting the env.variable MIDASSYS to the
"midas installation directory", but I do not have one, so what should I set
MIDASSYS to? I checkout MIDAS from cvs into /home/olchansk/daq/midas, build it
there, run it from there. I never do "make install" (I am not "root" on every
machine; I am not the only MIDAS user on every machine). What should I set
MIDASSYS to? K.O.

> Current MIDAS versions nag me about setting the env.variable MIDASSYS to the
> "midas installation directory", but I do not have one, so what should I set
> MIDASSYS to? I checkout MIDAS from cvs into /home/olchansk/daq/midas, build it
> there, run it from there. I never do "make install" (I am not "root" on every
> machine; I am not the only MIDAS user on every machine). What should I set
> MIDASSYS to? K.O.

Then set it to /home/olchansk/daq/midas. The reason for MIDASSYS is the same as
for ROOTSYS. Having it allows other packages like ROME to access the Midas source
code, include files and libraries.

I'm getting segfaults in yb_any_file_rclose (closing a file opened with
yb_any_file_ropen with type MIDAS).

I think there are bugs with freeing from uninitialized pointers my.pmagta,
my.pyh, and my.pylrl (which are only set when opening a YBOS file).  These
should be set to NULL in yb_any_file_ropen (case MIDAS).  Likewise, the MIDAS
format pointers my.pmp and my.pmrd should be NULLed for YBOS opens. 

It might be wise to also initialize the pointers in the "my" structure to null.

--Donald              

> I'm getting segfaults in yb_any_file_rclose (closing a file opened with
> yb_any_file_ropen with type MIDAS).
> 
> I think there are bugs with freeing from uninitialized pointers my.pmagta,
> my.pyh, and my.pylrl (which are only set when opening a YBOS file).  These
> should be set to NULL in yb_any_file_ropen (case MIDAS).  Likewise, the MIDAS
> format pointers my.pmp and my.pmrd should be NULLed for YBOS opens. 
> 
> It might be wise to also initialize the pointers in the "my" structure to null.

Do you see this crash even after my fix to (another?) double free?

K.O.

Dear midas users,

I have changed the ODB dump format to XML. As you might know, the logger writes
a special begin-of-run event to the .mid file which includes an ASCII dump of
the ODB. The same at the end-of-run. To read these ODB dumps back in offline
analysis, this requires setting up a ODB just to read back these values. In
order to avoid this, we switched the format to XML instead of the old ASCII
format. That way ROME can read the ODB dump and extract individual values from
it without setting up a shared memory.

A similar thing has been made for the ODB dumps to separate .odb files, which
are controlled by "/Logger/ODB Dump" and "/Logger/ODB Dump file". If the dump
file has the extension .xml, the file is dumped in XML format as well.

All the XML functionality is implemented in the new mxml.c/h library, which has
been added to the distribution, and which can be used in other projects as well
(XML configuration of ROODY?). It has already been successfully implemented in
ROME, so ROME is no longer dependent on libxml.

- Stefan

> All the XML functionality is implemented in the new mxml.c/h library, which has
> been added to the distribution, and which can be used in other projects as well
> (XML configuration of ROODY?). It has already been successfully implemented in
> ROME, so ROME is no longer dependent on libxml.

Since mxml.c/h is used in several projects (midas, ROME, elog), I separated it's
CVS tree. So in order to compile midas from scratch, you have to check out midas
AND mxml like

cd ~
cvs -d :ext:cvs@midas.psi.ch:/usr/local/cvsroot checkout midas
cvs -d :ext:cvs@midas.psi.ch:/usr/local/cvsroot checkout mxml

cd midas
make

so the "mxml" tree is ABOVE the "midas" tree. The midas Makefile has been adjusted
accordingly. If you decide to put the mxml somwhere else, you have to change
MXML_DIR in the Makefile accordingly.

- Stefan

> > All the XML functionality is implemented in the new mxml.c/h library
> 
> mxml.c/h ... I separated it's CVS tree.
>
> The midas Makefile has been adjusted accordingly.

Looks like the midas mxml Makefile bits did not make it to CVS. Current Makefile
revision 1.67 does not have them and building midas from cvs sources fails because it
does not find mxml.h and mxml.c

K.O.

> Looks like the midas mxml Makefile bits did not make it to CVS. Current Makefile
> revision 1.67 does not have them and building midas from cvs sources fails because it
> does not find mxml.h and mxml.c

I forgot to commit the new Makefile, thanks for reminding me. Now it should be fine.

Dear Midas users,

may I kindly invite you present your work at the Real-Time 2005 Conference in
Stockholm, June 4-10. The conference deals with all kinds of real time
applications like DAQ, control systems etc. It is a small conference with no
paralles sessions, and with two interesting short courses. The deadline has been
prolonged until March 13, 2005. If you are interested, please register under

http://www.physto.se/RT2005/

Here is the official letter from the chairman:

=====================================================================
               14th IEEE-NPSS Real Time Conference 2005
                  Stockholm, Sweden, 4-10 June, 2005
              Conference web site: www.physto.se/RT2005/

**********************************************************************
*                                                                    *
*        ABSTRACT SUBMISSION PROLONGED! DEADLINE: March 13, 2005     *
*                                                                    *
**********************************************************************

Considering that the Real Time conference is a highly meritorious and
multidisciplinary conference with purely plenary sessions and that the
accepted papers may be submitted to a special issue of the IEEE
Transactions on Nuclear Science we would like to give more people the
opportunity to participate. Therefore we have organized the program so
that there is now more time for talks than at the RT2003 and we are
extending the abstract submission to March 13. We strongly encourage
you to participate!

Submit your abstract and a summary through the conference web site
"Abstract submission" link. Please, make sure that your colleagues know
about the conference and invite them.

I would also like to take this opportunity to announce the two short
courses we have organized for Sunday 5/6:

- "Gigabit Networking for Data Acquisition Systems - A practical
introduction"
  Artur Barczyk, CERN

- "System On Programmable Chip - A design tutorial"
  Marco Riccioli, Xilinx

Please find the abstracts and more information about the conference on
www.physto.se/RT2005/

Thank you if you have already submitted an abstract.

Richard Jacobsson
General Chairman, RT2005 Conference

Email: RT2005@cern.ch
Phone: +41-22-767 36 19
Fax:   +41-22-767 94 25
CERN Meyrin
1211 Geneva 23
Switzerland

We stumbled upon and fixed a "double free" bug in src/ybos.c causing crashes in
lazylogger writing .mid files in the FORMAT_MIDAS format (why does it use
ybos.c? Pierre says- for generic file i/o). Why this code had ever worked before
remains a mystery. K.O.

is there a reason why histograms are not saved after a replay?

   /* save histos if requested */
   if (out_info.histo_dump && clp.online) {
                              ^^^^^^^^^^

perhaps the && should be ||?

> is there a reason why histograms are not saved after a replay?
> 
>    /* save histos if requested */
>    if (out_info.histo_dump && clp.online) {
>                               ^^^^^^^^^^
> 
> perhaps the && should be ||?

The original reason for that is the for running online, you want some histos for
monitoring after each run. For running offline, you specify a root output file via
"-o xxx.root" which contains trees AND histos. So the histos would there be twice
if you remove the "clp.online" from above.

Having "-o xxx.root" is IMHO a cleaner way, since you might want to analyze a run
in different ways (like using different calibrations). So what you do is specify
different "-o cal00123.root", "-o final00123.root" and so on, while with the
mechanism in eor() you always get the same file name. So try using "-o xxx.root"
and see if that fits your needs.

With recent changes to mana.c, creation of user ROOT objects in the MIDAS
analyser has changed. Here is the new example code for creating ROOT objects
that are visible in ROODY and are saved into the histogram file.

1) in the "global" context (outside of any function)

#include <TH1D.h>
#include <TProfile.h>

static TH1D* gMyHist1 = 0;
static TProfile* gMyHist2 = 0;

2) In the analyzer "init" or "begin run" method, create the histogram:

//extern TFolder *gManaHistosFolder; // from midas.h
gMyHist1 = new TH1D("gMyHist1",...);
gMyHist2 = new TProfile("gMyHist2",...);
gManaHistosFolder->Add(gMyHist1);
gManaHistosFolder->Add(gMyHist2);

(note: this will produce an warning about "possible memory leak")

3) In the per-event method, fill the histograms

gMyHist1->Fill(x);
gMyHist2->Fill(x,y);

4) In the Makefile, where you compile the frontend, add "-DUSE_ROOT" right after
"-I$(ROOTSYS)/include"

K.O.

> (preliminary, untested. I will keep this updated as I get testing feedback)
> 
> With recent changes to mana.c, creation of user ROOT objects in the MIDAS
> analyser has changed. Here is the new example code for creating ROOT objects
> that are visible in ROODY and are saved into the histogram file.
> 
> 1) in the "global" context (outside of any function)
> 
> #include <TH1D.h>
> #include <TProfile.h>
> 
> static TH1D* gMyHist1 = 0;
> static TProfile* gMyHist2 = 0;
> 
> 2) In the analyzer "init" or "begin run" method, create the histogram:
> 
> //extern TFolder *gManaHistosFolder; // from midas.h
> gMyHist1 = new TH1D("gMyHist1",...);
> gMyHist2 = new TProfile("gMyHist2",...);
> gManaHistosFolder->Add(gMyHist1);
> gManaHistosFolder->Add(gMyHist2);
> 
> (note: this will produce an warning about "possible memory leak")
> 
> 3) In the per-event method, fill the histograms
> 
> gMyHist1->Fill(x);
> gMyHist2->Fill(x,y);
> 
> K.O.


the book functions provide a convenient place to check against object duplication
and memory leaks etc., and a place to ensure that consistent subfolders are being
used.  eg. a while back we decided that TCutGs should be in a "cuts" subfolder.

To extend the booking to TProfile is fairly easy.  In fact if you want to
use the simple constructor TProfile::TProfile (const char *, const char *, Int_t,
Axis_t, Axis_t), then you could infact just use h1_book<TProfile>.

It now seems to me that the names h1_book, h2_book, cut_book are all too long
and even more upsetting are inconsistent.  Some of them are templates (most) and
some are not.  Perhaps they should all be templates, and all have the same name.
The attached patch accomplishes this (without deleting the old names).  With this
patch you can now do

gMyHist1 = book<TProfile>( "gMyHist2",...);

New book templates are needed when you (1) wish to change the subfolder, or (2)
need to use a different argument list in the constructor.  If you need help with
this for the TProfile constructors which are different from TH1D constructors then
let me know.  They should be easy to do.

For TGraph at lot depends on how you want to initialise the data points.

The current h1_book() macros (and the previous example analyzer code) have an
odd persistency problem: for example, the user wants to change some histogram
limits, edits the h1_book() calls, rebuilds and restarts the analyzer, starts a
new run, and observes that all histograms are filled using the old limits, his
changes "did not take". The user panics, I get paged during the Holy Lunch Hour,
everybody is unhappy.

This is what I think happens:

1) analyzer starts
2) LoadRootHistgrams() loads old histograms from file
3) user code calls h1_book()
4) h1_book template in midas.h does this (roughly):
      hist = (TH1X *) gManaHistosFolder->FindObjectAny(name);
      if (hist == NULL) {
         hist = new TH1X(name, title, bins, min, max);
5) since the histogram already exists (loaded from the file, with the old
limits), the TH1X constructor is not called at all, new histogram limits are
utterly ignored.

A possible solution is to unconditionally create the ROOT objects, like I do in
the example code posted at http://dasdevpc.triumf.ca:9080/Midas/191. That code
produces an annoying warning from ROOT about possible memory leaks. This could
be fixed by adding a two liner to "find and delete" the object before it is
created, trippling the number of user code lines per histogram (find & delete,
then create). Highly ugly.

midas.h macros (h1_book & co) can be fixed by adding checks for histogram limits
and such, but I would much prefer a generic solution/convention that would work
for arbitrary ROOT objects without MIDAS-specific wrappers (think TProfile,
TGraph, etc...).

Any suggestions?

K.O.

> The current h1_book() macros (and the previous example analyzer code) have an
> odd persistency problem: for example, the user wants to change some histogram
> limits, edits the h1_book() calls, rebuilds and restarts the analyzer, starts a
> new run, and observes that all histograms are filled using the old limits, his
> changes "did not take". The user panics, I get paged during the Holy Lunch Hour,
> everybody is unhappy.
> 
> This is what I think happens:
> 
> 1) analyzer starts
> 2) LoadRootHistgrams() loads old histograms from file

I can't get onto cvs@midas.psi.ch right now
(cvs update
cvs@midas.psi.ch's password: 
Permission denied, please try again.)

but when I changed LoadRootHistograms a few days ago I left it as:

    } else if (obj->InheritsFrom( "TH1")) {

      // still don't know how to do TH1s

so h1_book() is creating the first and only copy of the histograms.
I am able to create new histogram limits.
I don't get the memory leak problems.

However I have seen the memory leak problems before, and they are real.
They must be dealt with either by (1) first deleteing the old histogram
or (2) ensuring that histogram names are unique in the whole application
(different modules/folders can not use the same histogram names).

I will return to this once I can do a cvs update for midas.

John.

> 3) user code calls h1_book()
> 4) h1_book template in midas.h does this (roughly):
>       hist = (TH1X *) gManaHistosFolder->FindObjectAny(name);
>       if (hist == NULL) {
>          hist = new TH1X(name, title, bins, min, max);
> 5) since the histogram already exists (loaded from the file, with the old
> limits), the TH1X constructor is not called at all, new histogram limits are
> utterly ignored.
> 
> A possible solution is to unconditionally create the ROOT objects, like I do in
> the example code posted at <a
href="http://dasdevpc.triumf.ca:9080/Midas/191">http://dasdevpc.triumf.ca:9080/Midas/191</a>.
That code
> produces an annoying warning from ROOT about possible memory leaks. This could
> be fixed by adding a two liner to "find and delete" the object before it is
> created, trippling the number of user code lines per histogram (find & delete,
> then create). Highly ugly.
> 
> midas.h macros (h1_book & co) can be fixed by adding checks for histogram limits
> and such, but I would much prefer a generic solution/convention that would work
> for arbitrary ROOT objects without MIDAS-specific wrappers (think TProfile,
> TGraph, etc...).
> 
> Any suggestions?
> 
> K.O.

> I can't get onto cvs@midas.psi.ch right now
> (cvs update
> cvs@midas.psi.ch's password: 
> Permission denied, please try again.)

I had to upgrade midas.psi.ch today with Scientific Linux 3.03. Most things are back to work, but
 I failed to do the anonymous CVS account. I have to wait for next week when the experts are
there. I will let you know when it's working again.

- Stefan

> > I can't get onto cvs@midas.psi.ch right now
> > (cvs update
> > cvs@midas.psi.ch's password: 
> > Permission denied, please try again.)

cvs@midas.psi.ch should be up and running again.

So now that cvs is reachable again I have confirmed that
the code segment
 
     } else if (obj->InheritsFrom( "TH1")) {
 
       // still don't know how to do TH1s

is indeed still present.
If you want me to look at this some more, you need to provide some code to exhibit the problem.

John.

> > The current h1_book() macros (and the previous example analyzer code) have an
> > odd persistency problem: for example, the user wants to change some histogram
> > limits, edits the h1_book() calls, rebuilds and restarts the analyzer, starts a
> > new run, and observes that all histograms are filled using the old limits, his
> > changes "did not take". The user panics, I get paged during the Holy Lunch Hour,
> > everybody is unhappy.
> > 
> > This is what I think happens:
> > 
> > 1) analyzer starts
> > 2) LoadRootHistgrams() loads old histograms from file
> 
> I can't get onto cvs@midas.psi.ch right now
> (cvs update
> cvs@midas.psi.ch's password: 
> Permission denied, please try again.)
> 
> but when I changed LoadRootHistograms a few days ago I left it as:
> 
>     } else if (obj->InheritsFrom( "TH1")) {
> 
>       // still don't know how to do TH1s
> 
> so h1_book() is creating the first and only copy of the histograms.
> I am able to create new histogram limits.
> I don't get the memory leak problems.
> 
> However I have seen the memory leak problems before, and they are real.
> They must be dealt with either by (1) first deleteing the old histogram
> or (2) ensuring that histogram names are unique in the whole application
> (different modules/folders can not use the same histogram names).
> 
> I will return to this once I can do a cvs update for midas.
> 
> John.
> 
> > 3) user code calls h1_book()
> > 4) h1_book template in midas.h does this (roughly):
> >       hist = (TH1X *) gManaHistosFolder->FindObjectAny(name);
> >       if (hist == NULL) {
> >          hist = new TH1X(name, title, bins, min, max);
> > 5) since the histogram already exists (loaded from the file, with the old
> > limits), the TH1X constructor is not called at all, new histogram limits are
> > utterly ignored.
> > 
> > A possible solution is to unconditionally create the ROOT objects, like I do in
> > the example code posted at <a
> href="<a
href="http://dasdevpc.triumf.ca:9080/Midas/191">http://dasdevpc.triumf.ca:9080/Midas/191</a>">http://dasdevpc.triumf.ca:9080/Midas/191"><a
href="http://dasdevpc.triumf.ca:9080/Midas/191</a>">http://dasdevpc.triumf.ca:9080/Midas/191</a></a></a>.
> That code
> > produces an annoying warning from ROOT about possible memory leaks. This could
> > be fixed by adding a two liner to "find and delete" the object before it is
> > created, trippling the number of user code lines per histogram (find & delete,
> > then create). Highly ugly.
> > 
> > midas.h macros (h1_book & co) can be fixed by adding checks for histogram limits
> > and such, but I would much prefer a generic solution/convention that would work
> > for arbitrary ROOT objects without MIDAS-specific wrappers (think TProfile,
> > TGraph, etc...).
> > 
> > Any suggestions?
> > 
> > K.O.

Dealing with the NaN's in the history system in the past week, a question came
up at PSI about how to deal with invalid history data.

Assume you have several devices going into one history equipment, and one device
has a problem, such that it cannot be read. In the past, the device driver
system returned zero, which was written to the history file. While this is ok in
some cases, it might not be in others, where zero is maybe a valid measurement.
Furthermore, it might confuse some regulations loops.

An alternative is to keep the last correctly measured value. As long as the
device has its problem, the value is kept. However, values are written to the
history system which might look like valid, although they are not. So what about
writing explicitly NaNs to the history system? For the display routine the NaNs
could be omitted, leaving blank regions where no valid measurement is available.
Or one could explicitly mare the region as invalid. Konstantin, do you know how
to write NaN explicitly to a float variable? And what do the others think about
these possibilities?

- Stefan

I preliminary implemented NaNs into the history system. It works such that if a
device driver returns a read error status, the class driver writes a NaN
(Not-a-Number) into the corresponding variable via the new function ss_nan(). The
"mhist" utility directly displays these as "nan" (Linux) or "-1.#IND00" under
Windows, indicating the error status. The history display via mhttpd just skips
these values (see elog:/1). I think this is better than showing just zero values,
because in most cases zero is a valid measurement and could confuse people.

Of course it is not enough just having "gaps" in the history display, so it's
important that the corresponding device driver issues an error message, which could
even trigger an alarm.

I have tested this under Windows, but only compiled under Linux. The only class
driver I modified so far is "multi.c". People should have a look, make some tests,
and let me know if this is a good thing, or if we should change it somehow.

- Stefan

Could someone please explain to me how cm_msg, cm_msg1, etc. all work.  The
documentation is very terse.  

I want to setup a fairly significant set of debugging, and error messages for a
new frontend.  I need to get these messages to a logging file.  I also would
like to get the error messages to the user through whatever interface Midas
normally uses for error reporting.  

Jan

> Could someone please explain to me how cm_msg, cm_msg1, etc. all work.  The
> documentation is very terse.  
> 
> I want to setup a fairly significant set of debugging, and error messages for a
> new frontend.  I need to get these messages to a logging file.  I also would
> like to get the error messages to the user through whatever interface Midas
> normally uses for error reporting.  

For errors, use

  cm_msg(MERROR, "routine_name", "Your error message, code=%d", i);

This produces an error message which is logged to midas.log, and distributed to all
clients which have called cm_msg_register(). For example odbedit will just print
that message. The syntax of the second half of cm_msg is the same as for printf(),
so you can add format specifiers and variable arguments as you do for printf(). The
first argument is the message type (MDEBUG for example is only distributed but not
logged). 

For a more detailed list of message types, please refer to

http://midas.triumf.ca/doc/html/AppendixE.html#midas_macro

I am commiting MIDAS modification accumulated during the last few months of running TWIST:
1) system.c::ss_shm_open() fail if trying to map a file that is smaller than we expect.
2) midas.c::bm_lock_buffer(), el_submit(), el_delete_message(): do not wait for mutexes forever, use a 5 
minute timeout. If we can't get the lock, cm_msg()/abort().
The above helps dealing with complete midas freezes. I also have code to keep track of "who locked
the mutex *and* is still holding it?!?" but it is way too ugly to commit. I wish we had a "lockedByPid"
entry for all lockable objects.
K.O.
 

> I am commiting MIDAS modification accumulated during the last few months of running TWIST:

More:
- mfe.c: in error messages "cannot find statistics record", also print
  the name of the record we are looking for.
- mlogger.c: in warning message "Write operation took N ms", report the name
  of the offending data stream.
- system.c: do not chdir("/") in ss_daemon_init()- it prevents us from ever
  getting core dumps from midas daemons. The old behaviour is trivially
  restored by "cd /" before starting the daemon; or by "limit coredumpsize 0".
- odb.c: db_validate_db() detect and break infinite looping on free list corruption.

K.O.

> > I am commiting MIDAS modification accumulated...

mhttpd changes:

- Renee's improvements on http transaction logging
- Implement "minimum" and "maximum" clamping for history graphs. Unfortunately
  there is no GUI code for changing the "minimum" and "maximum" settings,
  other than directly frobbing the odb.
- When making history graphs, detect NaNs in the history data.
(- status page code for the TWIST event builder (precursor of the standard
   event builder) stays uncommited).

K.O.

> - system.c: do not chdir("/") in ss_daemon_init()- it prevents us from ever
>   getting core dumps from midas daemons. The old behaviour is trivially
>   restored by "cd /" before starting the daemon; or by "limit coredumpsize 0".

The chdir("/") is from one of the unix text books. They say you HAVE to do it. If you start a
daemon on an NFS file system, you cannot unmount that file system as long as the daemon is
running. I'm sure the same code is inside most other daemons (apache, ...). So if we go away
from that standard, we have to be aware of the consequences.

> > - system.c: do not chdir("/") in ss_daemon_init()- it prevents us from ever
> >   getting core dumps from midas daemons.
> 
> The chdir("/") is from one of the unix text books. They say you HAVE to do it. If you start a
> daemon on an NFS file system, you cannot unmount that file system as long as the daemon is
> running.

Right, I remember this NFS problem from a while back.

This problem does not exist in the current crop of Linux systems (since Red Hat 7.3 at least) - they
either kill off all user programs or use "umount -f" and "umount -l".

"umount -l" works in any case to unmount a "busy" filesystem.

For systems where the NFS problem does still exist, one should do this: "mlogger -D" becomes "(cd /; mlogger -D)".

So I suspect that the "cd /" advice from the unix programming book is no longer as necessary
as it used to be. (Perhaps a better advice would have been to "cd /tmp", so we could still get
core dumps from non-root daemons).

K.O.

When i compile the experiment example of 1.9.5 the problem happened:

adccalib.c: In function `INT adc_calib_init()':
adccalib.c:114: `H1_BOOK' undeclared (first use this function)
adccalib.c:114: (Each undeclared identifier is reported only once for each
   function it appears in.)
make: *** [adccalib.o] Error 1

my ROOT is 4.01 and Zlib is 1.2.2

> When i compile the experiment example of 1.9.5 the problem happened:
> 
> adccalib.c: In function `INT adc_calib_init()':
> adccalib.c:114: `H1_BOOK' undeclared (first use this function)
> adccalib.c:114: (Each undeclared identifier is reported only once for each
>    function it appears in.)
> make: *** [adccalib.o] Error 1
> 
> my ROOT is 4.01 and Zlib is 1.2.2

We're in the process of fixing in the proper manner this problem, in the mean time
please add to the analyzer makefile the definition: -DUSE_ROOT at the line:
...
ROOTCFLAGS += -DHAVE_ROOT -DUSE_ROOT

What is the api call to determine the index of the frontend when specifying the
-i parameter during execution of the frontend? 

> What is the api call to determine the index of the frontend when specifying the
> -i parameter during execution of the frontend? 

INT get_frontend_index();

- Stefan

   We've had mhttpd aborting regularly since upgrading from midas-1.9.3.  This
happens during elog queries, and is due to an elog file that was incorrectly
modified by hand.  The modification to the file occurred 6 months ago.
   el_retrieve(midas.c:15683) now has several assert statements, one of which
aborts the program on reading the bad entry.

   Why is assert used, instead of an error return from the function (if
necessary), and maybe an error message in the log file?  Assert statements are
often removed, using NDEBUG, for normal use.

Chris

   The problem elog entry had one character removed, so end-of-file came before
the end of the message.  This could probably occur without the file being
altered, if the disk containing the elog fills.

>    We've had mhttpd aborting regularly since upgrading from midas-1.9.3.  This
> happens during elog queries, and is due to an elog file that was incorrectly
> modified by hand.

(sorry for delayed reply, for reasons unknown, I did not get an email notice when this was posted)

Yes, I agree, error handling in midas elog code is insufficient (note missing error checks for
read() and lseek() system calls). Anything but "perfect" elog files would cause funny errors and
malfunctions.

>  The modification to the file occurred 6 months ago.
>    el_retrieve(midas.c:15683) now has several assert statements, one of which
> aborts the program on reading the bad entry.

I added those to fix problems with "broken last NN days" and with infinite looping in the elog code
that we observed in TWIST.

You are welcome to replace the assert() statements with proper error handling. I used to have some code
that could report the filename of the bad elog file. Can we also report the exact file location for broken
files.

Please send me the diff, I will commit it to midas cvs.

>    Why is assert used, instead of an error return from the function (if
> necessary), and maybe an error message in the log file?  Assert statements are
> often removed, using NDEBUG, for normal use.

I use assert() in several ways:

0) I want a core dump each time X happens. (This is the only reasonable action when facing memory/stack
corruption. The problems in the elog code were stack corruption).
1) "I am too lazy to write proper error handling code" so I just crash and burn. This includes the
case where "proper error handling" would be "too invasive".
2) the error is too bad (or too deep) and there is no reasonable way to recover. Print an error message
and dump core (for later analysis). I sometimes use "cm_msg(); abort()". (assert is "printf("error"); abort()")

Please refer to literature for philosophic discussions on uses of assert() (Argh! Stefan will have my
head again!), but I will mention that "abort() early, abort() often" I find very effective. BTW, this technique
is heavily used in the Linux kernel (oops(), bug(), panic()) with some good effect, too.

>    The problem elog entry had one character removed, so end-of-file came before
> the end of the message.  This could probably occur without the file being
> altered, if the disk containing the elog fills.

Yes, I think you are right. In TWIST, we have seen disk-full conditions break both elog and history.

K.O.