Back Midas Rome Roody Rootana
  Midas DAQ System, Page 3 of 45  Not logged in ELOG logo
Entry  31 Oct 2003, Konstantin Olchanski, , mana.c without ROOT and HBOOK 
Stephan, why did you prohibit building mana.c without ROOT and HBOOK
support? I think such a configuration is valid and should be allowed.

Also, this prohibition broke the Midas Makefile, it now bombs building
mana.c. The Makefile is setup for building hmana.c with HBOOK support,
rmana.c with ROOT support (if ROOTSYS is set) and mana.c without HBOOK and
ROOT support (currently bombs on #error in mana.c).

K.O.
    Reply  01 Nov 2003, Stefan Ritt, , mana.c without ROOT and HBOOK 
> Stephan, why did you prohibit building mana.c without ROOT and HBOOK
> support? I think such a configuration is valid and should be allowed.

Oops, sorry, my fault. I forgto that people use mana.c without ROOT and 
HBOOK. The reason I made the change was that people forgot the -DHVAE_HBOOK 
in their makefile. In that case, no HBOOK init is done in mana.c and the 
first histogram booking in the user code crashes HBOOK.

So please take the #error statement out of mana.c (I'm away in two hours for 
one week), but think about preventing the above mentionend problem. I don't 
know any way for the makefile or mana.c to figure out if there is any HF1 
call in the user code. Actually HF1 should return a "proper" error message 
than just crashing.

One possibility is that we put an additional layer on top of the histogram 
boooking/filling. These macros are converted to their HBOOK or ROOT 
equivalents depending on the HAVE_HBOOK/HAVE_ROOT. If none of both is 
present, the histogram booking macro can produce a runtime error. This has 
the additional advantage that users can switch from HBOOK to ROOT without 
change of their user code.
       Reply  01 Nov 2003, Konstantin Olchanski, , mana.c without ROOT and HBOOK 
> > Stephan, why did you prohibit building mana.c without ROOT and HBOOK
> > support? I think such a configuration is valid and should be allowed.
> 
> Oops, sorry, my fault. I forgto that people use mana.c without ROOT and 
> HBOOK. The reason I made the change was that people forgot the -DHVAE_HBOOK 
> in their makefile. In that case, no HBOOK init is done in mana.c and the 
> first histogram booking in the user code crashes HBOOK.

Ahem. There is only so much rope we can give out to prevent people from shooting
themselves in the foot...

> So please take the #error statement out of mana.c

Done.

> One possibility is that we put an additional layer on top of the histogram 
> boooking/filling. These macros are converted to their HBOOK or ROOT 
> equivalents depending on the HAVE_HBOOK/HAVE_ROOT. If none of both is 
> present, the histogram booking macro can produce a runtime error. This has 
> the additional advantage that users can switch from HBOOK to ROOT without 
> change of their user code.

I can't think of anything other than wrapping every HBOOK call with "if
(!hbook_is_initialized) initialize_hbook();". But then, where is PAWC
coming from anyway?!?

We could also print a warning message "This mana.c has no HBOOK support. If you
see HBOOK crashes, please relink with hmana,c". Ugly, but informative, plus it
points anybody who knows how to read towards a solution.

K.O.
Entry  31 Oct 2003, Konstantin Olchanski, , Disable "tab"s in xemacs 
The default C indentation style in xemacs uses "tab" characters, violating
the MIDAS coding convention. To disable this misfeature in xemacs (emacs
too?), put this incantation in your .xemacs/custom.el file:

(custom-set-variables
 '(indent-tabs-mode nil))

K.O.
Entry  30 Oct 2003, Stefan Ritt, , Fixed several potential problems for ODB corruption 
I just realized that db_set_value, db_set_data, db_set_num_values and 
db_merge_data do not check for num_values == 0. With such a parameter the 
ODB can become corrupted, since zero length ODB entries are not allowed. I 
fixed the according places in odb.c and committed the changes. Everyone 
with ODB corruption problems should update that code.
Entry  30 Oct 2003, Stefan Ritt, , 'umask' added to lazylogger for FTP connections 
I had to add a 'umask' opiton to the loggers (lazy and mlogger) for the new 
PSI archive. One can now put a filename into the settings like:

archive,21,user,pw,dir,run%05d.mid,026

where the optional last parameter is used for a "umask 026" command just 
sent to the FTP server after the connection has been established. This 
changes the mode bits of the newly transferred file. We needed that so that 
the files are group readable, since several people from one group want to 
read the data.

I committed mlogger.c and ybos.c which contains the ftp code (should 
actually go into lazylogger.c instead of ybos.c).
Entry  16 Oct 2003, David Morris, , Updated thread functions 
ss_thread_create now returns the thread ID on success, and zero on failure.
Previously returned SS_SUCCESS or SS_NO_THREAD. User must now test the
return value to determine result.

ss_thread_kill added to kill the passed thread ID. Returns SS_SUCCESS or
SS_NO_THREAD.

Any thread creation must be verified now, and old code must be examined to
ensure the return value is checked.
    Reply  28 Oct 2003, Stefan Ritt, , Updated thread functions 
> ss_thread_create now returns the thread ID on success, and zero on failure.
> Previously returned SS_SUCCESS or SS_NO_THREAD. User must now test the
> return value to determine result.
> 
> ss_thread_kill added to kill the passed thread ID. Returns SS_SUCCESS or
> SS_NO_THREAD.
> 
> Any thread creation must be verified now, and old code must be examined to
> ensure the return value is checked.

Thank you for that post. Internally, threads are not use in midas, so there 
should be no problem. Only experiments using threads explicitly should take 
care.
Entry  15 Oct 2003, Konstantin Olchanski, , test 
test
test
test
    Reply  15 Oct 2003, Konstantin Olchanski, , test 
> test
> test
> test

another test

K.O.
       Reply  15 Oct 2003, Stefan Ritt, , test 
> > test
> > test
> > test
> 
> another test
> 
> K.O.

I got the two email notifications, if you have tried that...
Entry  12 Oct 2003, Konstantin Olchanski, , mhttpd: add Elog text to outgoing email. 
This commit adds the elog message text to the outgoing email message. This
functionality has been requested a logn time ago, but I guess nobody got
around to implement it, until now. I also added assert() traps for the most
common array overruns in the Elog code.

Here is the cvs diff:

Index: src/mhttpd.c
===================================================================
RCS file: /usr/local/cvsroot/midas/src/mhttpd.c,v
retrieving revision 1.252
diff -r1.252 mhttpd.c
768a769
> #include <assert.h>
3740c3741
< char   mail_to[256], mail_from[256], mail_text[256], mail_list[256],
---
> char   mail_to[256], mail_from[256], mail_text[10000], mail_list[256],
3921a3923,3925
>         // zero out the array. needed because later strncat() does not
always add the trailing '\0'
>         memset(mail_text,0,sizeof(mail_text));
> 
3931a3936,3945
> 
>         assert(strlen(mail_text) + 100 < sizeof(mail_text)); // bomb out
on array overrun.
> 
>         strcat(mail_text+strlen(mail_text),"\n");
>         // this strncat() depends on the mail_text array being zeroed out:
>         // strncat() does not always add the trailing '\0'
>        
strncat(mail_text+strlen(mail_text),getparam("text"),sizeof(mail_text)-strlen(mail_text)-50);
>         strcat(mail_text+strlen(mail_text),"\n");
> 
>         assert(strlen(mail_text) < sizeof(mail_text)); // bomb out on
array overrun.
Index: src/midas.c
===================================================================
RCS file: /usr/local/cvsroot/midas/src/midas.c,v
retrieving revision 1.192
diff -r1.192 midas.c
604a605
> #include <assert.h>
16267a16269,16270
> 
>   assert(strlen(message) < sizeof(message)); // bomb out on array overrun.

K.O.
    Reply  13 Oct 2003, Stefan Ritt, , mhttpd: add Elog text to outgoing email. 
> around to implement it, until now. I also added assert() traps for the most
> common array overruns in the Elog code.

In addition to the assert() one should use strlcat() and strlcpy() all over 
the code to avoid buffer overruns. The ELOG standalone code does that already 
properly.

- Stefan
       Reply  13 Oct 2003, Konstantin Olchanski, , mhttpd: add Elog text to outgoing email. 
> > around to implement it, until now. I also added assert() traps for the most
> > common array overruns in the Elog code.
> 
> In addition to the assert() one should use strlcat() and strlcpy() all over 
> the code to avoid buffer overruns. The ELOG standalone code does that already 
> properly.
> 
> - Stefan

Yes, the original authors should have used strlcat(). Now that I uncovered this source of mhttpd 
memory corruption, maybe some volunteer will fix it up properly.

K.O.
          Reply  13 Oct 2003, Stefan Ritt, , mhttpd: add Elog text to outgoing email. 
> > > around to implement it, until now. I also added assert() traps for the 
most
> > > common array overruns in the Elog code.
> > 
> > In addition to the assert() one should use strlcat() and strlcpy() all 
over 
> > the code to avoid buffer overruns. The ELOG standalone code does that 
already 
> > properly.
> > 
> > - Stefan
> 
> Yes, the original authors should have used strlcat(). Now that I uncovered 
this source of mhttpd 
> memory corruption, maybe some volunteer will fix it up properly.
> 
> K.O.

I am the original author and will fix all that once I merged mhttpd and elog. 
Due to my current task list, this will happen probably in November.

- Stefan
Entry  12 Oct 2003, Konstantin Olchanski, , Array overruns in mhttpd.c::submit_elog() 
While adding new functionality to submit_elog() (add the message text to the
outgoing email), I noticed that the email text is being stored into an array
of size 256, mail_text[256], without any checks for array overrun. This
cannot be good. How should this be corrected?
K.O.
    Reply  12 Oct 2003, Konstantin Olchanski, , Array overruns in mhttpd.c::submit_elog() 
> While adding new functionality to submit_elog() (add the message text to the
> outgoing email), I noticed that the email text is being stored into an array
> of size 256, mail_text[256], without any checks for array overrun. This
> cannot be good. How should this be corrected?
> K.O.

Similar problem exists in midas.c::el_submit(). The array "message[10000]" is
easy to overrun by submitting a long elog message.

K.O.
       Reply  13 Oct 2003, Stefan Ritt, , Array overruns in mhttpd.c::submit_elog() 
> > While adding new functionality to submit_elog() (add the message text to 
the
> > outgoing email), I noticed that the email text is being stored into an 
array
> > of size 256, mail_text[256], without any checks for array overrun. This
> > cannot be good. How should this be corrected?
> > K.O.
> 
> Similar problem exists in midas.c::el_submit(). The array "message[10000]" 
is
> easy to overrun by submitting a long elog message.
> 
> K.O.

The whole elog functionality in mhttpd will be replaced (sometime) by the 
standalone ELOG package, linked against mhttpd. The ELOG functionality is 
much richer and does not conatin all the mentioned problems which have been 
fixed there some time ago. For the time being it might however be worth to 
fix the mentioned problems, but without spending too much time on it.
          Reply  13 Oct 2003, Konstantin Olchanski, , Array overruns in mhttpd.c::submit_elog() 
> > > While adding new functionality to submit_elog() ....
> 
> The whole elog functionality in mhttpd will be replaced (sometime) ...

I humbly submit that this has been the standard reply for the last 2 years since I was aware of 
the "last N days does not always work" problem (just saw it again yesterday).

K.O.
Entry  12 Oct 2003, Konstantin Olchanski, , Refuse to set run number zero 
I am debugging the frequent problem where the run number is mysteriously
reset to zero. As a first step, I am commiting changes to mhttpd.c and midas.c:
- abort on obviously corrupted "run number < 0"
- abort on cm_transition() to run 0 (the only place where the run number is
explicitely written to ODB)
- in the mhttpd "Start run" form, reject user setting the run number to <= 0.

Here is the CVS diff:

===================================================================
RCS file: /usr/local/cvsroot/midas/src/mhttpd.c,v
retrieving revision 1.253
diff -r1.253 mhttpd.c
2451a2452,2457
>   if (run_number < 0)
>     {
>     cm_msg(MERROR, "show_elog_new", "aborting on attempt to use invalid
run number %d",run_number);
>     abort();
>     }
> 
2506a2513,2519
> 
>     if (run_number < 0)
>       {
>       cm_msg(MERROR, "show_elog_new", "aborting on attempt to use invalid
run number %d",run_number);
>       abort();
>       }
> 
3582a3596,3602
> 
>   if (run_number < 0)
>     {
>     cm_msg(MERROR, "show_form_query", "aborting on attempt to use invalid
run number %d",run_number);
>     abort();
>     }
> 
5730a5751,5756
>   if (rn < 0) // value "zero" is okey
>     {
>     cm_msg(MERROR, "show_start_page", "aborting on attempt to use invalid
run number %d",rn);
>     abort();
>     }
> 
9684a9711,9719
>       if (i <= 0)
>         {
>         cm_msg(MERROR, "interprete", "Start run: invalid run number %d",i);
>         memset(str,0,sizeof(str));
>         snprintf(str,sizeof(str)-1,"Invalid run number %d",i);
>         show_error(str);
>         return;
>         }
> 
Index: src/midas.c
===================================================================
RCS file: /usr/local/cvsroot/midas/src/midas.c,v
retrieving revision 1.193
diff -r1.193 midas.c
3786c3786
<         status = cm_transition(_requested_transition | TR_DEFERRED, 0,
str, 256, SYNC, FALSE);
---
>         status = cm_transition(_requested_transition | TR_DEFERRED, 0,
str, sizeof(str), SYNC, FALSE);
3906a3907,3912
>   if (run_number <= 0)
>     {
>     cm_msg(MERROR, "cm_transition", "aborting on attempt to use invalid
run number %d",run_number);
>     abort();
>     }
> 
16069a16076,16081
>     }
> 
>   if (run_number < 0)
>     {
>     cm_msg(MERROR, "el_submit", "aborting on attempt to use invalid run
number %d", run_number);
>     abort();

K.O.
    Reply  12 Oct 2003, Konstantin Olchanski, , Refuse to set run number zero 
> I am debugging the frequent problem where the run number is mysteriously
> reset to zero. As a first step, I am commiting changes to mhttpd.c and midas.c:
> - abort on obviously corrupted "run number < 0"
> - abort on cm_transition() to run 0 (the only place where the run number is
> explicitely written to ODB)
> - in the mhttpd "Start run" form, reject user setting the run number to <= 0.

- abort on cm_transition() from run 0 to 1 during auto restart in mlogger.

Cvs diff:

RCS file: /usr/local/cvsroot/midas/src/mlogger.c,v
retrieving revision 1.65
diff -r1.65 mlogger.c
3277a3278,3283
>         if (run_number <= 0)
>           {
>           cm_msg(MERROR, "main", "aborting on attempt to use invalid run
number %d", run_number);
>           abort();
>           }
> 

K.O.
Entry  11 Aug 2003, Konstantin Olchanski, , mhttpd crash on corrupted ODB /RunInfo 
Invalid values of ODB /RunInfo/State cause mhttpd crash in
show_status_page() because of an out of bounds access to the array of state
names. Suggest this fix: remove array of state names, use existing ladder of
if/else statements to explicitely set state name. Verified the fix works for
TWIST. Will commit this into MIDAS CVS unless get feedback.

src/mhttpd.c:show_status_page() {
  ...
  rsprintf("<tr align=center><td>Run #%d", runinfo.run_number);

  if (runinfo.state == STATE_STOPPED)
    rsprintf("<td colspan=1 bgcolor=#FF0000>Stopped");
  else if (runinfo.state == STATE_PAUSED)
    rsprintf("<td colspan=1 bgcolor=#FFFF00>Paused");
  else if (runinfo.state == STATE_RUNNING)
    rsprintf("<td colspan=1 bgcolor=#00FF00>Running");
  else
    rsprintf("<td colspan=1 bgcolor=#FFFFFF>Unknown");

  if (runinfo.requested_transition)
  ...

K.O.
    Reply  10 Oct 2003, Konstantin Olchanski, , mhttpd crash on corrupted ODB /RunInfo 
There was no feedback. This code has been commited. K.O.

> Invalid values of ODB /RunInfo/State cause mhttpd crash in
> show_status_page() because of an out of bounds access to the array of state
> names. Suggest this fix: remove array of state names, use existing ladder of
> if/else statements to explicitely set state name. Verified the fix works for
> TWIST. Will commit this into MIDAS CVS unless get feedback.
> 
> src/mhttpd.c:show_status_page() {
>   ...
>   rsprintf("<tr align=center><td>Run #%d", runinfo.run_number);
> 
>   if (runinfo.state == STATE_STOPPED)
>     rsprintf("<td colspan=1 bgcolor=#FF0000>Stopped");
>   else if (runinfo.state == STATE_PAUSED)
>     rsprintf("<td colspan=1 bgcolor=#FFFF00>Paused");
>   else if (runinfo.state == STATE_RUNNING)
>     rsprintf("<td colspan=1 bgcolor=#00FF00>Running");
>   else
>     rsprintf("<td colspan=1 bgcolor=#FFFFFF>Unknown");
> 
>   if (runinfo.requested_transition)
>   ...
> 
> K.O.
Entry  02 Sep 2003, Pierre-André Amaudruz, , minor fix, window build 
- makefile.nt (/examples/experiment, /hbook)
  adjusted for local hmana.obj build  as for rmana.obj, add cvs tag for
  revision comment entry.
- drivers/class/hv.c
  change comment // to /* */
Entry  19 Aug 2003, Pierre-André Amaudruz, , minor fixes, new tarball 1.9.3-1 
- add pthread lib to examples/... makefile
- fix ybos_simfe.c for max_event_size
- fix camacnul.c for cam_inhibit_test(), cam_interrupt_test()
- update documentation (1.9.3)
- made midas-1.9.3-1.tar.gz on Triumf site
Entry  29 Jul 2003, Konstantin Olchanski, , Have to link with -lpthread? 
It appears that all midas applications are now required to link with the
pthreads library even if they do not use threads. This is caused by a
pthread_create() call from ss_thread_create() in system.c.

Is this the intended behaviour?

K.O.
    Reply  30 Jul 2003, David Morris, , Have to link with -lpthread? 
The change is required to support implementation of pthreads in the Linux
compile of Midas. This was added recently. I believe pthreads is also needed
for ROOT based compiles.

David

> It appears that all midas applications are now required to link with the
> pthreads library even if they do not use threads. This is caused by a
> pthread_create() call from ss_thread_create() in system.c.
> 
> Is this the intended behaviour?
> 
> K.O.
Entry  26 Jul 2003, Konstantin Olchanski, , use "odbedit -C" to connect to corrupted ODB 
Add switch "-C" to odbedit to allow it to connect to corrupted ODB. Then,
depending on corruption, the user can manually remove or correct the
corrupted entries. Also, some corruption is automatically fixed by "odbedit"
itself. I use this functionality to debug and fix broken ODBs.

K.O.

For your enjoyment, here is the diff:

diff -r1.64 odbedit.c
3058a3059
> BOOL          corrupted;
3063c3064
<   debug = cmd_mode = FALSE;
---
>   debug = corrupted = cmd_mode = FALSE;
3077a3079,3080
>     else if (argv[i][0] == '-' && argv[i][1] == 'C')
>       corrupted = TRUE;
3104c3107,3108
<         printf("               [-c Command] [-c @CommandFile] [-s size]
[-g (debug)]\n\n");
---
>         printf("               [-c Command] [-c @CommandFile] [-s size]\n");
>         printf("               [-g (debug)] [-C (connect to corrupted
ODB)]\n\n");
3123c3127,3133
<   if (status != CM_SUCCESS)
---
>   else if ((status == DB_INVALID_HANDLE)&&corrupted)
>     {
>     cm_get_error(status, str);
>     puts(str);
>     printf("ODB is corrupted, connecting anyway...\n");
>     }
>   else if (status != CM_SUCCESS)
Entry  26 Jul 2003, Konstantin Olchanski, , more ODB checks in src/odb.c 
Add more checks to db_validate_key() for pkey->total_size, item_size and
num_values. Automatically correct total_size to be item_size*num_values (we
saw this corruption and tested this fix).

K.O.

For your enjoyment, here is the diff:

RCS file: /usr/local/cvsroot/midas/src/odb.c,v
retrieving revision 1.64
diff -r1.64 odb.c
718a719,744
>   /* check key sizes */
>   if ((pkey->total_size < 0)||(pkey->total_size > pheader->key_size))
>     {
>     cm_msg(MERROR, "db_validate_key", "Warning: invalid key \"%s\"
total_size: %d", path, pkey->total_size);
>     return 0;
>     }
> 
>   if ((pkey->item_size < 0)||(pkey->item_size > pheader->key_size))
>     {
>     cm_msg(MERROR, "db_validate_key", "Warning: invalid key \"%s\"
item_size: %d", path, pkey->item_size);
>     return 0;
>     }
> 
>   if ((pkey->num_values < 0)||(pkey->num_values > pheader->key_size))
>     {
>     cm_msg(MERROR, "db_validate_key", "Warning: invalid key \"%s\"
num_values: %d", path, pkey->num_values);
>     return 0;
>     }
> 
>   /* check and correct key size */
>   if (pkey->total_size != pkey->item_size*pkey->num_values)
>     {
>     cm_msg(MINFO,  "db_validate_key", "Warning: corrected key \"%s\" size:
total_size=%d, should be %d*%d=%d", path, pkey->total_size, pkey->item_size,
pkey->num_values, pkey
->item_size*pkey->num_values);
>     pkey->total_size = pkey->item_size*pkey->num_values;
>     }
> 
Entry  26 Jun 2003, David Morris, , pthreads for Linux 
Added ss_create_thread support for Linux in system.c
Added pthread library in main make file
Entry  17 Jun 2003, Stefan Ritt, , example experiment makefile for NT 
I have added ROOT support to midas\examples\experiment\makefile.nt. To 
compile the example experiment under Windows, one needs

1) Installed version of ROOT
2) Having ROOTSYS environment variable defined
3) Invoke "nmake -f makefile.nt" in the midas\examples\experiment directory

Please note that in the current release 3.05 of ROOT, sockets are not yet 
working under Windows, so the histogram server built into the analyzer 
cannot be accessed. It is however possible to output the analyzed data into 
a .root file and visualize it with the root browser like

analyzer -i run00001.mid -o run00001.root
Entry  12 Jun 2003, Pierre-André Amaudruz, , Tape handling 
- remove ss_tape_get_blockn from lazylogger.c
- add ss_tape_get_blockn to system.c
- add ss_tape_get_blockn prototype into midas.h
- fix buffer size for "dir" in mtape.c
- add block# for "dir" in mtape if command successful.
- handle TID_STRUCT bank type by display as 8bit in ybos.c (mdump)
Entry  28 Sep 2004, Piotr Zolnierczuk, Forum, MIDAS/MVME167/Linux 
Hi,
 has anyone tried runnning midas frontend on a Linux running 
on a Motorola MVME167 motorola embedded CPU?
I have seen people running Linux on a MV167 
(http://www.sleepie.demon.co.uk/linuxvme/)
so in principle this can be done.

The reason I am asking is that we have a lot of them in house 
and we would like to avoid paying for VxWorks
(I have succesfully run Midas on a mvme167/VxWorks node)

Or maybe one has come up with a much better solution 
[short of dumping mv167 into a sewer :)]

Piotr
Entry  03 Oct 2004, Stefan Ritt, Info, Introduction of new transition scheme 
A new transition scheme has been implemented and committed. Previously, one had the
possibility to register for PRE/POST transitions, which was necessary in order to first
stop the frontends, then stop the logger to close the data file. While this scheme
long time has proven to be successful, it was now concluded that three levels
(PRESTROP/STOP/POSTSTOP for example) are not suffucient in some cases. Therefore,
a true sequence-based scheme has been introduced, implemented and committed.

The PRE/POST transition have been removed and an extra parameter "sequence_number"
has been added to cm_register_transition. If clients register with different
sequence numbers, their RPC transition function is executed according to their
sequnce number, smaller numbers being executed prior to larger numbers.

The frontends register at sequence number 500 for example, while the logger
registers with 200 for start and 800 for stop, making sure it's called after the
frontend(s) when stopping a run. The default numbers can be changed from within
the user code with the new function cm_set_transition_sequence(). This way, it is
for example possible to have all frontends being called in a certain sequence
when starting and stopping runs.

The modification will (hopefully) not have any influence of existing experiemnts,
as long as they don't call cm_register_transition directly. If so however, one has
to add the additional parameter to this function.
ELOG V3.1.4-2e1708b5