Back Midas Rome Roody Rootana
  Midas DAQ System, Page 121 of 137  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
IDdown Date Author Topic Subject
  321   11 Jan 2007 Steve HardyForumShared memory problems
Thanks for your help.  I tried again and it got me back to the initial problem I had.
 The frontend will start, and the analyzer starts (complains about there not being a
last.root, but other than that it's fine), and then when starting mlogger, I get:

[odb.c:860:db_validate_db] Warning: database corruption, first_free_key 0x0001A4
04
[odb.c:3666:db_get_key] invalid key handle
[midas.c:1970:cm_check_client] cannot delete client info
[odb.c:3666:db_get_key] invalid key handle
[midas.c:1970:cm_check_client] cannot delete client info
[odb.c:3666:db_get_key] invalid key handle


And it continues to shoot out error messages about invalid key handles until I kill
it.  Then trying to start the frontend again fails until I remove the .ODB.SHM file. 
Any other ideas?

> > Hello,
> > 
> > Just did a fresh install of MIDAS from the SVN repository under CentOS and
> > everything compiles fine, but when I go to run the frontend (using dio), I get
> > the following error message:
> > 
> > Connect to experiment ...[odb.c:868:db_open_database] Different database format:
> >  Shared memory is 14, program is 2
> > [midas.c:1763:cm_connect_experiment1] cannot open database
> > 
> > 
> > Any ideas on what the problem could be, or how to fix it?  
> 
> You have an old .ODB.SHM from a previous version in your directoy (note the '.' in
> front, so you need a 'ls -alg' to see it). Delete that file and try again.
  320   11 Jan 2007 Stefan RittForumShared memory problems
> Hello,
> 
> Just did a fresh install of MIDAS from the SVN repository under CentOS and
> everything compiles fine, but when I go to run the frontend (using dio), I get
> the following error message:
> 
> Connect to experiment ...[odb.c:868:db_open_database] Different database format:
>  Shared memory is 14, program is 2
> [midas.c:1763:cm_connect_experiment1] cannot open database
> 
> 
> Any ideas on what the problem could be, or how to fix it?  

You have an old .ODB.SHM from a previous version in your directoy (note the '.' in
front, so you need a 'ls -alg' to see it). Delete that file and try again.
  319   11 Jan 2007 Steve HardyForumShared memory problems
Hello,

Just did a fresh install of MIDAS from the SVN repository under CentOS and
everything compiles fine, but when I go to run the frontend (using dio), I get
the following error message:

Connect to experiment ...[odb.c:868:db_open_database] Different database format:
 Shared memory is 14, program is 2
[midas.c:1763:cm_connect_experiment1] cannot open database


Any ideas on what the problem could be, or how to fix it?  


~Steve
  318   08 Jan 2007 Stefan RittSuggestionAccess to out_info from mana.c
I changed out_info into a global structure definition ANA_OUTPUT_INFO and put it into
midas.h, so it can be accessed easily from the user analyzer source code.

> Would it be relevant to transform out_info into a *non-static* variable of a type
> defined by a *named* struct?
> Currently,  programs that  try to access out_info cannot do it anymore; and they
> typically copy the struct definition from mana.c, which is not robust against future
> changes in mana.c.
> 
> If mana.c could be changed in the way described above, that would be great . 
> Otherwise, is it safe to patch it myself for local use?  or is there a better way of
> accessing out_info from mana.c?
> 
> As always, any help would be much appreciated :)
> 
> EOL
> 
> > Hello,
> > 
> > Is it possible to access out_info (defined in mana.c) from another program?
> > 
> > In fact, out_info is now defined as an (anonymous) "static struct" in mana.c,
> > which it seems to me precludes any direct use in another program.  Is there an
> > indirect way of getting ahold of out_info?  or of the information it contains?
> > 
> > out_info used to be defined as a *non-static* struct, and the code I'm currently
> > modifying used to compile seamlessly: it now stops the compilation during
> > linking time, as out_info is now static and the program I have to compile
> > contains an "extern struct {} out_info".
> > 
> > Any help would be much appreciated!  I searched in vain in this forum for
> > details about out_info and I really need to access the information it contains!
> > 
> > EOL (a pure MIDAS novice)
  317   05 Jan 2007 Eric-Olivier LE BIGOTSuggestionAccess to out_info from mana.c
Would it be relevant to transform out_info into a *non-static* variable of a type
defined by a *named* struct?
Currently,  programs that  try to access out_info cannot do it anymore; and they
typically copy the struct definition from mana.c, which is not robust against future
changes in mana.c.

If mana.c could be changed in the way described above, that would be great . 
Otherwise, is it safe to patch it myself for local use?  or is there a better way of
accessing out_info from mana.c?

As always, any help would be much appreciated :)

EOL

> Hello,
> 
> Is it possible to access out_info (defined in mana.c) from another program?
> 
> In fact, out_info is now defined as an (anonymous) "static struct" in mana.c,
> which it seems to me precludes any direct use in another program.  Is there an
> indirect way of getting ahold of out_info?  or of the information it contains?
> 
> out_info used to be defined as a *non-static* struct, and the code I'm currently
> modifying used to compile seamlessly: it now stops the compilation during
> linking time, as out_info is now static and the program I have to compile
> contains an "extern struct {} out_info".
> 
> Any help would be much appreciated!  I searched in vain in this forum for
> details about out_info and I really need to access the information it contains!
> 
> EOL (a pure MIDAS novice)
  316   27 Dec 2006 Eric-Olivier LE BIGOTForumAccess to out_info from mana.c
Hello,

Is it possible to access out_info (defined in mana.c) from another program?

In fact, out_info is now defined as an (anonymous) "static struct" in mana.c,
which it seems to me precludes any direct use in another program.  Is there an
indirect way of getting ahold of out_info?  or of the information it contains?

out_info used to be defined as a *non-static* struct, and the code I'm currently
modifying used to compile seamlessly: it now stops the compilation during
linking time, as out_info is now static and the program I have to compile
contains an "extern struct {} out_info".

Any help would be much appreciated!  I searched in vain in this forum for
details about out_info and I really need to access the information it contains!

EOL (a pure MIDAS novice)
  315   26 Oct 2006 Hans FynboForumSetup of Ortec ADC AD413A in MIDAS
We are new to MIDAS and try to setup a simple system with one ortec camac ADC
AD413A and the hytec1331 controler. Has anyone used this module in MIDAS we
would be grateful for the corresponding frontend.c etc. 

It would be very useful to have somewhere examples of files used by various
experiments in addition to the example files provided in the installation.

Best regards,
Hans 
  314   16 Oct 2006 Stefan RittBug FixBuild error with mana.c while using CERNLIB, svn 3366
Committed, thanks.
  313   16 Oct 2006 Stefan RittBug Fix"make install" error on MacOS 10.4.7, svn 3366
> While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the 
> problem of "Makefile". I did some change to it and attach the diff file here.

I committed your patch. Thank you.
  312   16 Oct 2006 Exaos LeeBug Fix"make install" error on MacOS 10.4.7, svn 3366
While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the 
problem of "Makefile". I did some change to it and attach the diff file here.
  311   16 Oct 2006 Exaos LeeBug FixBuild error with mana.c while using CERNLIB, svn 3366
If you use CERNLIB to build hmana.o, you may encounter the following error:
src/mana.c: In function ‘write_event_hbook’:
src/mana.c:2881: error: invalid assignment
or somthing like this:
src/mana.c: In function ‘write_event_hbook’:
src/mana.c:2881: warning: target of assignment not really an lvalue; this will be a hard error in the future
So I checked the mana.c and found these lines
2880            /* shift data pointer to next item */
2881            (char *) pdata += key.item_size * key.num_values;
should be changed to
2880            /* shift data pointer to next item */
2881            pdata += key.item_size * key.num_values * sizeof(char) ;
  310   28 Sep 2006 Stefan RittBug Reportmhttpd elog corruption via double-edit
> I do not know how to "properly" fix this bug without changing the indexing
> scheme to something similar to what is used by elogd- message numbers instead of
> file indices. In the existing scheme, message editing also breaks URLs shown in
> the email notifications (they contain file indices that point to the wrong
> places after messages are moved around by editing) and "reply threading" links.

Well, the development of elogd with it's message numbers was actually stimulated by
the problem you mentioned. After that all those problems went away. Another
incarnation of that problem is if you edit an mhttpd log file manually. Afterwards
the file offsets are different and the system gets corrupted. To fix this properly,
one would have to backport the el_xxx functions from elogd to mhttpd, or, even
simpler, remove the elog functionality in mhttpd and "force" everybody to use elogd
(after doing elconv to convert the files into the new format).
  309   28 Sep 2006 Stefan RittSuggestionIncrease of maximum event size

K.O. wrote:
Now, we have per-buffer tunable size (see message
https://ladd00.triumf.ca/elog/Midas/283) and in the long run, I would prefer the
compiled-in limit to go away: already all memory is allocated dynamically and
the MAX_EVENT_SIZE is only useful as kind of a sanity check against frontend
misconfiguration or against malformed events.

If MAX_EVENT_SIZE goes away, the maximum event size becomes limited by the
largest SysV shared memory segment permitted by Linux (via sysctl kernel.shmmax).

To go beyound the limit on SysV shared memories, on can use mmap() based shared
memory: this is limited by available RAM+swap (and disk space for the
.SYSTEM.SHM file). Current MIDAS system.c has an experimental implementation of
mmap() shared memory, but AFAIK it has not been used in any production system, yet.


MAX_EVENT_SIZE is also used for the RPC layer, since the receiving buffer must hold at
least one event. It is right that this can and should be made dynamically. Concerning
the shared memory there is the problem that it cannot be increased when any program is
running and attached to the shared memory, so it can only be defined at startup of the
first program creating the shared memory.

The sanity check in the frontend is done against max_event_size defined in frontend.c which can be smaller than MAX_EVENT_SIZE (some front-ends have limited memory).

So I agree that this issue may need revision, maybe something for me next visit Wink
  308   27 Sep 2006 Konstantin OlchanskiSuggestionIncrease of maximum event size
> The current event size in midas is limited to 512k (MAX_EVENT_SIZE in midas.h)

Yes, 512 kBytes is rather small. For the T2K prototype TPC DAQ, I built and ran
MIDAS with 4 MByte events, and it worked fine.

Now, we have per-buffer tunable size (see message
https://ladd00.triumf.ca/elog/Midas/283) and in the long run, I would prefer the
compiled-in limit to go away: already all memory is allocated dynamically and
the MAX_EVENT_SIZE is only useful as kind of a sanity check against frontend
misconfiguration or against malformed events.

If MAX_EVENT_SIZE goes away, the maximum event size becomes limited by the
largest SysV shared memory segment permitted by Linux (via sysctl kernel.shmmax).

To go beyound the limit on SysV shared memories, on can use mmap() based shared
memory: this is limited by available RAM+swap (and disk space for the
.SYSTEM.SHM file). Current MIDAS system.c has an experimental implementation of
mmap() shared memory, but AFAIK it has not been used in any production system, yet.

K.O.
  307   27 Sep 2006 Konstantin OlchanskiBug Reportmhttpd elog corruption via double-edit
[quote="Stefan Ritt"][Quote="K.O.]Aparently the mhttpd elog will corrupt the
elog files if two (or more\?) elog entries are being edited at the same time.
K.O.[/quote]

The corruption is very simple. mhttpd elog indexes the elog entries by the elog
file and offset inside the file, i.e. "http://ladd00:8088/EL/060927.318",
"060927" corresponds to log file "060927.log", "318" is the offset inside the
file where the message is located.

During "edit", the code "remembers" the offset of the original message and in
el_submit() blindly writes the edited message into the file at the remembered
offset.

If another message was edited before the edit of the first message is submitted,
the remembered offset becomes invalid (messages have shifted inside the file)
and el_submit() writes the edited text into the wrong place in the file,
corrupting it.

I have now added a check for this and we crash instead of corrupting the elog
file (midas.c rev 3340).

I do not know how to "properly" fix this bug without changing the indexing
scheme to something similar to what is used by elogd- message numbers instead of
file indices. In the existing scheme, message editing also breaks URLs shown in
the email notifications (they contain file indices that point to the wrong
places after messages are moved around by editing) and "reply threading" links.

Here is how I reproduce this bug:

1) start with an empty elog
2) create two messages
3) "edit" the second message, but do not submit it yet.
4) "edit" the first message, change the text to make sure the message size
becomes different; submit this change.
5) submit the "edit" of the first message. !!BOOM!!

K.O.
  306   24 Sep 2006 Stefan RittBug Reportmhttpd elog corruption via double-edit

K.O. wrote:
Aparently the mhttpd elog will corrupt the elog files if two (or more\?) elog entries are being edited at the same time. K.O.


That's strange. Since mhttpd is single threaded, there should not be any multi-thread/process conflict there, since the elog files cannot be written simultaneously from two different browser sessions. If entries are edited at the same time, they get then submitted one after the other. Of course it is possible to edit the same entry, in which case the second submission "wins", overwriting the first one without notification. Withing the standalone elog server there is the option to lock entries ("use lock = 1") to prevent this, but this feature is not present in the mhttpd elog.
  305   23 Sep 2006 Konstantin OlchanskiBug Fixcommit latest ccusb.c CAMAC-USB driver
> I commited the latest driver for the Wiener CCUSB USB-CAMAC driver. It
> implements all functions from mcstd.h and has been tested to be plug-compatible
> with at least one of our CAMAC frontends. K.O.

This driver is known to not work with the latest CCUSB firmware (20x, 204, 30x, 303). I know what 
modifications are required and an updated driver will be available shortly. If there is a delay, and you need the 
driver ASAP, please drop me an email.

Also, I am thinking about dropping support for the very old CCUSB firmware revisions (before 204). (Any 
comments?)

K.O.
  304   23 Sep 2006 Konstantin OlchanskiBug Reportmhttpd elog corruption via double-edit
Aparently the mhttpd elog will corrupt the elog files if two (or more?) elog entries are being edited at the 
same time. K.O.
  303   20 Sep 2006 Stefan RittSuggestionIncrease of maximum event size
Since nobody complained so far, I increased MAX_EVENT_SIZE to 2MB. If anybody has problems with this setting, please report. Note that after updating to SVN revision 3327 it will be necessary to recompile all midas programs and to delete any old SYSTEM.SHM or .SYSTEM.SHM. I added some code which should check for inconsistent SYSTEM.SHM sizes, but I'm not sure if it works everywhere.
  302   20 Sep 2006 Stefan RittSuggestionIncrease of maximum event size
Dear midas users,

The current event size in midas is limited to 512k (MAX_EVENT_SIZE in midas.h). This is mainly due to old (pre 2.2) linux kernels which had only a very limited shared memory pool. These days this limit has increased considerably and I question if we should increase the default event size and to which size we should increase it.

The drawback of a larger event size is that the SYSTEM event buffer has to hold at least two events, and when the last midas program is stopped or started, this buffer has to be written to or read from the .SYSTEM.SHM file, which slows down the start/stop of the program. But writing/reading a few MB is fast these days anyhow so this again might now be a big problem. So what do you think how big we should make the default max event size?

- Stefan
ELOG V3.1.4-2e1708b5