13 Feb 2008, Konstantin Olchanski, Info, mhttpd history display updates
|
I now merged almost all the mhttpd changes from CERN AD-5/ALPHA. Only the code
for mhttpd HTTP:// access control list remains unmerged.
Changes to the history display code merged from ALPHA:
- add option to show latest values of history variables: "show values of
variables" check box on the history config panel
- add custom labels for each variables: instead of midas variable name, history
plots would show the text entered into the "label" text area
- show history errors on the plot: before, if one out of 10 history variables
could not be plotted, nothing was shown at all, now all variables are \
show, those that could not be read with hs_read() show the error code
- the selection of which variables to plot is alphanumerically sorted (adc11 >
adc9) [this code is not active for standard midas because mlogger sup\
port has not yet been committed]
- the selection of which variables to plot shows the last variable selected, not
the first one - useful when entering variables from a long list [th\
is code is not active for standard midas because mlogger support has not yet
been committed]
These changes have been extensively tested since last Summer at the AD-5 ALPHA
expt at CERN.
I could only do minimal testing for this merged code, so if there are any errors,
they would most likely be merge errors. This new code will be heavily used at
TRIUMF,
so if any errors got any, we hope to flush them out quickly.
As noted, mlogger support for some of the mhttpd functions is not in standard
midas yet. It will be committed shortly.
K.O. |
14 Feb 2008, Stefan Ritt, Info, mhttpd history display updates
|
You misspelled one ODB entry:
Line 9014:
sprintf(str, "/History/Display/%s/Label", path);
Line 9028:
sprintf(str, "/History/Display/%s/Labels", path);
---^
I wonder how you could have tested that code for 1/2 year without noticing this error.
I fixed and committed it.
|
21 Feb 2008, Konstantin Olchanski, Info, mhttpd history display updates
|
> You misspelled one ODB entry:
> Line 9014:
> sprintf(str, "/History/Display/%s/Label", path);
>
> Line 9028:
> sprintf(str, "/History/Display/%s/Labels", path);
> ---^
>
> I wonder how you could have tested that code for 1/2 year without noticing this error.
> I fixed and committed it.
It turns out that the program was tested as originally committed. With the above
modification, it corrupts ODB - originally, it used the wrong array element size to create
the wrong array. Corrected, it creates the right array with the wrong size, then
subsequent db_set_data_index() happily corrupts ODB.
Fix for mhttpd committed as svn revision 4128.
Fix for ODB corruption committed at svn revision 4129 (also fixes extract_key())
K.O. |
18 Feb 2008, Konstantin Olchanski, Bug Report, mhttpd safari 3.0.4 redirect problem
|
I now encountered a new problem with mhttpd - I connect using the Safari 3.0.4 browser, go to the
"Programs" page, press the button "Start feplc" (or any other "start" button) and instead of starting this
program, I get an error in the browser, funny entries in ODB in "/Programs", corrupted ODB and a spew
of messages in the midas log file about the mess. ODB has to be reloaded from backup to recover.
Investigation shows that the culprit is odd bahaviour of the "redirect()" function:
/* start command */
if (*getparam("Start")) {
/* for NT: close reply socket before starting subprocess */
- redirect2("?cmd=programs");
+ redirect2("/?cmd=programs");
The version without "/" makes Safari explode - it appends the "?cmd..." stuff to the existing URL, which
already has the "?cmd..." tags, making a mess.
Firefox accepts either version.
ODB corruption happens here:
sprintf(str, "/Programs/%s/Start command", name);
- db_get_value(hDB, 0, str, command, &size, TID_STRING, TRUE);
+ db_get_value(hDB, 0, str, command, &size, TID_STRING, FALSE);
if (command[0]) {
ss_system(command);
It looks like db_get_value() would corrupt ODB if given funny "str". When Safari explodes,
funny strings are generated.
The simple fix is to replace "TRUE" with "FALSE", then at least db_get_value() does not try to make bogus
entries in ODB.
The "Stop" command has the same problem, but does not currupt ODB - there is no db_get_value() in
that code path.
I am reporting this "fresh" as I made one of our daq systems work again.
I did not investigate the history of changes to this "redirect" command (perhaps it was broken in the
recent reorganisation of midas urls?), what versions of Safari work or not.
K.O. |
21 Feb 2008, Stefan Ritt, Bug Report, mhttpd safari 3.0.4 redirect problem
|
> /* start command */
> if (*getparam("Start")) {
> /* for NT: close reply socket before starting subprocess */
> - redirect2("?cmd=programs");
> + redirect2("/?cmd=programs");
The second version won't work if mhttpd is run under an Apache proxy. Assume the proxy redirects
http://proxy.ca/midas
to
http://daq.ca:8080
If you now do a redirect to "/?cmd=programs", you will end up at
http://proxy.ca/?cmd=programs
which is now what you want. I tried to put a "./?cmd=programs", and that bings you to
http://proxy.ca/midas/./?cmd=programs
which is correctly redirected to
http://daq.ca:8080/?cmd=programs
I tried with the windows version (ughhh) of Safari and it worked for me. So give it a try, the change is committed.
> ODB corruption happens here:
>
> sprintf(str, "/Programs/%s/Start command", name);
> - db_get_value(hDB, 0, str, command, &size, TID_STRING, TRUE);
> + db_get_value(hDB, 0, str, command, &size, TID_STRING, FALSE);
> if (command[0]) {
> ss_system(command);
>
> It looks like db_get_value() would corrupt ODB if given funny "str". When Safari explodes,
> funny strings are generated.
What happes is an endless redirect from xxxx -> xxxx?cmd=Programs. So in the end you have
http://url.ca?cmd=programs?cmd=programs?cmd=programs?cmd=programs....
and in the end you get a stack overflow, which busts all.
> The simple fix is to replace "TRUE" with "FALSE", then at least db_get_value() does not try to make bogus
> entries in ODB.
I changed both butting FALSE there and adding
if (strchr(name, '?'))
*strchr(name, '?') = 0;
which keeps the URL short.
So for me it looks fine at the moment, but I cannot guarantee that everything works, so keep an eye open on that. |
16 Oct 2006, Exaos Lee, Bug Fix, "make install" error on MacOS 10.4.7, svn 3366
|
While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the
problem of "Makefile". I did some change to it and attach the diff file here. |
16 Oct 2006, Stefan Ritt, Bug Fix, "make install" error on MacOS 10.4.7, svn 3366
|
> While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the
> problem of "Makefile". I did some change to it and attach the diff file here.
I committed your patch. Thank you. |
19 Feb 2008, Maggie Lee, Bug Fix, "make install" error on MacOS 10.4.7, svn 3366
|
> While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the
> problem of "Makefile". I did some change to it and attach the diff file here.
Thank you very much for your instructions for installing Midas on MacOSX.
I followed your instructions to change the Makefile but I still get the following error message:
...
... Installing programs and utilities to /usr/local/bin
...
install: darwin/bin/lazylogger exists but is not a directory
install: darwin/bin/mchart exists but is not a directory
install: darwin/bin/mcnaf exists but is not a directory
install: darwin/bin/mdump exists but is not a directory
install: darwin/bin/melog exists but is not a directory
install: darwin/bin/mhdump exists but is not a directory
install: darwin/bin/mhist exists but is not a directory
install: darwin/bin/mhttpd exists but is not a directory
install: darwin/bin/mlogger exists but is not a directory
install: darwin/bin/mlxspeaker exists but is not a directory
install: darwin/bin/mserver exists but is not a directory
install: darwin/bin/mstat exists but is not a directory
install: darwin/bin/mtape exists but is not a directory
install: darwin/bin/odbedit exists but is not a directory
install: darwin/bin/odbhist exists but is not a directory
install: darwin/bin/stripchart.tcl exists but is not a directory
install: darwin/bin/webpaw exists but is not a directory
make: *** [install] Error 71
Could you help me solve this problem? Thank you in advance =) |
19 Feb 2008, Maggie Lee, Bug Fix, "make install" error on MacOS 10.4.7, svn 3366
|
I forgot to mention that, the following (and similar) lines:
install -v -D -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
are changed into
install -v -d -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
since -D is an illegal option for install. I am not sure whether -D in Linux means the same thing for -d in MacOSX install.
> > While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the
> > problem of "Makefile". I did some change to it and attach the diff file here.
>
> Thank you very much for your instructions for installing Midas on MacOSX.
> I followed your instructions to change the Makefile but I still get the following error message:
>
> ...
> ... Installing programs and utilities to /usr/local/bin
> ...
> install: darwin/bin/lazylogger exists but is not a directory
> install: darwin/bin/mchart exists but is not a directory
> install: darwin/bin/mcnaf exists but is not a directory
> install: darwin/bin/mdump exists but is not a directory
> install: darwin/bin/melog exists but is not a directory
> install: darwin/bin/mhdump exists but is not a directory
> install: darwin/bin/mhist exists but is not a directory
> install: darwin/bin/mhttpd exists but is not a directory
> install: darwin/bin/mlogger exists but is not a directory
> install: darwin/bin/mlxspeaker exists but is not a directory
> install: darwin/bin/mserver exists but is not a directory
> install: darwin/bin/mstat exists but is not a directory
> install: darwin/bin/mtape exists but is not a directory
> install: darwin/bin/odbedit exists but is not a directory
> install: darwin/bin/odbhist exists but is not a directory
> install: darwin/bin/stripchart.tcl exists but is not a directory
> install: darwin/bin/webpaw exists but is not a directory
> make: *** [install] Error 71
>
> Could you help me solve this problem? Thank you in advance =) |
19 Feb 2008, Stefan Ritt, Bug Fix, "make install" error on MacOS 10.4.7, svn 3366
|
> I forgot to mention that, the following (and similar) lines:
> install -v -D -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
> are changed into
> install -v -d -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
>
> since -D is an illegal option for install. I am not sure whether -D in Linux means the same thing for -d in MacOSX install.
-D under linux means:
-D create all leading components of DEST except the last, then
copy SOURCE to DEST; useful in the 1st format
This means if you install the first time, and eithe SYSBIN_DIR or `basename is not existing, it will be created on-the-fly from
the install program. If OSX does not support this, you somehow have to crate these subdirectories manually. |
19 Feb 2008, Maggie Lee, Bug Fix, "make install" error on MacOS 10.4.7, svn 3366
|
Thank you for your help =)
Since SYSBIN_DIR is defined as /usr/local/bin in the Makefile and it exists in my computer, so I deleted the -D in the Makefile and tried to "make install" again and the
error message becomes:
...
... Installing programs and utilities to /usr/local/bin
...
/bin/sh: -c: line 2: syntax error: unexpected end of file
make: *** [install] Error 2
Can anyone help me solve this problem?
> > I forgot to mention that, the following (and similar) lines:
> > install -v -D -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
> > are changed into
> > install -v -d -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
> >
> > since -D is an illegal option for install. I am not sure whether -D in Linux means the same thing for -d in MacOSX install.
>
> -D under linux means:
>
> -D create all leading components of DEST except the last, then
> copy SOURCE to DEST; useful in the 1st format
>
> This means if you install the first time, and eithe SYSBIN_DIR or `basename is not existing, it will be created on-the-fly from
> the install program. If OSX does not support this, you somehow have to crate these subdirectories manually. |
19 Feb 2008, Petr Nomokonov, Info, Frontend - Backend c onnection
|
Backend computer with SLC4.4 Linux did'not work as mserver because some security
protection under iptables service (could not connect with frontend computers).
The connection established if to make ( under root ) iptables disable
by command: service iptables stop, or much more gently
just to accept mserver port with command (under root):
iptables -I INPUT -p tcp --dport 1175 -j ACCEPT
( in /etc/service
midas 1175/tcp #Midas server)
To check which ports is open
it is possible to use the command: "netstat -n" to see digital numbers of ports. |
18 Feb 2008, Jimmy Ngai, Bug Report, Analyzer cannot run as Daemon
|
Hi All,
I'm testing MIDAS SVN rev-4113 on Scientific Linux 5.1 (i386) and the Analyzer
can't start as Daemon. What I mean "can't" is that it stops running
immediately without leaving any error messages. However, it can run offline or
without becoming a Daemon. I have tested with ROOT 5.14e/5.16/5.18 and
the "Experiment" example coming with MIDAS and this problem always happens.
Any ideas?
Best Regards,
Jimmy |
05 Feb 2008, Stefan Ritt, Info, Implementation of relative paths in mhttpd
|
A major change was made to mhttpd, changing all internal URLs to relative paths.
This allows proxy access to mhttpd via an apache server for example, which might
be needed to securely access an experiment from outside the lab through a
firewall. Following setting can be places into the Apache configuration,
assuming the experiment runs on machine "online1.your.domain", and apache on a
publically available machine "www.your.domain":
Redirect permanent /online1 http://www.your.domain/online1
ProxyPass /online1/ http://online1.your.domain/
<Location "/online1">
AuthType Basic
AuthName ...
AuthUserFile ...
Require user ...
</Location>
If the the URL http://www.your.domain/online1 is accessed, it gets redirected
(after optional authentication) to http://online1.your.domain. If you click on
the mhttpd history page for example, mhttpd would normally redirect this to
http://online1.your.domain/HS/
but this is not correct since you want to go through the proxy www.your.domain.
The new relative redirection inside mhttpd now redirects the history page
correctly to
http://www.your.domain/onlin1/HS/
I had to change many places inside mhttpd to make this work, and I'm not 100%
sure if I covered all occurrences. So if you upgrade to mhttpd revision 4115 and
observe some error accessing some pages, please report it to me.
- Stefan |
13 Feb 2008, Konstantin Olchanski, Info, Implementation of relative paths in mhttpd
|
> A major change was made to mhttpd, changing all internal URLs to relative paths.
> This allows proxy access to mhttpd via an apache server for example, which might
> be needed to securely access an experiment from outside the lab through a
> firewall.
It is good to see improvements to the MIDAS URLs. We have been successfully running
mhttpd behind an apache SSL/HTTPS proxy without these changes, but our case was very
limited to one experiment, one mhttpd behind one proxy. I hope to test these changes
in the near future at CERN, I guess we will hear if things broke. I am especialloy
worried about the function for "split mhttpd history generator" via "/History/URL".
I remember it was hard to get it right and I hope if this function did not survive
this update, it will be easy to resurrect.
K.O. |
05 Feb 2008, qinzeng peng, Forum, rpc timeout, related to event_size and watch dog? need help
|
Dear all,
I'm trying to write a simulation code on midas. What I did is just modify the
frontend.c(pp) from experiment samples and made some parameters change on midas.h .
Because my simulation ask for about 4.5MB for each event, so I increase the
MAX_EVENT_SIZE and max_event_size accordingly.
in midas.h :
#define MAX_EVENT_SIZE 0xa00000 //0x400000 /**< maximum event size 4MB*/
#define BANKLIST_MAX 640 //64 /**< max # of banks in event */
#define DEFAULT_RPC_TIMEOUT 60000 //10000
#define WATCHDOG_INTERVAL 5000 //1000
#define DEFAULT_WATCHDOG_TIMEOUT 60000 /**< Watchdog */
in frontend.cpp :
BOOL frontend_call_loop = TRUE;
INT max_event_size = 5 * 1024 * 1024;
INT max_event_size_frag = 2* max_event_size;
INT event_buffer_size = 2 * max_event_size;
EQUIPMENT equipment[] = {
{"WFD_SIMU", /* equipment name */
{1, 0, /* event ID, trigger mask */
"SYSTEM", /* event buffer */
#ifdef USE_INT
EQ_INTERRUPT, /* equipment type */
#else
EQ_POLLED, /* equipment type */
#endif
LAM_SOURCE(0, 0xFFFFFF), /* event source crate 0, all stations */
"MIDAS", /* format */
TRUE, /* enabled */
RO_RUNNING, // | /* read only when running */
// RO_ODB, /* and update ODB */
5000, /* poll for 500ms */
0, /* stop run after this event limit */
0, /* number of sub events */
0, /* don't log history */
"", "", "",},
read_simu_event, /* readout routine */
},
......
}
INT frontend_loop()
{
/* if frontend_call_loop is true, this routine gets called when
the frontend is idle or once between every event */
ss_sleep(100);
return SUCCESS;
}
Compilation OK and running mlogger, odbedit, frontend is OK.
start the run -> no problem ( but there is a long waiting time in frontend if
starting the run. Before the run begins, frontend terminal popping up messages
frequently, say, every 10 seconds. When run starts, frontend terminal hang on
for a couple of minutes before popping up next bunch of messages.)
stop the run -> Problem -> rpc timeout
message from odbedit:
[qzpeng@phy2-dhcp140 simu]$ odbedit -s 10000000
12:54:27 [WFD Simu,INFO] Program WFD Simu on host phy2-dhcp140 started
12:54:37 [Logger,INFO] Program Logger on host phy2-dhcp140 started
[local:simu:S]/>start
Run number [1]: 7
Are the above parameters correct? ([y]/n/q):
Starting run #7
Run #7 started
[local:simu:R]/>stop
[midas.c:9231:rpc_client_call,ERROR] rpc timeout, routine = "rc_transition",
host = "phy2-dhcp140.bu.edu"
Error: Unknown error 504 from client 'WFD Simu' on host phy2-dhcp140.bu.edu
[local:simu:R]/>
runing message from frontend:
[qzpeng@phy2-dhcp140 simu]$ ./frontend
Frontend name : WFD Simu
Event buffer size : 10485760
System max event size : 10485760
User max event size : 5242880
User max frag. size : 10485760
# of events per buffer : 2
Connect to experiment...
OK
Init hardware...
...... |
06 Feb 2008, Stefan Ritt, Forum, rpc timeout, related to event_size and watch dog? need help
|
Most likely you changed the maximal event size in midas.h, but you did not re-compile all programs. The maximal event size goes into the size of the shared memory buffer, so all participating programs have to have the same setting, especially the mserver program. So do the following:
- update to the latest midas version, which is revision 4116
- modify in your midas.h only MAX_EVENT_SIZE. The other settings you modified might have bad side effects. If you increase the RPC timeout, the error will still happen, just later. It comes from the fact that you sent too big events the the server (or the logger), which refuses to take the big events or simply crashes, so the RPC call never returns and after the timeout you get the error.
- recompile all midas programs, don't forget the mserver program
- run the standard demo frontend from the distribution
I tried the above and it just worked fine for me. |
06 Feb 2008, qinzeng peng, Forum, rpc timeout, related to event_size and watch dog? need help
|
Stefan Ritt wrote: | Most likely you changed the maximal event size in midas.h, but you did not re-compile all programs. |
Every time I changed midas.h or system header files, I did the re-compile with following procedure:
ipcrm
rm .*.SHM
mcleanup
make clean
make
su
make install
Stefan Ritt wrote: | The maximal event size goes into the size of the shared memory buffer, so all participating programs have to have the same setting, especially the mserver program. |
Question here:
How to compile mserver seperately? I think make and make install under midas directory already did the job.
Stefan Ritt wrote: | update to the latest midas version, which is revision 4116 |
I used latest svn version, so I believe I'm using the latest version 4116.
svn co svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/midas/trunk midas
svn co svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/mxml/trunk mxml
I followed your instructions and did the following:
1)
svn co svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/midas/trunk midas
svn co svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/mxml/trunk mxml
2)
changed two parameters in midas.h
#define MAX_EVENT_SIZE 0xa00000 //0x400000 /**< maximum event size 4MB->10MB*/
#define BANKLIST_MAX 640 //64 /**< max # of banks in event, I need 356 */
3) make
su
make install
I don't know if I need compile mserver seperately.
4) running only two programs:
odbedit -s 10000000
./frontend
And I still got the same problem as before.
[qzpeng@phy2-dhcp140 simu]$ odbedit -s 10000000
[local:simu:S]/>ls
System
Programs
Experiment
Logger
Runinfo
Alarms
[local:simu:S]/>mkdir Equipment
12:44:12 [WFD Simu,INFO] Program WFD Simu on host phy2-dhcp140 started
[local:simu:S]/>start
Run number [1]:
Are the above parameters correct? ([y]/n/q):
Starting run #1
Run #1 started
[local:simu:R]/>stop
[midas.c:9231:rpc_client_call,ERROR] rpc timeout, routine = "rc_transition", host = "phy2-dhcp140.bu.edu"
Error: Unknown error 504 from client 'WFD Simu' on host phy2-dhcp140.bu.edu
[local:simu:R]/>
And I know that the run stopped after a while on frontend, but after the eroor message showed above. If I tried to stop again in odb after a while, it did stopped.
[local:simu:R]/>stop
Run #1 stopped
By the way, thanks for the quick response. I've been working on this for a couple of weeks and I am a newbie.
I also attached my frontend.cpp code and output of make with warning message but comilation completed. Thanks in advance.
In frontend.cpp I only use simulation and don't need any hardware realted issue or function calls but I jsut leave some of them there. |
06 Feb 2008, Stefan Ritt, Forum, rpc timeout, related to event_size and watch dog? need help
|
First of all, I would appreciate if you do not post your entry ten times. Each time you edit it, you produce an email notification going to everybody, so people might get annoyed to receive too many emails from you. Think what you want to write and then post once.
Second, I told you to use the frontend from the distribution, but you used your own code. Since I successfully ran the demo frontend with the large event size, the origin of your problem must be "in between". So start with the demo frontend, try it, then modify its buffer size in frontend.c, then try again. When I told to to recompile midas, I meant you should also recompile your front-end each time you change midas.h. The mserver is automatically recompiled when you recompile and install midas (just check the /usr/local/bin/mserver date and time to confirm that it got updated during your last "make install"). Then add things from your specific front-end program step by step to see at which step the problem occurs the first time. This gives you some hint where the real cause lies. |
04 Feb 2008, Robert Pattie, Forum, analyzer crashes at high rates
|
I'm using midas to read data from a waveform digitizer at event rates of
10-30kHz. To accomplish this the digitizer is read via Block transfers and the
raw data put into a single MIDAS event. Thus a MIDAS event could contain upto
250 physical events and at maximum 350kBytes. In the analyzer modules I had
been analyzing the first physics event contained in a MIDAS event with no
problem. Recently I tried to analyze all the physical events. At low rates,
100hz-1khz, this was no problem, 1-5 physical events in a MIDAS event. At
higher rates 10-20kHz, where there are about 40physical events per MIDAS event,
the analyzer keeps up for a few seconds then seg faults with " 'shared object
read from target memory' has disappear; keeping it symbols". Any suggestions as
to why the analyzer is crashing would be very helpful.
Thanks,
Robert |
05 Feb 2008, Stefan Ritt, Forum, analyzer crashes at high rates
|
> I'm using midas to read data from a waveform digitizer at event rates of
> 10-30kHz. To accomplish this the digitizer is read via Block transfers and the
> raw data put into a single MIDAS event. Thus a MIDAS event could contain upto
> 250 physical events and at maximum 350kBytes. In the analyzer modules I had
> been analyzing the first physics event contained in a MIDAS event with no
> problem. Recently I tried to analyze all the physical events. At low rates,
> 100hz-1khz, this was no problem, 1-5 physical events in a MIDAS event. At
> higher rates 10-20kHz, where there are about 40physical events per MIDAS event,
> the analyzer keeps up for a few seconds then seg faults with " 'shared object
> read from target memory' has disappear; keeping it symbols". Any suggestions as
> to why the analyzer is crashing would be very helpful.
I personally have never seen this error message. The analyzer is designed such that
it produces "back pressure" if the data rate is higher than the analysis rate and
you have "request all events" on. The only thing I can image are the following two
issues:
- At higher rate where you have more than 40 physical events per MIDAS event, there
is some bug in your analysis code which gets exploited only in that case. Maybe some
temporary array which is only 35 entries long or something like this.
- The back pressure mentioned above will slow down the frontend. If your computer
busy logic is not working correctly, you might get more triggers than you can
acquire. Maybe then the data gets screwed up and the analyzer chokes on it.
Finding the exact reason is not simple. For sure you have to run the analyzer inside
the debugger, to see exactly where the segfault happens. You then maybe have to
produce some dummy data in the frontend (like always sending the same event) to
disentangle some possible trigger problems from other problems.
Best regards,
Stefan |
27 Nov 2007, Stefan Ritt, Info, ODB links to array elements implemented
|
In revision 4090 I implemented ODB links to individual array elements. Now you
can have for example:
Key name Type #Val Size Last Opn Mode Value
---------------------------------------------------------------------------
array INT 10 4 2m 0 RWD
[0] 0
[1] 0
[2] 123
[3] 0
[4] 0
[5] 0
[6] 0
[7] 0
[8] 0
[9] 0
element2 -> /array[2] INT 1 4 3m 0 RWD 123
In this case, the link "element2" points to the third element of "array", but is
treated like a single value. This links are very useful for example for the
"Edit on start" parameters, which can now point to individual array elements.
The same is true for the "Links BOR" when the logger writes to a MySQL database.
This modification required major modifications in the ODB. I have carefully
tested the example experiment from the distribution to verify that everything is
fine, but I'm not 100% sure that I covered all possible situations. So if you
update to revision 4090+ and you observe some strange behavior related to links
in the ODB, please report.
There are following two new functions related to this change:
db_get_link()
db_get_link_data()
They are counterparts of db_get_key() and db_get_data(), respectively, but
without following links in the ODB. These functions are probably not of much use
outside odbedit and mhttpd, which are supposed to display links explicitly. Most
user applications want to follow links without even knowing that these are links. |
22 Jan 2007, Carl Metelko, Forum, Midas on a x86_64
|
Hi,
has anyone managed to get midas to work on a x86_64 processor. I followed the
instructions for the 64-bit opteron but i am getting runtime error when trying
the examples.
When running example/basic/odb_test I getting errors like
[odb.c:6818:db_get_record] struct size mismatch for "/Alarms/Alarms/Demo ODB"
(464 instead of 452)
[odb.c:6818:db_get_record] struct size mismatch for "/Alarms/Alarms/Demo ODB"
(464 instead of 452)
[midas.c:16576:al_check] Cannot get alarm record
Any ideas what is wrong? |
22 Jan 2007, Konstantin Olchanski, Forum, Midas on a x86_64
|
> has anyone managed to get midas to work on a x86_64 processor. I followed the
> instructions for the 64-bit opteron but i am getting runtime error when trying
> the examples.
We run 64-bit MIDAS on RHEL4 with 64-bit ROOT and everything generally works,
except for compatibility problems with 32-bit MIDAS.
Everything should work if you ensure that on your 64-bit machine everything is
compiled 64-bit (including the mserver - we always forget to install the correct version
to /usr/local/bin). 32-bit MIDAS programs running on other machine
can talk to 64-bit MIDAS via the mserver.
The big problem is that 64-bit and 32-bit ODB turned out to be incompatible - several data
fields have different sizes - and we did not decide yet how to fix this. Any fix will involve
breaking the binary ODB for one of the two platforms (we could break both, just to be fair, heh!)
> When running example/basic/odb_test I getting errors like
> [odb.c:6818:db_get_record] struct size mismatch for "/Alarms/Alarms/Demo ODB" (464 instead of 452)
Yes, data size mismatch errors indicates that you mixed 32-bit and 64-bit MIDAS. Recompile everyting
as 64-bit, remove all the dot-ODB files, remove all the shared memory segments (ipcrm),
then everything should work.
K.O. |
12 Jul 2007, Konstantin Olchanski, Forum, Midas on a x86_64 - incompatible with x86_32
|
> We run 64-bit MIDAS on RHEL4 with 64-bit ROOT and everything generally works,
> except for compatibility problems with 32-bit MIDAS.
>
> The big problem is that 64-bit and 32-bit ODB turned out to be incompatible ...
I have now identified 3 data structures that change size when compiled with "-m64":
EVENT_REQUEST: stores a pointer to a function. Pointer size is 4 bytes with -m32 and 8 bytes with -m64.
This structure is part of an array inside BUFFER_HEADER, resulting in a sizable size mismatch between 32
bit and 64 bit shared memory data buffers.
The fix is simple: the function pointer is not used anywhere. Replace is with a "DWORD unused_filler"
makes -m32 and -m64 data buffers compatible. (But breaks compatibility with previous -m64 compiled midas).
CHN_SETTINGS and CHN_STATISTICS: apparently, -m32 and -m64 GCC has different packing rules and in -m64
mode, 4 bytes of padding are added to these data structures. Size size mismatch appears to be benign,
but will result in "size mismatch" complaints from ODB.
The fix is simple: adding "__attribute__ ((__packed__))" to the definition of the data structure makes
-m64 identical to -m32.
The "svn diff" of changes involved is attached below.
The biggest problem here is that making 32-bit ODB and 64-bit ODB compatible requires breaking one or
the other (My proposed changes break the 64-bit version. Alternatively, one could add explicit padding
to these data structures and break the 32-bit ODB).
I think it is important to make 32-bit and 64-bit code compatible: at TRIUMF we have to use a mixed
environment because out latest host computers all run 64-bit Linux while all our VME processors and all
older machines can only run 32-bit code; this incompatibility causes us weekly headaches.
Any thoughts?
K.O.
(this output of svn diff is doctored for clarity)
ladd00:midas$ svn diff
Index: include/midas.h
===================================================================
--- include/midas.h (revision 3744)
+++ include/midas.h (working copy)
- void (*dispatch) (HNDLE, HNDLE, EVENT_HEADER *, void *);
+ INT unused; // was void (*dispatch) (HNDLE, HNDLE, EVENT_HEADER *, void *);
} EVENT_REQUEST;
--- include/msystem.h (revision 3744)
+++ include/msystem.h (working copy)
+#define PACKED __attribute__ ((__packed__)) <--- this goes into midas.h inside the #ifdef "we use GCC"
-typedef struct {
+typedef struct PACKED { ... CHN_SETTINGS
-typedef struct {
+typedef struct PACKED { ... CHN_STATISTICS |
13 Jul 2007, Stefan Ritt, Forum, Midas on a x86_64 - incompatible with x86_32
|
> The biggest problem here is that making 32-bit ODB and 64-bit ODB compatible requires breaking one or
> the other (My proposed changes break the 64-bit version. Alternatively, one could add explicit padding
> to these data structures and break the 32-bit ODB).
>
> I think it is important to make 32-bit and 64-bit code compatible: at TRIUMF we have to use a mixed
> environment because out latest host computers all run 64-bit Linux while all our VME processors and all
> older machines can only run 32-bit code; this incompatibility causes us weekly headaches.
>
> Any thoughts?
I agree to make 32-bit and 64-bit compatible. In the long run, everything will be 64-bit, so I would suggest
in breaking the 32-bit ODB, add some padding there where needed, probably with some conditional compiling.
This ensures to keep the native 64-bit packing, which probably will be somehow optimized for 64-bit
architectures and therefore might be a bit faster in the long run, when most systems are 64-bit. After this
has been implemented and well tested, I would go with an official announcement of the 32-bit break in the ODB,
and release a new version, so people can update from a TAR file if necessary. Existing ODB's can be converted
to the new format by exporting them in XML form and importing them again after the upgrade. |
12 Aug 2007, Konstantin Olchanski, Forum, Midas on a x86_64 - incompatible with x86_32
|
> I agree to make 32-bit and 64-bit compatible. In the long run, everything will be 64-bit, so I would suggest
> in breaking the 32-bit ODB, add some padding there where needed, probably with some conditional compiling.
I now have the patches to implement this. Changes turned out to be minimal:
1) midas.h: remove unused field "dispatch" from EVENT_REQUEST and bump DATABASE_VERSION from 2 to 3
2) msystem.h: add 32-bit padding to CHN_STATISTICS and CHN_SETTINGS
(Pedantic note: the C/C++ languages permit compilers to arbitrary pad data members inside structures and one is
not supposed to rely on the specific layout of "struct"s, they could changing from day to day depending on
compiler vendor, version, 32/64 bit, optimization level, etc. This is quite silly, but I guess it was the only way
"they" could agree on a standard)
In practice, compilers are will behaved and one can follow simple rules and stay out of trouble.
1) if all data members are of the same size -> no padding
2) do not use "double" (64-bit) and "short" (16-bit), make all char[] arrays divisible by 4 -> size of everything
is 32-bit, see rule 1
3) if you have to use "short", they have to come in pairs to keep everything else aligned to 32-bit
4) if you have to use "double" (or uint64_t), keep them aligned to 64-bit, i.e. struct { int a,b,c; double x;} is
*bad* (4-byte padding may be added between c and x). struct { int a,b,c,d; double x; } is good.
Below are is "svn diff include/midas.h include/msystem.h". These changes have been tested on SL4 32-bit and
64-bit, SL5 32/64, F7 32/64 and SL4/ICC (Intel compiler) 32 bit and 64 bit.
The testing was done by adding checks on sizes of all struct's kept on ODB, i.e.
assert(sizeof(CHN_SETTINGS ) == 640); // ODB v3 with padding
assert(sizeof(CHN_STATISTICS ) == 32); // ODB v3 with padding
... etc ...
K.O.
ladd03:midas$ svn diff include/midas.h include/msystem.h
Index: include/midas.h
===================================================================
--- include/midas.h (revision 3798)
+++ include/midas.h (working copy)
@@ -38,7 +38,7 @@
* @{ */
/* has to be changed whenever binary ODB format changes */
-#define DATABASE_VERSION 2
+#define DATABASE_VERSION 3
/* MIDAS version number which will be incremented for every release */
#define MIDAS_VERSION "2.0.0"
@@ -810,8 +810,6 @@
short int event_id; /**< event ID */
short int trigger_mask; /**< trigger mask */
INT sampling_type; /**< GET_ALL, GET_SOME, GET_FARM */
- /**< dispatch function */
- void (*dispatch) (HNDLE, HNDLE, EVENT_HEADER *, void *);
} EVENT_REQUEST;
typedef struct {
Index: include/msystem.h
===================================================================
--- include/msystem.h (revision 3798)
+++ include/msystem.h (working copy)
@@ -454,6 +454,7 @@
INT event_id;
INT trigger_mask;
DWORD event_limit;
+ INT pad; // FIXME 64-bit "double" should be 64-bit aligned
double byte_limit;
double tape_capacity;
char subdir_format[32];
@@ -465,6 +466,7 @@
double bytes_written;
double bytes_written_total;
INT files_written;
+ INT pad; // FIXME pad data structure to be 64-bit aligned
} CHN_STATISTICS;
typedef struct {
ladd03:midas$ |
20 Aug 2007, Konstantin Olchanski, Forum, Midas on a x86_64 - incompatible with x86_32
|
> > I agree to make 32-bit and 64-bit compatible. In the long run, everything will be 64-bit, so I would suggest
> > in breaking the 32-bit ODB, add some padding there where needed, probably with some conditional compiling.
>
> I now have the patches to implement this. Changes turned out to be minimal:
>
> 1) midas.h: remove unused field "dispatch" from EVENT_REQUEST and bump DATABASE_VERSION from 2 to 3
> 2) msystem.h: add 32-bit padding to CHN_STATISTICS and CHN_SETTINGS
The padding of CHN_STATISTICS and CHN_SETTINGS is not working right - somehow mhttpd and mlogger keep recreating the
data in ODB and erasing the padding fields. I am looking into this.
K.O. |
29 Aug 2007, Konstantin Olchanski, Forum, ODBv3, second try - Midas on a x86_64 - incompatible with x86_32
|
> > > I agree to make 32-bit and 64-bit compatible. In the long run, everything will be 64-bit, so I would suggest
> > > in breaking the 32-bit ODB, add some padding there where needed, probably with some conditional compiling.
> > 1) midas.h: remove unused field "dispatch" from EVENT_REQUEST and bump DATABASE_VERSION from 2 to 3
> > 2) msystem.h: add 32-bit padding to CHN_STATISTICS and CHN_SETTINGS
I am now trying a different solution of to fixing the issue of CHN_STATISTICS and CHN_SETTINGS changing size.
1) midas.h: (same as before) remove unused field "dispatch" from EVENT_REQUEST and bump DATABASE_VERSION from 2 to 3
2) msystem.h: in CHN_STATISTICS and CHN_SETTINGS change type of "event_limit" and "files_written" from int to "double".
Below are the latest ODBv3 meta patches:
ladd03:midas$ svn diff
Index: include/midas.h
===================================================================
--- include/midas.h (revision 3844)
+++ include/midas.h (working copy)
/* has to be changed whenever binary ODB format changes */
-#define DATABASE_VERSION 2
+#define DATABASE_VERSION 3
.........
short int trigger_mask; /**< trigger mask */
INT sampling_type; /**< GET_ALL, GET_SOME, GET_FARM */
- /**< dispatch function */
- void (*dispatch) (HNDLE, HNDLE, EVENT_HEADER *, void *);
} EVENT_REQUEST;
Index: include/msystem.h
===================================================================
--- include/msystem.h (revision 3845)
+++ include/msystem.h (working copy)
-"Event limit = DWORD : 0",\
+"Event limit = DOUBLE : 0",\
..................
-"Files written = INT : 0",\
+"Files written = DOUBLE : 0",\
..................
- DWORD event_limit;
+ double event_limit;
..................
- INT files_written;
+ double files_written;
K.O. |
21 Nov 2007, Konstantin Olchanski, Forum, ODBv3, second try - Midas on a x86_64 - incompatible with x86_32
|
These changes to make 32-bit and 64-bit ODB binary compatible with each other are now commited to midas svn, revision 4080.
Starting with this revision, ODB version changes from 2 to 3, breaking binary compatibility with previous releases.
Before upgrading to this revision, save your ODB as an XML file, *and* try to reload it, to catch any potential problems with parsing of the XML file.
Part of this commit are checks for sizes of important midas data structures stored in ODB shared memory - if the compiled size does not match the expected
value, binary compatibility is broken and the program will abort - to avoid further corruption of ODB shared memory. This feature is only enabled on Linux and
it is expected to trigger only on compiler malfunctions (generates wrong data size) and on accidental or intentional changes to important data structures in
midas, to warn the user that they broke ODB binary compatibility.
K.O.
> > > > I agree to make 32-bit and 64-bit compatible. In the long run, everything will be 64-bit, so I would suggest
> > > > in breaking the 32-bit ODB, add some padding there where needed, probably with some conditional compiling.
> > > 1) midas.h: remove unused field "dispatch" from EVENT_REQUEST and bump DATABASE_VERSION from 2 to 3
> > > 2) msystem.h: add 32-bit padding to CHN_STATISTICS and CHN_SETTINGS
>
> I am now trying a different solution of to fixing the issue of CHN_STATISTICS and CHN_SETTINGS changing size.
>
> 1) midas.h: (same as before) remove unused field "dispatch" from EVENT_REQUEST and bump DATABASE_VERSION from 2 to 3
> 2) msystem.h: in CHN_STATISTICS and CHN_SETTINGS change type of "event_limit" and "files_written" from int to "double".
>
> Below are the latest ODBv3 meta patches:
>
> ladd03:midas$ svn diff
> Index: include/midas.h
> ===================================================================
> --- include/midas.h (revision 3844)
> +++ include/midas.h (working copy)
> /* has to be changed whenever binary ODB format changes */
> -#define DATABASE_VERSION 2
> +#define DATABASE_VERSION 3
> .........
> short int trigger_mask; /**< trigger mask */
> INT sampling_type; /**< GET_ALL, GET_SOME, GET_FARM */
> - /**< dispatch function */
> - void (*dispatch) (HNDLE, HNDLE, EVENT_HEADER *, void *);
> } EVENT_REQUEST;
>
> Index: include/msystem.h
> ===================================================================
> --- include/msystem.h (revision 3845)
> +++ include/msystem.h (working copy)
> -"Event limit = DWORD : 0",\
> +"Event limit = DOUBLE : 0",\
> ..................
> -"Files written = INT : 0",\
> +"Files written = DOUBLE : 0",\
> ..................
> - DWORD event_limit;
> + double event_limit;
> ..................
> - INT files_written;
> + double files_written;
>
> K.O. |
26 Jan 2007, Carl Metelko, Forum, Midas on a x86_64
|
I upgraded from 1.9.5 to the latest on SVN an it works fine |
15 Mar 2007, Konstantin Olchanski, Info, mhdump: a standalone MIDAS history dump utility
|
While working on improvements to the MIDAS history system, I understood the data
format of the MIDAS .hst files and wrote a standalone program to extract data
from them, called mhdump.
mhdump is intended to be easier to use, compared to mhist. By default it reads
and decodes all the data in the given .hst files, with options to limit the
decoding to specified events and tags, and an option to omit the event and tag
names from the output.
mhdump is completely standalone and does not require MIDAS header files and
libraries.
The mhdump source code and a description of the .hst file format are here:
http://daq-plone.triumf.ca/SR/MIDAS/utils/mhdump/
I hope people find this program useful. If you have any feedback (patches, bug
reports, requests for improvements), please post them as replies to this forum
message.
K.O. |
15 Mar 2007, Stefan Ritt, Info, mhdump: a standalone MIDAS history dump utility
|
> I hope people find this program useful. If you have any feedback (patches, bug
> reports, requests for improvements), please post them as replies to this forum
> message.
I wouldn't mind putting this into the midas distribution. Put it under utils/, add
an entry to the Makefile, and fix that warning:
mhdump.cxx: In function `int readHstFile(FILE*)':
mhdump.cxx:161: warning: comparison between signed and unsigned integer expressions |
20 Nov 2007, Konstantin Olchanski, Info, mhdump: a standalone MIDAS history dump utility
|
> > I hope people find this program useful. If you have any feedback (patches, bug
> > reports, requests for improvements), please post them as replies to this forum
> > message.
>
> I wouldn't mind putting this into the midas distribution. Put it under utils/, add
> an entry to the Makefile, and fix that warning:
>
>
> mhdump.cxx: In function `int readHstFile(FILE*)':
> mhdump.cxx:161: warning: comparison between signed and unsigned integer expressions
Done and done.
The program mhdump, a standalone decoder for midas history files, is now in midas svn.
K.O. |
17 Oct 2007, Randolf Pohl, Forum, Adding MIDAS .root-files
|
Dear MIDAS users,
I want to add several .root-files produced by the MIDAS analyzer, in a fast
and convenient way. ROOT's hadd fails because it does not know how to treat
TFolders. I guess this problem is not unique to me, so I hope that somebody of
you might already have found a solution.
Why don't I just run "analyzer -r 1 10000"?
We have taken lots of runs under (rapidly) varying conditions, so it would be
lots of "-r". And the analysis is quite involved, so rerunning all data takes
about one hour on a fast PC making this quite painful.
Therefore, I would like to rerun all data only once, and then add the result
files depending on different criteria.
Of course, I tried to write a script that does the adding. But somehow it is
incredibly slow. And I am not the Master Of C++, too.
Is there any deeper reason for MIDAS using TFolders, not TDirectorys? ROOT's
hadd can treat TDirectory. Can I simply patch "my" MIDAS? Is there general
interest in a change like this? (Does anyone have experience with the speed of
hadd?)
Looking forward to comments from the Forum.
Cheers,
Randolf |
17 Oct 2007, John M O'Donnell, Forum, Adding MIDAS .root-files
|
The following program handles regular directories in a file, or folders (ugh).
Most histograms are added bin by bin.
For scaler events it is convenient to see the counts as a function of time (ala
sclaer history plots in mhttpd). If the histogram looks like a scaler plot versus
time, then new bins are added on to the end (or into the middle!) of the histogram.
All different versions of cuts are kept.
TTrees are not explicitly supported, so probably don't do the right thing...
John.
> Dear MIDAS users,
>
> I want to add several .root-files produced by the MIDAS analyzer, in a fast
> and convenient way. ROOT's hadd fails because it does not know how to treat
> TFolders. I guess this problem is not unique to me, so I hope that somebody of
> you might already have found a solution.
>
> Why don't I just run "analyzer -r 1 10000"?
> We have taken lots of runs under (rapidly) varying conditions, so it would be
> lots of "-r". And the analysis is quite involved, so rerunning all data takes
> about one hour on a fast PC making this quite painful.
> Therefore, I would like to rerun all data only once, and then add the result
> files depending on different criteria.
>
> Of course, I tried to write a script that does the adding. But somehow it is
> incredibly slow. And I am not the Master Of C++, too.
>
> Is there any deeper reason for MIDAS using TFolders, not TDirectorys? ROOT's
> hadd can treat TDirectory. Can I simply patch "my" MIDAS? Is there general
> interest in a change like this? (Does anyone have experience with the speed of
> hadd?)
>
> Looking forward to comments from the Forum.
>
> Cheers,
>
> Randolf |
17 Oct 2007, Randolf Pohl, Forum, Multi-core CPUs
|
Dear Forum,
I have this beautiful Intel Quadcore with fast disks, but MIDAS does obviously
only make use of one CPU at a time. Has anyboy of you already done some work
on making MIDAS parallel? Event-based data analysis should be the best
candidate for this.
Has anybody done this with PVM? There is some PVM-related stuff in the MIDAS
sources, but I got the impression this works only with HBOOK, not with ROOT.
Or am I wrong?
But then PVM is probably also not the most efficient thing one ONE machine
with multiple CPUs, right? And finally, with PVM we're back to
adding .root-files efficiently (see my previous post).
Any thoughts?
Cheers,
Randolf |
17 Oct 2007, Stefan Ritt, Forum, Multi-core CPUs
|
> I have this beautiful Intel Quadcore with fast disks, but MIDAS does obviously
> only make use of one CPU at a time. Has anyboy of you already done some work
> on making MIDAS parallel? Event-based data analysis should be the best
> candidate for this.
There are ring buffer routines rb_xxx for distributed event analysis, but this is
currently only implemented in the front-end framework. These routines are pretty
simple, and their integration into the analyzer should not be very difficult.
Unfortunately I don't have time for that right now. We do our analysis such that we
analyze four different runs in parallel on a quadcore machine.
- Stefan |
11 Oct 2007, Stefan Ritt, Bug Report, _syscall0 not available on gcc 4.1.1
|
Dear Stephan,
I am writting on behalf of the LiBeRACE collaboration
at Berkeley/Livermore.
We are trying to use midas (2.0.0) for our acquisition system.
However we had some difficulties to compile it on LINUX Fedora
Core 6 with gcc 4.1.1
I tried to trace back the problem and I found that _syscall0 in
system.c is actually an obsolete call (since gcc 4.x apparently).
Playing with assembly language being behond my competence, I would
like to know if you ever came across this situation recently and
if you have any suggestion(s).
With my best regards
Julien GIBELIN
------------------------------------------------------
GIBELIN Julien
Lawrence Berkeley National Laboratory
Nuclear Science Division
One Cyclotron Rd.
MS 88R0192
BERKELEY, CA 94720-8101
Tel: +1 (510) 495-2695
Fax: +1 (510) 486-7983
------------------------------------------------------ |
11 Oct 2007, Stefan Ritt, Bug Report, _syscall0 not available on gcc 4.1.1
|
> Dear Stephan,
>
> I am writting on behalf of the LiBeRACE collaboration
> at Berkeley/Livermore.
>
> We are trying to use midas (2.0.0) for our acquisition system.
> However we had some difficulties to compile it on LINUX Fedora
> Core 6 with gcc 4.1.1
> I tried to trace back the problem and I found that _syscall0 in
> system.c is actually an obsolete call (since gcc 4.x apparently).
> Playing with assembly language being behond my competence, I would
> like to know if you ever came across this situation recently and
> if you have any suggestion(s).
The '_syscall0' function call was replaced by 'syscall' in SVN revision 3583. I
would recommend that you switch to the current SVN version (see
http://ladd00.triumf.ca/~daqweb/doc/midas/html/quickstart.html on how to obtain
the SVN version). If the problem still persists, please let us know.
- Stefan |
08 Oct 2007, Carl Metelko, Bug Report, Error in data format- ending blocks on 32bit boundary x86_64
|
Hi,
I found that midas banks can be given an extra 32 bits of zeros when
trying to keep to 32bit boundary on my x86_64.
This can be fixed by changing (in midas.h)
#define ALIGN8(x) (((x)+7) & ~7)
to
#define ALIGN8(x) (((x)+3) & ~3)
Is there any bad consequences doing this? |
08 Oct 2007, Stefan Ritt, Bug Report, Error in data format- ending blocks on 32bit boundary x86_64
|
> Hi,
> I found that midas banks can be given an extra 32 bits of zeros when
> trying to keep to 32bit boundary on my x86_64.
>
> This can be fixed by changing (in midas.h)
> #define ALIGN8(x) (((x)+7) & ~7)
> to
> #define ALIGN8(x) (((x)+3) & ~3)
>
> Is there any bad consequences doing this?
Yes. ALIGN8 means 'align to 8-byte boundary' (64-bit), and if you change that, you
break the code at various locations. Furthermore, 8-byte aligned access is faster
on x86_64 than 4-byte aligned access, so you will get a performance penalty. If
course if you have very many small banks, the zero padding can cause some
overhead, but in that case you could combine some data into a single bank. |
02 Oct 2007, Konstantin Olchanski, Info, ROODY, ROOTANA updates
|
The ROODY online histogram viewer and the ROOTANA midas analyzer toolkit have been updated to work
with ROOT version 5.16 and tested on Linux (SL4.4) and MacOS (10.4.10/PPC).
This update includes the library called "TNetDirectory" for access to remote ROOT objects. This library is
still under development, but is complete enough for use with ROODY. To try it, please specify -P9091 in
rootana and -Plocalhost:9091 in ROODY.
K.O. |
06 Sep 2007, Stefan Ritt, Info, Introduction of MIDAS_MAX_EVENT_SIZE
|
We had the problem that different experiments used different MAX_EVENT_SIZE
values (the MEG experiment actually 10 MB!). If each experiment changes the
value in midas.h and accidentally commits it, other experiments are affected.
Therefore I modified midas.h and the Makefile to accept a new environment
variable MIDAS_MAX_EVENT_SIZE. If this value is set, the Makefile passes it's
value to midas.h where it supersedes the default value which is currently at 4 MB.
PAA: Can you pleas add this to the documentation at the right spot? Thanks. |
20 Aug 2007, Konstantin Olchanski, Bug Report, how to handle end of run?
|
I am having problems with handling the end-of-run situation in my midas
frontend. I have a device that continuously sends data (over USB) and I read
this data in my "read_event" function.
Everything is good until the end-of-run, at which time this happens:
0) mfe.c calls my read_event() to read the data (loop until the end-of-run
transition)
1) mfe.c calls my end_of_run()
2) here, I tell the device "please stop sending data"
3) all seems good, but wait!!!
4) there is all this data generated between step 0 and step 2 still sitting
inside the device and it has nowhere to go: the run is ended, the output file is
closed, my read_event() will never be called ever again (well, until the next run).
It seems to me mfe.c needs to have one more function, something like
"pre_end_of_run()" that works like this:
0) mfe.c calls my read_event() to read the data (loop until the end-of-run
transition)
1) mfe.c calls pre_end_of_run(), here I tell the device to stop sending data
2) mfe.c calls read_event() for the very last time, to give me the opportunity
to read and send away any data I still may have.
3) mfe.c calls the end_of_run(). The run is truely finished.
Any thoughts?
K.O. |
03 Sep 2007, Stefan Ritt, Bug Report, how to handle end of run?
|
> I am having problems with handling the end-of-run situation in my midas
> frontend. I have a device that continuously sends data (over USB) and I read
> this data in my "read_event" function.
>
> Everything is good until the end-of-run, at which time this happens:
> 0) mfe.c calls my read_event() to read the data (loop until the end-of-run
> transition)
> 1) mfe.c calls my end_of_run()
> 2) here, I tell the device "please stop sending data"
> 3) all seems good, but wait!!!
> 4) there is all this data generated between step 0 and step 2 still sitting
> inside the device and it has nowhere to go: the run is ended, the output file is
> closed, my read_event() will never be called ever again (well, until the next run).
>
> It seems to me mfe.c needs to have one more function, something like
> "pre_end_of_run()" that works like this:
> 0) mfe.c calls my read_event() to read the data (loop until the end-of-run
> transition)
> 1) mfe.c calls pre_end_of_run(), here I tell the device to stop sending data
> 2) mfe.c calls read_event() for the very last time, to give me the opportunity
> to read and send away any data I still may have.
> 3) mfe.c calls the end_of_run(). The run is truely finished.
>
> Any thoughts?
You can achieve the desired functionality without changing mfe.c:
0) mfe.c calls read_event
1) mfe.c calls end_of_run. Your end_of_run tells the device to stop data and flushes
the remaining data. At this point you have to re-make actually a part of the mfe.c
functionality, but basically you need a bm_compose_event() and a bm_send_event(), so
just a few lines of code. If you want to have the final event number right in your
equipment, you also need to update eq->events_sent accordingly.
Given the fact that 99% of the experiments do not need this functionality, I propose
that we keep mfe.c and you add the few lines of code into your user part of the
specific frontend.
Stefan |
29 Aug 2007, Konstantin Olchanski, Info, Added data compression to mlogger
|
I now commited the changes to mlogger (mlogger.c, msystem.h) implementing data
compression using zlib (svn revision 3845)
To enable compression, observe that mlogger is compiled with -DHAVE_ZLIB (see
the Makefile), in "/Logger/Channels/NNN/Settings", set "compression" to "1" and
the filename to "run%05d.mid.gz" (note the suffix ".gz").
In the Makefile, I only enabled HAVE_ZLIB for Linux, as that is the only
platform I tested. If somebody can test compression on Windows, please do and
let us know.
My ROOT analyzer (rootana) package can read compressed MIDAS files directly and
if one wants to add this capability to other MIDAS-related packages, one is
welcome to use my TMidasFile.cxx as an example
(http://ladd00.triumf.ca/viewcvs/rootana/trunk/TMidasFile.cxx?view=markup).
K.O. |
08 Jun 2006, Konstantin Olchanski, Bug Fix, commit latest ccusb.c CAMAC-USB driver
|
I commited the latest driver for the Wiener CCUSB USB-CAMAC driver. It
implements all functions from mcstd.h and has been tested to be plug-compatible
with at least one of our CAMAC frontends. K.O. |
23 Sep 2006, Konstantin Olchanski, Bug Fix, commit latest ccusb.c CAMAC-USB driver
|
> I commited the latest driver for the Wiener CCUSB USB-CAMAC driver. It
> implements all functions from mcstd.h and has been tested to be plug-compatible
> with at least one of our CAMAC frontends. K.O.
This driver is known to not work with the latest CCUSB firmware (20x, 204, 30x, 303). I know what
modifications are required and an updated driver will be available shortly. If there is a delay, and you need the
driver ASAP, please drop me an email.
Also, I am thinking about dropping support for the very old CCUSB firmware revisions (before 204). (Any
comments?)
K.O. |
22 Aug 2007, Konstantin Olchanski, Bug Fix, commit latest ccusb.c CAMAC-USB driver
|
> > I commited the latest driver for the Wiener CCUSB USB-CAMAC driver. It
> > implements all functions from mcstd.h and has been tested to be plug-compatible
> > with at least one of our CAMAC frontends. K.O.
Well, it took almost a year to finish an updated driver, which has now been
commited to MIDAS SVN (see http://savannah.psi.ch/viewcvs/trunk/drivers/camac/ccusb/?root=midas).
This supports ccusb firmware release 0x402. With earlier firmware, simple CAMAC operations should work,
but to use the readout list feature one has to have the latest main firmware (0x402 as of today) and the latest CPLD
firmware.
The driver kit includes:
- the "ccusb" driver which implements the MIDAS mcstd.h CAMAC interface;
- test_ccusb to probe the interface and generally make the lights flash;
- ccusb_flash for updating the ccusb main firmware (assembled from bits and pieces found on the CCUSB driver CD);
- feccusb, an example midas frontend, which uses the ccusb readout list feature and has extensive error handling,
should be good enough for production use (unlike the Wiener libxxusb drivers, which lack basic error handling).
- analyzer.cxx, an rootana-based example on how to decode the ccusb data;
- README file with release notes.
If you use this driver, please drop me an email (even if it works perfectly for you, hah!) - the ccusb device is very
nice but can be hard to use and I would like to hear about problems other people have with it.
Today's version of the README files is attached below:
MIDAS driver for the Wiener/JTec CC-USB CAMAC-USB interface.
Date: 22-AUG-2007/KO
Note 1: The CC-USB interface comes with a CD which contains manuals,
firmware files, Windows and Linux software. The Wiener/JTec driver
is called "libxxusb". These MIDAS/musbstd drivers were written before
libxxusb bacame available and do not use libxxusb.
This driver implements the MIDAS CAMAC interafce "mcstd.h" using
the MIDAS USB interface musbstd.h.
Note 2: There exist many revisions of CCUSB firmware. Basic CAMAC
access works in all of them, but the "readout list" feature seems
to be only functional with firmware revision 0x402 or older and
with CPLD revisions CC_atmmgr_101406.jed, CC_datamgr_021905.jed,
CC_lammgr_brdcst_041906.jed or older.
To upgrade the main CCUSB firmware, follow instructions from
the CCUSB manual. On Linux, one can use the ccusb_flash
program included with these MIDAS drivers. It is a copy
of ccusb_flash from the Wiener CD, with all the pieces
assembled into one place and with a working Makefile. (I am too
lazy to add the flashing bits to the ccusb.c driver).
To upgrade the CPLD firmware, one needs a Xilinx JTag programmer
cable (we use a "parallel port to JTag" cable provided by Wiener),
and the Xilinx software (on Linux, we use Xilinx91i). For successful
upgrade, follow instructions from Xilinx and Wiener.
Note 3: Before starting to use the CCUSB interface, one should obtain
the latest version of the CCUSB manual and firmware by downloading
the latest version the CCUSB driver CD from the Wiener web
site (registration required)
Note 4: The example CCUSB frontend assumes this hardware configuration:
LeCroy 2249A 12 channel ADC in slot 20, Kinetic Systems 3615 6 channel
scaler in slot 12. NIM trigger input connected to CCUSB input "I1"
firing at 10-100 Hz. Without the external trigger CCUSB will not
generate any data and the frontend will only give "data timeout"
errors. With the trigger, the LED on the scaler should flash at 1 Hz
and the LEDs on the CCUSB should flash at the trigger rate.
Note 5: The CCUSB interface does not reliably power up in some CAMAC
crates (this has something to do with the sequence in which
different voltages start at different times with different CAMAC
power supplies). Some newer CCUSB modules may have this
problem fixed in the hardware and in the CPLD firmware. For modules
exhibiting this problem (i.e. no USB communication after power up),
try to cycle the power several time, or implement the "hardware reset
switch" (ask Wiener).
Note 6: The CCUSB firmware is very fickle and would crash if you look
at it the wrong way. This MIDAS driver tries to avoid all known crashers
and together with the example frontend, can recover from some
of them. Other crashes cannot be recovered from other than by
a hardware reset or power cycle.
//end |
|