> The current versions of mlogger SVN 4215 is flooding our message system with
> stuff like
> 
> > Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> > Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> > Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events

Problem confirmed on the M11 DAQ system at TRIUMF. We definitely do nothing funny
there, so what is going on? Will investigate.

K.O.

Since the very beginning midas had the problem that the last event of a run was
sometimes missing in the data. While for most experiments this is not an issue,
it starts to hurt on experiments using event building (front-end 1 and front-end
2 in the example below). A missing event can screw up the event builder on the
next begin of run, where the "missing event" would show up as the first event of
the new run, triggering an event mismatch error in the event builder.

After some analysis, we identified the problem as follows. Assume FE1 controls
the trigger, while FE2 generates the second event fragment.

1) Stop is requested to FE1
2) tr_stop gets called on FE1
3) tr_stop calls end_of_run() use code
4) end_of_run() disables the trigger
5) FE1 finishes stop transition
6) Stop is requested to FE2
7) FE2 finishes stop transition

What can now happen is the following: An additional event occurs between 2) and
4). This event triggers ADCs and TDCs, and is then stored in the front-end
hardware. FE2 sees this event, since it has not yet done the stop transition,
and reads it out. FE1 is however already in the end_of_run() routine, and simply
disables the trigger, without reading this last event, and thus causing the
event mismatch at the beginning of the next run.

To fix the problem, the framework in mfe.c was changed:

1) Stop is requested to FE1
2) tr_stop gets called on FE1
3) tr_stop calls end_of_run() use code
4) end_of_run() disables the trigger
4b) tr_stop calls check_polled_events()
5) FE1 finishes stop transition
6) Stop is requested to FE2
7) FE2 finishes stop transition

The new routine check_polled_events checks if there is any more event in the
hardware by calling the user polling routine. If there is one more event, calls
the user readout routine and sends it to the back-end before concluding the run
transition. 

This modification solved our problem at the MEG experiment at PSI, but it might
be good that all experiments using event building update midas to revision 4225.

I do not expect any bad side effect, but one never knows. So if there are new
problems caused by this modification, please report.

- Stefan

> The current versions of mlogger SVN 4215 is flooding our message system with
> stuff like
> 
> > Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> > Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> > Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events
> 
> This is fatal to us and blowing up the midas.log like hell. I would prefer if
> one could flag these kind of messages (ODB /Logger/..), i.e. enable and disable
> it. At the moment I have to comment it out in the source code since we cannot
> work with it.

I just sent the attached message to Stefan - please read it.

Before we take any action, we need to understand why history is being
reconfigured every 10 seconds (according to your logfile snippet).

Are you starting a new run every 10 seconds?

If that is what you do and that is your intent, I guess it is atypical usage of
MIDAS and the message from the mlogger is offensive and should be removed/disabled.

If something else is going on, we need to understand it before we sweep trouble
under the carpet by disabling this message.

K.O.

Stefan - there is more bad news - the message is produced when the history
is being reconfigured. This only is supposed to happen when the mlogger
starts or at the begin of run.

So these messages are just a tip of an iceberg of some other trouble.

The logic of when history is reconfigured I did not change. So likely
the trouble existed before, but you did not know about it.

We can kill the message, but why is the history being reconfigured
at a rate that "floods the log file"? That cannot possibly be good.

K.O.

The current versions of mlogger SVN 4215 is flooding our message system with
stuff like

> Tue Jun 10 16:42:01 2008 [Logger,INFO] Configured history with 22 events
> Tue Jun 10 16:42:14 2008 [Logger,INFO] Configured history with 22 events
> Tue Jun 10 16:42:26 2008 [Logger,INFO] Configured history with 22 events

This is fatal to us and blowing up the midas.log like hell. I would prefer if
one could flag these kind of messages (ODB /Logger/..), i.e. enable and disable
it. At the moment I have to comment it out in the source code since we cannot
work with it.

Cheers,
  Andreas 

Hi All,

I am testing the libraries provided by CAEN with the sample softwares in the 
bundle CD. The Windows sample program works fine, but I cannot get started with 
the Linux sample program. When I run CAENVMEDemo in Scientific Linux 5.1, it 
gives me a message "Error opening the device". I have followed the instructions 
in CAENVMElibReadme.txt: 
- compile and load the device driver v1718.ko
- install the library libCAENVME.so

Does anyone have any experience of using V1718 in Scientific Linux? Thanks.

Regards,
Jimmy


> Hi All,
> 
> Is there any example code for using MIDAS with the CAEN VME-USB Bridge V1718? 
> Thanks.
> 
> Regards,
> Jimmy

Hi All,

Is there any example code for using MIDAS with the CAEN VME-USB Bridge V1718? 
Thanks.

Regards,
Jimmy

> > Here is the list of known problems I am aware of and of fixes not yet committed to midas svn:
> > 1) added variable /equiment/foo/common/PerVariableHistory
> 
> corrected in svn revision 4203, read
> http://savannah.psi.ch/viewcvs/trunk/src/mlogger.c?root=midas&rev=4203&sortby=rev&view=log

Was still broken - all should work in revision 4207.

> > 2) writing compressed midas files (foo.mid.gz) crashes the mlogger when file
> > size reaches 2 GBytes. This problem could be new in SL5.1.

It turns out that on SL5 and SL5.1 (and others?) the 32-bit version of ZLIB opens the
compressed output file without the O_LARGEFILE flag, this limits the file size to 2 GB.

Fixed by opening the file ourselves, then attach compression stream using gzdopen().

Revision 4207. (Not tested on Windows - may be broken!)

> > 5) mhttpd history "export" button needs to be fixed (by request from ALPHA). At
> > present it either does not return all exiting data or crashes mhttpd. (no fix)
> 
> (no change)
> 
> > 6) mhttpd ODB editor in "set value" page, the "cancel" button is broken (needs
> > to be corrected for "relative URL").
> 
> Apply this patch to src/mhttpd.c
> 
> @@ -11156,10 +11190,7 @@
>           sprintf(str, "SC/%s/%s", eq_name, group);
>           redirect(str);
>        } else {
> -         strlcpy(str, path, sizeof(str));
> -         if (strrchr(str, '/'))
> -            strlcpy(str, strrchr(str, '/')+1, sizeof(str));
> -         redirect(str);
> +         redirect("./");
>        }
> 
> > 7) mhttpd needs AJAX-style methods for reading and writing ODB. (no fix)
> 
> (no change)
> 
> K.O.

> Here is the list of known problems I am aware of and of fixes not yet committed
> to midas svn:
> 
> 1) added variable /equiment/foo/common/PerVariableHistory

corrected in svn revision 4203, read
http://savannah.psi.ch/viewcvs/trunk/src/mlogger.c?root=midas&rev=4203&sortby=rev&view=log

> 2) writing compressed midas files (foo.mid.gz) crashes the mlogger when file
> size reaches 2 GBytes. This problem could be new in SL5.1.

(no change)

> 3) when a midas client becomes unresponsive, runs cannot be stopped using the
> "stop" button in mhttpd. This is because cm_transition() loops over all attached
> clients, but never removes clients that are known to be dead. Proposed fix is to
> call cm_check_client() for each client before calling their rpc transition handler.

Fixed in SVN revision 4198, read
http://savannah.psi.ch/viewcvs/trunk/src/midas.c?root=midas&rev=4201&sortby=rev&view=log

> 4) the discussed before fix for reading broken history files (skip bad data).

Fixed in SVN revision 4202, read https://ladd00.triumf.ca/elog/Midas/482

> 5) mhttpd history "export" button needs to be fixed (by request from ALPHA). At
> present it either does not return all exiting data or crashes mhttpd. (no fix)

(no change)

> 6) mhttpd ODB editor in "set value" page, the "cancel" button is broken (needs
> to be corrected for "relative URL").

Apply this patch to src/mhttpd.c

@@ -11156,10 +11190,7 @@
          sprintf(str, "SC/%s/%s", eq_name, group);
          redirect(str);
       } else {
-         strlcpy(str, path, sizeof(str));
-         if (strrchr(str, '/'))
-            strlcpy(str, strrchr(str, '/')+1, sizeof(str));
-         redirect(str);
+         redirect("./");
       }

> 7) mhttpd needs AJAX-style methods for reading and writing ODB. (no fix)

(no change)

K.O.

> > But to make things more interesting we had another history outage this week...
> > Anyhow, I now have a patch to allow hs_read() to "skip the bad spots" in history files.
> 
> [Stefan suggested]
>
>   if ((irec.time - last_irec_time) > 3600*24)


Yes, your stronger check works quite nicely. The whole patch is now committed into SVN,
revision 4202.

This is how it all works:

0) teach hs_gen_index() to skip over bad data. This is important because hs_read() only
looks at data records listed in the index file: if bad data is omitted from the index,
hs_read() will never see it and we do not need to worry about it in hs_read().
0a) because hs_gen_index() does not check validity of time stamps, we still need to check
them in hs_read().
1) in hs_read(), if we detect bad data (invalid headers, bad time stamps, etc), we
regenerate the index files - this removes a while class of bad data. We also look at time
stamps carefully and ignore records where time goes backwards (usually bad data) and ignore
records with time in the future beyound the end of the current history file (each history
file only contains 24*60*60 seconds = 1 day's worth of data).

While certainly not bullet-proof, these changes should make it easier to deal with
corruption of history files.

K.O.

Here is the list of known problems I am aware of and of fixes not yet committed
to midas svn:

1) added variable /equiment/foo/common/PerVariableHistory breaks stuff (mostly
mhttpd). It is not clear how this problem escaped my pre-commit checks. This
per-equipment variable enables the per-variable history for the given equipment.
Local consensus is that this variable should not be in "common" and should not
be in "settings". Probably in "/history"? Or have only one variable to enable
this for all equipments at once (like we do in ALPHA).

2) writing compressed midas files (foo.mid.gz) crashes the mlogger when file
size reaches 2 GBytes. This problem could be new in SL5.1.

3) when a midas client becomes unresponsive, runs cannot be stopped using the
"stop" button in mhttpd. This is because cm_transition() loops over all attached
clients, but never removes clients that are known to be dead. Proposed fix is to
call cm_check_client() for each client before calling their rpc transition handler.

4) the discussed before fix for reading broken history files (skip bad data).

5) mhttpd history "export" button needs to be fixed (by request from ALPHA). At
present it either does not return all exiting data or crashes mhttpd. (no fix)

6) mhttpd ODB editor in "set value" page, the "cancel" button is broken (needs
to be corrected for "relative URL"). (no fix)

7) mhttpd needs AJAX-style methods for reading and writing ODB. (no fix)

K.O.

Here is the list of known problems I am aware of and of fixes not yet committed
to midas svn:

FYI - in conjunction with replacement of ladd00.triumf.ca, this MIDAS ELOG has been updated to the latest 
version 2.7.3-2058. Please report any problems or anomalies. K.O.

> > I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
> > access functions db_xxx(), i.e.
> > INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
> > becomes
> > INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
>
> I fully approve your idea.

Committed revision 4172.

K.O.

> Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
> problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
> access functions db_xxx(), i.e.
> INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
> becomes
> INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
> 
> If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
> rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.
> 
> P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
> the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
> complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

I fully approve your idea. You are absolutely right that it also will help to prevent errors such as modifying
fixed strings. I was just too lazy to do that, because it requires some additional code like:

func(const char *p)
{
char str[256];

  strlcpy(str, p, sizeof(str));
  strlcat(str, ...)
} 

So if you do it, it's great!

Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
access functions db_xxx(), i.e.
INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
becomes
INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)

If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.

P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

K.O.

Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
access functions db_xxx(), i.e.
INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
becomes
INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)

If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.

P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

K.O.

Before approving the code, two conditions have to be fulfilled:

1) The code has to work on PSI experiments
2) The code must work without any SQL database

Concerning point 1), you correctly mentioned that the event numbering does not work
if there are more than 1000 variables per event. What I do not want is that there
will be a special T2K midas version and a special PSI version. This would make
maintenance horrible in the future. One could make the formula variable with id =
ev_id*n+var_n, where n is not fixed to 1000, but variable (stored in the ODB). The
down side would be that if you analyze your history files offline (outside the
experiment) you have to know a priori n in order to read back the data. If you have
990 variables, then you add 20, then you modify n from 1000 to 1500, then you would
screw up yourself since you cannot read the old data any more. 

Taking all this into account, I see no clean way to fix this except to modify the
database format (which you change anyhow "somehow" going to per-variable mode). Use a
32-bit ID for the event (16-bit) and the variable (16-bit). This will increase the
overhead, but only marginally, since there is already a 32-bit time stamp. But this
method would then work for all experiments at all times. I suspect that even in T2K
you will come at some point to a configuration where you have move than n variables
per event, whatever n is. So even you would benefit.

Concernign ponit 2), I like your ODBC approach. I never used it, but if you tell me
it works on all supported OSes it's fine with me, but make sure it compiles under
Windows (with the help of Pierre). One thing I would make sure however is that it
runs by default without setting up a database. There are many experiments out there
which do not need a SQL database, and it would be a hassle for them all to set up a
database, just to continue running. So by default I would use either the current flat
file system, and then per configuration enable ODBC, with bindings to MySQL pgSQL and
maybe SQLite3.

Cheers,

  Stefan

I would like to hear from potential users on which SQL database would be
preferable for storage of MIDAS history data.

My current preference is to use the ODBC interface, leaving the choice of
database engine to the user. While ODBC is not pretty, it appears to be adequate
for the job, permits "funny" databases (i.e. flat files) and I already have
prototype implementations for reading (mhttpd) and writing (mhdump/mlogger)
history data using ODBC.

In practice, MySQL and PgSQL are the main two viable choices for using with the
MIDAS history system. We tested both (no change in code - just tell ODBC which
driver to use) and both provide comparable performance and disk space use. We
were glad to see that the disk space use by both SQL databases is very
efficient, only slightly worse than uncompressed MIDAS history files.

At TRIUMF, for T2K/ND280, we now decided to use MySQL - it provides a better
match to MIDAS data types (has 1-byte and 2-byte integers, etc) and appears to
have working database replication (required for our use).

With mlogger already including support for MySQL, and MySQL being a better match
for MIDAS data, this gives them a slight edge and I think it would be reasonable
choice to only implement support for MySQL.

So I see 3 alternatives:

1) use ODBC (my preference)
2) use MySQL exclusively
3) implement a "midas odbc layer" supporting either MySQL or PgSQL.

Before jumping either way, I would like to hear from you folks.

K.O.

> The changes to mlogger implementing per-variable history have been committed to
> svn. Revision 4145.

To make code changes more clear, the commit was done in 3 stages:

revision 4142+4143 are minor fixes, refactoring (switch the code to use helper
functions) and implementation of history for structured banks
revision 4144 implements the per-variable history
revision 4145 is minor cleanup.

K.O.

The changes to mlogger implementing per-variable history have been committed to
svn. Revision 4145.

The rationale for these changes is roughly described in
https://ladd00.triumf.ca/elog/Midas/347

The main user-visible effect is reduction of data volume written to history
files and better integration with the history plot system in mhttpd.

The new functionality is disabled by default, pending review by Stefan (Except
for /history/tags stuff, which will be created by mlogger and used by mhttpd).
To enable it, set "/equipment/xxx/Common/PerVariableHistory" to 1 (type TID_INT).

In the "per-variable" mode, each entry in /equipment/xxx/variables is assigned
it's own event id and creates it's own events in the history file. In the
"classical" (or per-equipment) mode, all variables are assigned the same event
id (equal to the equipment id) and are written to disk at the same time.

In other words, in per-equipment mode, if there are 100 variables and 1 of them
is updated, all 100 numbers are written to disk. In per-variable mode, only the
one updated variable is written out.

The one point for review in this implementation is the assignment of event id's.
Committed code uses the formula "1000*eq_id + n" (i.e. variables in equipment id
2 get 2001, 2002, etc..., equipment id 3 get 3001, 3002, ...). This formula
works for most experiments, but as I understand is no good for some experiments
at PSI. Other than inventing a better formula that would work for everybody in
every case, one can also assign event id's manually by creating appropriate
entries in "/history/events".

This code has been used at CERN for running ALPHA since last Summer and it will
be used extensively at TRIUMF for T2K/ND280 slow controls. Per-variable history
is also required for the pending implementation of "history logged directly to
an SQL database", to be used at T2K/ND280.

If history (ahem) is any guide, we will now have a brief period of fixing merge
errors and "works for me" mistakes.

K.O.