In revision 4090 I implemented ODB links to individual array elements. Now you
can have for example:

Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
array                           INT     10    4     2m   0   RWD
                                        [0]             0
                                        [1]             0
                                        [2]             123
                                        [3]             0
                                        [4]             0
                                        [5]             0
                                        [6]             0
                                        [7]             0
                                        [8]             0
                                        [9]             0
element2 -> /array[2]           INT     1     4     3m   0   RWD  123

In this case, the link "element2" points to the third element of "array", but is
treated like a single value. This links are very useful for example for the
"Edit on start" parameters, which can now point to individual array elements.
The same is true for the "Links BOR" when the logger writes to a MySQL database.

This modification required major modifications in the ODB. I have carefully
tested the example experiment from the distribution to verify that everything is
fine, but I'm not 100% sure that I covered all possible situations. So if you
update to revision 4090+ and you observe some strange behavior related to links
in the ODB, please report.

There are following two new functions related to this change: 

  db_get_link()
  db_get_link_data()

They are counterparts of db_get_key() and db_get_data(), respectively, but
without following links in the ODB. These functions are probably not of much use
outside odbedit and mhttpd, which are supposed to display links explicitly. Most
user applications want to follow links without even knowing that these are links.

The midas history system always had the problem that the database can get
corrupted if the disk gets full where the history records (*.hst & *.idx) are
stored. This can happen if a history event can only be written partially on the
almost full disk. If later some space is freed up (by deleting other files), the
writing continues at the old position, leaving the partial event in the data
base. In that case the whole history data of the current day cannot be read
because it is corrupted.

To solve the problem, a roll-back system has been implemented in the
hs_write_event() function. If an event cannot be written fully, the history file
is restored to the old state, so the partial event is removed from the end of
the file via truncation. This way only the data which could not be written to
the disk is missing in the history file, but the other data from that day is
still valid and readable. The change has been committed in revision 4107.

A major change was made to mhttpd, changing all internal URLs to relative paths.
This allows proxy access to mhttpd via an apache server for example, which might
be needed to securely access an experiment from outside the lab through a
firewall. Following setting can be places into the Apache configuration,
assuming the experiment runs on machine "online1.your.domain", and apache on a
publically available machine "www.your.domain":

Redirect permanent /online1 http://www.your.domain/online1
ProxyPass /online1/ http://online1.your.domain/

<Location "/online1">
  AuthType Basic
  AuthName ...
  AuthUserFile ...
  Require user ...
</Location>

If the the URL http://www.your.domain/online1 is accessed, it gets redirected
(after optional authentication) to http://online1.your.domain. If you click on
the mhttpd history page for example, mhttpd would normally redirect this to 

http://online1.your.domain/HS/

but this is not correct since you want to go through the proxy www.your.domain.
The new relative redirection inside mhttpd now redirects the history page
correctly to

http://www.your.domain/onlin1/HS/

I had to change many places inside mhttpd to make this work, and I'm not 100%
sure if I covered all occurrences. So if you upgrade to mhttpd revision 4115 and
observe some error accessing some pages, please report it to me.

- Stefan

Hi!

I have released pymidas - Python binding to Midas.
It includes support for Online Database, Buffer, event
construction and parsing. 

We have used it for a couple years now here at CMD. (http://cmd.inp.nsk.su)
One of principal DAQ applications here (Slow Control Frontend) is
written in Python using pymidas.

http://cmd.inp.nsk.su/~bilenko/projects/pymidas/pymidas.html

> A major change was made to mhttpd, changing all internal URLs to relative paths.
> This allows proxy access to mhttpd via an apache server for example, which might
> be needed to securely access an experiment from outside the lab through a
> firewall.

It is good to see improvements to the MIDAS URLs. We have been successfully running
mhttpd behind an apache SSL/HTTPS proxy without these changes, but our case was very
limited to one experiment, one mhttpd behind one proxy. I hope to test these changes
in the near future at CERN, I guess we will hear if things broke. I am especialloy
worried about the function for "split mhttpd history generator" via "/History/URL".
I remember it was hard to get it right and I hope if this function did not survive
this update, it will be easy to resurrect.
K.O.

> The midas history system always had the problem that the database can get
> corrupted if the disk gets full where the history records (*.hst & *.idx) are
> stored.

Stefan - big thanks for fixing this problem - it is one of those cases "how come I
did not think of do it!".

This change should fix the last remaining problem with history at CERN - we seem to
be unable to avoid running out of disk space once in a while (run away scripts, fat
fingers, etc) and history got corrupted every time.

But to make things more interesting we had another history outage this week - we
happen to write history files to an NFS server (not recommened! do not do this!) and
when the NFS server had a glitch, history files got corrupted - because during the
glitch NFS was not available, I think this roll-back feature would not have helped.

Anyhow, I now have a patch to allow hs_read() to "skip the bad spots" in the history
files. (hs_gen_index() also needs a patch).

In the nutshell, if invalid history data is detected, the code continues to read the
data one byte at a time, looking for valid event_id markers (etc).

The code looks sane by inspection, and if nobody objects, I would like to commit it
in the next few days.

Here is the diff against src/history.c rev 4114

Index: history.c
===================================================================
--- history.c	(revision 4118)
+++ history.c	(working copy)
@@ -129,6 +129,7 @@
    HIST_RECORD rec;
    INDEX_RECORD irec;
    DEF_RECORD def_rec;
+   int recovering = 0;
 
    printf("Recovering index files...\n");
 
@@ -171,7 +172,7 @@
 
          /* skip tags */
          lseek(fh, rec.data_size, SEEK_CUR);
-      } else {
+      } else if (rec.record_type == RT_DATA) {
          /* write index record */
          irec.event_id = rec.event_id;
          irec.time = rec.time;
@@ -180,6 +181,15 @@
 
          /* skip data */
          lseek(fh, rec.data_size, SEEK_CUR);
+      } else {
+
+         if (!recovering)
+            cm_msg(MERROR, "hs_gen_index", "broken history file %d, trying to
recover", (int)ltime);
+
+	 recovering = 1;
+         lseek(fh, -sizeof(rec)+1, SEEK_CUR);
+
+         continue;
       }
 
    } while (TRUE);
@@ -220,6 +230,7 @@
    time_t lt;
    int fh, fhd, fhi;
    struct tm *tms;
+   int idxsize = 0;
 
    if (*ltime == 0)
       *ltime = ss_time();
@@ -250,12 +261,15 @@
    hs_open_file(*ltime, "idf", O_RDONLY, &fhd);
    hs_open_file(*ltime, "idx", O_RDONLY, &fhi);
 
+   if (fhi >= 0)
+     idxsize = lseek(fhi, 0, SEEK_END);
+
    close(fh);
    close(fhd);
    close(fhi);
 
    /* generate them if not */
-   if (fhd < 0 || fhi < 0)
+   if (fhd < 0 || fhi < 0 || idxsize == 0)
       hs_gen_index(*ltime);
 
    return HS_SUCCESS;
@@ -1480,12 +1494,33 @@
             i = -1;
             M_FREE(cache);
             cache = NULL;
-         } else
+         } else {
+
+	 try_again:
+
             i = sizeof(irec);
-
-         if (cp < cache_size) {
             memcpy(&irec, cache + cp, sizeof(irec));
             cp += sizeof(irec);
+
+	    /* if history file is broken ... */
+	    if (irec.time < last_irec_time) {
+	      //printf("time %d -> %d, cache_size %d, cp %d\n", last_irec_time, irec.time,
cache_size, cp);
+
+	      //printf("Seeking next record...\n");
+
+	      while (cp < cache_size)
+		{
+		  DWORD* evidp = (DWORD*)(cache + cp);
+		  if (*evidp == event_id) {
+		    //printf("Found at cp %d\n", cp);
+		    goto try_again;
+		  }
+
+		  cp++;
+		}
+
+	      i = -1;
+	    }
          }
       } else
          i = read(fhi, (char *) &irec, sizeof(irec));

K.O.

I now merged almost all the mhttpd changes from CERN AD-5/ALPHA. Only the code
for mhttpd HTTP:// access control list remains unmerged.

Changes to the history display code merged from ALPHA: 
- add option to show latest values of history variables: "show values of
variables" check box on the history config panel 
- add custom labels for each variables: instead of midas variable name, history
plots would show the text entered into the "label" text area 
- show history errors on the plot: before, if one out of 10 history variables
could not be plotted, nothing was shown at all, now all variables are \
show, those that could not be read with hs_read() show the error code 
- the selection of which variables to plot is alphanumerically sorted (adc11 >
adc9) [this code is not active for standard midas because mlogger sup\
port has not yet been committed] 
- the selection of which variables to plot shows the last variable selected, not
the first one - useful when entering variables from a long list [th\
is code is not active for standard midas because mlogger support has not yet
been committed] 
 
These changes have been extensively tested since last Summer at the AD-5 ALPHA
expt at CERN. 
I could only do minimal testing for this merged code, so if there are any errors, 
they would most likely be merge errors. This new code will be heavily used at
TRIUMF, 
so if any errors got any, we hope to flush them out quickly. 
 
As noted, mlogger support for some of the mhttpd functions is not in standard
midas yet. It will be committed shortly. 

K.O.

> But to make things more interesting we had another history outage this week - we
> happen to write history files to an NFS server (not recommened! do not do this!) and
> when the NFS server had a glitch, history files got corrupted - because during the
> glitch NFS was not available, I think this roll-back feature would not have helped.

Actually I put our history data on a separate file system, on a separate disk controlled
by a separate RAID controller! If you write bulk data with the logger, and want to read
history files at the same time with mhttpd, you get a bottleneck if both data are at the
same physical disk. Separating this (and even the controller) speeded things up
dramatically.

The rollback will not work for NFS, since it requires truncating the file if an event
gets only partially written. While on a full file system you always can *delete* data,
this does not work if NFS is down. This explains the behavior.

> Anyhow, I now have a patch to allow hs_read() to "skip the bad spots" in the history
> files. (hs_gen_index() also needs a patch).
> 
> In the nutshell, if invalid history data is detected, the code continues to read the
> data one byte at a time, looking for valid event_id markers (etc).
> 
> The code looks sane by inspection, and if nobody objects, I would like to commit it
> in the next few days.

Great. I was thinking of something like this myself. Having a quick look at your code
looks good. The best of course would be if we would have some "magic number" for
re-synchronizating the data stream, but that would blow up the file length. So searching
for the right event id is good, but will not work 100%. Also the check

  if (irec.time < last_irec_time)

to see if the history is broken is very weak. If you take random data, it will be true
50% and false 50%. If one makes however a check

  if ((irec.time - last_irec_time) > 3600*24)

this would work correctly with random data in >99% of all cases (3600*24/2^32). Maybe
you should change that.

You misspelled one ODB entry:

Line 9014:
            sprintf(str, "/History/Display/%s/Label", path);

Line 9028:

            sprintf(str, "/History/Display/%s/Labels", path);
                                                ---^

I wonder how you could have tested that code for 1/2 year without noticing this error.
I fixed and committed it.
 

Backend computer with SLC4.4 Linux did'not work as mserver because some security
protection under iptables service (could not connect with frontend computers).
The connection established if to make ( under root ) iptables disable
by command: service iptables stop, or much more gently
just to accept mserver port with command (under root):
iptables -I INPUT -p tcp --dport 1175 -j ACCEPT
( in /etc/service 
  midas    1175/tcp       #Midas server)
To check which ports is open
it is possible to use the command: "netstat -n" to see digital numbers of ports.

> You misspelled one ODB entry:
> Line 9014:
>             sprintf(str, "/History/Display/%s/Label", path);
> 
> Line 9028:
>             sprintf(str, "/History/Display/%s/Labels", path);
>                                                 ---^
> 
> I wonder how you could have tested that code for 1/2 year without noticing this error.
> I fixed and committed it.


It turns out that the program was tested as originally committed. With the above
modification, it corrupts ODB - originally, it used the wrong array element size to create
the wrong array. Corrected, it creates the right array with the wrong size, then
subsequent db_set_data_index() happily corrupts ODB.

Fix for mhttpd committed as svn revision 4128.
Fix for ODB corruption committed at svn revision 4129 (also fixes extract_key())


K.O.

> > Affected files:
> > Makefile (add cnaf_callback.o)
> That's a good idea.
> To make things a bit easier, I modified the midas\examples\experiment\fronted.c to
> contain this call, so people should be guided by that. I also added cnaf_callback.c
> to the Makefile of the example frontend.

A request was made to remove cnaf_callback.o from libmidas as it creates a unwanted dependency on the CAMAC 
hardware driver when libmidas.so is used in programs that do not use CAMAC.

After looking around, it appears that removing cnaf_callback.o from libmidas would not break anything critical, 
other than CAMAC frontends that would fail to link with an obvious and easy to fix error.

I am leaving cnaf_callback.o in the Makefile - so it will be built and placed in linux/lib/cnaf_callback.o for anybody 
who wants to use it.

svn revision 4130.

K.O.

The changes to mlogger implementing per-variable history have been committed to
svn. Revision 4145.

The rationale for these changes is roughly described in
https://ladd00.triumf.ca/elog/Midas/347

The main user-visible effect is reduction of data volume written to history
files and better integration with the history plot system in mhttpd.

The new functionality is disabled by default, pending review by Stefan (Except
for /history/tags stuff, which will be created by mlogger and used by mhttpd).
To enable it, set "/equipment/xxx/Common/PerVariableHistory" to 1 (type TID_INT).

In the "per-variable" mode, each entry in /equipment/xxx/variables is assigned
it's own event id and creates it's own events in the history file. In the
"classical" (or per-equipment) mode, all variables are assigned the same event
id (equal to the equipment id) and are written to disk at the same time.

In other words, in per-equipment mode, if there are 100 variables and 1 of them
is updated, all 100 numbers are written to disk. In per-variable mode, only the
one updated variable is written out.

The one point for review in this implementation is the assignment of event id's.
Committed code uses the formula "1000*eq_id + n" (i.e. variables in equipment id
2 get 2001, 2002, etc..., equipment id 3 get 3001, 3002, ...). This formula
works for most experiments, but as I understand is no good for some experiments
at PSI. Other than inventing a better formula that would work for everybody in
every case, one can also assign event id's manually by creating appropriate
entries in "/history/events".

This code has been used at CERN for running ALPHA since last Summer and it will
be used extensively at TRIUMF for T2K/ND280 slow controls. Per-variable history
is also required for the pending implementation of "history logged directly to
an SQL database", to be used at T2K/ND280.

If history (ahem) is any guide, we will now have a brief period of fixing merge
errors and "works for me" mistakes.

K.O.

> The changes to mlogger implementing per-variable history have been committed to
> svn. Revision 4145.

To make code changes more clear, the commit was done in 3 stages:

revision 4142+4143 are minor fixes, refactoring (switch the code to use helper
functions) and implementation of history for structured banks
revision 4144 implements the per-variable history
revision 4145 is minor cleanup.

K.O.

I would like to hear from potential users on which SQL database would be
preferable for storage of MIDAS history data.

My current preference is to use the ODBC interface, leaving the choice of
database engine to the user. While ODBC is not pretty, it appears to be adequate
for the job, permits "funny" databases (i.e. flat files) and I already have
prototype implementations for reading (mhttpd) and writing (mhdump/mlogger)
history data using ODBC.

In practice, MySQL and PgSQL are the main two viable choices for using with the
MIDAS history system. We tested both (no change in code - just tell ODBC which
driver to use) and both provide comparable performance and disk space use. We
were glad to see that the disk space use by both SQL databases is very
efficient, only slightly worse than uncompressed MIDAS history files.

At TRIUMF, for T2K/ND280, we now decided to use MySQL - it provides a better
match to MIDAS data types (has 1-byte and 2-byte integers, etc) and appears to
have working database replication (required for our use).

With mlogger already including support for MySQL, and MySQL being a better match
for MIDAS data, this gives them a slight edge and I think it would be reasonable
choice to only implement support for MySQL.

So I see 3 alternatives:

1) use ODBC (my preference)
2) use MySQL exclusively
3) implement a "midas odbc layer" supporting either MySQL or PgSQL.

Before jumping either way, I would like to hear from you folks.

K.O.

Before approving the code, two conditions have to be fulfilled:

1) The code has to work on PSI experiments
2) The code must work without any SQL database

Concerning point 1), you correctly mentioned that the event numbering does not work
if there are more than 1000 variables per event. What I do not want is that there
will be a special T2K midas version and a special PSI version. This would make
maintenance horrible in the future. One could make the formula variable with id =
ev_id*n+var_n, where n is not fixed to 1000, but variable (stored in the ODB). The
down side would be that if you analyze your history files offline (outside the
experiment) you have to know a priori n in order to read back the data. If you have
990 variables, then you add 20, then you modify n from 1000 to 1500, then you would
screw up yourself since you cannot read the old data any more. 

Taking all this into account, I see no clean way to fix this except to modify the
database format (which you change anyhow "somehow" going to per-variable mode). Use a
32-bit ID for the event (16-bit) and the variable (16-bit). This will increase the
overhead, but only marginally, since there is already a 32-bit time stamp. But this
method would then work for all experiments at all times. I suspect that even in T2K
you will come at some point to a configuration where you have move than n variables
per event, whatever n is. So even you would benefit.

Concernign ponit 2), I like your ODBC approach. I never used it, but if you tell me
it works on all supported OSes it's fine with me, but make sure it compiles under
Windows (with the help of Pierre). One thing I would make sure however is that it
runs by default without setting up a database. There are many experiments out there
which do not need a SQL database, and it would be a hassle for them all to set up a
database, just to continue running. So by default I would use either the current flat
file system, and then per configuration enable ODBC, with bindings to MySQL pgSQL and
maybe SQLite3.

Cheers,

  Stefan

Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
access functions db_xxx(), i.e.
INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
becomes
INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)

If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.

P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

K.O.

Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
access functions db_xxx(), i.e.
INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
becomes
INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)

If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.

P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

K.O.

> Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
> problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
> access functions db_xxx(), i.e.
> INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
> becomes
> INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
> 
> If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
> rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.
> 
> P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
> the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
> complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

I fully approve your idea. You are absolutely right that it also will help to prevent errors such as modifying
fixed strings. I was just too lazy to do that, because it requires some additional code like:

func(const char *p)
{
char str[256];

  strlcpy(str, p, sizeof(str));
  strlcat(str, ...)
} 

So if you do it, it's great!

> > I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
> > access functions db_xxx(), i.e.
> > INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
> > becomes
> > INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
>
> I fully approve your idea.

Committed revision 4172.

K.O.