> I suspect mlogger uses ASYNC transactions exactly to avoid
> this type of deadlock (mlogger used ASYNC transactions since svn revision 2, the
> beginning of time).

That's exactly the case. If you would have asked me, I would have told you 
immediately, but it is also good that you re-confirmed the deadlock behavior with 
the SYNC flag. I didn't check this for the last ten years or so.

Making the buffers bigger is only a partial solution. Assume that the disk gets 
slow for some reason, then any buffer will fill up and you get the dead lock.

The only real solution is to put the logic into a separate thread. So the thread 
does all the RPC communication with the clients, while the main logger thread logs 
data as usual in parallel. The problem is that the RPC layer is not yet completely 
tested to be thread safe. I put some mutex and you correctly realized that these 
are system wide, but you want a local mutex just for the logger process. You need 
also some basic communication between the "run stop thread" and the "logger main 
thread". Maybe Pierre remembers that once there was the problem that the logger did 
not know when all events "came down the pipe" and could close the file. He added 
some delay which helped most of the time. But if we would have some communication 
from the "run stop thread" telling the main thread that all programs except the 
logger have stopped the run, then the logger only has to empty the local system 
buffer and knows 100% that everything is done.

In the MEG experiment we have the same problem. We need a certain sequence 
(basically because we have 9 front-ends and one event builder, which has to be 
called after the front-ends). We realized quickly that the logger cannot stop the 
run, so we wrote a little tool "RunSubmit", which is a run sequence with scripting 
facility. So you write a XML file, telling RunSubmit to start 10 runs, each with 
5000 events. RunSubmit now watches the run statistics and stops the run. Since it's 
outside the logger process, there is no dead lock. Unfortunately RunSubmit was 
written by one of our students and contains some MEG specific code. Otherwise it 
could be committed to the distribution.

So I feel that a separate thread for run stop (and maybe even start) would be a 
good thing, but I'm not sure when I will have time to address this issue.

- Stefan

I would like to share with you a silly problem with mktime() and daylight savings time (Summer 
time/Winter time) that I have run into while working on the mhttpd history query page.

I am implementing 1 hour granularity for the queries (was 1 day granularity) and somehow all my queries 
were off by 1 hour.

It turns out that the mktime() and localtime() functions for converting between time_t and normal time 
units (days, hours) are not exact inverses of each other.

Daylight savings time (DST) is to blame.

While localtime() always applies the current DST,  mktime() will return the wrong answer unless tm_isdst is 
set correctly.

For tm_isdst, the default value 0 is wrong 50% of the time in most locations as it means "DST off" (whether 
that's Summer time or Winter time depends on your location).

Today in Vancouver, BC, DST is in effect, and localtime(mktime()) is off by 1 hour.

If I were doing this in January, I would not see this problem.

"man mktime" talks about "tm_isdst" special value "-1" that is supposed to fix this. But the wording of 
"man mktime" on Linux and on MacOS is different (I am amused by the talk about "attempting to divine 
the DST setting"). Wording at http://pubs.opengroup.org/onlinepubs/007908799/xsh/mktime.html is 
different again. MS Windows (Visual Studio) documentation says different things for different versions.

So for mhttpd I use the following code. First mktime() gets the approximate time, a call to localtime() 
returns the DST setting in effect for that date, a second mktime() with the correct DST setting returns the 
correct time. (By "correct" I mean that localtime(mktime(t)) == t).

time_t mktime_with_dst(const struct tm* ptms)
{
   // this silly stuff is required to correctly handle daylight savings time (Summer time/Winter time)
   // when we fill "struct tm" from user input, we cannot know if daylight savings time is in effect
   // and we do not know how to initialize the value of tms->tm_isdst.
   // This can cause the output of mktime() to be off by one hour.
   // (Rules for daylight savings time are set by national and local govt and in some locations, changes 
yearly)
   // (There are no locations with 2 hour or half-hour daylight savings that I know of)
   // (Yes, "man mktime" talks about using "tms->tm_isdst = -1")
   //
   // We assume the user is using local time and we convert in two steps:
   //
   // first we convert "struct tm" to "time_t" using mktime() with unknown tm_isdst
   // second we convert "time_t" back to "struct tm" using localtime_r()
   // this fills "tm_isdst" with correct value from the system time zone database
   // then we reset all the time fields (except for sub-minute fields not affected by daylight savings)
   // and call mktime() again, now with the correct value of "tm_isdst".
   // K.O. 2013-09-14

   struct tm tms = *ptms;
   struct tm tms2;
   time_t t1 = mktime(&tms);
   localtime_r(&t1, &tms2);
   tms2.tm_year = ptms->tm_year;
   tms2.tm_mon  = ptms->tm_mon;
   tms2.tm_mday = ptms->tm_mday;
   tms2.tm_hour = ptms->tm_hour;
   tms2.tm_min  = ptms->tm_min;
   time_t t2 = mktime(&tms2);
   //printf("t1 %.0f, t2 %.0f, diff %d\n", (double)t1, (double)t2, (int)(t1-t2));
   return t2;
}

K.O.

I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().

/Stefan

> I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().

I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().

K.O.

> > I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> > this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> > variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().
> 
> I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().
> 
> K.O.

tzset() not only sets the time zone, but also DST.

> > > I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> > > this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> > > variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().
> > 
> > I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().
> > 
> > K.O.
> 
> tzset() not only sets the time zone, but also DST.

I found following code in elogd.c, maybe it helps:

/* workaround for wong timezone under MAX OSX */
long my_timezone()
{
#if defined(OS_MACOSX) || defined(__FreeBSD__) || defined(__OpenBSD__)
   time_t tp;
   time(&tp);
   return -localtime(&tp)->tm_gmtoff;
#else
   return timezone;
#endif
}



void get_rfc2822_date(char *date, int size, time_t ltime)
{
   time_t now;
   char buf[256];
   int offset;
   struct tm *ts;

   /* switch locale temporarily back to english to comply with RFC2822 date format */
   setlocale(LC_ALL, "C");

   if (ltime == 0)
      time(&now);
   else
      now = ltime;
   ts = localtime(&now);
   assert(ts);
   strftime(buf, sizeof(buf), "%a, %d %b %Y %H:%M:%S", ts);
   offset = (-(int) my_timezone());
   if (ts->tm_isdst)
      offset += 3600;
   snprintf(date, size - 1, "%s %+03d%02d", buf, (int) (offset / 3600),
            (int) ((abs((int) offset) / 60) % 60));
}

I need to look at histograms inside a ROOT file, but all the old ways for doing this no longer work. (in theory I can scp the ROOT file to 
the computer I am sitting in front of, but this assumes I have a working ROOT there. anyhow it is pointless to fight this, all modern 
packages are written to only work on the developer's laptop).

- root new TBrowser starts a web server, tries to open firefox (and fails)
- root --web=off new TBrowser using ssh X11 tunnel no longer works, ROOT X11 graphics refresh is broken
- macos root binary kit is built without X11 support, root --web=off does not work at all
- root7 recommended "rootssh" prints an error message (and fails)

What does work well is JSROOT which we use to look at manalyzer live histograms (through apache and mhttpd web proxies).

So I wrote mjsroot.exe. It opens a ROOT file and starts JSROOT to look at it (plus a bit of dancing around to make it actually work):

mjsroot.exe -R8082 root_output_files/output00371.root

To actually see the histograms:

a) if you sitting in front of the same computer, open http://localhost:8082
b) if you are somewhere else, start an ssh tunnel: ssh daq13 -L8082:localhost:8082, open http://localhost:8082
c) if daq13 is running mhttpd, setup http proxy:
set ODB /webserver/proxy/mjsroot to http://localhost:8082
open https://daq13.triumf.ca/proxy/mjsroot/
also
set ODB /alias/mjsroot to "/proxy/mjsroot/"
reload MIDAS status page, observe "mjsroot" in listed in the left-hand side, open it.

K.O.
 

The JavaScript function

mjsonrpc_db_save / mjonrpc_db_load

have been dropped from the API because they were not considered safe. Users 
should use now the new function

file_save_ascii()

and

file_load_ascii()

These function have the additional advantage that the file is not loaded 
directly into the ODB but goes into the JavaScript code in the browser, which 
can check or modify it before sending it to the ODB via mjsonrpc_db_paste(). 

Access of these functions is limited to <experiment>/userfiles/* where 
<experiment> is the normal MIDAS experiment directory defined by "exptab" or 
"MIDAS_DIR". This ensures that there is no access to e.g. system-level files. If 
you need to access a directory not under "userfile", us symbolic links.

These files can be combined with file_picker(), which lets you select files on 
the server interactively.

Stefan

Dear MIDAS experts,

I'm using MIDAS javascript interface (mjsonrpc) to communicate with a frontend from a custom web page 
and observe that the if the frontend's response exceeds certain number of bytes, it is always truncated. 

MIDAS C/C++ RPC interface allows users to specify the max response length :

https://daq00.triumf.ca/MidasWiki/index.php/Remote_Procedure_Calls_(RPC)#C++_2

How would one do the same from with mjsonrpc ? 

-- many thanks, regards, Pasha

> I'm using MIDAS javascript interface (mjsonrpc) to communicate with a frontend from a custom web page 
> and observe that the if the frontend's response exceeds certain number of bytes, it is always truncated. 
> 
> MIDAS C/C++ RPC interface allows users to specify the max response length :
> 
> https://daq00.triumf.ca/MidasWiki/index.php/Remote_Procedure_Calls_(RPC)#C++_2
> 
> How would one do the same from with mjsonrpc ? 

I just documented the max_reply_length (javascript) and max_len (python) parameters on that page. Both are optional and default to 1024 bytes.

I also added a link to the full mjsonrpc schema https://daq00.triumf.ca/MidasWiki/index.php/Mjsonrpc#Schema_(List_of_all_RPC_methods) . You can also find the auto-generated schema on any midas installation by going to the "Help" webpage served by mhttpd and clicking the "JSON-RPC schema" > "text table format" link.

I would like to document a few problems I ran into while setting up a new
experiment (two USB interfaces to Alice TPC electronics, plus maybe a USB
interface to CAMAC). I am using a midas cvs checkout from last October, so I am
not sure if these problems exist in the very latest code. I have fixes for all
of them and I will commit them after some more testing and after I figure out
how to commit into this new svn thingy.

- mxml: writing xml into an in-memory buffer probably produces invalid xml
because one of the mxml functions always writes "/>" into writer->fh, which is 0
for in-memory writers, so the "/>" tag goes to the console instead of the xml
data stream.

- hs_write_event() closes fd 0 (standard input), which confuses ss_getch(),
which makes mlogger not work (at least on my machine). I traced this down to the
history file file descriptors being initialized to zero and hs_write_event()
closing files without checking that it ever opened them.

- mevb: event builder did not work with a single frontend (a two-liner fix, once
Pierre showed me where to look. Why? My second TPC-USB interface did not yet
arrive and I wanted to test my frontend code. Yes, it had enough bugs to prevent
the event builder from working).

- mevb: consumes 100% CPU. Fix: add a delay in the main busy-loop.

- mlogger ROOT tree output does not work for data banks coming through the event
builder: mlogger looks for the bank definition under the event_id of mevb, in 
/equipment/evb/variables, which is empty, as the data banks are under
/equipment/frontendNN/variables. This may be hard to fix: bank "TPCA" may be
under "fe01", "TPCB" under "fe02" and mlogger knows nothing about any of this.
Fix: go back to .mid files.

K.O.

FYI - committed the last changes from TRIUMF DAQ systems for PIENU and T2K/ND280 FGD and TPC 
tests:

- mhttpd: add <odb xxx format="%d">xxx</odb>, similar to AJAX ODBget() method
- alarm.c: if alarm stops the run, log a message (sometimes it is hard to tell "why did this run stop?!?") 
use DETACH transition (was ASYNC - does not follow requested transition sequencing, now calls 
mtransition helper). Also verified that alarm handler always runs on the main computer - for remote 
clients, alarms are processed inside the corresponding mserver process.
- midas.c: event buffer fixes:
-- mserver 100% cpu busy loop if event buffer is full
-- consolidate event buffer cleanup into one routine. do things similar to odb cleanup - check for client 
pid, etc.
-- do not kill clients that have the watchdog timeout set to zero.

svn rev 4541
K.O.

> FYI - committed the last changes from TRIUMF DAQ systems for PIENU and T2K/ND280 FGD and TPC 
> tests:
> svn rev 4541

Also:

- add traps to event buffer code to catch event buffer (shared memory) corruption observed in PIENU
- dynamically allocate some RPC network data buffers to permit better communication between MIDAS clients built with different values of 
MAX_EVENT_SIZE (in T2K/ND280 the default 4 Mbytes is too small for some users, while other users use the default size - this change permits all 
these programs to talk to each other).

K.O.

- add pthread lib to examples/... makefile
- fix ybos_simfe.c for max_event_size
- fix camacnul.c for cam_inhibit_test(), cam_interrupt_test()
- update documentation (1.9.3)
- made midas-1.9.3-1.tar.gz on Triumf site

- makefile.nt (/examples/experiment, /hbook)
  adjusted for local hmana.obj build  as for rmana.obj, add cvs tag for
  revision comment entry.
- drivers/class/hv.c
  change comment // to /* */

Minor changes to run transitions code:
- improve debug messages
- fail transition if cannot connect to one of the clients
K.O.

> Minor changes to run transitions code:
> - fail transition if cannot connect to one of the clients

This change introduced a problem:
1) a run is happily taking data
2) a frontend crashes
3) the web interface cannot stop the run (cannot contact the crashed frontend)
until  it is removed by the timeout (10-60 seconds?).

I am now considering allowing the run to end even if some clients cannot be
contacted. The begin, pause and resume transitions would continue to fail if
clients cannot be contacted.

K.O.

> I am now considering allowing the run to end even if some clients cannot be
> contacted. The begin, pause and resume transitions would continue to fail if
> clients cannot be contacted.

Sounds like a good idea.

- Stefan

When I compile current cvs-head midas, I get errors about undefined function
min(). I do not think min() is in the list of standard C functions, so
something else should be used instead, like a MIN(a,b) macro. To make life
more interesting, in a few places, there is also a variable called "min".
Here is the error:

src/mana.c: In function `INT write_event_ascii(FILE*, EVENT_HEADER*, 
   ANALYZE_REQUEST*)':
src/mana.c:2571: `min' undeclared (first use this function)
src/mana.c:2571: (Each undeclared identifier is reported only once for each 
   function it appears in.)
make: *** [linux/lib/rmana.o] Error 1

K.O.

> When I compile current cvs-head midas, I get errors about undefined function
> min(). I do not think min() is in the list of standard C functions, so
> something else should be used instead, like a MIN(a,b) macro. To make life
> more interesting, in a few places, there is also a variable called "min".
> Here is the error:
> 
> src/mana.c: In function `INT write_event_ascii(FILE*, EVENT_HEADER*, 
>    ANALYZE_REQUEST*)':
> src/mana.c:2571: `min' undeclared (first use this function)
> src/mana.c:2571: (Each undeclared identifier is reported only once for each 
>    function it appears in.)
> make: *** [linux/lib/rmana.o] Error 1

This is really a miracle to me. The min/max macros are defined both in midas.h
and msystem.h and worked the last ten years or so. However, I agree that macros
should follow the standard and use capital letters, so I changed that.