> I through to improve this by fixing a bug in cm_msg_log() (where the messages are coming from)

The periodic messages about broken semaphore actually come from al_check(). I put some whining there, too.

K.O.

In some experiments (i.e. DEAP), we see spurious symlinks to midas.log scattered just about everywhere. I 
now traced this to an uninitialized variable in cm_msg_log() and it should be fixed now. K.O.

I made some tests of MIDAS support for "international characters" and we seem to be in a reasonable 
shape.

The standard standard is UTF-8 encoding of Unicode and the MIDAS core is believed to be UTF-8 clean - 
one can use "international characters" in ODB names, in ODB values, in filenames, etc.

The web interface had some problems with percent-encoding of ODB URLs, but as of current git version, 
everything seems to work okey, as long as the web browser is in the UTF-8 encoding mode. The default 
mode is "Western ISO-8859-1" and javascript encodeURIComponent() is mangling some stuff making the 
ODB editor not work. Switching to UTF-8 mode seems to fix that.

Perhaps we should make the UTF-8 encoding the default for mhttpd-generated web pages. This should be 
okey for TRIUMF - we use English language almost exclusively, but need to check with other labs before 
making such a change. I especially worry about PSI because I am not sure if and how they any of the special 
German-language characters.

On the minus side, odbedit does not seem to accept non-English characters at all. Maybe it is easy to fix.

K.O.

The capability to generate doxygen documentation of MIDAS was restored.

Use "make dox" and "make cleandox",
find generated documentation in ./html,
look at it via "firefox html/index.html".

The documentation is not generated by default - it takes quite a long time to build all the call graphs.

And the call graphs is what makes this documentation useful - without some visual graphical 
representation it is quite difficult to understand some parts of MIDAS. Both caller and callee graphs are 
generated.

Note that doxygen documentation for the javascript functions in mhttpd.js is also generated, making a 
handy reference in addition to the full documentation on the MIDAS Wiki.

K.O.

> Since several projects at PSI need MSCB but not MIDAS, I decided to separate the two repositories. So if you 
> need MIDAS with MSCB support inside mhttpd, you have to clone MIDAS, MXML and MSCB from bitbucket 
> (or the local clone at TRIUMF) as described in
> 
> https://midas.triumf.ca/MidasWiki/index.php/Main_Page#Download
> 
> I tried to fix all Makefiles to link to the new locations, but I'm not sure if I got all. So if something does not 
> compile please let me know.
> 
> -Stefan

After this split, Makefiles used to build experiment frontends need to be modified for the new location of the mscb tree:

replace
$(MIDASSYS)/mscb
with
$(MIDASSYS)/../mscb

K.O.

I am not sure what to do with the javascript snippet - I understand it should be somehow connected to /Custom/Header, but if I create the /Custom/Header string, I cannot put this snippet 
into this string using odbedit - if I try to cut&paste it into odbedit, it is truncated to the first line - nor using the mhttpd odb editor - when I cut&paste it into the odb editor text entry box, it 
is truncated to the first 519 bytes (must be a hard limit somewhere). K.O.

> As reported in the bug tracker, the proposed header does not work if no specific (= different from the default 60 sec.) update period is specified, 
> since then no cookie is present. Here is the updated code which works for all cases:
> 
> 
> 
> <div id="footerDiv" class="footerDiv">
> <div style="display:inline; float:left;">MIDAS experiment "Test"</div>
> <div id="refr" style="display:inline; float:right;"></div>
> </div>
> <script type="text/javascript">
> var r = document.getElementById('refr');
> var now = new Date();
> var refr;
> if (document.cookie.search('midas_refr') == -1)
>    refr = 60;
> else {
>    var c = document.cookie.split('midas_refr=');
>    refr = c.pop().split(';').shift();
> }
> r.innerHTML = now.toString() + '&nbsp;&nbsp;&nbsp;' + 'Refr:' + refr;
> </script>
> 
> 
> 
> /Stefan

> I'm looking into using MIDAS for an experiment that creates one large event
> (20MB or more) every second.

Hi, there - 20 Mbyte event at 1/sec is not so large these days. (Well, depending on your hardware).

Using typical 1-2 year old PC hardware, 20 M/sec to local disk should work right away. Sending data from a 
remote front end (through the mserver), or writing to a remote disk (NFS, etc) - will of course requre a GigE 
network connection.

By default, MIDAS is configured for using about 1-2 Mbyte events, so for your case, you will need to:

- increase the event size limits in your frontend,
- increase /Experiment/MAX_EVENT_SIZE in ODB
- increase the size of the SYSTEM event buffer (/Experiment/Buffer sizes/SYSTEM in ODB)

I generally recommend sizing the SYSTEM event buffer to hold a few seconds worth of data (ot 
accommodate any delays in writing to local disk - competing  reads, internal delays of the disks, etc).

So for 20 M/s, the SYSTEM buffer size should be about 40-60 Mbytes.

For your case, you also want to buffer 3-5-10 events, so the SYSTEM buffer size would be between 100 and 
200 MBytes.

Assuming you have between 8-16-32 GBytes of RAM, this should not be a problem.

One the other hand, if you are running on a low-power ("green") ARM system with 1 Gbyte of RAM and a 
1GHz CPU, you should be able to handle the data rate of 20 Mbytes/sec, as long as your network and 
storage can handle it - I see GigE ethernet running at about 30-40 Mbytes/sec, so you should be okey,
but local storage to SD flash is only about 10 Mbytes/sec - too slow. You can try USB-attached HDD or SSD, 
this should run at up to 30-40 Mbytes/sec. I would expect no problems with this rate from MIDAS, as long 
as you can fit into your 1 GByte of RAM - obviously your SYSTEM buffer will have to be a little smaller than 
on a full-featured PC.

More information on MIDAS event size limits is here (as already reported by Stefan)
https://midas.triumf.ca/MidasWiki/index.php/Event_Buffer

Let us know how it works out for you.

K.O.

> > The capability to generate doxygen documentation of MIDAS was restored.
> > 
> > Use "make dox" and "make cleandox",
> > find generated documentation in ./html,
> > look at it via "firefox html/index.html".
> > 
> 
> To generate the files, you need doxygen installed which not everybody has.

On most Linux systems, doxygen is easy to install. Red Hat instructions are here: 
http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Install_packages_needed_for_QUARTUS.2C_ROOT.2C_EPICS_and_MIDAS_DAQ

On MacOS, doxygen is easy to install via macports: sudo port install doxygen

> Is there a web site where one can see the generated graphs?

http://ladd00.triumf.ca/~olchansk/midas/index.html

there is no cron job to update this, but I may update it infrequently.

K.O.

I made a few minor modifications to the ODB JSON encoder and implemented a javascript "ls" function to 
report full ODB directory information as available from odbedit "ls -l" and the mhttpd odb editor page.

Using the new ODBMLs(), the existing ODBMCreate(), ODBMDelete() & etc a complete ODB editor can be 
written in Javascript (or in any other AJAX-capable language).

While implementing this function, I found some problems in the ODB JSON encoder when handling 
symlinks, also some problems with handling symlinks in odbedit and in the mhttpd ODB editor - these are 
now fixed.

Changes to the ODB JSON encoder:
- added the missing information to the ODB KEY (access_mode, notify_count)
- added symlink target information ("link")
- changed encoding of simple variable (i.e. jcopy of /experiment/name) - when possible (i.e. ODB KEY 
information is omitted), they are encoded as bare values (before, they were always encoded as structures 
with variable names, etc). This change makes it possible to implement ODBGet() and ODBMGet() using the 
AJAX jcopy method with JSON data encoding. Bare value encoding in ODBMCopy()/AJAX jcopy is enabled by 
using the "json-nokeys-nolastwritten" encoding option.

All these changes are supposed to be backward compatible (encoding used by ODBMCopy() for simple 
values and "-nokeys-nolastwritten" was previously not documented).

Documentation was updated:
https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js

K.O.

> If I am not mistaken, the mhttpd.css is hard coded (path/name) into the mhttpd.

mhttpd.css is served from $MIDASSYS/resources/mhttpd.css. The actual path is reported on the mhttpd 
"help" page.

(I think the internal mhttpd.css and mhttpd.js should be removed as no longer useful - nothing will work 
right if the real mhttpd.js and mhttpd.css cannot be served).

> Especially the look and feel of the runlog is unsatisfactorily from my point of view.

persons in charge of implementing the CSS stuff failed to convert quite a few pages, for example, the elog 
and the history editor pages were left completely broken. (mostly fixed now).

so thank you for reporting the runlog breakage, I hope Stefan & co can fix it quickly. (I cannot do - I have 
have no runlog pages on any of my test experiments).

> the old style was much more readable.

I think the new style is not too bad, except for a few visual artefacts here and there, the general comment 
that CSS is too complicated and hard to debug and the fact that over-subtle colouring yields inconsistent 
visuals between different monitors and ambient lighting conditions. (persons who select the colours always 
respond that "but to me, it looks just fine on my laptop", making it hard to resolve any issues).

> I could recover the old style look and feel by slightly changing the mhttpd.cxx

If you post the patches that fix it for you, I can commit them to midas. (git diff | mail olchansk@triumf.ca).

K.O.

> 
> I don't know if this actually fits the Bug Report category. I've been trying to install Midas on my Mac OS 
> Mavericks and I keep getting errors like "conflicting types for '___builtin____strlcpy_chk' ..." and similarly for 
> strlcat. I googled a bit and I think the problem might be that in Mavericks strlcat and strlcpy are already 
> defined in string.h, and so there might be a redundant definition somewhere. I'm not sure what the best 
> way to fix this would be though. Any help would be appreciated.
> 

We have run into this problem - MacOS 10.9 plays funny games with definitions of strlcpy() & co - and it has been fixed since last Summer.

For the record, current MIDAS builds just fine on MacOS 10.9.2.

For a pure test, try the instructions posted at midas.triumf.ca:

cd $HOME
mkdir packages
cd packages
git clone https://bitbucket.org/tmidas/midas
git clone https://bitbucket.org/tmidas/mscb
git clone https://bitbucket.org/tmidas/mxml
cd midas
make

K.O.

The midas wiki at https://midas.triumf.ca was updated to mediawiki 1.22.4 - the latest production version. 
If you see any problems, please report them to this elog. K.O.

> In the old SVN midas world it was typically such that the Windows dll's and
> exe's were ready to be used when checking out.

The Windows executables are no longer included in the midas git repository. Old versions are still available in 
the git repository - they got pulled in during conversion from svn.

One reason for removing them is that neither myself, nor Pierre, nor Stefan have ready access to a Windows 
development environment and we cannot keep Windows binaries up to date. Theoretically we can setup a 
Windows machine just for compiling MIDAS, but then there is a question of which Windows we should use and 
how much priority we should put into it. I do not think there is any demand for MIDAS on Windows at TRIUMF.

(Personally, I think Windows is no longer a viable platform for any business use - with Microsoft focusing on 
"experiences", "tiles", touch screens, portable devices, and other gimmicks - rather than on providing a solid OS 
to get work done)

> I am not so sure this is the case
> for the current version, since when I use the packed dll's  and exe's (e.g.
> odbedit.exe) I get the warning that this is running midas 2.0.0 but the current
> version (on the linux server) is 2.1. What does this mean?

You can ignore this message. Stefan incremented the MIDAS version when we migrated to git, but
there are no changes to the MIDAS RPC mechanism and we are still fully compatible with old versions,
at least in the MIDAS RPC and in the mserver.

So tools like odbedit.exe should still work okey when connecting from Windows to MIDAS running on Linux or 
MacOS.

But old frontend programs may cause some trouble because the ODB layout changed somewhat with new things 
added to /eq/xxx/common. Simplest is to try, if it works, it works.

> 1) A little bug in the packed windows part, but up-to-date dll's and exe's?
> 2) The dll's and exe's are not bundled any more to up-to-date version?

Case (2) is the case. Personally I do not have any capability to build Windows binaries. Same for Pierre and I think 
for Stefan.

> If 2) is the case, I would like to get a hint how to build midas under Windows
> (Windows 7), since we still have some few Windows clients.  

I do not think pre-built executables will ever return - the new way of things is to "cut-and-paste" the "git clone" 
command from a web page, type "make", and be done with it. If your OS does not have "git", "make" & etc, you 
should switch to a real OS.

On the MIDAS software side, we have no problem with supporting Windows - same as on any other platform, 
please try to build and run it, report any problems, fixes, patches and improvements - we will commit them into 
the midas repository.

K.O.

> I stumbled over a problem which I cannot pin point and would appreciate suggestions.
> 
> [nemu@lem00 2014]$ odbedit -e nemu
> odbedit: src/odb.c:753: db_update_open_record: Assertion `xkey->notify_count == pkey-
>notify_count' failed.
> Aborted

I think this is a real bug in MIDAS - I will have to take a look to figure out where this is coming from. At the 
least, if I cannot replace the assert with some corrective action, I may replace it with an error message.

I am glad you could recover by reloading odb.

K.O.

> > > odbedit can now save ODB in JSON-formatted files.
> encode NaN, Inf and -Inf as JSON string values "NaN", "Infinity" and "-Infinity". (Corresponding to the respective Javascript values).

A new standard just came out - Oasis OData JSON format 4.0 - 
http://docs.oasis-open.org/odata/odata-json-format/v4.0/os/odata-json-format-v4.0-os.html

Section 7.1 reads:

> Values of types [...] Edm.Single, Edm.Double, and Edm.Decimal are represented as JSON numbers, except for NaN, INF, and –INF which are represented as strings.

This is consistent with what we do in MIDAS - encode special numbers as strings. For now I think we stay with Javascript-standard "Infinity", "-Infinity",
but if more standards start using "INF", "-INF", maybe we will switch. It is easy enough to support both encodings in the JSON parser and in the ODB decoder.

https://xkcd.com/927/
K.O.

> I'm trying to get a frontend running on an arduino yun single board computer
> (cpu is Atheros AR9331 and OS is a linux derivate
> http://arduino.cc/en/Main/ArduinoBoardYun )

What you want to do should be possible.

Here, the smallest machine we used to run a MIDAS frontend was a 300MHz PowerPC processor inside a 
Virtex4 FPGA with 256 Mbytes of RAM. Looks like your machine is a 400MHz MIPS with 64 Mbytes of RAM 
so there should be enough hardware available to run a MIDAS frontend underLinux.

One source of trouble could be if your MIPS CPU is running in big-endian mode (MIPS can do either big-
endian or little-endian). MIDAS supports big-endian frontends connecting to little-endian x86 PC hosts, 
but with big-endian machines getting less common, this code does not get much testing. If you run into 
trouble with this, please let us know and we will fix it for you.

> The idea is to use this device for some slow control for our experiment (ASACUSA
> Antihydrogen) we are using midas as main DAQ system and we would like to
> integrate the slow control with this small boards.

> My question is: How can I compile the midas library with the openwrt crosscompiler?

In the MIDAS Makefile, looks for the "crosscompile" target which we use to cross-build MIDAS for our 
PowerPC target using the regular GCC cross compiler chain. If you have very new MIDAS, you will also see 
some make targets for ARM Linux machines, also using GCC cross compilers.

> the system discspace is very limited (6 MB) therefore I don't want to have mysql, zlib an so on.

The MIDAS Makefile crosscompiler builds a very minimalistic version of MIDAS - no mysql, no sqlite, etc 
requirements for the MIDAS libraries and frontend. zlib may be required but it is not used by frontend 
code, so you may try to disable it.

If that is still too big, there is a possibility for building a super-minimal version of MIDAS just for running 
cross-compiled frontends. We use this function to build MIDAS for VxWorks. If you want to try that, I 
think it is not in the main Makefile, but in the VxWorks Makefile. Let me know if you want this and I can 
probable restore this function into the main Makefile fairly quickly.

> Do you have any suggestions on how to realize something like that?

1) cross compile MIDAS (see  the Makefile "make crosscompile" target)
2) cross compile your frontend
3) run it, with luck, it will fit into your 64 Mbytes of RAM

If you run into problems, please post them here (so other people can see the problems and the solutions)

K.O.

> Ok, I'm currently trying to get things running, setting up a crosscompiler toolchain for the Arduino Yun is fairly
> easy, just follow the tutorial on the  OpenWrt webpage.
> 
> The main problem is that openwrt uses the uClibc library instead of glibc this produces lots of difficulties
>

Okey, I see. I do not think we used uClibc with MIDAS yet.

>
> one is that building of the shared library is complaining about symbol name mismatches, but I guess this can be
> fixed somehow, I wont use the midas-shared library, therefore I just disabled it in the Makefile. 
> 

The shared library is generally not used. The Makefile builds it as a convenience for things like pymidas, etc.

> 
> The next problem is the backtrace functions tjhat are used within system.c, the functions backtrace and
> backtrace_symbols are only available in glibc for a quick fix I just changed the #ifdef directive in a way that this
> code is not built.
>

Yes. They should probably be behind an #ifdef GLIBC (whatever the GLIBC identifier is)

> 
> There is a more tricky problem, the compiler complains about mismatched function defintions:
> 
> error: declaration of 'size_t strlcat(char*, const char*, size_t) throw ()' has a different exception specifier
> error: declaration of 'size_t strlcpy(char*, const char*, size_t) throw ()' has a different exception specifier
> 
> This can be solved by editing the midas.h file:
> size_t EXPRT strlcpy(char *dst, const char *src, size_t size); -> size_t EXPRT strlcpy(char *dst, const char *src,
> size_t size) __THROW __nonnull ((1, 2));
> 

No need to edit anything, this is controlled by NEED_STRLCPY in the Makefile - to enable our own strlcpy on systems that do not provide it (hello, GLIBC!)

> 
> After changing this midas compiles with the crosscompiler and the resulting programs are executable on the Arduino
> Yun. I'll report back if I got my frontend to run and connect to the midas server.

Congratulations!

K.O.

We have tested operation of MIDAS using a 10GigE network connection. Using a dummy frontend 
generating fake data, we can record MIDAS data to disk at at least 700 Mbytes/sec as reported by 
the MIDAS status page.

Two configurations were tested, both run at at least 700Mbytes/sec sustained:

1) MIDAS mhttpd, mserver, mlogger running on the disk server machine (mlogger writes to local 
disk), frontend running on remote machine (10GigE mserver connection).
2) MIDAS mhttpd, mserver, mlogger, frontend running on remote machine (mlogger writes data to 
an NFS-mounted disk over a 10GigE connection).

In addition, for configuration (2), I simulated online analysis reading fresh midas files at the same 
time as MIDAS writes new data. The resulting observation is that Linux seems to be giving main 
priority to disk write traffic (700 Mbytes/sec) with the remaining disk bandwidth given to read traffic 
(50-100Mbytes/sec). In other words, when running online data analysis on fresh data files, mlogger 
continues to run at full speed (analysis does not slow down data taking).

A few problems with MIDAS were observed during this test:

a) mlogger data compression using gzip-1 has to be turned off (limits data rate to about 
200Mbytes/sec). We plan to implement high speed LZO/LZ4 data compression that we expect to 
keep up with a 10GigE network interface.
b) CPU use by mserver and mlogger is rather high (about 40% CPU)
c) when writing to the NFS disk, mlogger has a pause of 1-2 seconds when closing and reopening 
subrun data files. To avoid a interruption in data taking, the SYSTEM event buffer has to be big 
enough to ride through this pause, but stock MIDAS limits the maximum size of event buffer to 1GB 
(too small), this can be easily increased to 2GB (almost big enough) and with some more work it can 
be increased to 4GB, but no more because the buffer length is a 32-bit integer.
d) when writing to the NFS disk, we also see periodic 3-5 second interruptions ("write operation took 
5123 ms") and we had one death of mlogger by a timeout of 60 sec.

Details of the hardware:

1) the disk server machine CPU is 3.4GHz Intel i7-4770, mobo is ASUS Z87 WS (10 SATA, 2xGigE), 
RAM is 32GB DDR3-1600.
2) disk array is 8x4TB Seagate ST4000VN000-1H4168 NAS disks RAID0 (striped) configuration, raw 
data read/write rate is around 1 GByte/sec, disks are directly attached to mobo (no raid card), linux 
software raid.
3) the frontend machine CPU is 3.7GHz Intel i7-4820, mobo is ASUS P9X79 WS, RAM is 32GB DDR3-
1600.
4) 10GigE network is Solarflare Communications SFC9120 (both machines) with a cross-over fiber 
cable (direct connection,no switches)
5) OS is up-to-date SL6.5 (both machines)

K.O.

> We have tested operation of MIDAS using a 10GigE network connection. Using a dummy frontend 
> generating fake data, we can record MIDAS data to disk at at least 700 Mbytes/sec as reported by 
> the MIDAS status page.
>
> Details of the hardware:
> 
> 1) the disk server machine CPU is 3.4GHz Intel i7-4770, mobo is ASUS Z87 WS (10 SATA, 2xGigE), 
> RAM is 32GB DDR3-1600.
> 2) disk array is 8x4TB Seagate ST4000VN000-1H4168 NAS disks RAID0 (striped) configuration, raw 
> data read/write rate is around 1 GByte/sec, disks are directly attached to mobo (no raid card), linux 
> software raid.
>

These tests were done using a raid0 array (striped), which is not suitable for production use.

For production use, RAID5 and RAID6 is recommended. But their default configuration has severely reduced performance (50% of 
RAID0) this is because internally the raid driver issues disk read operations that compete against and severely slow down the disk write 
requests. This is easy to see with "iostat -x 1" - when writing to the raid array, there should be no reads from the disks. Following 
changes are required to achieve maximum performance:

echo 32000 > /sys/block/md6/md/stripe_cache_size # increase internal memory buffers - because "raid write" is always "read-
modify-write", bigger buffers ensure that the reads are done from cache, not from phsyical disk
mdadm --grow --bitmap=/md6bitmap /dev/md6 # use external bitmap - if bitmap is internal, there is a large number of disk reads 
competing against writes. external bitmap seems to help quite a bit.

With these settings, my RAID6 array can read and write at about 700-900 Mbytes/sec - this is comparable to RAID0 (minus 2 disks).

With this, I repeated the MIDAS performance tests - (but without 10GigE) - MIDAS can write 700 Mbytes/sec of fake data to a local 
RAID6 data array. (hardware configuration is listed above).

K.O.

In the ALPHA experiment at CERN I found a problem in mfe.c handling of multithreaded equipments. This problem was in 
some forms introduced around May 2013 and around Aug 2013 (commit 
https://bitbucket.org/tmidas/midas/src/45984c35b4f7/src/mfe.c) (I hope I got it right).

The effect was very odd - if event rate of multithreaded equipment was more than 100 Hz, the event counters on the midas 
status page would not increment and the frontend will crash on end of run. Other than that, all the events from the 
multithreaded equipment seem to appear in the SYSTEM buffer and in the data file normally.

This happened: in mfe.c::receive_trigger_event() a loop was introduced (previously,
there was no loop there - there was and still is a loop outside of receive_trigger_event()):

while (1)
   wait 10 ms for an event
   process event, loop back
   if there is no event, exit
}

Obviously, if the event rate is more than 100 Hz (repetition rate less than 10 ms),
the 10 ms wait will always return an event and we will never exit this loop.

So the mfe.c main loop is now stuck here and will not process any periodic activity
such as updating the equipment statistics (event counters on the midas status page)
or running periodic equipments in the same front end program.

The crash at the end of run will be caused by a timeout in responding to the "end of run" RPC call.

I have a patch in testing that solves this problem by restoring receive_trigger_event() to the original configuration, i.e. 
https://bitbucket.org/tmidas/midas/src/6899b96a4f8177d4af92035cd84aadf5a7cbc875/src/mfe.c?at=develop

K.O.