Back Midas Rome Roody Rootana
  Midas DAQ System, Page 88 of 142  Not logged in ELOG logo
ID Date Authordown Topic Subject
  1157   28 Jan 2016 Konstantin OlchanskiSuggestionscript command limited to 256 characters; remove limit?
Thank you for reporting this problem:

a) ODB key *names* are restricted to 31 characters (32 bytes, last byte is a NUL), not 256 characters.
b) ODB string length is unlimited (32-bit length field)
c) ODB C API "db_get_value" & co require fixed length buffer and most users of this API provide a 256-byte fixed buffer for strings, some of them also do not 
check the status code, resulting in silent truncation. (I think the ODB functions themselves report truncation to midas.log, so not completely silent).

We try to fix this where we must - but it is cumbersome with the current ODB API - as in your fix on has to:
- get the ODB key, extract size
- allocate buffer
- call db_get_value() & co
- use the data
- remember to free the buffer on each and every return path

The first three steps could become one if we had an ODB "get_data" function that automatically allocated the data buffer.

But the main source of bugs will be the last step - remember to free the buffer, always.

P.S.

We are not alone in pondering how to do this best. If you want to see it "done right",
read the fresh-off-the-presses book "Go Programming Language" by Alan Donovan and Brian Kernighan,
http://www.gopl.io/

Brian Kernighan is the "K" in K&R "C programming language", still around and kicking, now at Google.
Sadly the "R" passed away in 2011 - http://www.nytimes.com/2011/10/14/technology/dennis-ritchie-programming-trailblazer-dies-at-70.html

K.O.

> Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
> script via Midas - which silently truncates command strings longer than 
> 256 characters.
> 
> I'd prefer that Midas place no limit on string length.  Failing that, it would be
> helpful to have character limits called out in the documentation 
> (https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
> https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).
> 
> As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
> (Although key *names* are restricted to 256 characters.)  I've submitted one 
> possible version of an arbitrary-length exec_script() as a pull request 
> (https://bitbucket.org/tmidas/midas/pull-requests/).
> 
> Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
> strings in the ODB as limited to 256 characters?
  1166   26 Feb 2016 Konstantin OlchanskiSuggestionscript command limited to 256 characters; remove limit?
> Using low-level memory allocation routines in higher-level programs like mhttpd makes me nervous.

It should not, people have used malloc() for decades now without much injury to themselves. (Thomas corrects me: some people had big injury to their pride, me included).

> We could use vector arrays to allow variable-sized allocation, and use the data() member function to access the char* needed for functions like strlcat,
> db_get_data, and db_sprintf.

I thought auto_ptr was the correct tool to allocate "I just need a few bytes for a few minutes" arrays, but there is some discrepancy
between delete and delete[] (with brackets) and auto_ptr p(new char[i]) is verboten (even though it compiles just fine).

I ended up writing a custom replacement for auto_ptr called auto_string - now in mhttpd.cxx available for use in other places like this.

Still I think a db_get_data() that returns allocated memory is the correct solution. But this memory still needs to be released and lacking auto_ptr it opens the door for memory leaks.

> This conforms to the c++ standard, but doesn't require explicit freeing by the user - at least, not when you're allocating std::vector<char>

I do not think std::vector<char> can be cast into "char*" and used as replacement of "char str[100]" or "char* str = malloc(i);"

In other new, the limit on the command length is now removed.

K.O.

> 
> Amy
> 
> > Thank you for reporting this problem:
> > 
> > a) ODB key *names* are restricted to 31 characters (32 bytes, last byte is a NUL), not 256 characters.
> > b) ODB string length is unlimited (32-bit length field)
> > c) ODB C API "db_get_value" & co require fixed length buffer and most users of this API provide a 256-byte fixed buffer for strings, some of them also do not 
> > check the status code, resulting in silent truncation. (I think the ODB functions themselves report truncation to midas.log, so not completely silent).
> > 
> > We try to fix this where we must - but it is cumbersome with the current ODB API - as in your fix on has to:
> > - get the ODB key, extract size
> > - allocate buffer
> > - call db_get_value() & co
> > - use the data
> > - remember to free the buffer on each and every return path
> > 
> > The first three steps could become one if we had an ODB "get_data" function that automatically allocated the data buffer.
> > 
> > But the main source of bugs will be the last step - remember to free the buffer, always.
> > 
> > P.S.
> > 
> > We are not alone in pondering how to do this best. If you want to see it "done right",
> > read the fresh-off-the-presses book "Go Programming Language" by Alan Donovan and Brian Kernighan,
> > http://www.gopl.io/
> > 
> > Brian Kernighan is the "K" in K&R "C programming language", still around and kicking, now at Google.
> > Sadly the "R" passed away in 2011 - http://www.nytimes.com/2011/10/14/technology/dennis-ritchie-programming-trailblazer-dies-at-70.html
> > 
> > K.O.
> > 
> > > Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
> > > script via Midas - which silently truncates command strings longer than 
> > > 256 characters.
> > > 
> > > I'd prefer that Midas place no limit on string length.  Failing that, it would be
> > > helpful to have character limits called out in the documentation 
> > > (https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
> > > https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).
> > > 
> > > As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
> > > (Although key *names* are restricted to 256 characters.)  I've submitted one 
> > > possible version of an arbitrary-length exec_script() as a pull request 
> > > (https://bitbucket.org/tmidas/midas/pull-requests/).
> > > 
> > > Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
> > > strings in the ODB as limited to 256 characters?
  1168   09 Mar 2016 Konstantin OlchanskiInfo/Experiment/Edit on start/Edit Run number
The MIDAS documentation here:
  https://midas.triumf.ca/MidasWiki/index.php/Edit-on-start_Parameters
is missing informaiton about this ODB entry:
  /Experiment/Edit on start/Edit Run number (TID_BOOL)

This is what it does in mhttpd:
a) if it exists and is of type TID_BOOL and set to "n", run number is not editable
b) "Edit run number" itself is hidden, will not show up on the web page

This is what it does in odbedit:
a) it is hidden, will not show up in the list of run parameters
b) it's value has no effect, run number is always editable.

K.O.
  1172   22 Mar 2016 Konstantin OlchanskiInfoemacs web-mode.el
For those who use emacs to edit web pages - the built-in CSS and Javascript modes seem to work 
just fine for editing files.css and files.js, but the built-in html modes fall flat on modern web pages
which contain a mix of html, javascript inside <script> tags and javascript inside button "onclick" 
attributes.

So I looked at several emacs "html5 modes" and web-mode.el works well for me - html is indented 
correctly (default indent level is easy to change), javascript inside <script> tags is indented 
correctly (default indent level is easy to change), javascript inside "onclick" attributes has to be 
indented manually.

web-mode code repository and instructions are here, the author is very responsive and fixed my 
one request (permit manual intentation of javascript inside html attributes):
https://github.com/fxbois/web-mode

I now edit the html files in the MIDAS repository using these emacs settings:

8s-macbook-pro:web-mode 8ss$ more ~/.emacs
(setq-default indent-tabs-mode nil)
(setq-default tab-width 3)
(add-to-list 'load-path "~/git/web-mode")
(require 'web-mode)
(add-to-list 'auto-mode-alist '("\\.html\\'" . web-mode))
(setq web-mode-markup-indent-offset 2)
(setq web-mode-css-indent-offset 2)
(setq web-mode-code-indent-offset 3)
(setq web-mode-script-padding 0)
(setq web-mode-attr-indent-offset 2)
8s-macbook-pro:web-mode 8ss$ 

K.O.
  1179   17 May 2016 Konstantin OlchanskiInfoopenssl situation, MacOS 10.11 (El Capitan) openssl compilation errors
> I recently upgraded my macbook to MacOS 10.11. 
> [ and midas would not compile ]
> It seems that MacOS has now fully removed openssl ...

My read of tea leaves - the macos version of openssl was so old it was almost useless, did not support any of the modern HTTPS 
features. So to use mhttpd with https you pretty much had to install openssl from macports anyway. For macos 10.11 maybe they 
looked at upgrading to newer version, but since the openssl kerfuffle last year, there is several forks of openssl (the OpenBSD fork 
named libressl is the best, IMO), so rather than picking and choosing, they deleted the whole thing.

Now back to MIDAS.

We use the mongoose web server module and I have expected by now for them to make a move on improving HTTPS support, but no 
move happened.

Right now mongoose support OpenSSL only (I would expect the OpenBSD LibreSSL fork to work to of the box, too). Other then that, 
they have:
a) their own mickey-mouse https library (krypton) which does not support any modern cryptography (RC4 only - when RC4 is known to 
be useless).
b) an adapter library (polar) for interfacing with PolarSSL (mbedtls)

At this point I would rather abandon the implicit dependency on the system-provided openssl and have an explicit dependancy on a 
modern https crypto library.

Option (b) would work for us - 
1) add "git clone mbedtls; cd mbedtls; make" to midas build instructions
2) add polarssl_compat.c to midas git (from cessanta/polar repo)
3) retest mhttpd against ssllabs https scanner, retest against all web browsers.

The downside of this route is loss of automatic nightly updates to the https crypto library (for better or for worse).

K.O.

P.S. Because on MacOS use of openssl from macports is pretty much required, it should be moved from the "tricks" page to the 
standard midas installation instructions ("install required packages").
  1180   13 Jun 2016 Konstantin OlchanskiInforunning mhttpd on port 443
mhttpd running as non-root cannot bind to standard https port 443. By default, mhttpd uses port 
8443 and it works just fine, but some applications such as the SSLlabs https tester insist on using 
port 443.

To connect mhttpd with port 443, I use the tcpproxy package from
git://git.spreadspace.org/tcpproxy.git

./tcpproxy -D -U -p 443 -r localhost4 -o 8443

(you can run this from rc.local)

(to remember, for best security one should run mhttpd behind an industry-standard https proxy)

K.O.
  1181   13 Jun 2016 Konstantin OlchanskiBug Fixexample ssl certificate removed
I removed the example ssl certificate from the midas git repository (ssl_cert.pem). Now every midas 
installation must generate their own certificate - because to have any security at all each encryption 
private key has to be unique (and it has to be secret).

The command for generating a self-signed certificate is printed by mhttpd on startup:

openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl 
x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat 
ssl_cert.key >> ssl_cert.pem

K.O.
  1182   13 Jun 2016 Konstantin OlchanskiInfomongoose v6.4 is ready for use
latest version of mongoose web server library (v6.4) is now implemented in midas. To try it out, edit 
the Makefile, comment-out USE_MONGOOSE4, uncomment USE_MONGOOSE6, make clean, 
make.

After some more testing mongoose v6 will be made the default. (if you see problems, please report 
them here).

Main user-visible change is implementation of pipelined http requests, where the same socket 
connection is reused for many requests (instead of opening a new connection for each request). 
This is supposed to significantly speed up things like ajax requests over https (ssl handshake is 
done only once). (As a buglet, some midas web pages do not generated the "ContentLength" 
header, and force connection reset).

Special features: (implemented in mhttpd.cxx)

- https support (same as mongoose v4)
- https score A- at SSLlabs (if ignore whining about self-signed certificate)
- CORS support (same as v4) (cross-origin AJAX requests - web pages loaded from some other 
web server can make requests into midas)
- password protection (same as v4, uses http digest authentication)
- http-to-https redirect (same as v4)
- setuid-root mode for binding to port 80 (special request from PSI).

K.O.
  1189   08 Aug 2016 Konstantin OlchanskiReleaseMerged - new pure html web pages: programs and alarms.
The code for the new pure html and javascript web pages was merged into main midas.

In this release, the "programs" and "alarms" pages are implemented as html files, see 
resources/programs.html and alarms.html.

Eventually we hope to implement all midas web pages in html, so this is just a start.

If you see problems with the new html code, you can revert to the old mhttpd-generated web 
pages by removing the files programs.html and alarms.html.

The new code for starting and stopping runs (start.html and transition.html) is also merged, but not 
yet enabled, pending a few more tests.

K.O.
  1199   13 Sep 2016 Konstantin OlchanskiInfomongoose v6.4 is ready for use
> latest version of mongoose web server library (v6.4) is now implemented in midas.

A number of bugs were found in the mongoose v6 implementation of HTTP digest authentication:

- unusual URL in the form "https://blah:8443/?" (notice trailing "?") were rejected. These URLs are sometimes generated by 
MIDAS.
- URLs longer than 200 bytes were rejected
- a check for matching URIs between the HTTP request and in digest authentication was missing (required by specs)

If you are using mhttpd with mongoose v6 https, please update mhttpd.cxx to the latest version.

We continue to recommend that mhttpd be used behind a proper HTTPS proxy with password protection (i.e. apache httpd).

mongoose v4 does not seem to have the same bugs, old server does not support https so does not have these bugs.

K.O.
  1200   16 Sep 2016 Konstantin OlchanskiInfoNew rootana forum | rootana web display tools
> We have started a new elog for discussions of the ROOTANA MIDAS analyzer package

Posting there is almost like talking to oneself - barely anybody is subscribed, not even me.

Hence this reminder.

If you use ROOTANA, click the "config" link, then click the "rootana" checkbutton, then "save".

K.O.
  1202   26 Sep 2016 Konstantin OlchanskiInfomongoose v6.4 is ready for use
> Since updating to the most recent midas commit, we get the following error if we try running mhttpd without su privileges: 
> 
> >mhttpd -e CR --http 8081
> mhttpd is running in setuid-root mode.
> mhttpd is listening on port 80
> Mongoose version 4 cannot listen to port 80 in setuid mode. Please use mongoose version 6. Sorry, bye!
> [mhttpd,ERROR] [midas.c:1960:,ERROR] cm_disconnect_experiment not called at end of program
> 
> It works if we run it as root, but that creates other problems. Is there a flag to turn off setuid-root mode? Or some other fix?
> 


From these messages, it looks like you really are using the setuid-root mode. And indeed it is not usable with the mongoose version 4 implementation in MIDAS.

I can suggest several fixes:

1) the setuid-root mode was only ever intended for use at PSI because of peculiar network configuration of the PSI corporate firewall. It is not intended for general 
use.
1a) I as an author of MIDAS recommend against using the setuid-root mode and against installing mhttpd as setuid-root because it is not secure. (normally you 
would run mhttpd behind an apache https proxy providing https encryption and password protection).
1b) if you follow the midas installation instructions at https://midas.triumf.a you will see that we do not login as root and run "make install" to install mhttpd as 
setuid-root.
1c) if you follow these instructions, or if you run mhttpd from the midas build directory ($MIDASSYS/linux/bin/mhttpd), the setuid-root mode will not activate and 
everything will work ok.

2) you can run in the "old server" mode, but this more does not implement the JSON-RPC methods, so the "programs" and "alarms" pages will not work.
3) you can build mhttpd with the mongoose version 6 implementation, it will work even with the setuid-root mode. To do this, edit the Makefile, comment-out 
"USE_MONGOOSE4=1" and uncomment "USE_MONGOOSE6=1", then make clean, make.

K.O.
  1203   30 Sep 2016 Konstantin OlchanskiSuggestionAJAX jmsg "get messages since t" ability - add to docs?
> I recently needed to watch the Midas messages for a particular error - and 
> thus needed a command to "get all the messages since a time t".
> 
> The documentation (https://midas.triumf.ca/MidasWiki/index.php/AJAX#jmsg) 
> documents a way to "get the most recent n messages" - but when I dug into the 
> code, I was delighted to find that the existing Midas code also supports the 
> "get all messages since t" query.
> 
> For the "get all messages since t" query, the parameter t should be the unix 
> timestamp in seconds, and the parameter n should be zero: curl -X GET 
> "http://localhost:8081/?cmd=jmsg&n=0&t=1473437918".
> 
> Pretty useful!  Perhaps this should be added to the AJAX documentation?

The "jmsg" methods are obsolete - please use the JSON-RPC method "cm_msg_retrieve" as shown in resources/example.html. It takes all the same parameters as the midas.h 
cm_msg_retrieve() function, see the snipped from example.html below.

To see the full list of JSON-RPC methods, go to the "help" page and press the button for "json-rpc schema in text table format".

The entry for "cm_msg_retrieve" has this:

------------------------------------------------------------------------------------
cm_msg_retrieve?      | Retrieve midas messages using cm_msg_retrieve2()
                      | ------------------------------------------------------------
                      | params   | facility?           | string         | message facility, default is "midas"
                      |          | min_messages?       | integer        | get at least this many messages, default is 1
                      |          | time?               | number         | start from given timestamp, value 0 means give me newest messages, default is 0
                      | ------------------------------------------------------------
                      | result   | num_messages        | integer        | number of messages returned
                      |          | messages            | string         | messages separated by \n
                      |          | status              | integer        | return status of cm_msg_retrieve2()
------------------------------------------------------------------------------------

Snippet from resources/example.html: (to add "time" parameter, put "time":12345 next to "min_messages").

<input type=button value='Get last 10 midas messages'
          onClick='mjsonrpc_call("cm_msg_retrieve", { "min_messages": 10 })
                   .then(function(rpc) {
                   document.getElementById("cm_msg_retrieve_num_messages").innerHTML = JSON.stringify(rpc.result.num_messages);
                   document.getElementById("cm_msg_retrieve_messages").innerHTML = JSON.stringify(rpc.result.messages);
                   //mjsonrpc_debug_alert(rpc);
                   })
                   .catch(function(error) {
                   mjsonrpc_error_alert(error);
                   });'></input>
  1204   30 Sep 2016 Konstantin OlchanskiBug Reportcontrol characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail
> I've recently run into issues when using JSON.parse on ODB keys containing 
> 8-bit data.

I am tempted to take a hard line and say that in general MIDAS TID_STRING data should be valid 
UTF-8 encoded Unicode. In the modern mixed javascript/json/whatever environment I think
it is impractical to handle or permit invalid UTF-8 strings.

Certainly in the general case, replacing all control characters with something else or escaping them or 
otherwise changing the value if TID_STRING data would wreck *valid* UTF-8 strings, which I would 
assume to be the normal use.

In other words, non-UTF-8 strings are following non-IEEE-754 floating point values into oblivion - as 
we do not check the TID_FLOAT and TID_DOUBLE is valid IEEE-754 values, we should not check 
that TID_STRING is valid UTF-8.

But in your specific case, why do you have random control characters in your TID_STRING data? 
Maybe you are using TID_STRING as general storage instead of arrays of TID_CHAR or 
TID_DWORD?

K.O.



> 
> For JSON.parse to successfully parse a string, (A) the string must be valid 
> UTF-8, (B) several whitespace characters, control characters, and the 
> characters " and \ must be escaped, and (C) you've got to follow the key-
> value rules laid out in http://www.json.org/.
> 
> The web browser takes care of (A), and I verified that for this key Midas 
> handled (C) correctly.  In principle, the function json_write in odb.c 
> handles (B) - but json_write does not escape control characters.
> 
> To manage this problem, I modified json_write (in odb.c) to replace any 
> control character with the more-inocuous character, 'C'.  My default case 
> now looks like:
> 
> default:
>          {
>            // if a char is a control character,
>            // print 'C' in its place
>            // note that this loses data:
>            // a more-correct method would be to print
>            // \uXXXX, where XXXX is the character in hex
>            if(iscntrl(*s)){
>              (*buffer)[(*buffer_end)++] = 'C';
>              s++;
>            } else {
>              (*buffer)[(*buffer_end)++] = *s++;
>            }
>          }
>       
> Where the call to iscntrl(*s) requires the addition of the ctype.h header 
> file.
> 
> I'm guessing a blanket replacement of control characters with 'C' isn't 
> something all Midas users would want to do.  Replacing the control character 
> with its hex value seems like a good choice - but not without adding bounds 
> checking!
> 
> An alternative to changing odb.c could be to add a regex to Midas response 
> text which removes all control characters (U+0000 - U+001F): 
> 
> var resp_lint = req.response.replace(/[\u{0000}-\u{001F}]/gmu, '');
> var json_obj = JSON.parse(resp_lint);
> 
> Unfortunately, the 'u' regex flax doesn't work on the Firefox version 
> included in Scientific Linux 6.8.  
  1205   30 Sep 2016 Konstantin OlchanskiForumODB as JSON file
> Hi. Is it currently possible to automatically save the MIDAS ODB as a JSON file?
> I can do it manually in odbedit, but it looks like the only option for the
> automatic ODB save for each run is the standard .ODB format. Is there a way to
> change this?

I think today it makes sense to make all ODB dump in the JSON format - in my experience it is much 
easier to work with JSON data compared to XML data.

To write the ODB dump file in JSON format, set "/logger/ODB Dump File" to "run%05d.json".

In the midas data files, the ODB dump made into the begin-of-run and end-of-run events is presently 
unconditionally done in XML format.

Perhaps the data file dump should match the format of the odb dump file (both XML or both JSON or 
both ODB).

But at the moment our standard analyzer ROOTANA does not have the code to process JSON ODB 
dumps, so I am hesitant to make this change today, Maybe tomorrow when there is a VirtualODB 
JsonOdb class in ROOTANA. The requires JSON parser is already part of MIDAS (mjson.h/mjson.cxx).

K.O.
  1206   30 Sep 2016 Konstantin OlchanskiForumAlarm/Warning
> > Midas has a nice alarm system. I am wondering whether it is easily possible to
> > get the Alarm/Warning banner also on top of custom pages?!
> 
> K.O. made nice JavaScript routines to access the alarm status. The new alarm page is completely 
> made dynamically from JavaScript code (mhttpd does not supply any HTML code any more, only 
> functions to obtain ODB values etc). Part of this new dynamic page must be some code to display 
> the alarm status. You just need to copy this to your custom page. K.O. can tell you details.
> 

Yes, please look at resources/alarm.html and the "get_alarms" JSON-RPC method. The "get_alarms" example in 
resources/example.html probably already does exactly what you need. Also note the presence of "al_reset_alarm" and 
"al_trigger_alarm" JSON_RPC methods.

K.O.
  1207   30 Sep 2016 Konstantin OlchanskiSuggestionFrontend crush on high event rate
> 
> More suggestions from John and my reply.
> 
> > we also don't use the VME back plane - it's just too slow - mixing VME commands to plain modules and digitizer modules is unreliable....
> 
> > We use CAEN fiberoptic version 2 to talk to the digitizers directly, we have upto 12 digitizers, and can use all channels for several hours, and can fill to about 75% 
> of the A3818 bandwidth... 
> 
> So far we are limitted to 30 MB/s, if tested with CAEN examples, for example, the wavedump program by CAEN.
> I think is kind of the limit by IDE hard drive.
> Unfortunately we are still far from that limit, only ~ 1 MB/s now.  :(
>

From writing MIDAS frontends for many years, I am starting to form an opinion that this type of problem is undebuggable
in the current midas frontend framework - it is impossible to separate problems in vendor-supplied libraries and linux kernel modules
from problems with midas (i.e. incorrectly created data banks, too-small event buffers getting full) from problems with
bad interaction (collision over the SIGALARM handlers).

I am pondering on a new scheme for midas frontend writing. Perhaps such a new scheme should have a "no midas" mode where you can
compile and link a midas frontend "without midas", leaving you to debug just your code and the vendor code and their interactions.

K.O.
  1209   13 Oct 2016 Konstantin OlchanskiSuggestionchecksums for midas data files
Confirmed, this is a bug in mlogger. It should be creating *2* files, one with the before-compression checksum and one with the after-compression checksum. At 
least both checksums are written to midas.log, so you can grep them from there. K.O.

> Hi
> 
> > On one side, such checksums help me confirm that uncompressed data contents is the same as original 
> > data (compression/decompression is okey).
> > 
> 
> > I can write the computed checksums into midas.log, or into runNNN.crc32, runNNN.sha256, etc files. (or 
> > both).
> > 
> 
> Just a thought on my side. I have been using a checksum, on data produced  by our experiments via mlogger, the runxxxx.mid.gz, in 
> the same manner you proposed and I see now implemented. 
> 
> I have a slight, objection, if I may call it that, to how the checksum is saved to disk, in 
> run00007.mid.gz.sha256 as an example.
> 
> 
> $ cat ~/Data/run00007.mid.gz.sha256
> f315af7caf6ca204cc082132862cb4227d77066cb60c6e2b1039d6dc5b04d1ee 650597 Data/run00007.mid.gz
> 
> 
> It seems a little misleading to have the gzip'd filename paired with the checksum of the uncompressed content.
> 
> May I suggest that the pairing should be ,
> 
> f315af7caf6ca204cc082132862cb4227d77066cb60c6e2b1039d6dc5b04d1ee  run00007.mid as an example.
> 
> As I find, this information will sit in an archive, database in my case for a long period, and it might
> be confusing later on, when verification of the checksum is required.
  1210   13 Oct 2016 Konstantin OlchanskiInfonew odbinit utility
odbinit is a new utility program to initialize new ODB and to recover from corrupted ODB.

Right now, midas odb has some strange properties different from typical behavior of other 
database packages:

a) a new odb of default size is automatically create run running *any* midas program (surprise: now 
way to specify the size of odb).
b) the size of ODB is not saved anywhere. If your experiment requires an ODB of big size, one 
always forgets to use "odbedit -s" when recovering from odb corruption, leading to massive 
confusion: nothing works, odb is corrupted? (maybe not), recreate odb (of default size instead of 
large size), reload odb, (reload fails, odb is too small), now really for sure nothing works. Been 
there, done that myself 100 times. Tired.
c) there is no midas tool to automatically recover from odb corruption (or any generic ODB 
malfunction, such as stuck ODB semaphore): shared memory has to be deleted, old .ODB.SHM 
has to be deleted, old semaphore has to be deleted. Some of these steps are different on Linux 
and MacOS (hello Apple, where is MacOS "ls -l /dev/shm"?!?).

The new odbinit tool corrects these problems:

1) ODB size is saved to .ODB_SIZE.TXT, then is used to recreate ODB after corruption recovery
2) "odbinit -s different_size_from_saved_size" will ask "are you sure?". No way to unintentionally 
change size of ODB.
3) if you already have an ODB, it will insist that you say "odbinit --cleanup"
4) there is a "-n" mode, to report what will be done, but "do nothing"
5) "odbinit --cleanup" tries very hard to recover from any and all possible ODB problems.
6) old .ODB.SHM is never deleted, always renamed to .ODB.SHM.timestamp
7) if "odbinit" gets to "Done!", you have a working ODB, 100% guaranteed, for sure.
8) output of "odbinit" is very verbose for pasting into this forum here to make it possible to debug 
your problem. (in the unlikely case odbinit fails).

Next step will be to remove the automatic creation of ODB (and event buffers) and require running 
"odbinit" to create a new experiment. ("odbedit -s nnn" will be removed).

But not today, as all that requires changes to the midas internal APIs: ss_shm_open() needs to 
return the size of connected shared memory, there needs to be ss_shm_create() and 
db_create_database(), etc.

This will make ODB to work more like a normal database: with a tool to create a new database and 
a tool to recover from corruption/malfunction.

K.O.
  1212   14 Oct 2016 Konstantin OlchanskiForumWiener PCIVME link
> Hello, 
> I'm trying to make Wiener PCIVME link work with MIDAS. 
> In documentation/VME dirvers/ it's saying: "wevmemm.c PCI/VME Wiener board
> supported. (see Wiener PCI)".
> Provided link is dead. Does anyone have that file? I would appreciate very very
> much if someone could send it to me.
> 
> Thank you and best regards, 
> L.P.

Hi, I am not familiar with this module, I am pretty sure I have never seen one.
I do not see any code for it in the midas distribution.
I do not see any reference to it on the wiener web site (http://www.wiener-d.com/)

For obsolete modules, they direct us to http://file.wiener-d.com/ which is dead.

The next best step is to contact Wiener customer support. They usually reply very quickly.

If you have no luck getting answer directly from Wiener, you can ask me to contact them through
our sales representative. He is always super very helpful.

K.O.
ELOG V3.1.4-2e1708b5