ID |
Date |
Author![up up](up.png) |
Topic |
Subject |
874
|
11 Apr 2013 |
Konstantin Olchanski | Forum | Persistent ipcrm error | > [system.c:308:ss_shm_open,ERROR] Shared memory segment with key 0x4d008002 already exists,
please remove it manually: ipcrm -M 0x4d008002
> [midas.c:1950:cm_connect_experiment1,ERROR] cannot open database
> Unexpected error #304
For the record, the SYSV shared memory with it's keys and segments has always been brittle and hard to
debug with problems such as you describe.
Also SYSV shared memory suffers from key aliasing - shared memory segments created with different
names all map into the same key, collide and nothing works. You may not see this if all the files are
located on a local disk, but if the .SHM files are located on an NFS disk, it can happen (and did happen in
T2K).
For this reason, since around August 2010, MIDAS also implements the POSIX shared memory and for new
MIDAS installations, POSIX shared memory is the default. (On MacOS, POSIX shared memory was always
the default because MacOS has very small maximum SYSV shared memory size).
The type of shared memory is set by the contents of .SHM_TYPE.TXT and it is possible to switch between
SYSV and POSIX shared memory at will. (Ask me).
MIDAS still uses SYSV semaphores because they have a built-in feature to automatically unlock the
semaphore if the program that locked it dies for any reason. POSIX semaphores do not have this built-in
feature and we would have to implement some kind of detection and recovery for the case when a
semaphore is locked by a program that died (and will never unlock it back).
K.O.
P.S. I will address the rest of Prof. Thorsten's question in a private email.
P.P.S. Please post elog messages in the "plain" format. NOT HTML or ELCODE. |
881
|
30 Apr 2013 |
Konstantin Olchanski | Info | ROOT switched to GIT | Latest news - the ROOT project switched from SVN to GIT.
Announcement:
http://root.cern.ch/drupal/content/root-has-moved-git
Fons's presentation with details on the conversion process, repository size and performance
improvements:
https://indico.cern.ch/getFile.py/access?contribId=0&resId=0&materialId=slides&confId=246803
"no switch yard" work flow:
http://root.cern.ch/drupal/content/suggested-work-flow-distributed-projects-nosy
GIT cheat sheet:
http://root.cern.ch/drupal/content/git-tips-and-tricks
K.O. |
882
|
06 May 2013 |
Konstantin Olchanski | Info | Recent-ish SVN changes at PSI | A little while ago, PSI made some changes to the SVN hosting. The main SVN URL seems to remain the
same, but SVN viewer moved to a new URL (it seems a bit faster compared to the old viewer):
https://savannah.psi.ch/viewvc/meg_midas/trunk/
Also the SSH host key has changed to:
savannah.psi.ch,192.33.120.96 ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEAwVWEoaOmF9uggkUEV2/HhZo2ncH0zUfd0ExzzgW1m0HZQ5df1OYIb
pyBH6WD7ySU7fWkihbt2+SpyClMkWEJMvb5W82SrXtmzd9PFb3G7ouL++64geVKHdIKAVoqm8yGaIKIS0684
dyNO79ZacbOYC9l9YehuMHPHDUPPdNCFW2Gr5mkf/uReMIoYz81XmgAIHXPSgErv2Nv/BAA1PCWt6THMMX
E2O2jGTzJCXuZsJ2RoyVVR4Q0Cow1ekloXn/rdGkbUPMt/m3kNuVFhSzYGdprv+g3l7l1PWwEcz7V1BW9LNPp
eIJhxy9/DNUsF1+funzBOc/UsPFyNyJEo0p0Xw==
Fingerprint: a3:18:18:c4:14:f9:3e:79:2c:9c:fa:90:9a:d6:d2:fc
The change of host key is annoying because it makes "svn update" fail with an unhelpful message (some
mumble about ssh -q). To fix this fault, run "ssh svn@savannah.psi.ch", then fixup the ssh host key as
usual.
K.O. |
883
|
06 May 2013 |
Konstantin Olchanski | Info | TRIUMF MIDAS page moved to DAQWiki | The MIDAS web page at TRIUMF (http://midas.triumf.ca) moved from the daq-plone site to the DAQWiki
(MediaWiki) site. Links were updated, checked and corrected:
https://www.triumf.info/wiki/DAQwiki/index.php/MIDAS
Included is the link to our MIDAS installation instructions. These are more complete compared to the
instructions in the MIDAS documentation:
https://www.triumf.info/wiki/DAQwiki/index.php/Setup_MIDAS_experiment
K.O. |
884
|
07 May 2013 |
Konstantin Olchanski | Info | Updated: javascript custom page examples | I updated the MIDAS javascript examples in examples/javascript1. All existing mhttpd.js functions are
now exampled. (yes).
Here is the full list of functions, with notes:
ODBSet(path, value, pwdname);
ODBGet(path, format, defval, len, type);
ODBMGet(paths, callback, formats); --- doc incomplete - no example of callback() use
ODBGetRecord(path);
ODBExtractRecord(record, key);
new ODBKey(path); --- doc incomplete, wrong - one has to use "new ODBKey" - last_used was added.
ODBCopy(path, format); -- no doc
ODBRpc_rev0(name, rpc, args); --- doc refer to example
ODBRpc_rev1(name, rpc, max_reply_length, args); --- same
ODBGetMsg(n);
ODBGenerateMsg(m);
ODBGetAlarms(); --- no doc
ODBEdit(path); --- undoc - forces page reload
As annotated, the main documentation is partially incomplete and partially wrong (i.e. ODBKey() has to be
invoked as "new ODBKey()"). I hope this will be corrected soon. In the mean time, I recommend that
everybody uses this example as best documentation available.
http://ladd00.triumf.ca/~daqweb/doc/midas/html/RC_mhttpd_custom_js_lib.html
svn rev 5360
K.O. |
885
|
10 May 2013 |
Konstantin Olchanski | Bug Fix | Fixed: crash if alarm "write elog message" is enabled | If the MIDAS Alarm property "write elog message" is enabled, an uninitialized variable "tag" is passed to
el_submit() and depending on your luck, cause a crash. "tag" is supposed to be and is now a NUL-
terminated string. The only other use of el_submit() is in mhttpd.cxx and mserver.c, where it is called
correctly.
alarm.c svn rev 5361
K.O. |
886
|
10 May 2013 |
Konstantin Olchanski | Info | mhttpd JSON support | > odbedit can now save ODB in JSON-formatted files.
> Added functions:
> INT EXPRT db_save_json(HNDLE hDB, HNDLE hKey, const char *file_name);
> INT EXPRT db_copy_json(HNDLE hDB, HNDLE hKey, char **buffer, int *buffer_size, int *buffer_end, int save_keys, int follow_links);
>
Added JSON encoding format to Javascript ODBCopy() ("jcopy"). Use format="json", Javascript example updated with an example example.
Also updated db_copy_json():
- always return NUL-terminated string
- "save_keys" values: 0 - do not save any KEY data, 1 - save all KEY data, 2 - save only KEY.last_written
odb.c, mhttpd.cxx, example.html
svn rev 5362
K.O. |
887
|
10 May 2013 |
Konstantin Olchanski | Info | Updated: javascript custom page examples | > ODBCopy(path, format); -- no doc
Updated example of ODBCopy:
format="" returns data in traditional ODB save format
format="xml" returns data in XML encoding
format="json" returns data in JSON encoding.
K.O. |
888
|
17 May 2013 |
Konstantin Olchanski | Info | mhttpd JSON-P support | >
> Added JSON encoding format to Javascript ODBCopy(path,format) ("jcopy"). Use format="json", Javascript example updated with an example example.
>
More ODBCopy() expansion: format="json-p" returns data suitable for JSON-P ("script tag") messaging.
Also implemented multiple-paths for "jcopy" (similar to "jget"/ODBMGet()). An example ODBMCopy(paths,callback,format) is present in example.html (will move to mhttpd.js).
Added JSON encoding options:
- format="json-nokeys" will omit all KEY information except for "last_written"
- "json-nokeys-nolastwritten" will also omit "last_written"
- "json-nofollowlinks" will return ODB symlink KEYs instead of following them (ODBGet/ODBMGet always follows symlinks)
- "json-p" adds JSON-P encapsulation
All these JSON format options can be used at the same time, i.e. format="json-p-nofollowlinks"
To see how it all works, please look at examples/javascript1/example.html.
The new code seems to be functional enough, but it is still work in progress and there are a few problems:
- ODBMCopy() using the "xml" format returns gibberish (the MIDAS XML encoder has to be told to omit the <?xml> header)
- example.html does not actually parse any of the XML data, so we do not know if XML encoding is okey
- JSON encoding has an extra layer of objects (variables.Variables.foo instead of variables.foo)
- ODBRpc() with JSON/JSON-P encoding not done yet.
mhttpd.cxx, example.html
svn rev 5364
K.O. |
889
|
31 May 2013 |
Konstantin Olchanski | Info | mhttpd JSON-P support | > To see how it all works, please look at examples/javascript1/example.html.
>
> - JSON encoding has an extra layer of objects (variables.Variables.foo instead of variables.foo)
>
This is now fixed. See updated example.html. Current encoding looks like this:
{
"System" : {
"Clients" : {
"24885" : {
"Name/key" : { "type" : 12, "item_size" : 32, "last_written" : 1370024816 },
"Name" : "ODBEdit",
"Host/key" : { "type" : 12, "item_size" : 256, "last_written" : 1370024816 },
"Host" : "ladd03.triumf.ca",
"Hardware type/key" : { "type" : 7, "last_written" : 1370024816 },
"Hardware type" : 44,
"Server Port/key" : { "type" : 7, "last_written" : 1370024816 },
"Server Port" : 52539
}
},
"Tmp" : {
...
odb.c, example.html
svn rev 5368
K.O. |
890
|
10 Jun 2013 |
Konstantin Olchanski | Forum | ladd00.triumf.ca https ssl certificate update, elogd update, relocation. | The HTTPS SSL certificate on ladd00.triumf.ca has been updated. Same as the old
certificate, the new one is self-signed and your web browser may complain about
that and ask you to "save a security exception".
When you save the new certificate, you can verify that you are connected to the
real ladd00.triumf.ca by comparing the "SHA1 fingerprint" reported by your web
browser to the one given below (as reported by "svn update"):
Certificate information:
- Hostname: ladd00.triumf.ca
- Valid: from Mon, 10 Jun 2013 18:43:19 GMT until Tue, 10 Jun 2014 18:43:19 GMT
- Issuer: DAQ, TRIUMF, Vancouver, BC, CA
- Fingerprint: 3f:15:d0:1d:68:ae:f0:73:10:78:84:66:f3:af:c7:67:5c:70:00:62
At the same time, elogd was updated to latest version from Stefan (elog-2.9.2-1.i386).
Also as part of local computer updates, elogd is temporarily running on ladd03.triumf.ca. This should be
transparent to all users but please let me know if there are strange artefacts, etc. (ladd03 will become the
new ladd00 "soon").
K.O. |
891
|
22 Jul 2013 |
Konstantin Olchanski | Info | MidasWiki at TRIUMF | We are happy to announce the creation of the MidasWiki at TRIUMF (https://midas.triumf.ca) as the
new location of MIDAS documentation, user instructions, examples, etc.
https://midas.triumf.ca
K.O. |
892
|
22 Jul 2013 |
Konstantin Olchanski | Info | MIDAS source code converted from SVN to GIT | The MIDAS source code repository was converted from SVN to GIT, hosted as bitbucket:
https://bitbucket.org/tmidas.
A clonable copy of the repository is located at TRIUMF: git clone
http://daq.triumf.ca/~daqweb/git/midas.git (and mxml.git).
The documentation is being slowly updated with GIT instructions (git clone) instead of SVN (svn
checkout).
The MIDAS code history goes all the way to CVS/SVN rev 1 dated Thu Oct 8 13:46:02 1998.
K.O. |
894
|
24 Jul 2013 |
Konstantin Olchanski | Info | MidasWiki at TRIUMF | > We are happy to announce the creation of the MidasWiki at TRIUMF
> https://midas.triumf.ca
We are running MediaWiki in the world-readable, authenticated-user-writable mode.
New user registration is done by the "confirm new user" extension (https://www.mediawiki.org/wiki/Extension:ConfirmAccount):
- go to https://midas.triumf.ca/MidasWiki/index.php/Special:RequestAccount
- fill the form, submit request - request goes to wiki administrator for confirmation
- wait for email about email address confirmation, follow instructions to confirm your email address
- wait for email about account confirmation
- try to login to the wiki.
K.O. |
895
|
26 Jul 2013 |
Konstantin Olchanski | Bug Report | abort on buffer overflow in odb.c::merge_records() | The odb.c function merge_records() has a fixed size array of 10000 bytes to handle the data and it
aborts with an assert() if passed data bigger than that. It is called from db_create_record() which
already allocates a data buffer of correct size for it's operations.
K.O. |
896
|
26 Jul 2013 |
Konstantin Olchanski | Bug Report | odbedit fixed size buffer overrun | odbedit uses a fixed size buffer for ODB data. If an array in ODB is bugger than this size,
db_get_data() correctly returns DB_TRUNCATED and there is no memory overwrite, but the following
code for printing the data does not know about this truncation and proceeds printing memory
values contained after the end of the fixed size data buffer - in the current case, this memory
somehow had the contents of the shell history - this very confused the MIDAS users as they though
that the funny printout was actually in ODB. This is in the print_key() function. If db_get_data()
returns DB_TRUNCATED, it should allocate a buffer of bigger size. This (and the previous) bug found
by the TIGRESS experiment. K.O. |
897
|
02 Aug 2013 |
Konstantin Olchanski | Bug Fix | multithreaded run transitions work! | As of commit
https://bitbucket.org/tmidas/midas/commits/dfa5fb1a93cae11a2960d441044c7fd277e1f0ec
(we are now liberated from the tyranny of SVN IDs),
multithreaded run transitions seem to work reliably and are now the default in mhttpd.
In odbedit and mtransition, the default is the old sequential transitions. "-m" and "-a" flags activate
the new multithread run transitions. mhttpd now uses the equivalent of "mtransition -a"
(mutithreaded asynchronous).
This is one of the new features implemented by Stefan while at TRIUMF.
K.O.
(We hope to write up all the recent changes soon). |
898
|
21 Aug 2013 |
Konstantin Olchanski | Info | Documentation for ODBGet() & co, Javascript and AJAX functions. | The bulk of the MIDAS AJAX and Javascript functions is now documented on the MIDAS Wiki:
https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js
https://midas.triumf.ca/MidasWiki/index.php/AJAX
Enjoy,
K.O. |
899
|
22 Aug 2013 |
Konstantin Olchanski | Info | Documentation for ODBGet() & co, Javascript and AJAX functions. | > The bulk of the MIDAS AJAX and Javascript functions is now documented on the MIDAS Wiki:
>
> https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js
> https://midas.triumf.ca/MidasWiki/index.php/AJAX
>
The documentation was updated again.
All functions and AJAX methods except jset, jget, jrpc (ODBGet, ODBSet, ODBRpc) and inline edit are now fully
documented. AJAX methods jset/jget and their javascript wrappers ODBSet/ODBGet/ODBMGet/ODBGetRecord() are
partially documented. Inline edit will have to be documented by Stefan.
When using these functions please read the "BUG" sections carefully.
When using the Javascript functions (ODBGet, ODBSet, ODBMCopy, etc) please pay special attention to the rules for URI-
encoding function arguments - some functions require that some arguments be pre-encoded by
"encodeURIComponent()", some functions require some arguments to NOT be pre-encoded. The examples in
examples/javascript1/example.html are mostly correct.
Special confusion is created by special handling in mhttpd of URI-encoding of parameters named "format".
Special confusion is created by ODBSet(path, value), where "path" should be pre-encoded, while "value" is now encoded
internally, which is a recent change introduced with the inline edit function. Older versions of mhttpd.js still require that
"value" be URI-encoded.
Going forward, I hope to resolve most of the confusion by providing a cleaner interface for reading ODB
- ODBMCopy() already looks good with full async JSON/JSONP support (already implemented)
- ODBMKey() to read just the keys, with async JSON/JSONP support (to be added)
- ODBMCreate() to create ODB keys (RPC for db_create()) (to be added)
- ODBMSet() to write into ODB. (to be added)
K.O. |
900
|
26 Aug 2013 |
Konstantin Olchanski | Bug Fix | Enable cross-site requests in mhttpd | Javascript "AJAX" functions (and their MIDAS wrappers - ODBGet/ODBSet) are subject to something called
"same origin policy" intended to prevent something called "cross-site scripting attacks", i.e. see
http://en.wikipedia.org/wiki/Same-origin_policy
In practice it means that if you load the MIDAS custom web page from test.foo.com and try to access
mhttpd at midas.foo.com, ODBSet/ODBGet will not work.
I always thought that this meant that the requests are blocked by the browser and are a form of
protection of the web server - only scripts loaded from mhttpd can do AJAX (ODBGet/ODBSet) to mhttpd.
It turns out that I was wrong. This is what actually happens: the "cross-site" requests are still sent to the
server (mhttpd), the response it received, parsed and discarded if "same origin" conditions are not met.
This means that the "same origin" policy does not protect mhttpd at all - any script from any page
anywhere can issue AJAX requests into any mhttpd, these requests will be successfully sent, received
and processed by mhttpd, including requests for writing into ODB ("jset" command using the HTTP GET
method).
So for the case of MIDAS, "same origin" does not prevent malicious (or buggy) scripts from writing into the
wrong mhttpd of the wrong experiment.
All it does is prevent desired and intentional access to mhttpd (ODBGet) from scripts that happen to have
been loaded outside of mhttpd (i.e. from a developer own test page).
Then it turns out that there is an "official" way to disable this unwanted protection policy, called CORS, see
http://www.w3.org/TR/cors/
I have now implemented this in mhttpd and added an mhttpd.js function ODBSetURL() to explicitly set the
URL of mhttpd that we want to talk to.
This work is on the feature/ajax branch, to be merged soon. For the impatient, here is what you need to
do in mhttpd:
diff --git a/src/mhttpd.cxx b/src/mhttpd.cxx
index 1d9d1cc..0460cec 100755
--- a/src/mhttpd.cxx
+++ b/src/mhttpd.cxx
@@ -1070,6 +1070,7 @@ void show_text_header()
{
rsprintf("HTTP/1.0 200 Document follows\r\n");
rsprintf("Server: MIDAS HTTP %d\r\n", mhttpd_revision());
+ rsprintf("Access-Control-Allow-Origin: *\r\n");
rsprintf("Pragma: no-cache\r\n");
rsprintf("Expires: Fri, 01 Jan 1983 00:00:00 GMT\r\n");
rsprintf("Content-Type: text/plain; charset=iso-8859-1\r\n\r\n");
K.O. |
|