ID |
Date |
Author |
Topic |
Subject |
889
|
31 May 2013 |
Konstantin Olchanski | Info | mhttpd JSON-P support | > To see how it all works, please look at examples/javascript1/example.html.
>
> - JSON encoding has an extra layer of objects (variables.Variables.foo instead of variables.foo)
>
This is now fixed. See updated example.html. Current encoding looks like this:
{
"System" : {
"Clients" : {
"24885" : {
"Name/key" : { "type" : 12, "item_size" : 32, "last_written" : 1370024816 },
"Name" : "ODBEdit",
"Host/key" : { "type" : 12, "item_size" : 256, "last_written" : 1370024816 },
"Host" : "ladd03.triumf.ca",
"Hardware type/key" : { "type" : 7, "last_written" : 1370024816 },
"Hardware type" : 44,
"Server Port/key" : { "type" : 7, "last_written" : 1370024816 },
"Server Port" : 52539
}
},
"Tmp" : {
...
odb.c, example.html
svn rev 5368
K.O. |
888
|
17 May 2013 |
Konstantin Olchanski | Info | mhttpd JSON-P support | >
> Added JSON encoding format to Javascript ODBCopy(path,format) ("jcopy"). Use format="json", Javascript example updated with an example example.
>
More ODBCopy() expansion: format="json-p" returns data suitable for JSON-P ("script tag") messaging.
Also implemented multiple-paths for "jcopy" (similar to "jget"/ODBMGet()). An example ODBMCopy(paths,callback,format) is present in example.html (will move to mhttpd.js).
Added JSON encoding options:
- format="json-nokeys" will omit all KEY information except for "last_written"
- "json-nokeys-nolastwritten" will also omit "last_written"
- "json-nofollowlinks" will return ODB symlink KEYs instead of following them (ODBGet/ODBMGet always follows symlinks)
- "json-p" adds JSON-P encapsulation
All these JSON format options can be used at the same time, i.e. format="json-p-nofollowlinks"
To see how it all works, please look at examples/javascript1/example.html.
The new code seems to be functional enough, but it is still work in progress and there are a few problems:
- ODBMCopy() using the "xml" format returns gibberish (the MIDAS XML encoder has to be told to omit the <?xml> header)
- example.html does not actually parse any of the XML data, so we do not know if XML encoding is okey
- JSON encoding has an extra layer of objects (variables.Variables.foo instead of variables.foo)
- ODBRpc() with JSON/JSON-P encoding not done yet.
mhttpd.cxx, example.html
svn rev 5364
K.O. |
887
|
10 May 2013 |
Konstantin Olchanski | Info | Updated: javascript custom page examples | > ODBCopy(path, format); -- no doc
Updated example of ODBCopy:
format="" returns data in traditional ODB save format
format="xml" returns data in XML encoding
format="json" returns data in JSON encoding.
K.O. |
886
|
10 May 2013 |
Konstantin Olchanski | Info | mhttpd JSON support | > odbedit can now save ODB in JSON-formatted files.
> Added functions:
> INT EXPRT db_save_json(HNDLE hDB, HNDLE hKey, const char *file_name);
> INT EXPRT db_copy_json(HNDLE hDB, HNDLE hKey, char **buffer, int *buffer_size, int *buffer_end, int save_keys, int follow_links);
>
Added JSON encoding format to Javascript ODBCopy() ("jcopy"). Use format="json", Javascript example updated with an example example.
Also updated db_copy_json():
- always return NUL-terminated string
- "save_keys" values: 0 - do not save any KEY data, 1 - save all KEY data, 2 - save only KEY.last_written
odb.c, mhttpd.cxx, example.html
svn rev 5362
K.O. |
885
|
10 May 2013 |
Konstantin Olchanski | Bug Fix | Fixed: crash if alarm "write elog message" is enabled | If the MIDAS Alarm property "write elog message" is enabled, an uninitialized variable "tag" is passed to
el_submit() and depending on your luck, cause a crash. "tag" is supposed to be and is now a NUL-
terminated string. The only other use of el_submit() is in mhttpd.cxx and mserver.c, where it is called
correctly.
alarm.c svn rev 5361
K.O. |
884
|
07 May 2013 |
Konstantin Olchanski | Info | Updated: javascript custom page examples | I updated the MIDAS javascript examples in examples/javascript1. All existing mhttpd.js functions are
now exampled. (yes).
Here is the full list of functions, with notes:
ODBSet(path, value, pwdname);
ODBGet(path, format, defval, len, type);
ODBMGet(paths, callback, formats); --- doc incomplete - no example of callback() use
ODBGetRecord(path);
ODBExtractRecord(record, key);
new ODBKey(path); --- doc incomplete, wrong - one has to use "new ODBKey" - last_used was added.
ODBCopy(path, format); -- no doc
ODBRpc_rev0(name, rpc, args); --- doc refer to example
ODBRpc_rev1(name, rpc, max_reply_length, args); --- same
ODBGetMsg(n);
ODBGenerateMsg(m);
ODBGetAlarms(); --- no doc
ODBEdit(path); --- undoc - forces page reload
As annotated, the main documentation is partially incomplete and partially wrong (i.e. ODBKey() has to be
invoked as "new ODBKey()"). I hope this will be corrected soon. In the mean time, I recommend that
everybody uses this example as best documentation available.
http://ladd00.triumf.ca/~daqweb/doc/midas/html/RC_mhttpd_custom_js_lib.html
svn rev 5360
K.O. |
883
|
06 May 2013 |
Konstantin Olchanski | Info | TRIUMF MIDAS page moved to DAQWiki | The MIDAS web page at TRIUMF (http://midas.triumf.ca) moved from the daq-plone site to the DAQWiki
(MediaWiki) site. Links were updated, checked and corrected:
https://www.triumf.info/wiki/DAQwiki/index.php/MIDAS
Included is the link to our MIDAS installation instructions. These are more complete compared to the
instructions in the MIDAS documentation:
https://www.triumf.info/wiki/DAQwiki/index.php/Setup_MIDAS_experiment
K.O. |
882
|
06 May 2013 |
Konstantin Olchanski | Info | Recent-ish SVN changes at PSI | A little while ago, PSI made some changes to the SVN hosting. The main SVN URL seems to remain the
same, but SVN viewer moved to a new URL (it seems a bit faster compared to the old viewer):
https://savannah.psi.ch/viewvc/meg_midas/trunk/
Also the SSH host key has changed to:
savannah.psi.ch,192.33.120.96 ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEAwVWEoaOmF9uggkUEV2/HhZo2ncH0zUfd0ExzzgW1m0HZQ5df1OYIb
pyBH6WD7ySU7fWkihbt2+SpyClMkWEJMvb5W82SrXtmzd9PFb3G7ouL++64geVKHdIKAVoqm8yGaIKIS0684
dyNO79ZacbOYC9l9YehuMHPHDUPPdNCFW2Gr5mkf/uReMIoYz81XmgAIHXPSgErv2Nv/BAA1PCWt6THMMX
E2O2jGTzJCXuZsJ2RoyVVR4Q0Cow1ekloXn/rdGkbUPMt/m3kNuVFhSzYGdprv+g3l7l1PWwEcz7V1BW9LNPp
eIJhxy9/DNUsF1+funzBOc/UsPFyNyJEo0p0Xw==
Fingerprint: a3:18:18:c4:14:f9:3e:79:2c:9c:fa:90:9a:d6:d2:fc
The change of host key is annoying because it makes "svn update" fail with an unhelpful message (some
mumble about ssh -q). To fix this fault, run "ssh svn@savannah.psi.ch", then fixup the ssh host key as
usual.
K.O. |
881
|
30 Apr 2013 |
Konstantin Olchanski | Info | ROOT switched to GIT | Latest news - the ROOT project switched from SVN to GIT.
Announcement:
http://root.cern.ch/drupal/content/root-has-moved-git
Fons's presentation with details on the conversion process, repository size and performance
improvements:
https://indico.cern.ch/getFile.py/access?contribId=0&resId=0&materialId=slides&confId=246803
"no switch yard" work flow:
http://root.cern.ch/drupal/content/suggested-work-flow-distributed-projects-nosy
GIT cheat sheet:
http://root.cern.ch/drupal/content/git-tips-and-tricks
K.O. |
880
|
12 Apr 2013 |
Stefan Ritt | Forum | Persistent ipcrm error | > [odb.c:6038:db_paste,ERROR] found string exceeding MAX_STRING_LENGTH
Ok, so here is what probably happened. Some user program wrote a long string into the ODB and somehow corrupted it. This corruption persists as long as you work with
binary data. Indeed "rebuilding" the ODB helps in that case. What we do actually is at the beginning of every run, the ODB contents is dumped into the data file via
/Logger/Channels/0/Setting/ODB dump
in case we get ODB corruption, we clear all *.shm files as well as the shared memory segments, create a fresh ODB, extract the ODB from the last successful run via
odbhist -e runxxx.mid
and load it via odbedit. I put some additional code in most midas functions to prevent this corruption (and thus your saw the above error "found string exceeding
MAX_STRING_LENGTH"), but since the ODB is physically in the address space of each midas program, they can theoretically bypass the midas functions and write accidentally
into the ODB with an uninitialized pointer or so.
Best regards,
Stefan |
879
|
12 Apr 2013 |
Thorsten Lux | Forum | Persistent ipcrm error | Hi,
it seems that I solved the problem in a quite brutal way.
I opened the database with odbedit and saved first the whole database as a ASCII
file and then I did the same for each section separately. Then I closed odbedit.
Afterwards I deleted all .*.SHM files including .ODB.SHM and rebooted the system.
After the restart I opened odbedit and started mhttpd. With this blank system
the problem had disappeared. Afterwards I loaded section by section from the
previous created ASCII files. After each section I tested if I can start and
stop runs and it worked without problems. At the end I also loaded the ASCII
file which contained the whole database. In this case I got the following error
message:
[odb.c:6038:db_paste,ERROR] found string exceeding MAX_STRING_LENGTH
However, after a reboot everything worked fine. I can start and stop runs, with
and without frontend, without any error message. Only the mlogger resisted to
work again.
But also this problem we solved. It seems it was related to a missing library
path. It is strange since while in the mhttpd web page the command does not work
and is not giving any error message, copying the same command to a terminal and
to start it manually does the job. We solved it by putting the start of mlogger
in a simple shell script and to execute it then from the mhttpd web page.
Probably not an elegant solution but it does the job.
Well, with this I can enjoy my weekend to start over with data taking next week!
Thanks a lot!
Thorsten |
878
|
12 Apr 2013 |
Thorsten Lux | Forum | Persistent ipcrm error | >
> > Hi Stefan,
> >
> > under /Programs/Logger/Start command I have
> > /home/next/MIDAS/midas/linux/bin/mlogger -D . This command does not work if I
> > press the "Start Logger" button on the mhttpd webpage but when I copy and paste
> > this command to a terminal window, it does the job.
> >
> > Well, thanks to you both for the fast response. I wrote Konstantin an email with
> > the results of the tests he suggested me to do.
> >
> > Ciao
>
> Let me guess: mhttpd is started under root (to be able to connect to port 80), and for root the mlogger program
> is not in the path. Try to put into the odb the full path:
>
> /Programs/Logger Start command = /usr/local/bin/mlogger -D
Yes, mhttpd is started as sudo, but I have the full path in the start command. And every user has the right to
execute mlogger. But okay, I will concentrate first to get the rest working again and then I will fight this problem.
Thanks! |
877
|
12 Apr 2013 |
Stefan Ritt | Forum | Persistent ipcrm error |
> Hi Stefan,
>
> under /Programs/Logger/Start command I have
> /home/next/MIDAS/midas/linux/bin/mlogger -D . This command does not work if I
> press the "Start Logger" button on the mhttpd webpage but when I copy and paste
> this command to a terminal window, it does the job.
>
> Well, thanks to you both for the fast response. I wrote Konstantin an email with
> the results of the tests he suggested me to do.
>
> Ciao
Let me guess: mhttpd is started under root (to be able to connect to port 80), and for root the mlogger program
is not in the path. Try to put into the odb the full path:
/Programs/Logger Start command = /usr/local/bin/mlogger -D |
876
|
12 Apr 2013 |
Thorsten Lux | Forum | Persistent ipcrm error | [quote="Stefan Ritt"][quote="Thorsten Lux"]In addition now I cannot start
anymore the mlogger from the web interface but only manually. However, I can
stop it from the web interface.[/quote]
At least that one can be fixed easily. Each program has a certain command with
which one can start it. This has to be put into the ODB under
/Programs/<program>. In your case you probably need
/Programs/Logger/Start command = mlogger -D
to start the logger from the Web page. To debug your run stop problems, I would
recommend to start all programs in a terminal window and look which one crashes
on the run end.
/Stefan[/quote]
Hi Stefan,
under /Programs/Logger/Start command I have
/home/next/MIDAS/midas/linux/bin/mlogger -D . This command does not work if I
press the "Start Logger" button on the mhttpd webpage but when I copy and paste
this command to a terminal window, it does the job.
Well, thanks to you both for the fast response. I wrote Konstantin an email with
the results of the tests he suggested me to do.
Ciao |
875
|
11 Apr 2013 |
Stefan Ritt | Forum | Persistent ipcrm error |
Thorsten Lux wrote: | In addition now I cannot start anymore the mlogger from the web interface but only manually. However, I can stop it from the web interface. |
At least that one can be fixed easily. Each program has a certain command with which one can start it. This has to be put into the ODB under /Programs/<program>. In your case you probably need
/Programs/Logger/Start command = mlogger -D
to start the logger from the Web page. To debug your run stop problems, I would recommend to start all programs in a terminal window and look which one crashes on the run end.
/Stefan |
874
|
11 Apr 2013 |
Konstantin Olchanski | Forum | Persistent ipcrm error | > [system.c:308:ss_shm_open,ERROR] Shared memory segment with key 0x4d008002 already exists,
please remove it manually: ipcrm -M 0x4d008002
> [midas.c:1950:cm_connect_experiment1,ERROR] cannot open database
> Unexpected error #304
For the record, the SYSV shared memory with it's keys and segments has always been brittle and hard to
debug with problems such as you describe.
Also SYSV shared memory suffers from key aliasing - shared memory segments created with different
names all map into the same key, collide and nothing works. You may not see this if all the files are
located on a local disk, but if the .SHM files are located on an NFS disk, it can happen (and did happen in
T2K).
For this reason, since around August 2010, MIDAS also implements the POSIX shared memory and for new
MIDAS installations, POSIX shared memory is the default. (On MacOS, POSIX shared memory was always
the default because MacOS has very small maximum SYSV shared memory size).
The type of shared memory is set by the contents of .SHM_TYPE.TXT and it is possible to switch between
SYSV and POSIX shared memory at will. (Ask me).
MIDAS still uses SYSV semaphores because they have a built-in feature to automatically unlock the
semaphore if the program that locked it dies for any reason. POSIX semaphores do not have this built-in
feature and we would have to implement some kind of detection and recovery for the case when a
semaphore is locked by a program that died (and will never unlock it back).
K.O.
P.S. I will address the rest of Prof. Thorsten's question in a private email.
P.P.S. Please post elog messages in the "plain" format. NOT HTML or ELCODE. |
873
|
11 Apr 2013 |
Thorsten Lux | Forum | Persistent ipcrm error | Hello,
I have a problem with our DAQ which is based on Midas. Until now, for about 3 years, it worked quite well but since I tried to restart data taking after a break of 2 months, I get always the following error message:
[system.c:308:ss_shm_open,ERROR] Shared memory segment with key 0x4d008002 already exists, please remove it manually: ipcrm -M 0x4d008002
[midas.c:1950:cm_connect_experiment1,ERROR] cannot open database
Unexpected error #304
Then I tried the following to fix the problem:
-) I first checked with ipcs the shared memory segments:
0x4d008002 3244040 next 666 1077248 1
0x4d00006e 3276809 next 666 116444 1
Sometimes there is an additional line which I also delete.
-) I deleted with ipcrm -M 0x4d008002 / 0x4d00006e the shared memory segments
-) I removed the .SYS*.SHM files:
-rw-r--r-- 1 next users 0 Mar 16 2010 MIDAS/online/.ALARM.SHM
-rw-r--r-- 1 next users 0 Mar 16 2010 MIDAS/online/.ELOG.SHM
-rw-r--r-- 1 next users 0 Mar 16 2010 MIDAS/online/.HISTORY.SHM
-rw-r--r-- 1 next users 0 Mar 16 2010 MIDAS/online/.MSG.SHM
-rw-r--r-- 1 next users 1089536 Apr 11 15:46 MIDAS/online/.ODB.SHM
-rw-r--r-- 1 next users 116444 Apr 11 15:43 MIDAS/online/.SYSMSG.SHM
-rw-r--r-- 1 next users 16793660 Apr 11 15:43 MIDAS/online/.SYSTEM.SHM
-) I reboot the PC
-) I start the midas daemon using a shell script with the following lines:
cd /home/next/CAEN/A2818Drv/
sudo sh a2818_load
mhttpd -p 8080 -D
-) Normally I can start then a run but when I try to stop it I get again the error message from above.
In addition I get from time to time the following error messages:
[mhttpd,INFO] Client 'unknown' on buffer 'SYSMSG' removed by cm_watchdog because client pid 3287 does not exist
[NEXT DAQ,INFO] Client 'unknown' on buffer 'SYSMSG' removed by bm_wait_for_free_space because client pid 3280 does not exist
[mtransition,INFO] Client 'mhttpd' (PID 3229) on buffer 'ODB' removed by cm_watchdog (idle 47.4s,TO 10s)
Since all this did not help and although there was no update of the operation system, I decided the recompile the whole midas framework on this machine.
It compiled and I installed but the error persisted. In addition now I cannot start anymore the mlogger from the web interface but only manually. However, I can stop it from the web interface.
Do you have an idea what could be the problem? I start to be a bit desperate. Also because I am user of the DAQ system but the person who developed the system in the past, left already some years ago.
I am using a midas version from the 15.03.2010 (midas20100315.tar.gz) as it seems. In principle there is only one frontend device, a CAEN V1740 digitizer, connected to Midas.
Thanks! |
872
|
05 Apr 2013 |
Konstantin Olchanski | Info | ODB JSON support | odbedit can now save ODB in JSON-formatted files. (JSON is a popular data encoding standard associated
with Javascript). The intent is to eventually use the ODB JSON encoder in mhttpd to simplify passing of
ODB data to custom web pages. In mhttpd I also intend to support the JSON-P variation of JSON (via the
jQuery "callback=?" notation).
JSON encoding implementation follows specifications at:
http://json.org/
http://www.json-p.org/
http://api.jquery.com/jQuery.getJSON/ (seek to JSONP)
The result passes validation by:
http://jsonlint.com/
Added functions:
INT EXPRT db_save_json(HNDLE hDB, HNDLE hKey, const char *file_name);
INT EXPRT db_copy_json(HNDLE hDB, HNDLE hKey, char **buffer, int *buffer_size, int *buffer_end, int
save_keys, int follow_links);
For example of using this code, see odbedit.c and odb.c::db_save_json().
Example json file:
Notes:
1) hex numbers are quoted "0x1234" - JSON does not permit "hex numbers", but Javascript will
automatically convert strings containing hex numbers into proper integers.
2) "double" is encoded with full 15 digit precision, "float" with full 7 digit precision. If floating point values
are actually integers, they are encoded as integers (10.0 -> "10" if (value == (int)value)).
3) in this example I deleted all the "name/key" entries except for "stringvalue" and "sbyte2". I use the
"/key" notation for ODB KEY data because the "/" character cannot appear inside valid ODB entry names.
Normally, depending on the setting of "save_keys" argument, KEY data is present or absent for all entries.
ladd03:midas$ odbedit
[local:testexpt:S]/>cd /test
[local:testexpt:S]/test>save test.js
[local:testexpt:S]/test>exit
ladd03:midas$ more test.js
# MIDAS ODB JSON
# FILE test.js
# PATH /test
{
"test" : {
"intarr" : [ 15, 0, 0, 3, 0, 0, 0, 0, 0, 9 ],
"dblvalue" : 2.2199999999999999e+01,
"fltvalue" : 1.1100000e+01,
"dwordvalue" : "0x0000007d",
"wordvalue" : "0x0141",
"boolvalue" : true,
"stringvalue" : [ "aaa123bbb", "", "", "", "", "", "", "", "", "" ],
"stringvalue/key" : {
"type" : 12,
"num_values" : 10,
"item_size" : 1024,
"last_written" : 1288592982
},
"byte1" : 10,
"byte2" : 241,
"char1" : "1",
"char2" : "-",
"sbyte1" : 10,
"sbyte2" : -15,
"sbyte2/key" : {
"type" : 2,
"last_written" : 1365101364
}
}
}
svn rev 5356
K.O. |
871
|
03 Apr 2013 |
Randolf Pohl | Info | Review of github and bitbucket | > > * "git bisect" for finding which commit introduced a (reproducible) bug.
>
> I did not know this command, so I read about it. This IS WONDERFUL! I had once (actually with MSCB) the case that a bug was introduced i the last 100
> revisions, but I did not know in which. So I checked out -1, -2, -3 revisions, then thought a bit, then tried -99, -98, then had the bright idea to try -50, then
> slowly converged. Later I realised that I should have done a binary search, like -50, if ok try -25, if bad try -37, and so on to iteratively find the offending
> commit. Finding that there is a command it git which does this automatically is great news.
even more so considering the nonlinear history (due to branching) in a regular git repo. |
870
|
03 Apr 2013 |
Stefan Ritt | Info | Review of github and bitbucket | > * "git bisect" for finding which commit introduced a (reproducible) bug.
I did not know this command, so I read about it. This IS WONDERFUL! I had once (actually with MSCB) the case that a bug was introduced i the last 100
revisions, but I did not know in which. So I checked out -1, -2, -3 revisions, then thought a bit, then tried -99, -98, then had the bright idea to try -50, then
slowly converged. Later I realised that I should have done a binary search, like -50, if ok try -25, if bad try -37, and so on to iteratively find the offending
commit. Finding that there is a command it git which does this automatically is great news.
Stefan |
|