> The history of at least bm.cxx seems to be screwed up. Bitbucket shows this:
>   https://bitbucket.org/tmidas/midas/annotate/fa649e6e9bb7447b4da01e964c6dc14b01e64052/src/bm.cxx?at=test%2Fsplit-midas-cxx
> and CLion shows what I attached here. Most lines are tagged "Today Olchanski" which cannot be.

yup, I see the same nonsense for bm.cxx from gitea.

back to the drawing board, will try branch, rename, merge as described here:
https://devblogs.microsoft.com/oldnewthing/20190916-00/?p=102892/

the naming of split files looks ok or we prefer more descriptive names "event_buffer.cxx",
"transition.cxx", "message.cxx", etc?

K.O.

MIDAS Migration

We have transitioned from hosting MIDAS on bitbucket to TRIUMF's self-hosted gitlab instance (https://gitlab.triumf.ca/midas). Apologies for the early transition.

Do I need an account?

To push to the new server you will need an account on that server, and be added as a developer to the group. To create issues (or work items), you will also need an account. Otherwise, you can pull and clone anonymously.

Self-registration works on a whitelisting scheme: either your instiutional hostname or your full address needs to be whitelisted. If your regisration fails, please make a post in this forum or send me an email. 

How to get access to MIDAS?

You can pull to or push from the gitlab server. You can also pull from the bitbucket server - it has been set up as a mirror. Any changes to the gitlab repos will be pushed within 5 mins of the change. Long-term the bitbucket mirrors will be depreciated. 

Push permissions have been rescinded on bitbucket to enforce this.  

How to point your repos to the new server: 

git checkout develop
git pull --recurse-submodule
git remote set-url origin git@gitlab.triumf.ca:midas/midas.git # or use the html url: https://gitlab.triumf.ca/midas/midas.git
git submodule sync

Developer notes: 

Actions taken: 

• Removed write permissions from developer group to all MIDAS-related repos
• Transferred developers into a "Former Developers" group
• Re-synced gitlab and bitbucket repos
• Setup gitlab as push repos
  • This will expire Sep 25, 2027 (max duration)
  • Push mirroring happens after any push to the gitlab repo, with a minimum of 5 mins duration between mirror pushes
• MIDAS README updated

Stefan uncovered a bug in mhttpd URI decoding, the following did not work:
cd midas, cd resources, cp example.html c++.html, open http://midas/example.html (works), open c++.html and it bombs with error about "c  .html" (two pluses replaced by two spaces).

Stefan did a quick fix for this and I now finished the follow up.

here are the rules for encoding URLs for mhttpd: (using this example URL)

https://experiment.triumf.ca/vslice/?cmd=ODB&odb_path=foo

the query parameter value "foo" should be percent-encoded using encodeURIComponent(). this replaces spaces with %20, slashes with %2F and pluses with %2B, etc. a plain bare plus character "+" is permitted here, it will seen as a space by mhttpd (odb_path=A+B becomes "A B").

the query parameter name "odb_path" should also be percent-encoded, but standard mhttpd parameters only use ascii character for parameter names and it is okey to skip encodeURIComponent() and hardcode them. if custom parameter names use special characters or other funny characters, they should be percent-encoded.

the elements of the URL path "vslice" should be percent-encoded if they contain special URI characters like "?" and "&". internally mhttpd runs the URL path through decodeURIComponent(). this expands all percent-encoded characters. (plus and space characters are not changed, URLs like c++.html work).

one exception for percent-encoded URLs is the percent-encoded slash ("%2F" instead  of "/"). for security reasons apache httpd and other web servers do not permit this. for consistency, mhttpd also rejects "%2F" in the URL path with the "404 Not Found" error.

TL&DR:

1) replaced old url-encode and decode functions with encodeURIComponent() and decodeURIComponent(). the mhttpd encoder is more aggressive compared to most javascript encoders (they leave characters like "_" and "(" unencoded). this is safe.

2) rewrote decode_query() to remove a strdup(), remove strtok() and manually parse the query string using bare URI query delimiters "&" and "=". malformed queries (mismatched "=" and "&") are silently truncated. in parameter values, plus is decoded as a space, decodeURIComponent() is consistently applied to parameter names and values.

3) added a trap in handle_http_message() against percent-encoded slash ("%2F" is rejected with "404 Not Found").

Commit 01336247eb4fa5bb6ed1a7187fa2061d6d4e9086
and 590d2b0812eacebd313efbe1c0c38a34f6eaf78f

K.O.

As for issue reporting, please use the gitlab work item tracker: https://gitlab.triumf.ca/groups/midas/-/work_items

You will need an account on the TRIUMF gitlab, but self-registration is enabled. If there are issues registering, please make a post here or send me an email. 

Note that we will not be monitoring issues on the bitbucket jira tracker. 

Thanks, 

Derek

gitlab notifications do not send email on each commit and push (like bitbucket did).
to enable this, add your email address to the gitlab "emails on push integration": 
https://gitlab.triumf.ca/groups/midas/-/settings/integrations/emails_on_push/edit
K.O.

Hi,
 has anyone tried runnning midas frontend on a Linux running 
on a Motorola MVME167 motorola embedded CPU?
I have seen people running Linux on a MV167 
(http://www.sleepie.demon.co.uk/linuxvme/)
so in principle this can be done.

The reason I am asking is that we have a lot of them in house 
and we would like to avoid paying for VxWorks
(I have succesfully run Midas on a mvme167/VxWorks node)

Or maybe one has come up with a much better solution 
[short of dumping mv167 into a sewer :)]

Piotr

I would like to know whether all parameters used by the frontend code have to be in the "Experiment/
Run Parameters" section.  This section can become big and difficult to maintain, because it is one single 
big section of experim.h (EXP_PARAM_DEFINED).  I have parameters the various frontends read at the 
beginning of each run, which set the hardware settings of various devices.  I would like to place these in 
a section all their own, organized by device.  Is this doable? 

Hi Jan,

I usually keep under /Experiment/Run Parameters only those settings which are kind of "global" and thus of
interest to frontend *and* analyzer, like a run mode (data/calibration/cosmic/...). Settings more specific to a
frontend I keep under /Equipment/<name>/Settings where <name> is the equipment name the specific frontend
produces. In your case each frontend will then get its own tree (related to each fragment). Please note that
both discussed trees can contain a whole tree with subdirectories, which lets you organize your data better.

Best regards, Stefan.

   We've had mhttpd aborting regularly since upgrading from midas-1.9.3.  This
happens during elog queries, and is due to an elog file that was incorrectly
modified by hand.  The modification to the file occurred 6 months ago.
   el_retrieve(midas.c:15683) now has several assert statements, one of which
aborts the program on reading the bad entry.

   Why is assert used, instead of an error return from the function (if
necessary), and maybe an error message in the log file?  Assert statements are
often removed, using NDEBUG, for normal use.

Chris

   The problem elog entry had one character removed, so end-of-file came before
the end of the message.  This could probably occur without the file being
altered, if the disk containing the elog fills.

>    We've had mhttpd aborting regularly since upgrading from midas-1.9.3.  This
> happens during elog queries, and is due to an elog file that was incorrectly
> modified by hand.

(sorry for delayed reply, for reasons unknown, I did not get an email notice when this was posted)

Yes, I agree, error handling in midas elog code is insufficient (note missing error checks for
read() and lseek() system calls). Anything but "perfect" elog files would cause funny errors and
malfunctions.

>  The modification to the file occurred 6 months ago.
>    el_retrieve(midas.c:15683) now has several assert statements, one of which
> aborts the program on reading the bad entry.

I added those to fix problems with "broken last NN days" and with infinite looping in the elog code
that we observed in TWIST.

You are welcome to replace the assert() statements with proper error handling. I used to have some code
that could report the filename of the bad elog file. Can we also report the exact file location for broken
files.

Please send me the diff, I will commit it to midas cvs.

>    Why is assert used, instead of an error return from the function (if
> necessary), and maybe an error message in the log file?  Assert statements are
> often removed, using NDEBUG, for normal use.

I use assert() in several ways:

0) I want a core dump each time X happens. (This is the only reasonable action when facing memory/stack
corruption. The problems in the elog code were stack corruption).
1) "I am too lazy to write proper error handling code" so I just crash and burn. This includes the
case where "proper error handling" would be "too invasive".
2) the error is too bad (or too deep) and there is no reasonable way to recover. Print an error message
and dump core (for later analysis). I sometimes use "cm_msg(); abort()". (assert is "printf("error"); abort()")

Please refer to literature for philosophic discussions on uses of assert() (Argh! Stefan will have my
head again!), but I will mention that "abort() early, abort() often" I find very effective. BTW, this technique
is heavily used in the Linux kernel (oops(), bug(), panic()) with some good effect, too.

>    The problem elog entry had one character removed, so end-of-file came before
> the end of the message.  This could probably occur without the file being
> altered, if the disk containing the elog fills.

Yes, I think you are right. In TWIST, we have seen disk-full conditions break both elog and history.

K.O.

What is the api call to determine the index of the frontend when specifying the
-i parameter during execution of the frontend? 

> What is the api call to determine the index of the frontend when specifying the
> -i parameter during execution of the frontend? 

INT get_frontend_index();

- Stefan

When i compile the experiment example of 1.9.5 the problem happened:

adccalib.c: In function `INT adc_calib_init()':
adccalib.c:114: `H1_BOOK' undeclared (first use this function)
adccalib.c:114: (Each undeclared identifier is reported only once for each
   function it appears in.)
make: *** [adccalib.o] Error 1

my ROOT is 4.01 and Zlib is 1.2.2

> When i compile the experiment example of 1.9.5 the problem happened:
> 
> adccalib.c: In function `INT adc_calib_init()':
> adccalib.c:114: `H1_BOOK' undeclared (first use this function)
> adccalib.c:114: (Each undeclared identifier is reported only once for each
>    function it appears in.)
> make: *** [adccalib.o] Error 1
> 
> my ROOT is 4.01 and Zlib is 1.2.2

We're in the process of fixing in the proper manner this problem, in the mean time
please add to the analyzer makefile the definition: -DUSE_ROOT at the line:
...
ROOTCFLAGS += -DHAVE_ROOT -DUSE_ROOT

Could someone please explain to me how cm_msg, cm_msg1, etc. all work.  The
documentation is very terse.  

I want to setup a fairly significant set of debugging, and error messages for a
new frontend.  I need to get these messages to a logging file.  I also would
like to get the error messages to the user through whatever interface Midas
normally uses for error reporting.  

Jan

> Could someone please explain to me how cm_msg, cm_msg1, etc. all work.  The
> documentation is very terse.  
> 
> I want to setup a fairly significant set of debugging, and error messages for a
> new frontend.  I need to get these messages to a logging file.  I also would
> like to get the error messages to the user through whatever interface Midas
> normally uses for error reporting.  

For errors, use

  cm_msg(MERROR, "routine_name", "Your error message, code=%d", i);

This produces an error message which is logged to midas.log, and distributed to all
clients which have called cm_msg_register(). For example odbedit will just print
that message. The syntax of the second half of cm_msg is the same as for printf(),
so you can add format specifiers and variable arguments as you do for printf(). The
first argument is the message type (MDEBUG for example is only distributed but not
logged). 

For a more detailed list of message types, please refer to

http://midas.triumf.ca/doc/html/AppendixE.html#midas_macro