> in this case, delete previous odb contents using the "rm" command (then say "y" 
> to delete all keys, this is the equivalent of unix "rm -rf /").
> 
> this will delete all odb contents except some for some entries in /system, 
> /experiment, /alarms, /logger and /runinfo. some of these entries are delete-
> protected, some are watched and some are auto-re-created by al_check() and 
> others.

checking that this works uncovered an old bug/problem in the odb delete function:

"rm /experiment/security/rpc hosts" corrupted the free list because of incorrect 
delete of "rpc hosts/allowed hosts" which cannot be deleted because it is watched.

fixing this uncovered a second problem: odb "rm -rf /" cannot remove entries in 
/system because they are delete-protected (db_delete() should return DB_NO_ACCESS) 
and it cannot remove watched items (i.e. "allowed hosts", db_delete() should return 
DB_OPEN_RECORD).

but db_delete() cannot return both error codes at the same time! a problem.

previous code returned:

DB_OPEN_RECORD for db_delete("/expt/.../allowed hosts"),
DB_SUCCESS (and corrupted the free list) for db_delete("/expt/...")
DB_NO_ACCESS for db_delete() of delete-protected items
DB_SUCCESS for db_delete() of subdirectory that contains delete-protected items (no 
free list corruption, no delete of protected items, no delete of the subdirectory)

updated code returns:

DB_SUCCESS if everything requested was deleted
DB_NO_ACCESS or DB_OPEN_RECORD if some items were not deleted

names of items that could not be deleted are reported via cm_msg().

hopefully, these are improvements.

suggestion on how to make it even better are very welcome.

K.O.

after midas crash and odb corruption, midas and odbedit now automatically repair 
all (yea, right!) structural corruption - fix the free list, fix/remove invalid 
subdirectories and data entries, etc.

after this automatic repair, midas user should be able to reload odb from a 
backup json or xml file without having to delete and recreate .ODB.SHM and 
/dev/shm files, etc.

but sometimes odb corruption is too confusing or too corrupted and a backup file 
will not load cleanly.

in this case, delete previous odb contents using the "rm" command (then say "y" 
to delete all keys, this is the equivalent of unix "rm -rf /").

this will delete all odb contents except some for some entries in /system, 
/experiment, /alarms, /logger and /runinfo. some of these entries are delete-
protected, some are watched and some are auto-re-created by al_check() and 
others.

an odb backup file should load cleanly after this.

K.O.

automatic free list repair is now enabled by default. after fixing a few more minor problems (plus free list 
corruption in "rm /experiment/security/rpc hosts/allowed hosts", cannot be deleted because it is watched).

in case this causes problems, disable it via "setenv MIDAS_ODB_DO_NOT_REPAIR_FREE_LIST 1"

commit a0e86c45ef6bf028316d75e93095b48459c73533

K.O.

I'm pleased to announce a new documentation site for midas.

https://midas.triumf.ca/docs replaces the old MidasWiki. We'll keep the
MidasWiki around for a while, but the content has been replaced with links to
the relevant place in the new docs.

Most people will be interested in the "Feature guide", but there are also intro
pages for new users, reference pages for power-users, and auto-generated API
documentation based on the source code.

The pages for each feature are generally split into an overview (useful for
everyone), a configuration page (explaining ODB keys etc), and a programming
page (which shows examples in C++, Python and JavaScript).

The new system has some very nice usability improvements compared to MidasWiki.
One thing I particularly like is if you hover over a link, you get a preview of
the target page. Very useful for quickly checking the reference without losing
the context of your current page.

The search system has some useful filters based on the audience of each page. If
you know you want to see programming examples, for example, click the
"Developer" filter.


The new documentation is stored in the midas repository itself, in $MIDASSYS/
docs/mkdocs. This means you can: 
* build the docs yourself if you wish (see https://midas.triumf.ca/docs/develop/
reference/internals/documentation/)
* point your friendly AI assistant towards the markdown files to kickstart its
knowledge


Let me know if you have any issues with the new documentation
(bsmith@triumf.ca). There is a lot of new content, and a lot has been
restructured compared to the MidasWiki. I hope it will prove useful once you get
used to the new structure.

Cheers,
Ben

I was debugging a core dump from a crash inside ODB free_data() and after 
refactoring free_data(), I may have fixed a bug in merging of free data blocks.

the case where freed data closed a gap between two free data blocks maybe had a 
mistake, it should be fixed now. "free_block_1 | freed data | free_block_2" 
becomes "free_block".

commit 9b54a1264b252b4e88c259c82afdf7e16b5a89eb

P.S.

1) to remember, odbedit now has a check and repair for the ODB free list, and I 
just added one more test against ODB free list corruption (infinite loop if a 
free block points at itself).

2) to test ODB edge cases, run "./bin/torture_odb --all". I now added an "odb 
full" test which exercises most ODB free list edge cases. 

K.O.

I restored the bitbucket builds. currently build for Ubuntu 22 and 24, Alma Linux 
9 and 10. "make test" replaced by "make test-old". K.O.

> I looked into this now and it seems to work but the issue is that even when using only one sequencer with -c MyUniqueName
> the page does not work anymore. We tried to create our own custom page for this now but maybe a nice addition would be to
> simply change the custom page to also support multiple sequencers.

The sequencer webpage does support multiple sequencers (and that's how it supports both the original sequencer and the python-
based sequencer). The choice of sequencer is based on the `SeqODB` URL parameter.

To create a link that will show up in the menu, create an entry in `/Alias` in the ODB.

e.g. for original sequencer:
odbedit -c "create string '/Alias/My new sequencer'"
odbedit -c "set '/Alias/My new sequencer' '?cmd=Sequencer&SeqODB=/SequencerMyUniqueName'"

e.g. for pysequencer:
odbedit -c "create string '/Alias/My new pysequencer'"
odbedit -c "set '/Alias/My new pysequencer' '?cmd=Sequencer&SeqODB=/PySequencerMyUniqueName'"

You can of course create this entry using the ODB web interface instead.

> > > > I was wondering if one can use multiple pysequencers at the same time and if not
> > > > if this feature is planned in the future? Our experiment (mu3e) has one frontend
> > > > per detector and each sequencer would only access a subset of ODB entries.
> > > 
> > > I believe sequencer design supports this, same as mlogger supports multiple output 
> > > files and mhttpd supports multiple listener ports.
> > > 
> > > But how to do this with pysequencer, I am not sure. Simplest is to wait a few more 
> > > weeks for Ben to return from vacation.
> > > 
> > > K.O.
> > 
> > I looked into this, and it has been possible for at least a year. Just start the sequencer with `-c MyUniqueName` and 
> > then all of the state etc. can be controlled from the ODB location `/PySequencerMyUniqueName/`.
> > And obviously use different names for each instance.
> 
> that's what I remember, too. but was not sure. K.O.

I looked into this now and it seems to work but the issue is that even when using only one sequencer with -c MyUniqueName
the page does not work anymore. We tried to create our own custom page for this now but maybe a nice addition would be to
simply change the custom page to also support multiple sequencers.

gitlab notifications do not send email on each commit and push (like bitbucket did).
to enable this, add your email address to the gitlab "emails on push integration": 
https://gitlab.triumf.ca/groups/midas/-/settings/integrations/emails_on_push/edit
K.O.

As for issue reporting, please use the gitlab work item tracker: https://gitlab.triumf.ca/groups/midas/-/work_items

You will need an account on the TRIUMF gitlab, but self-registration is enabled. If there are issues registering, please make a post here or send me an email. 

Note that we will not be monitoring issues on the bitbucket jira tracker. 

Thanks, 

Derek

Stefan uncovered a bug in mhttpd URI decoding, the following did not work:
cd midas, cd resources, cp example.html c++.html, open http://midas/example.html (works), open c++.html and it bombs with error about "c  .html" (two pluses replaced by two spaces).

Stefan did a quick fix for this and I now finished the follow up.

here are the rules for encoding URLs for mhttpd: (using this example URL)

https://experiment.triumf.ca/vslice/?cmd=ODB&odb_path=foo

the query parameter value "foo" should be percent-encoded using encodeURIComponent(). this replaces spaces with %20, slashes with %2F and pluses with %2B, etc. a plain bare plus character "+" is permitted here, it will seen as a space by mhttpd (odb_path=A+B becomes "A B").

the query parameter name "odb_path" should also be percent-encoded, but standard mhttpd parameters only use ascii character for parameter names and it is okey to skip encodeURIComponent() and hardcode them. if custom parameter names use special characters or other funny characters, they should be percent-encoded.

the elements of the URL path "vslice" should be percent-encoded if they contain special URI characters like "?" and "&". internally mhttpd runs the URL path through decodeURIComponent(). this expands all percent-encoded characters. (plus and space characters are not changed, URLs like c++.html work).

one exception for percent-encoded URLs is the percent-encoded slash ("%2F" instead  of "/"). for security reasons apache httpd and other web servers do not permit this. for consistency, mhttpd also rejects "%2F" in the URL path with the "404 Not Found" error.

TL&DR:

1) replaced old url-encode and decode functions with encodeURIComponent() and decodeURIComponent(). the mhttpd encoder is more aggressive compared to most javascript encoders (they leave characters like "_" and "(" unencoded). this is safe.

2) rewrote decode_query() to remove a strdup(), remove strtok() and manually parse the query string using bare URI query delimiters "&" and "=". malformed queries (mismatched "=" and "&") are silently truncated. in parameter values, plus is decoded as a space, decodeURIComponent() is consistently applied to parameter names and values.

3) added a trap in handle_http_message() against percent-encoded slash ("%2F" is rejected with "404 Not Found").

Commit 01336247eb4fa5bb6ed1a7187fa2061d6d4e9086
and 590d2b0812eacebd313efbe1c0c38a34f6eaf78f

K.O.

MIDAS Migration

We have transitioned from hosting MIDAS on bitbucket to TRIUMF's self-hosted gitlab instance (https://gitlab.triumf.ca/midas). Apologies for the early transition.

Do I need an account?

To push to the new server you will need an account on that server, and be added as a developer to the group. To create issues (or work items), you will also need an account. Otherwise, you can pull and clone anonymously.

Self-registration works on a whitelisting scheme: either your instiutional hostname or your full address needs to be whitelisted. If your regisration fails, please make a post in this forum or send me an email. 

How to get access to MIDAS?

You can pull to or push from the gitlab server. You can also pull from the bitbucket server - it has been set up as a mirror. Any changes to the gitlab repos will be pushed within 5 mins of the change. Long-term the bitbucket mirrors will be depreciated. 

Push permissions have been rescinded on bitbucket to enforce this.  

How to point your repos to the new server: 

git checkout develop
git pull --recurse-submodule
git remote set-url origin git@gitlab.triumf.ca:midas/midas.git # or use the html url: https://gitlab.triumf.ca/midas/midas.git
git submodule sync

[EDIT: use the html url to pull without a TRIUMF gitlab account]

Developer notes: 

Actions taken: 

• Removed write permissions from developer group to all MIDAS-related repos
• Transferred developers into a "Former Developers" group
• Re-synced gitlab and bitbucket repos
• Setup gitlab as push repos
  • This will expire Sep 25, 2027 (max duration)
  • Push mirroring happens after any push to the gitlab repo, with a minimum of 5 mins duration between mirror pushes
• MIDAS README updated

Thanks for reporting this. We need the 64 to keep the array size constant at n*64. The problem was the read of the v.c_str() beyond its length, 
which triggered the address sanitizer. I fixed that now and committed to develop.

Stefan

Hi,
I was seeing a crash when connecting remotely with MIDAS_SERVER_HOST. Debugging, it seems to be 
because hs_define_panel assumes all of its `var` are 64 bytes (at https://gitlab.triumf.ca/midas/midas/-
/blob/develop/src/history.cxx?ref_type=heads#L3507). But I'm passing in a shorter one. It doesn't happen 
when using a local Midas because the local path uses mstrlcpy which stops at the null terminator.

Is there a reason why it has to be 64 bytes? Or can I change it to the minimum of 64 and the string size? 
E.g.

-      db_set_data_index(hDB, hKeyVar, v.c_str(), 64, i, TID_STRING);
+      db_set_data_index(hDB, hKeyVar, v.c_str(), std::min(64, v.size()), i, TID_STRING);


Output from memory sanitiser is below.


==19362==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60400001d93a at pc 
0x5555556af004 bp 0x7ffffffc0d70 sp 0x7ffffffc0538
READ of size 64 at 0x60400001d93a thread T0
    #0 0x5555556af003 in __interceptor_memcpy 
(/build/online/switching_pc/midas_fe/switch_fe_central+0x15b003) (BuildId: 
069e8fae8f012afe15b42dd7e819397d7c530852)
    #1 0x55555593377c in rpc_call_encode(__va_list_tag (&) [1], RPC_LIST const&, NET_COMMAND**) 
/code/midas/src/midas.cxx:13612:13
    #2 0x55555591b412 in rpc_call(unsigned int, ...) /code/midas/src/midas.cxx:14295:7
    #3 0x555555969585 in db_set_data_index(int, int, void const*, int, int, unsigned int) 
/code/midas/src/odb.cxx:8249:14
    #4 0x555555a090a6 in hs_define_panel(char const*, char const*, 
std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 
std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) 
/code/midas/src/history.cxx:3507:7
    #5 0x555555762c6c in setup_history() /code/online/switching_pc/midas_fe/switch_fe.cpp:454:5

> The history of at least bm.cxx seems to be screwed up. Bitbucket shows this:
>   https://bitbucket.org/tmidas/midas/annotate/fa649e6e9bb7447b4da01e964c6dc14b01e64052/src/bm.cxx?at=test%2Fsplit-midas-cxx
> and CLion shows what I attached here. Most lines are tagged "Today Olchanski" which cannot be.

yup, I see the same nonsense for bm.cxx from gitea.

back to the drawing board, will try branch, rename, merge as described here:
https://devblogs.microsoft.com/oldnewthing/20190916-00/?p=102892/

the naming of split files looks ok or we prefer more descriptive names "event_buffer.cxx",
"transition.cxx", "message.cxx", etc?

K.O.

The history of at least bm.cxx seems to be screwed up. Bitbucket shows this:

  https://bitbucket.org/tmidas/midas/annotate/fa649e6e9bb7447b4da01e964c6dc14b01e64052/src/bm.cxx?at=test%2Fsplit-midas-cxx

and CLion shows what I attached here. Most lines are tagged "Today Olchanski" which cannot be.

Best,
Stefan

> > A split of src/midas.cxx was discussed during Stefan & Thomas R. visit to TRIUMF.
> 
> A test of split src/midas.cxx is now pushed to branch test/split-midas-cxx
> 
> please take a look at this branch, run git log and git blame using your preferred tool of choice (vscode, clion, etc)
> and let me know if tracking of revision history works through the split commit and if there
> is any other oddities.
>

bitbucket web gui for src/rb.cxx
- show history - only to split commit (ouch!)
- show blame - all the way to the beginning of time (good)

K.O.

> A split of src/midas.cxx was discussed during Stefan & Thomas R. visit to TRIUMF.

A test of split src/midas.cxx is now pushed to branch test/split-midas-cxx

midas.cxx -> deleted, as required by git (see below)

cm_msg.cxx - will be all the cm_msg functions
bm.cxx     - all the event buffer functions
bk.cxx     - will be all the event bank functions
rb.cxx     - ring buffer functions (obsolete?)
rpc.cxx    - will be all the RPC functions
cm.cxx     - everything else

when I do the split for real, I think I will split run transitions to cm_transition.cxx

please take a look at this branch, run git log and git blame using your preferred tool of choice (vscode, clion, etc)
and let me know if tracking of revision history works through the split commit and if there
is any other oddities.

also, better names for the split files are most welcome.

only rb.cxx and bm.cxx are actually reduced in size, the rest of the files are still copies of midas.cxx (this is only a test of the 
split).

K.O.

P.S. More details below:

> 
> Main reason for the split is to simplify analysis using AI tools, they work best on smaller files that contain closely related 
> code.
> 
> Main difficulty with the split is "git" tooling, which does not have a concept of file copy or file split. (still this is better 
> than older tools like cvd and svn, which had no concept of branching and merging).
> 
> My first step is to do a strawman split of midas.cxx on a branch, push it out and ask people to check that they git tooling 
> (vscode, clion, bitbucket, github, gitea, etc) see the split correctly.
> 
> Ideally, git log and git blame should be able to trace the revision of each split file
> back to the beginning of time, instead of to the point of split from midas.cxx.
> 
> According to google AI, I should do this:
> 
> step 1: midas.cxx removed, split files are exact copies of midas.cxx, this allows git to identify the split point
> 
> cp midas.cxx split1.cxx
> cp midas.cxx split2.cxx
> ...
> git rm midas.cxx
> git commit
> 
> step 2: edit the split files to remove duplicated code, commit.
> 
> step 3: check
> 
> git blame split1.cxx -> shows correct revision history for me. maybe need to use "-C"
> git log split1.cxx -> shows correct revision history to split commit
> git log --follow split1.cxx -> shows correct revision history for me
> 
> TD;DR continue below.
> 
> Identify all function names in midas.cxx:
> 
> cat midas.cxx | grep -v -e "^ " -e "^{" -e "^}" -e "^$" -e "^/" -e '^\\' -e "^@" -e "^*" -e "^#" | sort
> 
> ^void functions:
> 
> void bk_xxx()
> void bm_xxx()
> void cm_xxx()
> void dbg_xxx()
> void rpc_xxx()
> 
> ^std::string functions:
> 
> cm_xxx()
> msprintf()
> rpc_xxx()
> 
> ^static functions:
> 
> bm_xxx
> rpc_xxx
> cm_xxx
> 
> ^int functions:
> 
> bk_xxx
> bm_xxx
> cm_xxx
> cm_msg_xxx
> rb_xxx
> rpc_xxx
> 
> ^const data
> 
> cm_xxx
> rpc_xxx
> 
> ^bool functions:
> 
> rpc_xxx
> 
> ^INT functions
> 
> bk_xxx
> bm_xxx
> cm_xxx
> cm_msg_xxx
> rpc_xxx
> 
> ^BOOL functions
> 
> bk_xxx
> cm_xxx
> 
> LC_ALL=C sort midas.cxx | grep -v -e "^/" -e "^\s" -e "^W" -e "^w" -e ^} -e ^{ -e ^[a-v] -e '^\\' -e ^[A-U] -e ^@ -e ^* -e ^# -e ^$
> 
> Stawman split:
> 
> midas.cxx -> deleted
> 
> cm_msg.cxx - all the cm_msg functions
> bm.cxx     - all the event buffer functions
> bk.cxx     - all the event bank functions
> rb.cxx     - ring buffer functions (obsolete?)
> rpc.cxx    - all the RPC functions
> 
> cm.cxx     - everything else
> 
> K.O.

A split of src/midas.cxx was discussed during Stefan & Thomas R. visit to TRIUMF.

Main reason for the split is to simplify analysis using AI tools, they work best on smaller files that contain closely related 
code.

Main difficulty with the split is "git" tooling, which does not have a concept of file copy or file split. (still this is better 
than older tools like cvd and svn, which had no concept of branching and merging).

My first step is to do a strawman split of midas.cxx on a branch, push it out and ask people to check that they git tooling 
(vscode, clion, bitbucket, github, gitea, etc) see the split correctly.

Ideally, git log and git blame should be able to trace the revision of each split file
back to the beginning of time, instead of to the point of split from midas.cxx.

According to google AI, I should do this:

step 1: midas.cxx removed, split files are exact copies of midas.cxx, this allows git to identify the split point

cp midas.cxx split1.cxx
cp midas.cxx split2.cxx
...
git rm midas.cxx
git commit

step 2: edit the split files to remove duplicated code, commit.

step 3: check

git blame split1.cxx -> shows correct revision history for me. maybe need to use "-C"
git log split1.cxx -> shows correct revision history to split commit
git log --follow split1.cxx -> shows correct revision history for me

TD;DR continue below.

Identify all function names in midas.cxx:

cat midas.cxx | grep -v -e "^ " -e "^{" -e "^}" -e "^$" -e "^/" -e '^\\' -e "^@" -e "^*" -e "^#" | sort

^void functions:

void bk_xxx()
void bm_xxx()
void cm_xxx()
void dbg_xxx()
void rpc_xxx()

^std::string functions:

cm_xxx()
msprintf()
rpc_xxx()

^static functions:

bm_xxx
rpc_xxx
cm_xxx

^int functions:

bk_xxx
bm_xxx
cm_xxx
cm_msg_xxx
rb_xxx
rpc_xxx

^const data

cm_xxx
rpc_xxx

^bool functions:

rpc_xxx

^INT functions

bk_xxx
bm_xxx
cm_xxx
cm_msg_xxx
rpc_xxx

^BOOL functions

bk_xxx
cm_xxx

LC_ALL=C sort midas.cxx | grep -v -e "^/" -e "^\s" -e "^W" -e "^w" -e ^} -e ^{ -e ^[a-v] -e '^\\' -e ^[A-U] -e ^@ -e ^* -e ^# -e ^$

Stawman split:

midas.cxx -> deleted

cm_msg.cxx - all the cm_msg functions
bm.cxx     - all the event buffer functions
bk.cxx     - all the event bank functions
rb.cxx     - ring buffer functions (obsolete?)
rpc.cxx    - all the RPC functions

cm.cxx     - everything else

K.O.

Dear MIDAS Community,

In a little over a week from today, on September 28, we will migrate the MIDAS repository away from bitbucket and to TRIUMF's self-hosted gitlab instance.

What does this mean for you?

Developers: If you want to push changes to the repository, you will need reconfigure your git settings to point to the new host. You will also need accounts on TRIUMF's gitlab (self-registration open).

Users: The bitbucket instance will remain as a mirror so everyone doesn't have to change every experiment. You will still be able to pull from bitbucket to fetch the latest updates. You may need to re-sync your submodules to have everything point to the right host. 

Full migration instructions will be provided here, at this forum, on September 28. 

Thanks, 

Derek