Hi Konstantin,

> > * No central repo. Have all the history with you on the train.
> > * Branching and merging, with stable branches and feature branches.
> >   Happy hacking while my students do analysis on a stable version.
> >   Or multiple development branches for several features.
> 
> This is the part that worries me the most. Without a "central" "authoritative" repository,
> in just a few quick days, everybody will have their own incompatible version of midas.

No! This is probably one of the biggest misunderstandings of the git workflow.

You can of course _define_ one central repo: This is the one that you and Stefan decide to be "the source" (as
Linus does for the kernel). It's like the central svn repo: Only Stefan and you can push to it, and everybody
else will pull from it. Why should I pull MIDAS from some obscure source, when your "public" repo is available.

Look at the Linux Kernel: Linus' version is authoritative, even though everybody and his best friend has his
own kernel repo.

So, the main workflow does not change a lot: You collect patches, commit them, and "push" them to the central
repo. All users "pull" from this central repo. This is very much what svn offers.

> 
> I guess I am okey with your private midas diverging from mainstream, but when *I* end up
> with 10 different incompatible versions just in *my* repository, can that be good?

See above: _You_ define what the central repo is.

But: I _bet_ you will very soon have 10 versions in your personal repo, because _you choose_ to do so. It's
just SO much easier. The non-linear history with many branches is a _feature_. I can't live without it any more:


Looking at my MIDAS analyzer:

I have a "public" repo in /pub/git/lamb.git. This is where I publish my analyzer versions. All my collaborators
pull from this.

Then I have my personal repo in ~/src/lamb. 
This is where I develop. When I think something is ready for the public, I merge this branch into the public repo. 

Whenever I start to work on a new feature, I create a branch in my _local_ repo (~/src/lamb).  I can fiddle and
play, not affecting anybody else, because it never sees the public repo.
OK, collaborator A finds a bug. I switch to my local copy of the public version, fix the bug, and push the fix
to the publix repo. Then I go back to my (local) feature branch, merge the bug fix, and continue hacking.
Only when the feature is ready, I push it to the public repo.

Things get moe interesting as you work on several features simultaneously. You have e.g. 3 topic branches:
(a) is nearly ready, and you want a bunch of people to test it.
    push branch "feature (a)" to the public repo and tell the people which branch to pull.
(b) is WIP, you hack on it without affecting (a).
(c) is bug fixes which may or may not affect (a) or (b).
And so on.

You will soon discover the beauty of several parallel branches.

Plus, git merges are SO simple that you never think about "how to merge"

> 
> >   And merging really works, including fixing up merge conflicts.
> 
> But somebody still has to do it. With a central repository, the problem takes care of
> itself - each developer has to do their own merging - with svn, you cannot commit
> to the head without merging the head into your code first. But with git, I can just throw
> my changes int some branch out there hoping that somebody else would do the merging.
> But guess what, there aint anybody home but us chickens. We do not have a mad finn here
> to enforce discipline and keep us in shape...

See above: You will have the exact same workflow in git, if you like.




> As an example, look at the HADOOP/HDFS code development, they have at least 3 "mainstream"
> branches going, neither has all the features combined together and each branch has bugs with
> the fixes in a different branch. What a way to run a railroad.

I haven't look at this. All I can say: Branches are one of the best features.

> 
> > * "git bisect" for finding which commit introduced a (reproducible) bug.
> > * "gitk --all"
> >
> > Go for git. :-)
> 
> Absolutely. For me, as soon as I can wrap my head around this business of "who does all the merging".

Easy: YOU do it.

Keep going as in svn: Collect patches, and send them out.

And then, try "git checkout -b my_first_branch", hack, hack, hack,
"git merge master".

Best,

Randolf


> 
> K.O.

> * "git bisect" for finding which commit introduced a (reproducible) bug.

I did not know this command, so I read about it. This IS WONDERFUL! I had once (actually with MSCB) the case that a bug was introduced i the last 100 
revisions, but I did not know in which. So I checked out -1, -2, -3 revisions, then thought a bit, then tried -99, -98, then had the bright idea to try -50, then 
slowly converged. Later I realised that I should have done a binary search, like -50, if ok try -25, if bad try -37, and so on to iteratively find the offending 
commit. Finding that there is a command it git which does this automatically is great news.

Stefan

> > * "git bisect" for finding which commit introduced a (reproducible) bug.
> 
> I did not know this command, so I read about it. This IS WONDERFUL! I had once (actually with MSCB) the case that a bug was introduced i the last 100 
> revisions, but I did not know in which. So I checked out -1, -2, -3 revisions, then thought a bit, then tried -99, -98, then had the bright idea to try -50, then 
> slowly converged. Later I realised that I should have done a binary search, like -50, if ok try -25, if bad try -37, and so on to iteratively find the offending 
> commit. Finding that there is a command it git which does this automatically is great news.

even more so considering the nonlinear history (due to branching) in a regular git repo.

Hi All,

I set "/Analyzer/ODB Load" to true and analyzed a run in offline mode. After
that, I found the start time and stop time in /RunInfo did not reflect the
correct time as in online. How do I retrieve the correct start/stop time from
the ODB in offline mode?

Thanks!

Jimmy

> I set "/Analyzer/ODB Load" to true and analyzed a run in offline mode. After
> that, I found the start time and stop time in /RunInfo did not reflect the
> correct time as in online. How do I retrieve the correct start/stop time from
> the ODB in offline mode?

Most trees in the ODB are not loaded with "/Analyzer/ODB Load", since you might 
want to have the start/stop time of the offline analysis there for example 
(although I agree that the online start/stop time is more interesting). So you 
have several options:

- modify mana.c. There is a function odb_load(), which first locks the whole ODB 
and then unprotects "/Experiment/Run Parameters" for example. Just add three more 
lines for "/Runinfo".

- write a run summary when running online. After each run, write a summary with 
start/stop time, number of events, settings etc. into some file. I usually do this 
in the EOR routine of the online analyzer and write directly into a CSV file which 
I can import directly into Excel. There I can make filtering depending on certain 
parameters, like show me all runs with more than x events where setting y was 10.

- extract the ODB from the .mid file with "odbhist -e filename.mid" and look into 
that.

- The time stamp of each event is in UNIX time form (seconds since 1.1.1970), so 
you now exactly when each event was recorded.

Hope one of this helps...

- Stefan

Replacing the MIDAS status page with a custom status page documented at

https://midas.triumf.ca/MidasWiki/index.php/Custom_Page_Features#Replace_Status_Page_by_a_Custom_page

does not appear to be supported in the current MIDAS version. 

As two of my experiments use this feature may I suggest its reinstatement?

Suzannah

> Replacing the MIDAS status page with a custom status page documented at
> 
> https://midas.triumf.ca/MidasWiki/index.php/Custom_Page_Features#Replace_Status_Page_by_a_Custom_page
> 
> does not appear to be supported in the current MIDAS version. 
> 
> As two of my experiments use this feature may I suggest its reinstatement?

It still works, but is actually simpler. The status page is now a "dynamic" page, meaning mhttpd just servers an html file to 
the browser and everything is done in JavaScript there. The file for the status page is under midas/resources/status.html. 
You can easily change that file or replace it with a completely different (custom) file without having to change the ODB. 

There is only one potential problem. All midas html pages now have a certain structure, as written in 

https://midas.triumf.ca/MidasWiki/index.php/Custom_Page#How_to_use_the_standard_MIDAS_navigation_bars_on_your_cust
om_page

So if you have an existing custom status page, you might have to change it slightly to include the standard elements 
"mheader" and "msidenav". But this allows you to have the standard menu on your custom page and alerts displayed at the 
top row of your custom page (which was not possible before).

Once this works for you, it would be nice to adjust the documentation to reflect this new way.

Stefan

> > I think there sneaked in a little bug in the mhttpd: when starting an experiment
> > from scratch and starting the mhttpd, the Menu Buttons are missing

Ok, the original problem with a small bug in the javascript code for the menu buttons (fixed now),
but I was moved to implement something I wanted to do for a long time.

The menu configuration is now done through a subdirectory /experiment/menu. Each entry corresponds to 
one menu button. Set to "y" to show it, set to "n" to hide it.

Buttons are displayed in the same order as they are in ODB, to change the order of buttons,
change their order in ODB (odbedit command "move").

This fixes the long standing problem with adding new midas pages - they were not automatically added to 
the existing "menu buttons" lists. So for example when the "chat" page was added, I did not know about it 
for a long time (and some people still do not know about it's existence) because it is was not included in 
my  "/experiment/menu buttons" list in all my already existing experiments. When the "start" and 
"transition" pages were added, probably nobody knows that they exist.

Now new buttons for new pages are automatically added to the list (via mhttpd.cxx::init_menu_buttons()), 
the users have an option to hide them by setting their values to "n".

K.O.

> > > I think there sneaked in a little bug in the mhttpd: when starting an experiment
> > > from scratch and starting the mhttpd, the Menu Buttons are missing
> 
> Ok, the original problem with a small bug in the javascript code for the menu buttons (fixed now),
> but I was moved to implement something I wanted to do for a long time.
> 

Is this change back-wards compatible with an old ODB?  Ie, if I upgrade MIDAS, will it notice that I have the old-style key "/Experiment/Menu Buttons" 
and replace it equivalently set keys in /Experiment/Menu?  Or will it just continue to use the old-style ODB key?

> > > > I think there sneaked in a little bug in the mhttpd: when starting an experiment
> > > > from scratch and starting the mhttpd, the Menu Buttons are missing
> > 
> > Ok, the original problem with a small bug in the javascript code for the menu buttons (fixed now),
> > but I was moved to implement something I wanted to do for a long time.
> > 
> 
> Is this change back-wards compatible with an old ODB?  Ie, if I upgrade MIDAS, will it notice that I have the old-style key "/Experiment/Menu Buttons" 
> and replace it equivalently set keys in /Experiment/Menu?  Or will it just continue to use the old-style ODB key?

I am trying to keep some compatibility between the web pages and mhttpd. I think in most cases, old mhttpd should continue to work
against new web pages (assuming matching mhttpd.js & co). But old web pages would probably break against new mhttpd, mostly due
to the rapid pace of their development.

Anyhow, the midas web page forms menu buttons in this order:
/Experiment/Menu, if it does not exist, then:
/Experiment/menu buttons, if it does not exist, then
built in list of menu buttons, which includes all possible buttons, hardcoded in mhttpd.js.

In cooperation with mhttpd: new mhttpd
- will automatically create the tree /experiment/menu with all buttons disabled
- will complain about the existence of /expriment/menu buttons, instruct user to delete it.

So to answer the question:
after git pull, make, restart mhttpd, you will see all possible menu buttons and you will have to go
into the odb editor to disable the buttons you do not want to see (i.e. the mscb button).

I did it this way on purpose, to give old-time midas users an opportunity to discover
some of the newly added buttons and pages, like the "chat" page, or the "example" page. If I migrated
the existing "menu buttons" verbatim, to the new tree, I would not even today know
that the "chat" page exists (I do not think it was ever announced or described on this forum
or anywhere in the documentation).

K.O.

Hi Folks,

In last year's updates to MIDAS, the MIDAS sequencer has been broken out as a
separate program (rather than running as part of mhttpd).  We hope that this
change will make the sequencer operation more stable.

Before anyone gets too used to using the new sequencer program, I would like to
rename it.  Currently the program is called 'sequencer'; I would like to rename
it 'msequencer', to make it consistent with most other MIDAS programs.  If you
object to making this change, please say so in the next two weeks.

Documentation on the MIDAS sequencer can be found on the wiki:

https://midas.triumf.ca/MidasWiki/index.php/Sequencer

Note that there are still some tweaks that need to be made to the sequencer
webpage and mhttpd in order to handle this new sequencer program.

Cheers,
Thomas

Hearing no objections I changed the name of the program to msequencer.  Wiki
documentation updated.

> Hi Folks,
> 
> In last year's updates to MIDAS, the MIDAS sequencer has been broken out as a
> separate program (rather than running as part of mhttpd).  We hope that this
> change will make the sequencer operation more stable.
> 
> Before anyone gets too used to using the new sequencer program, I would like to
> rename it.  Currently the program is called 'sequencer'; I would like to rename
> it 'msequencer', to make it consistent with most other MIDAS programs.  If you
> object to making this change, please say so in the next two weeks.
> 
> Documentation on the MIDAS sequencer can be found on the wiki:
> 
> https://midas.triumf.ca/MidasWiki/index.php/Sequencer
> 
> Note that there are still some tweaks that need to be made to the sequencer
> webpage and mhttpd in order to handle this new sequencer program.
> 
> Cheers,
> Thomas

I am debugging the frequent problem where the run number is mysteriously
reset to zero. As a first step, I am commiting changes to mhttpd.c and midas.c:
- abort on obviously corrupted "run number < 0"
- abort on cm_transition() to run 0 (the only place where the run number is
explicitely written to ODB)
- in the mhttpd "Start run" form, reject user setting the run number to <= 0.

Here is the CVS diff:

===================================================================
RCS file: /usr/local/cvsroot/midas/src/mhttpd.c,v
retrieving revision 1.253
diff -r1.253 mhttpd.c
2451a2452,2457
>   if (run_number < 0)
>     {
>     cm_msg(MERROR, "show_elog_new", "aborting on attempt to use invalid
run number %d",run_number);
>     abort();
>     }
> 
2506a2513,2519
> 
>     if (run_number < 0)
>       {
>       cm_msg(MERROR, "show_elog_new", "aborting on attempt to use invalid
run number %d",run_number);
>       abort();
>       }
> 
3582a3596,3602
> 
>   if (run_number < 0)
>     {
>     cm_msg(MERROR, "show_form_query", "aborting on attempt to use invalid
run number %d",run_number);
>     abort();
>     }
> 
5730a5751,5756
>   if (rn < 0) // value "zero" is okey
>     {
>     cm_msg(MERROR, "show_start_page", "aborting on attempt to use invalid
run number %d",rn);
>     abort();
>     }
> 
9684a9711,9719
>       if (i <= 0)
>         {
>         cm_msg(MERROR, "interprete", "Start run: invalid run number %d",i);
>         memset(str,0,sizeof(str));
>         snprintf(str,sizeof(str)-1,"Invalid run number %d",i);
>         show_error(str);
>         return;
>         }
> 
Index: src/midas.c
===================================================================
RCS file: /usr/local/cvsroot/midas/src/midas.c,v
retrieving revision 1.193
diff -r1.193 midas.c
3786c3786
<         status = cm_transition(_requested_transition | TR_DEFERRED, 0,
str, 256, SYNC, FALSE);
---
>         status = cm_transition(_requested_transition | TR_DEFERRED, 0,
str, sizeof(str), SYNC, FALSE);
3906a3907,3912
>   if (run_number <= 0)
>     {
>     cm_msg(MERROR, "cm_transition", "aborting on attempt to use invalid
run number %d",run_number);
>     abort();
>     }
> 
16069a16076,16081
>     }
> 
>   if (run_number < 0)
>     {
>     cm_msg(MERROR, "el_submit", "aborting on attempt to use invalid run
number %d", run_number);
>     abort();

K.O.

> I am debugging the frequent problem where the run number is mysteriously
> reset to zero. As a first step, I am commiting changes to mhttpd.c and midas.c:
> - abort on obviously corrupted "run number < 0"
> - abort on cm_transition() to run 0 (the only place where the run number is
> explicitely written to ODB)
> - in the mhttpd "Start run" form, reject user setting the run number to <= 0.

- abort on cm_transition() from run 0 to 1 during auto restart in mlogger.

Cvs diff:

RCS file: /usr/local/cvsroot/midas/src/mlogger.c,v
retrieving revision 1.65
diff -r1.65 mlogger.c
3277a3278,3283
>         if (run_number <= 0)
>           {
>           cm_msg(MERROR, "main", "aborting on attempt to use invalid run
number %d", run_number);
>           abort();
>           }
> 

K.O.

In close cooperation with Stefan, I refactored and rewrote the MIDAS event
buffering code (bm_send_event, bm_flush_cache, bm_receive_event and bm_push_event).

The main goal of this update is to make sure the event buffering code does not
have any infinite loops: in the past, we have seen mlogger and some frontends
loop forever consuming 100% CPU in the event buffering code. This should now be
completely fixed.

As additional bonuses, the refactored code is easier to read, has less code
duplication and should be more robust. A few potential logical problems have
been corrected and one case of reproducible infinite looping has been fixed.

The new code has passed the low-level consumer-producer tests, but has not yet
been used in anger in any real experiment. One hopes any new bugs introduced
would cause outright failures and core dumps (rather than silent data corruption).

All are welcome to try the new code. If it explodes, please send me the error
messages, stack traces and core dumps.

K.O.

> In close cooperation with Stefan, I refactored and rewrote the MIDAS event
> buffering code (bm_send_event, bm_flush_cache, bm_receive_event and bm_push_event).
>
> All are welcome to try the new code. If it explodes, please send me the error
> messages, stack traces and core dumps.

Stefan quickly found one new error (a typoe in a check against infinite looping) and
then I found one old error present in the old code that caused event loss when the
buffer became exactly 100% full (0 bytes free).

Both errors are now fixed in svn commit 3294.

K.O.

The custom and alias links in the standard midas status page were shown as HTML 
links so far. If there are many links with names having spaces in their names, 
it's a bit hard to distinguish between them. Therefore, they are packed now into 
individual buttons (see attachment) starting from SVN revision 4633 on. This makes 
also the look more homogeneous. If there is any problem with that, please report.

> The custom and alias links in the standard midas status page were shown as HTML 
> links so far. If there are many links with names having spaces in their names, 
> it's a bit hard to distinguish between them. Therefore, they are packed now into 
> individual buttons (see attachment) starting from SVN revision 4633 on. This makes 
> also the look more homogeneous. If there is any problem with that, please report.

Would you consider using a different colour for the alias buttons (or background
colour)? At present it's hard to know whether a button is an alias link, a custom page
link or a user-button especially if you are not familiar with the button layout. 

> > The custom and alias links in the standard midas status page were shown as HTML 
> > links so far. If there are many links with names having spaces in their names, 
> > it's a bit hard to distinguish between them. Therefore, they are packed now into 
> > individual buttons (see attachment) starting from SVN revision 4633 on. This makes 
> > also the look more homogeneous. If there is any problem with that, please report.
> 
> Would you consider using a different colour for the alias buttons (or background
> colour)? At present it's hard to know whether a button is an alias link, a custom page
> link or a user-button especially if you are not familiar with the button layout. 

Ok, I changed the background colors for the button rows. There are now four different 
colors: Main menu buttons, Scripts, Manually triggered events, Alias & Custom pages. Hope 
this is ok. Of course one could have each button in a different color, but then it gets 
complicated... In that case I would recommend to make a dedicated custom page with all these 
buttons, which you can then tailor exactly to your needs.

Since the column on the main midas status page with fraction of analyzed events is 
barely used, I decided to drop it. Anyhow it does not make sense for all slow 
control events. If this feature is required in some experiment, I propose to move it 
into a custom page and calculate this ratio in JavaScript, where one has much more 
flexibility. 

This modification frees up more space on the status page for the "Status" column, where 
front-end programs can report errors etc.