> Could somebody please give me a boost?

no need to shout into the void, it is pretty easy to identify the author of manalyzer and ask me directly.

> we are planning to migrate from mana to manalyzer. I started to have a look into it and realized that I have some lose ends.
> Is there a clear migration docu somewhere?

README.md and examples in the manalyzer git repository.

If something is missing, or unclear, please ask.

> Currently I understand it the following way (which might be wrong):
> The class TARunObject is used to write analyzer modules which are registered by TAFactory. I hope this is right?

Please read the README file. It explains what is going on there.

Design of manalyzer had 2 main goals:
a) lifetime of all c++ objects (and ROOT objects) is well defined (to fix a design defect in rootana)
b) event flow and data flow are documentable (problem in mfe.c frontends, etc)

> However, in mana there is an analyzer implemented by the user which binds the modules and has additional routines:
> analyzer_init(), analyzer_exit(), analyzer_loop()
> ana_begin_of_run(), ana_end_of_run(), ana_pause_run(), ana_resume_run()
> which we are using.

I have never used the mana analyzer, I wrote the c++ rootana analyzer very early on (first commit in 2006).

But the basic steps should all be there for you:
- initialization (create histograms, open files) can be done in the module constructor or in BeginRun()
- finalization (fit histograms, close files) should be done in EndRun()
- event processing (obviously) in Analyze()
- pause run, resume run and switch to next subrun file have corresponding methods
- all the "flow" and multithreading stuff you can ignore to first order.

To start the migration, I recommend you take manalyzer_example_root.cxx and start stuffing it with your code.

If you run into any problems, I am happy to help with them. Ask here or contact me directly.

> This part I somehow miss in manalyzer, most probably due to lack of understanding, and missing documentation.

True, I wrote a migration guide for the frontend mfe.c to c++ tmfe, because we do this migration
quite often. But I never wrote a migration guide from mana.c analyzer, because we never did such
migration. Most experiments at TRIUMF are post-2006 and use rootana in it's different incarnations.

P.S. I designed the C++ TMFe frontend after manalyzer and I think it came out quite better, I especially
value the design input from Stefan, Thomas, Pierre, Joseph and Ben.

P.P.S. Be free to ignore all this manalyzer business and write your own analyzer based
on the midasio library:

int main()
{
   TMReaderInteraface* f = TMNewReader(file.mid.gz");
   while (1) {
      TMEvent* e = TMReadEvent(f);
      dwim(e);
      delete e;
   }
   delete f;
}

For online processing I use the TMFe class, it has enough bits to be a frontend and an analyzer,
or you can use the older TMidasOnline from rootana.

Access to ODB is via the mvodb library, which is new in midas, but has been part of rootana
and my frontend toolkit since at least 2011 or earlier, inspired by Peter Green's even
older "myload" ODB access library.

K.O.

> > Is there a clear migration docu somewhere?

I can also give you links to the alpha-g analyzer (very complex) and the DarkLight trigger TDC analyzer (very simple),
there is also analyzer examples of in-between complexity.

K.O.

Hi,
I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
Midas during online running.  At the end of each run it saves out custom data in the 
TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
choice to ignore all of TMFeRpcHandlerInterface when offline?

Thanks,

Mark.

Hi,

Sorry, we have been slammed with a couple projects on the TRIUMF side in the past weeks and haven't found time for a 
response.  I am hopeful that we will be able to answer this question (and the cache size question) within the next 10 
days.  

Again apologies,
Thomas

> Hi,
> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?
> 
> Thanks,
> 
> Mark.

Hello,

Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

thanks,
Zaher

We have a problem with over aggressive AI bots. They are scanning everything and 
are putting a crazy load on the mediawiki mysql database. This makes all out 
wikis very slow and unresponsive, which is a big problem.

These AI scanners do not seem to know what they are doing and are trying to load 
all the special pages, like "what links here" and "recent changes" and complete 
revision histories for each page. I am not impressed. Old school google and bing 
scanner bots are much more respectful and do not cause problems.

AI bots are also scanning this elog, and to Stefan's credit elogd is holding the 
load just fine.

To reduce load on the MidasWiki mysql database, I now restricted access to most 
special pages to logged in users. It seems to have helped.

If this causes big difficulty or if you have suggestion on better ways to deal 
with aggressive AI scanner bots, please speak up.

One alternative is to put MidasWiki behind a generic login page with a well 
known password. Downside is it will block google and bing search engines, too.

K.O.

> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.

Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
and sends the numeric data as an ASCII string. The first four chars of the data is the name
of the MIDAS data bank, second number is the data timestamp in seconds.

LCRY 1234567 1.1 2.2 3.3

A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
labview component. Please contact him directly for more information.

I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).

K.O.

Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

Zaher

> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.
> 
> Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
> and sends the numeric data as an ASCII string. The first four chars of the data is the name
> of the MIDAS data bank, second number is the data timestamp in seconds.
> 
> LCRY 1234567 1.1 2.2 3.3
> 
> A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
> labview component. Please contact him directly for more information.
> 
> I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).
> 
> K.O.

> Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

https://bitbucket.org/expalpha/a2daq/src/alpha/src/felabview.cxx

this is code circa 2006, there are now better ways to do some of that coding.

if you want bidirectional communication with labview, read/write odb, etc, simplest is probably
to write the "mjserver" that talks midas json-rpc over plain tcp, without all the http/https
gunk you need to go through mhttpd.

K.O.

> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?

apologies for delayed response.

I saw the question, completely did not understand it, only now got around to figure out what is going on.

according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t

no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
API and they indeed are only used when running online.

correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().

you could also save your data in the module destructor (although good programming recommendation is to use 
destructor only for unavoidable things, like freeing memory, etc).

K.O.

> Hello,
> 
> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> thanks,
> Zaher

We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.

Stefan

> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
>

Our parallel experience with the CERN ALPHA anti-hydrogen experiment: they have developed a whole labview empire
to control the cryogenics, the magnets, the positron source, the anti-proton trap, anti-hydrogen trap, etc.

At some point there was a wall of monitors in the counting room - each labview computer controlled one or two things -
so there is very many computers and each had to have a monitor (and mouse and keyboard).

All the data from this labview empire is logged to MIDAS history via felabview and feGEM, and they use
the MIDAS history to look and monitor almost everything. Control is done via Labview and Labview
based FPGA sequencers (National Instruments PXI hardware, $$$$$).

This works reasonably well to publish several papers in Nature.

But not 100%:

1) difficulties with labview source control (cannot be trivially managed by git, I guess)
2) unending fight against Microsoft and CERN IT trying to reboot the computers at the wrong time
3) more recently, forced Microsoft updates require trashing perfectly good machines and buy new ones

At TRIUMF there is very little Labview. All experiments use MIDAS and EPICS for most things.

Based on this experience, I agree with Stefan, today's sweet spot is RaspberryPi machines with USB attached
gizmos to control stuff. On the software side, drive the mess with MIDAS and custom web pages.

K.O.

Thanks Stefan, I would be very interested to see your code. At moment we have magnets and cryostats (3He and dilution) being delivered with Labview.



> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
> 
> Stefan

Hi,
The question was about the TMFeRpcHandlerInterface, not the TARunObject interface.  Derived classes of TARunObject do indeed work as expected in our 
environment.  We have worked around the issue by using an implementation of TARunObject as well as the (separate) implementation of 
TMFeRpcHandlerInterface.

Thanks,

Mark.

> > I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> > Midas during online running.  At the end of each run it saves out custom data in the 
> > TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> > However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> > never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> > but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> > choice to ignore all of TMFeRpcHandlerInterface when offline?
> 
> apologies for delayed response.
> 
> I saw the question, completely did not understand it, only now got around to figure out what is going on.
> 
> according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
> always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
> running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t
> 
> no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
> API and they indeed are only used when running online.
> 
> correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().
> 
> you could also save your data in the module destructor (although good programming recommendation is to use 
> destructor only for unavoidable things, like freeing memory, etc).
> 
> K.O.

> The question was about the TMFeRpcHandlerInterface, not the TARunObject interface.  Derived classes of TARunObject do indeed work as expected in our 
> environment.  We have worked around the issue by using an implementation of TARunObject as well as the (separate) implementation of 
> TMFeRpcHandlerInterface.

then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.

maybe if you point me to your source code, I can see what you are doing?

K.O.

This was exactly the question, should I expect it to run?  There's no point in the HandleBinaryRpc method offline, but there's an argument that the HandleBeginRun/HandleEndRun methods have a use.
I have the answer and we have a workaround, thanks.

> then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
> picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.
> 
> maybe if you point me to your source code, I can see what you are doing?
> 
> K.O.

I do not understand what you are doing. If you are offline, there is no TMFE singleton instance,
there is nothing TMFeRpcHandlerInterface to attach to, there is nobody to call TMFeRpcHandlerInterface methods.

Maybe what you are asking for is a mode where you analyze data from a file, but you want your analysis code
to think that it is online and that it is analyzing live data. This requires creating a fake TMFE singleton, attaching
TMFeRpcHandlerInterface to this fake TMFE singleton and using ProcessMidasOnlineTmfe(), driven by all this fake stuff:
a fake OpenBuffer() that actually opens a file, a fake ReceiveEvent() that actually reads from a file, fake callbacks
for begin and end run, etc.

That's a lot of work, but for what purpose? What is it about the existing offline and online modes that you do not like
and how all this fake stuff will make it better for you?

P.S. This is a 3rd version of my reply. Wrote and deleted 2 version. I think I completely do not understand
what you are doing and you completely do not understand what I am saying. Communication is not happening.

P.P.S. Simplest if you show me your code (email, elog), I am quite good at reading code and divining what
people are trying to do. You do not have to show me any of your secret secret stuff.

K.O.

> This was exactly the question, should I expect it to run?  There's no point in the HandleBinaryRpc method offline, but there's an argument that the HandleBeginRun/HandleEndRun methods have a use.
> I have the answer and we have a workaround, thanks.
> 
> > then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
> > picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.
> > 
> > maybe if you point me to your source code, I can see what you are doing?
> > 
> > K.O.

Dear experts,
      in the CYGNO experiment, we readout CMOS cameras for optical readout of GEM-TPCs. So far, we only developed the readout for a single camera. In the future, we will have multiple cameras to read out (up to 18 in the next phase of the experiment), and we are investigating how to optimize the readout by exploiting parallelization.

One idea is to start parallel threads within a single equipment. Alternatively, one could associate different equipment with each camera and run an Event Builder. Perhaps other solutions did not come to mind. Which one would you regard as the most effective and elegant?

Thank you very much,
           Francesco

> Dear experts,
>       in the CYGNO experiment, we readout CMOS cameras for optical readout of GEM-TPCs. So far, we only developed the readout for a single camera. In the future, we will have multiple cameras to read out (up to 18 in the next phase of the experiment), and we are investigating how to optimize the readout by exploiting parallelization.
> 
> One idea is to start parallel threads within a single equipment. Alternatively, one could associate different equipment with each camera and run an Event Builder. Perhaps other solutions did not come to mind. Which one would you regard as the most effective and elegant?
> 
> Thank you very much,
>            Francesco

In principle both will work. It's kind of matter of taste. In the multi-threaded approach one has a single frontend to start and stop, and in the second case you have to start 18 individual frontends and make sure that they are running. 

For the multi-threaded frontend you have to ensure proper synchronization between the threads (like common run start/stop), and in the end you also have to do some event building, sending all 18 streams into a single buffer. As you know, multi-thread programming can be a bit of an art, using mutexes or semaphores, but it can be more flexible as the event builder which is a given piece of software.

Best,
Stefan

I am updating TWIST to the latest MIDAS and when I load a saved .odb file, I get
these messages. Their text ought to say where and what strings it does not like.
K.O.



[twistonl@midtwist ~/online]$ odbedit
Please define environment variable 'MIDASSYS'
pointing to the midas installation directory.
[local:twist:S]/>load /twist/data_onl/current/run17548.odb
[odb.c:5600:db_paste] found string exceeding MAX_STRING_LENGTH
[odb.c:5600:db_paste] found string exceeding MAX_STRING_LENGTH
[odb.c:5600:db_paste] found string exceeding MAX_STRING_LENGTH