Hi K.O., your clarification is much appreciated! 
"
> I am not sure what you are trying to do. It is always easier to suggest a solution to a specific problem.

I think, I owe you an explanation :) :

Consider ~ 40 nodes with two FPGAs (PCIE cards) per node, talking to the detector hardware. 
One of those FPGAs, in addition to reading the data, performs the global timing synchronization.
The high-bandwidth data readout is not controlled by MIDAS, so all frontends perform only 'slow control'-type functions.
In MIDAS language, an FPGA implements two different units of slow control equipment: 
one - configuring and controlling a single FPGA (equipment type A), and another one - synchronizing 
multiple FPGAs (equipment type B). On one of the nodes, unit A and unit B share the FPGA card, 
so they better be controlled by the same frontend. 

For one, I need to make sure that all type A equipment units, managed by multiple frontends, 
are initialized before the [single] type B unit which shares the frontend with the type A unit. 
And, of course, the end of a run transition has to be handled in the opposite order - type B unit 
shuts down first. 

As 'periodic' actions for all registered pieces of equipment are performed in the same loop [thread], 
registering the equipment in the needed order - first A, then B - should give a solution - thanks for making that clear.     

> 
> 1) "time ordering of run transitions" - of course midas transitions are ordered by transition sequence numbers 
> and the tmfe class provides methods to control this. ditto for the mfe.cxx frontends.
> 
> 2) for one TMFrontend, the order of calling HandleBeginRun() is the order in which equipments were added to the 
> equipment using FeAddEquipment(). HandleEndRun() is called in reverse order. (I better check this).

the ordering of the rpc handler calls in tmfe's tr_stop/tr_pause/tr_resume functions is ok.

> 
> 3) to have multiple TMFrontends in one program would be unusual (mfe.cxx frontends completely do not support 
> this), but should work. Everything was coded to support this, but it was never tested in practice because we 
> cannot invent any useful use-case for it. HandleBeginRun() handlers are likely to be called in the frontends are 
> created. (I could check this and confirm it works, as long as you have a valid use-case for this configuration).

agreed, I don't think there is a good use case for that, so no need to spend time checking.

> 
> 4) Frontend X has EquipmentA and EquipmentB, you want EqA::HandleBeginRun() to be called at run transition 200 
> and EqB::HandleBeginRun() to be called at run transition 400.
> 
> This is not directly supported by mfe.cxx frontends (the begin_run() handler is a global function) and I did not 
> directly implement it in the TMFE frontend.
> 
> But I think this would be a useful improvement. I will look into this. 

In the simplest case, registering the equipment units in the right order is definitely the answer. 
However a single FPGA can perform multiple logically independent tasks and thus represent 
multiple logical units of equipment. Those units however are not independent: they share the hardware (FPGA) 
and thus do depend on each other. Giving users a full control over the sequence in which those logical units 
execute their run transitions is quite likely to be needed, for example, to work around peculiarities of the 
custom-made kernel drivers.
 
> 
> Likely I will add per-equipment data members fEqConfBeginRunSeqNo, fEqConfEndRunSeqno, etc. Value 0 would 
> unregister the corresponding run transition handler. This would cleanup the code quite a bit, a bunch
> of RegisterTranstionXXX functions could go away.

this also makes sense. -- thanks again, regards, Pasha

> 
> K.O.

Hi Pavel,

have you looked into 

  cm_set_transition_sequence()

which let's you define the sequence number for every midas client. You give any number between 1 and 1000 (default is 500 for frontends I believe).

The default value of 500 is defined in mfe.cxx:2641 where you have 500 for all four transitions, but it can be overwritten in the frontend_init function via 
cm_set_transition_sequence(). Since you have separate values for the start and stop transition, you can get the different sequencing for both transitions as you need. Like set 
all type A to 400, type B to 600 for TR_START, and set type A to 600 and type B to 400 for TR_STOP.

Since this works on the midas client level, it should also work for the tmfe.cxx framework. I'm however not sure if you have a similar default of 500 as in mfe.cxx:2641. But K.O. 
should know.

Best,
Stefan 

Dear MIDAS experts,

I'm using MIDAS javascript interface (mjsonrpc) to communicate with a frontend from a custom web page 
and observe that the if the frontend's response exceeds certain number of bytes, it is always truncated. 

MIDAS C/C++ RPC interface allows users to specify the max response length :

https://daq00.triumf.ca/MidasWiki/index.php/Remote_Procedure_Calls_(RPC)#C++_2

How would one do the same from with mjsonrpc ? 

-- many thanks, regards, Pasha

> I'm using MIDAS javascript interface (mjsonrpc) to communicate with a frontend from a custom web page 
> and observe that the if the frontend's response exceeds certain number of bytes, it is always truncated. 
> 
> MIDAS C/C++ RPC interface allows users to specify the max response length :
> 
> https://daq00.triumf.ca/MidasWiki/index.php/Remote_Procedure_Calls_(RPC)#C++_2
> 
> How would one do the same from with mjsonrpc ? 

I just documented the max_reply_length (javascript) and max_len (python) parameters on that page. Both are optional and default to 1024 bytes.

I also added a link to the full mjsonrpc schema https://daq00.triumf.ca/MidasWiki/index.php/Mjsonrpc#Schema_(List_of_all_RPC_methods) . You can also find the auto-generated schema on any midas installation by going to the "Help" webpage served by mhttpd and clicking the "JSON-RPC schema" > "text table format" link.

Dear MIDAS experts,

I have a time series of slow control measurements in an ASCII format - 
data records in a format (run_number, time, temperature, voltage1, ..., voltageN), 
and, if possible, would like to convert them into a MIDAS history format. 

Making MIDAS events out of that data is easy, but is it possible to preserve 
the time stamps?  - Logically, this boils down to whether it is possible to have  
the event time set by a user frontend

-- as always - many thanks, regards, Pasha

I think I found an answer to my question: a user-controlled event header does have a time stamp: 

https://daq00.triumf.ca/MidasWiki/index.php/Event_Structure#Event_Header

-- apologies for the spam, regards, Pasha

Hi,

we are planning to migrate from mana to manalyzer. I started to have a look into it and realized that I have some lose ends.
Is there a clear migration docu somewhere?

Currently I understand it the following way (which might be wrong):
The class TARunObject is used to write analyzer modules which are registered by TAFactory. I hope this is right?

However, in mana there is an analyzer implemented by the user which binds the modules and has additional routines:
analyzer_init(), analyzer_exit(), analyzer_loop()
ana_begin_of_run(), ana_end_of_run(), ana_pause_run(), ana_resume_run()
which we are using.

This part I somehow miss in manalyzer, most probably due to lack of understanding, and missing documentation.

Could somebody please give me a boost?

Hi Andreas,

please find in elog:2938/1 a short introduction that I wrote sometime ago.
I'm glad to offer additional support, if needed.

> Hi,
> 
> we are planning to migrate from mana to manalyzer. I started to have a look into it and realized that I have some lose ends.
> Is there a clear migration docu somewhere?
> 
> Currently I understand it the following way (which might be wrong):
> The class TARunObject is used to write analyzer modules which are registered by TAFactory. I hope this is right?
> 
> However, in mana there is an analyzer implemented by the user which binds the modules and has additional routines:
> analyzer_init(), analyzer_exit(), analyzer_loop()
> ana_begin_of_run(), ana_end_of_run(), ana_pause_run(), ana_resume_run()
> which we are using.
> 
> This part I somehow miss in manalyzer, most probably due to lack of understanding, and missing documentation.
> 
> Could somebody please give me a boost?

> Could somebody please give me a boost?

no need to shout into the void, it is pretty easy to identify the author of manalyzer and ask me directly.

> we are planning to migrate from mana to manalyzer. I started to have a look into it and realized that I have some lose ends.
> Is there a clear migration docu somewhere?

README.md and examples in the manalyzer git repository.

If something is missing, or unclear, please ask.

> Currently I understand it the following way (which might be wrong):
> The class TARunObject is used to write analyzer modules which are registered by TAFactory. I hope this is right?

Please read the README file. It explains what is going on there.

Design of manalyzer had 2 main goals:
a) lifetime of all c++ objects (and ROOT objects) is well defined (to fix a design defect in rootana)
b) event flow and data flow are documentable (problem in mfe.c frontends, etc)

> However, in mana there is an analyzer implemented by the user which binds the modules and has additional routines:
> analyzer_init(), analyzer_exit(), analyzer_loop()
> ana_begin_of_run(), ana_end_of_run(), ana_pause_run(), ana_resume_run()
> which we are using.

I have never used the mana analyzer, I wrote the c++ rootana analyzer very early on (first commit in 2006).

But the basic steps should all be there for you:
- initialization (create histograms, open files) can be done in the module constructor or in BeginRun()
- finalization (fit histograms, close files) should be done in EndRun()
- event processing (obviously) in Analyze()
- pause run, resume run and switch to next subrun file have corresponding methods
- all the "flow" and multithreading stuff you can ignore to first order.

To start the migration, I recommend you take manalyzer_example_root.cxx and start stuffing it with your code.

If you run into any problems, I am happy to help with them. Ask here or contact me directly.

> This part I somehow miss in manalyzer, most probably due to lack of understanding, and missing documentation.

True, I wrote a migration guide for the frontend mfe.c to c++ tmfe, because we do this migration
quite often. But I never wrote a migration guide from mana.c analyzer, because we never did such
migration. Most experiments at TRIUMF are post-2006 and use rootana in it's different incarnations.

P.S. I designed the C++ TMFe frontend after manalyzer and I think it came out quite better, I especially
value the design input from Stefan, Thomas, Pierre, Joseph and Ben.

P.P.S. Be free to ignore all this manalyzer business and write your own analyzer based
on the midasio library:

int main()
{
   TMReaderInteraface* f = TMNewReader(file.mid.gz");
   while (1) {
      TMEvent* e = TMReadEvent(f);
      dwim(e);
      delete e;
   }
   delete f;
}

For online processing I use the TMFe class, it has enough bits to be a frontend and an analyzer,
or you can use the older TMidasOnline from rootana.

Access to ODB is via the mvodb library, which is new in midas, but has been part of rootana
and my frontend toolkit since at least 2011 or earlier, inspired by Peter Green's even
older "myload" ODB access library.

K.O.

> > Is there a clear migration docu somewhere?

I can also give you links to the alpha-g analyzer (very complex) and the DarkLight trigger TDC analyzer (very simple),
there is also analyzer examples of in-between complexity.

K.O.

Hi,
I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
Midas during online running.  At the end of each run it saves out custom data in the 
TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
choice to ignore all of TMFeRpcHandlerInterface when offline?

Thanks,

Mark.

Hi,

Sorry, we have been slammed with a couple projects on the TRIUMF side in the past weeks and haven't found time for a 
response.  I am hopeful that we will be able to answer this question (and the cache size question) within the next 10 
days.  

Again apologies,
Thomas

> Hi,
> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?
> 
> Thanks,
> 
> Mark.

Hello,

Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

thanks,
Zaher

We have a problem with over aggressive AI bots. They are scanning everything and 
are putting a crazy load on the mediawiki mysql database. This makes all out 
wikis very slow and unresponsive, which is a big problem.

These AI scanners do not seem to know what they are doing and are trying to load 
all the special pages, like "what links here" and "recent changes" and complete 
revision histories for each page. I am not impressed. Old school google and bing 
scanner bots are much more respectful and do not cause problems.

AI bots are also scanning this elog, and to Stefan's credit elogd is holding the 
load just fine.

To reduce load on the MidasWiki mysql database, I now restricted access to most 
special pages to logged in users. It seems to have helped.

If this causes big difficulty or if you have suggestion on better ways to deal 
with aggressive AI scanner bots, please speak up.

One alternative is to put MidasWiki behind a generic login page with a well 
known password. Downside is it will block google and bing search engines, too.

K.O.

> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.

Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
and sends the numeric data as an ASCII string. The first four chars of the data is the name
of the MIDAS data bank, second number is the data timestamp in seconds.

LCRY 1234567 1.1 2.2 3.3

A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
labview component. Please contact him directly for more information.

I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).

K.O.

Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

Zaher

> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.
> 
> Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
> and sends the numeric data as an ASCII string. The first four chars of the data is the name
> of the MIDAS data bank, second number is the data timestamp in seconds.
> 
> LCRY 1234567 1.1 2.2 3.3
> 
> A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
> labview component. Please contact him directly for more information.
> 
> I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).
> 
> K.O.

> Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

https://bitbucket.org/expalpha/a2daq/src/alpha/src/felabview.cxx

this is code circa 2006, there are now better ways to do some of that coding.

if you want bidirectional communication with labview, read/write odb, etc, simplest is probably
to write the "mjserver" that talks midas json-rpc over plain tcp, without all the http/https
gunk you need to go through mhttpd.

K.O.

> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?

apologies for delayed response.

I saw the question, completely did not understand it, only now got around to figure out what is going on.

according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t

no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
API and they indeed are only used when running online.

correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().

you could also save your data in the module destructor (although good programming recommendation is to use 
destructor only for unavoidable things, like freeing memory, etc).

K.O.

> Hello,
> 
> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> thanks,
> Zaher

We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.

Stefan

> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
>

Our parallel experience with the CERN ALPHA anti-hydrogen experiment: they have developed a whole labview empire
to control the cryogenics, the magnets, the positron source, the anti-proton trap, anti-hydrogen trap, etc.

At some point there was a wall of monitors in the counting room - each labview computer controlled one or two things -
so there is very many computers and each had to have a monitor (and mouse and keyboard).

All the data from this labview empire is logged to MIDAS history via felabview and feGEM, and they use
the MIDAS history to look and monitor almost everything. Control is done via Labview and Labview
based FPGA sequencers (National Instruments PXI hardware, $$$$$).

This works reasonably well to publish several papers in Nature.

But not 100%:

1) difficulties with labview source control (cannot be trivially managed by git, I guess)
2) unending fight against Microsoft and CERN IT trying to reboot the computers at the wrong time
3) more recently, forced Microsoft updates require trashing perfectly good machines and buy new ones

At TRIUMF there is very little Labview. All experiments use MIDAS and EPICS for most things.

Based on this experience, I agree with Stefan, today's sweet spot is RaspberryPi machines with USB attached
gizmos to control stuff. On the software side, drive the mess with MIDAS and custom web pages.

K.O.