> The format of .hst midas history files is pretty simple and mhdump.cxx is an easy to read 
> illustration on how to read it from basic principles (without going through the midas library, 
> which can be somewhat complicated). The newer "FILE" format for history is even simpler 
> to read because it is just fixed-record-size binary data prepended by a text header.
> 
> You can also use the mh2sql program to import history data into an sql database (mysql 
> and sqlite should work) or to convert .hst files to "FILE" format files. This works well
> for "archiving" history data, because the "FILE" format works better for looking at old data,
> and for looking at data in "months" or "years" timescale.
> 
> Back to your question, you can certainly use "mhdump" as is, using a pipe (popen()), or 
> you can package mhdump.cxx as a c++ class and use it in your application. If you go this 
> route, your contribution of such a c++ class back to midas would be very welcome.
> 
> You can also use mhist, but the mhist code cannot be trivially packaged as a c++ class
> to use in your application.
> 
> You can also suggest that we write an easier to use history utility, we are always open to 
> suggested improvements.
> 
> Let us know how it works out for you. Good luck!
> 
> K.O.

Dear Konstantin,
thank you very much for the wealth of information you provided.
I have thought about it and I see two options:

- One is to convert to SQL format and then use a SQLite library to import the data in my 
application.

- The other is to encapsulate the mhdump.cxx code into a C++ class, as you say.

I am leaning towards the first option for three reasons.
1. I have never used a SQLite database so it is a good learning opportunity for me.
2, The SQLite database format is very well known and widespread, so there are tons of tools to 
handle it
3. I have taken a look at the mhdump.cxx source code and I think it is a beautiful piece of code, 
but has a very "functional" taste with little encapsulation. Basically, all the fun is happening 
inside the readHstFile function and there is no trivial way to get the data out of it. I don't mean 
that it would be difficult to wrap it around a C++ class, but I feel that I can learn more by going 
the SQL way.

PS some time ago, I don't remember if you or Stefan, recommended CLion as C++ IDE. I have tried it 
(together with PyCharm) and I must admit that it is really good. It took me years to configure Emacs 
as a IDE, while it took me minutes to have much better results in CLion. Thank you very much for 
your recommendation.

> (But note that back when I implemented the SQLITE history writer, sqlite database corruption
> recovery instructions were "delete the file, restore from backup". And indeed in every test
> experiment I tried, the sqlite history databases eventually corrupted themselves. You see
> same thing with google-chrome, lots of sqlite errors (bad locking, corrupted table, etc)
> in it's terminal output).

Thank you for the info. But I do not quite understand the comment above.
Do you mean that there is something wrong with the SQLite library itself or with the way that MIDAS creates the SQLite 
database?

All this is very good news. I really wish this were available some months ago: it would have helped me immensely. The old C API was clunky at best.
I really like the idea and looking forward to using it (even if at the moment I do not have the need to) ...

Eventually, I have settled for the SQLite format.
I could convert the MIDAS history files .hst to SQLite
database .sqlite3 using the utility mh2sql.
It worked out nicely, thank you for the advice.

However, as Konstantine predicted I did notice some
database corruption when a couple of problematic .hst
files were read. I solved the issue by just deleting
those .hst files (I think they were empty anyway).

Now I am developing a piece of code to read the
database using the SOCI library and integrate it
into a TTree but this is not relevant for MIDAS I think.

Thank you again for the discussion.

> I’m beginner of Midas, and trying to develop the slow control front-end with the latest Midas.
> I found the scfe.cxx in the “example”, but not enough to refer to write the front-end for my own devices 
> because it contains only nulldevice and null bus driver case...
> (I could have succeeded to run the HV front-end for ISEG MPod, because there is the device driver...)
> 
> Can I get some frontend examples such as simple TCP/IP and/or RS232 devices?
> Hopefully, I would like to have examples of frontend and device driver.
> (if any device driver which is included in the package is similar, please tell me.)
> 
> Thanks a lot.

Dear Yoshida-san,
my name is Giorgio and I am a Ph.D. student working on the T2K experiment.
I had to write many MIDAS frontends recently, so I think that my code could be of some help to you.

As you might already know, the MIDAS slow control system is structured into three layers/levels.

 - The highest layer is the "class" layer that directly interfaces with the user and the ODB. It is called 
"class" layer because it refers to a class of devices (for example all the high voltage power supplies, 
etc...). The idea is that in the same experiment you can have many different models of power supplies but 
all of them can be controlled with a single class driver.

 - Then there is the "device" layer that implements the functions specific to the particular device.

 - Finally, there is the "BUS" layer that directly communicates with the device. The BUS can be Ethernet 
(TCP/IP), Serial (RS-232 / RS-422 / RS-485), USB, etc ...

You can read more about the MIDAS slow control system here:
https://midas.triumf.ca/MidasWiki/index.php/Slow_Control_System

Anyway, you need to write code for all those layers. If you are lucky you can reuse some of the already 
existing MIDAS code. Keep in mind that all the examples that you find in the MIDAS documentation and the 
MIDAS source code are written in C (even if it is then compiled with g++). But, you can write a frontend in 
C++ without any problem so choose whichever language you are familiar the most with.

I am attaching an archive with some sample code directly taken from our experiment. It is just a small 
fraction of the code not meant to be compilable. The code is disclosed with the GPL3 license, so you can use 
it as you please but if you do, please cite my name and the WAGASCI-T2K experiment somewhere visible.

In the archive, you can find two example frontends with the respective drivers. The "Triggers" frontend is 
written in C++ (or C+ if you consider that the mfe.cxx API is very C-like). The "WaterLevel" frontend is 
written in plain C. The "Triggers" frontend controls our trigger board called CCC and the "WaterLevel" 
frontend controls our water level sensors called PicoLog 1012. They share a custom implementation of the 
TCP/IP bus. Anyway, this is not relevant to you. You may just want to take a look at the code structure.

Finally, recently there have been some very interesting developments regarding the ODB C++ API. I would 
definitely take a look at that. I wish I had that when I was developing these frontends.

Good luck

--
Pintaudi Giorgio, Ph.D. student
Neutrino and Particle Physics Minamino Laboratory
Faculty of Science and Engineering, Yokohama National University
giorgio.pintaudi.kx@ynu.jp
TEL +81(0)45-339-4182

> Something must be wrong on your side. If you take the example frontend under
> 
> midas/examples/experiment/frontend.cxx
> 
> and let it run to produce dummy events, you get about 90 Hz. This is because we have a
> 
>   ss_sleep(10);
> 
> in the read_trigger_event() routine to throttle things down. If you remove that sleep, 
> you get an event rate of about 500'000 Hz. So the framework is really quick.
> 
> Probably your routine which looks for a 'lam' takes really long and should be fixed.
> 
> Stefan

Sorry if I am going off-topic but, because the ss_sleep function was mentioned here, I 
would like to take the chance and report an issue that I am having.

In all my slow control frontends, the CPU usage for each frontend is close to 100%. This 
means that each frontend is monopolizing a single core. When I did some profiling, I 
noticed that 99% of the time is spent inside the ss_sleep function. Now, I would expect 
that the ss_sleep function should not require any CPU usage at all or very little.

So my two questions are:
Is this a bug or a feature?
Would you able to check/reproduce this behavior or do you need additional info from my 
side?

Hello!
Sometimes when I mess around with the history plots I get the following error:

[mhttpd,ERROR] [history.cxx:97:xread,ERROR] Error: Unexpected end-of-file when 
reading file "/home/wagasci-ana/Data/online/210219.hst"

I have tried the following without success:

- Remove the MIDAS history files
- Restart mhttpd and mlogger

I do not know what triggers the error but when it triggers the above message is 
printed hundres of times a second, completely spamming the message log.

It happened again today after I set the label of a frontend too long making 
mlogger crash. After fixing the label length, the above message appeared and it 
does not seem to go away.

It appears that the issue is trigger by a nonexisting Event and Variable as shown 
in the attached picture. This issue can arise when restoring the ODB from a 
previous version or importing ODB values from other MIDAS instances.
It might be useful if the error message were more clear about the source of the 
problem.

> Hello!
> Sometimes when I mess around with the history plots I get the following error:
> 
> [mhttpd,ERROR] [history.cxx:97:xread,ERROR] Error: Unexpected end-of-file when 
> reading file "/home/wagasci-ana/Data/online/210219.hst"
> 
> I have tried the following without success:
> 
> - Remove the MIDAS history files
> - Restart mhttpd and mlogger
> 
> I do not know what triggers the error but when it triggers the above message is 
> printed hundres of times a second, completely spamming the message log.
> 
> It happened again today after I set the label of a frontend too long making 
> mlogger crash. After fixing the label length, the above message appeared and it 
> does not seem to go away.

Before we try to merge the different access scheme for the different VME hardware,
we present the "optimal" configuration for the VMIC setup. This is a first shot so take it
with caution.
From these definitions, we should be able to workout a compromise and come up with
a satisfactory standard.

A) The VMIC vme_slave_xxx() options are not considered.
B) The interrupt handling can certainly match the 4 entries required in the user frontend
    code i.e. Attach, Detach, Enable, Disable.

I don't understand your argument that the handle should be hidden. In case of multiple
interfaces, how do you refer to a particular one if not specified? 
The following scheme does require a handle for refering to the proper (device AND window).

1 ) deviceHandle = vme_init(int devNumber);
    Even though the VMIC doesn't deal with multiple devices,
    the SIS/PCI does and needs to init on a specific PCI card.
    Internally:
      opening of the device (/dev/sisxxxx_1) (ignored in case of VMIC).
      Possible including a mapping to a default VME region of default size with default AM
      (VMIC :16MB, A24). This way in a single call you get a valid handle for full VME access
      in A24 mode. Needs to be elaborate this option. But in principle you need to declare the 
     VME region that you want to work on (vme_map).

2) mapHandle = vme_map(int deviceHandle, int vmeAddress, int am, int size);
    Return a mapHandle specific to a device and window. The am has to be specified.
    What ever are the operation to get there, the mapHandle is a reference to thas setting.
    It could just fill a map structure.
    Internally:
      WindowHandle[deviceHandle] = vme_master_create(BusHandle[deviceHandle], ...
      WindowPtr[WindowHandle] = vme_master_window(BusHandle[deviceHandle]
                                                                           , WindowHandle[deviceHandle]...

3) vme_setmode(mapHandle, const int DATA_SIZE, const int AM
                           , const BOOL ENA_DMA, const BOOL ENA_FIFO);
    Mainly used for the vme_block_read/write. Define for following read the data size and 
    am in case of DMA (could use orther DMA mode than window definition for optimal
    transfer).

    Predefine the mode of access:
    DATA_SIZE : D8, D16, D32
    AM             : A16, A24, A32, etc...
    enaDMA     : optional if available.
    enaFIFO     : optional for block read for autoincrement source pointer.

Remark:
PAA- I can imagine this function to be a vme_ioctl (int mapHandle, int *param)
        such that extension of functionality is possible. But by passing cons int
        arguments, the optimizer is able to substitute and reduce the internal code.

4)   
   uint_8Value   = vme_readD8  (int mapHandle, uint_64 vmeSrceOffset)
   uint_16Value = vme_readD16 (int mapHandle, uint_64 vmeSrceOffset)
   uint_32Value = vme_readD32 (int mapHandle, uint_64 vmeSrceOffset)
   Single VME read access. In the VMIC case, this access is always through mapping.
   Value = *(WindowPtr[WindowHandle] + vmeSrceOffset) 
   or 
   Value = *(WindowStruct->WindowPtr[WindowHandle] + vmeSrceOffset) 
 
5)   
   status  = vme_writeD8   (int mapHandle, uint_64 vmeSrceOffset, uint_8 Value)
   status  = vme_writeD16 (int mapHandle, uint_64 vmeSrceOffset, uint_16 Value)
   status  = vme_writeD32 (int mapHandle, uint_64 vmeSrceOffset, uint_32 Value)
   Single VME write access.

6)
   nBytes = vme_block_read(mapHandle, char * pDest, uint_64 vmeSrceOffset, int size);
   Multiple read access. Can be done through standard do loop or DMA if available.
   nBytes < 0 :  error
   Incremented pDest  = (pDest + nBytes); Don't need to pass **pDest for autoincrement.

7)
   nBytes = vme_block_write(mapHandle, uint_64 vmeSrceOffset, char *pSrce, int size);
   Multiple write access.
   nBytes < 0 :  error
   Incremented pSrce  = (pSrce + nBytes); Don't need to pass **pSrce for autoincrement.

8) status = vme_unmap(int mapHandle)
   Cleanup internal pointers or structure of given mapHandle only.

9) status = vme_exit()
   Cleanup deviceHandle and release device.

Dear Midas users,

As you certainly aware, ELOG (Electronic Logbook) has been written
by Stefan Ritt and its functionality is part of the Midas package too.
This web site using Elog is replacing the W-Agora Forum previously setup.

You will need to register to this forum in order to gain Write access and 
possible Email notification.

We would like to encourage you to post your questions or comments at
this Midas Elog site instead of using private Email to the authors as your 
remarks are surely of interest to the other users too.

 
 

- remove ss_tape_get_blockn from lazylogger.c
- add ss_tape_get_blockn to system.c
- add ss_tape_get_blockn prototype into midas.h
- fix buffer size for "dir" in mtape.c
- add block# for "dir" in mtape if command successful.
- handle TID_STRUCT bank type by display as 8bit in ybos.c (mdump)

The current and future situation of the Midas analyzer is summarized in the
attachment below.

Box explanation:
================
Front end:
---------
Midas code for accessing/gathering the hardware information into the Midas
format.

Midas SHM:
---------
Midas back end shared memory where the front end data are sent to.

mlogger:
-------
Data logger collecting the midas events and storing them on a physical
logging device (Disk, Tape)

Midas Analyzer:
--------------
Midas client for event-by-event analysis. Incoming data can be either online
or offline.

mserver:
-------
Subprocess interfacing external (remote) midas client to the centralized
data collection and database system.

PAW:
---
Standalone physics data analyzer (CERN).

ROOT:
----
Standalone Physics data analyser (CERN).


This diagram represents the data path from the Frontend to the analyzer in
online and offline mode. Each data path is annoted with a circled number
discussed below. In all cases, the data will flow from the front end
application to the midas back end data buffers which reside in a specific
share memory for a given experiment.

Path:
(1): From the shared memory, the midas analyzer can request events directly
and process them for output to divers destination.

(2): The data logger is a specific application which stores all the data to
 a storage media such as a disk or tape. This path is specific to the
creation of file.mid file format. The actual storage file in this .mid
format can be readout later on by the midas analyzer.

(3): The Midas analyzer has been developed originally for interfacing to the
PAW analyzer which uses its own shared memory segment for online display.
The analyzer can also save the data into a specific data format consistent
with PAW (HBOOK and Ntuples, extension .rz).

(4): Presently the data logger support a creation of the ROOT file format.
This file contains in the form of a Tree the midas event-by-event data. This
file is fully compatible with ROOT and therefore can be read out by the
standard ROOT application.

(5): Equivalent to the data logger, the analyzer receiving from the data
buffer or reading from a .mid file data can apply an event-by-event analysis
and on request produce a compliant ROOT file for further analysis. This
.root file can be composed of Trees as well as histograms.

(6): The possibility of ONLINE ROOT analysis has been implemented in a first
stage through the TMapFile (ROOT shared memory). While this configuration is
still in use an experiment, the intention is to deprecate it and replace it
with the data path (7).

(7): This path uses the network socket channel to transfer data out of the
analyzer to the ROOT environment. The current analyzer has a limited support
for ROOT analysis by only publishing on request the Midas analysis built in
histograms. No mean is yet implemented for Tree passing mechanism.

(8): The pass has not been yet investigated, but ROOT does provide
accessibility to external function calls which makes this option possible.
The ROOT framework will then perform dedicated event call to the main midas
data buffer using the standard midas communication scheme. The data format
translation from Midas banks to ROOT format will have to be taken care at
the user level in the ROOT environment.


Discussion:
==========
Presently the Socket communication between Midas and ROOT (7) is under
revision by Stefan Ritt and René Brun. This revision will simplify the
remote access of an object such as an histogram. For the Tree itself, the
requirement would be to implement a "ring buffer" mechanism for remote tree
request. This is currently under discussion.

The path (8) has been suggested by Triumf to address small experiment setup
where only a single analyzer is required. This path minimize the DAQ
requirements by moving all the data analysis handling to the user.
The same ROOT analysis code would be applicable to a ONLINE as well as
OFFLINE analysis.

Cons:
- Necessity of publishing raw data through the network for every instance of
the remote analyzer.
- Result sharing of the analysis cannot be done yet in real time.

Pros:
- No need of extra task for data translation (midas/root).
- Unique data unpacking code part of the user code.
- Less CPU requirement.

Other issues:
============
- The current necessity of the Midas shared memory for the midas analyzer to
run is a concern in particular for offline analysis where a priori no midas
is available. 

- The handling of the run/analyzer parameters. Possible parameter extraction
from file.odb.

- add pthread lib to examples/... makefile
- fix ybos_simfe.c for max_event_size
- fix camacnul.c for cam_inhibit_test(), cam_interrupt_test()
- update documentation (1.9.3)
- made midas-1.9.3-1.tar.gz on Triumf site

1) Prior upgrading midas to 1.9.3, make sure you've saved your ODB in ASCII
   format using "odbedit> save my_odb.odb", as the internal structure is
   incompatible with previous version. You will be able to restore it once
   the new odb is up using "odbedit> load my_odb.odb".

2) since version 1.9.2, the analyzer supports ROOT and PAW packages.
   The general Midas makefile build the analyzer core system mana.c
   differently depending on presence of the environment variable $ROOTSYS.

   In the case $ROOTSYS is not defined, the Makefile will create:
   ~/os/lib/mana.o, build for NO HBOOK calls.
   ~/os/lib/hmana.o, build with HBOOK calls for PAW analyzer
    (requires /cern/pro/lib to be present).

   In the case $ROOTSYS is defined and pointing to a valid root directory:
   ~/os/lib/mana.o, build for NO HBOOK calls.
   ~/os/lib/rmana.o, build for ROOT analyzer.

3) Since 1.9.2, the ~/examples/experiment contains the ROOT
   analyzer example instead of HBOOK. The local Makefile uses the source
   examples and the ~/os/lib/rmana.o for building the final user
   application.
   
   The previous HBOOK(PAW) analyzer has been moved into ~examples/hbookexpt
   directory. The analyzer is build using the ~/os/lib/hmana.o 

4) A new application "rmidas" is available when the system is build with
   ROOT support. This application is an initial "pure" ROOT GUI implementing
   TSocket for remote ROOT histogram display. 
   Once a ONLINE ROOT analyzer is up and running, by invoking "rmidas"
   you will be prompt for a host name. Enter the node name hosting the
   analyzer. You will be presented with a list of histogram which can
   be display in a ROOT frame environment (see attachment). 

5) The support of ROOT is also available for the logger by changing  
   the data format and the destination file name in the ODB structure.
   This option will save on file the Midas banks converted into ROOT Tree.
   This file can be opened with ROOT (see attachment).

------- ODB structure of /Logger/Channels/0/Settings
   [local:midas:R]Settings>ls
    Active                          y
    Type                            Disk
    Filename                        run%05d.root    <<<<<<<<< new extension
    Format                          ROOT            <<<<<<<<< new format
    Compression                     0
    ODB dump                        y
    Log messages                    0
    Buffer                          SYSTEM
    Event ID                        -1
    Trigger mask                    -1
    Event limit                     0
    Byte limit                      0
    Tape capacity                   0
    Subdir format                   
    Current filename                run00211.root
-------   

.

- makefile.nt (/examples/experiment, /hbook)
  adjusted for local hmana.obj build  as for rmana.obj, add cvs tag for
  revision comment entry.
- drivers/class/hv.c
  change comment // to /* */

- Remove temporary "/Programs/Lazy" creation.
- Fix Rate calculation for Web display.
- Change FTP channel description (see help).

The current Triumf DAQ standard (Midas) since the second quarter of this
year (2003) has the capability to deal with ROOT histograms. The internal
midas logger can save data files in ROOT format and the analyzer can book
and fill ROOT histograms. These features triggered a new project started
during summer 2003 for building a Triumf GUI ROOT/Midas display utility.

The initial requirements for this utility are:
1) Solely based on ROOT (VirtualX, no Qt)
2) Similar overall functionality than PAW.
   - Open concurrent ROOT files.
   - Open connection to a single Midas Online experiment (requires analyzer
                                                          as server)
   - Optional Auto-update in ONLINE mode.
   - Zoning, Zooming option display.
   - Simple Historgram gaphic manipulation. (based on current ROOT
                                             implementation)
   - Tree manipulation ( use of TBrowser())
   - Simple user script invocation.
   - Optional experiment specific customization.
3) Session configuration save/restore option.

An initial version has been developed and currently is under evaluation.
Improvement and further development will based on the local experimenters
responses. 

This utility will be available for external use around the second quarter of
the 2004 at the latest.

.

> Is the midas CVS server set-up so that I can pull the newest 
> version off the CVS server?
> 
> What would be my CVSROOT?
> pserver:anoncvs@midas.psi.ch:/cvs/midas *this did not work* :)
> 
> Piotr
> 
>  

In the Midas doc under "Quick Start"
http://midas.triumf.ca/doc/html/quickstart.html
you will find the proper cvs command for accessing the latest cvs Midas
version. The public pwd is cvs. You will only be able to checkout/update the
package.