ELOG Midas

Back Midas Rome Roody Rootana

Midas DAQ System, Page 140 of 144

Not logged in

Find | Login | Help

Full | Summary | Threaded | Hide attachments

2862 Entries

Goto page Previous 1, 2, 3 ... 139, 140, 141, 142, 143, 144 Next

Add support for rtsp camera streams in mlogger (history_image.cxx)

mlogger (history_image) now supports rtsp cameras, in ALPHA we have 
acquisitioned several new network connected cameras. Unfortunately they dont 
have a way of just capturing a single frame using libcurl


========================================
Motivation to link to OpenCV libraries
========================================

After looking at the ffmpeg libraries, it seemed non trivial to use them to 
listen to a rtsp stream and write a series of jpgs.

OpenCV became an obvious choice (it is itself linked to ffmpeg and 
gstreamer), its a popular, multiplatform, open source library that's easy to 
use. It is available in the default package managers in centos 7 and ubuntu 
(an is installed by default on lxplus).

========================================
How it works:
========================================

The framework laid out in history_image.cxx is great. A separate thread is 
dedicated for each camera. This is continued with the rtsp support, using 
the same periodicity:

if (ss_time() >= o["Last fetch"] + o["Period"]) {
An rtsp camera is detected by its URL, if the URL starts with ‘rtsp://’ its 
obvious its using the rtsp protocol and the cv::VideoCapture object is 
created (line 147).

If the connection fails, it will continue to retry, but only send an error 
message on the first 10 attempts (line 150). This counter is reset on 
successful connection
If MIDAS has been built without OpenCV, mlogger will send an error message 
that OpenCV is required if a rtsp URL is given (line 166)
The VideoCapture ‘stays live' and will grab frames from the camera based on 
the sleep, saving to file based on the Period set in the ODB.

If the VideoCapture object is unable to grab a frame, it will release() the 
camera, send an error message to MIDAS, then destroy itself, and create a 
new version (this destroy and create fully resets the connection to a 
camera, required if its on flaky wifi)
If the VideoCapture gets an empty frame, it also follows the same reset 
steps.
If the VideoCaption fills a cv::Frame object successfully, the image is 
saved to disk in the same way as the curl tools.

========================================
Concerns for the future:
========================================

VideoCapture is decoding the video stream in the background, allowing us to 
grab frames at will. This is nice as we can be pretty agnostic to the video 
format in the stream (I tested with h264 from a TP-LINK TAPO C100, but the 
CPU usage is not negligible.

I noticed that this used ~2% of the CPU time on an intel i7-4770 CPU, given 
enough cameras this is considerable. In ALPHA, I have been testing with 10 
cameras:
elog:2220/1 

My suggestion / request would be to move the camera management out of 
mlogger and into a new program (mcamera?), so that users can choose to off 
load the CPU load to another system (I understand the OpenCV will use GPU 
decoders if available also, which can also lighten the CPU load).

Attachment 1: unnamed.png

Add support for rtsp camera streams in mlogger (history_image.cxx)

> mlogger (history_image) now supports rtsp cameras

my goodness, we will drive the video surveillance industry out of business.

> My suggestion / request would be to move the camera management out of 
> mlogger and into a new program (mcamera?), so that users can choose to off 
> load the CPU load to another system (I understand the OpenCV will use GPU 
> decoders if available also, which can also lighten the CPU load).

every 2 years I itch to separate mlogger into two parts - data logger
and history logger.

but then I remember that the "I" in MIDAS stands for "integrated",
and "M" stands for "maximum" and I say, "nah..."

(I guess we are not maximum integrated enough to have mhttpd, mserver
and mlogger to be one monolithic executable).

There is also a line of thinking that mlogger should remain single-threaded
for maximum reliability and ease of debugging. So if we keep adding multithreaded
stuff to it, perhaps it should be split-apart after all. (anything that makes
the size of mlogger.cxx smaller is a good thing, imo).

K.O.

316

27 Dec 2006

Eric-Olivier LE BIGOT

Forum

Access to out_info from mana.c

Hello,

Is it possible to access out_info (defined in mana.c) from another program?

In fact, out_info is now defined as an (anonymous) "static struct" in mana.c,
which it seems to me precludes any direct use in another program.  Is there an
indirect way of getting ahold of out_info?  or of the information it contains?

out_info used to be defined as a *non-static* struct, and the code I'm currently
modifying used to compile seamlessly: it now stops the compilation during
linking time, as out_info is now static and the program I have to compile
contains an "extern struct {} out_info".

Any help would be much appreciated!  I searched in vain in this forum for
details about out_info and I really need to access the information it contains!

EOL (a pure MIDAS novice)

317

05 Jan 2007

Eric-Olivier LE BIGOT

Suggestion

Access to out_info from mana.c

Would it be relevant to transform out_info into a *non-static* variable of a type
defined by a *named* struct?
Currently,  programs that  try to access out_info cannot do it anymore; and they
typically copy the struct definition from mana.c, which is not robust against future
changes in mana.c.

If mana.c could be changed in the way described above, that would be great . 
Otherwise, is it safe to patch it myself for local use?  or is there a better way of
accessing out_info from mana.c?

As always, any help would be much appreciated :)

EOL

> Hello,
> 
> Is it possible to access out_info (defined in mana.c) from another program?
> 
> In fact, out_info is now defined as an (anonymous) "static struct" in mana.c,
> which it seems to me precludes any direct use in another program.  Is there an
> indirect way of getting ahold of out_info?  or of the information it contains?
> 
> out_info used to be defined as a *non-static* struct, and the code I'm currently
> modifying used to compile seamlessly: it now stops the compilation during
> linking time, as out_info is now static and the program I have to compile
> contains an "extern struct {} out_info".
> 
> Any help would be much appreciated!  I searched in vain in this forum for
> details about out_info and I really need to access the information it contains!
> 
> EOL (a pure MIDAS novice)

Access to out_info from mana.c

I changed out_info into a global structure definition ANA_OUTPUT_INFO and put it into
midas.h, so it can be accessed easily from the user analyzer source code.

> Would it be relevant to transform out_info into a *non-static* variable of a type
> defined by a *named* struct?
> Currently,  programs that  try to access out_info cannot do it anymore; and they
> typically copy the struct definition from mana.c, which is not robust against future
> changes in mana.c.
> 
> If mana.c could be changed in the way described above, that would be great . 
> Otherwise, is it safe to patch it myself for local use?  or is there a better way of
> accessing out_info from mana.c?
> 
> As always, any help would be much appreciated :)
> 
> EOL
> 
> > Hello,
> > 
> > Is it possible to access out_info (defined in mana.c) from another program?
> > 
> > In fact, out_info is now defined as an (anonymous) "static struct" in mana.c,
> > which it seems to me precludes any direct use in another program.  Is there an
> > indirect way of getting ahold of out_info?  or of the information it contains?
> > 
> > out_info used to be defined as a *non-static* struct, and the code I'm currently
> > modifying used to compile seamlessly: it now stops the compilation during
> > linking time, as out_info is now static and the program I have to compile
> > contains an "extern struct {} out_info".
> > 
> > Any help would be much appreciated!  I searched in vain in this forum for
> > details about out_info and I really need to access the information it contains!
> > 
> > EOL (a pure MIDAS novice)

14 Jan 2004

Razvan Stefan Gornea

Access to hardware in the MIDAS framework

I am just starting to explore MIDAS, i.e. reading the manual and trying 
some examples. For the moment I would like to make a simple frontend that 
access a portable multimeter through RS-232 port. I think this could help 
me understand how to access hardare inside MIDAS framework. Initially I've 
started from the MiniFE.c example and tried to initialize the serial port 
on run start transition and build a readout loop in the main function. I 
know that this is not a full frontend but I was just interested in getting 
some experience with the drivers available in the distribution, in this 
case RS-232. The portable multimeter is very simple in principle, one just 
has to configure the port settings and then send character 'R' and read 14 
ASCII characters from the device. Unfortunately I could not understand how 
to invoke the driver services so I changed and started again with the 
slowcont/frontend.c example. From this example and after reading the "Slow 
Control System" section in the MIDAS manual I think that all I need to do 
is to define my own equipment structure based on the multi.c class driver 
with a single input channel (and replace the null driver with the RS-232).

Here I got stuck. I see from the code source that there is a relationship 
between drivers at all levels (even bus) and the ODB but I don't yet fully 
understand how they work. Actually for a couple of days now I am in a loop 
going from class to device to bus and then back again to class drivers 
trying to see how to create my own device driver and especially how to call 
the bus driver. It could be that the framework is invoking the drivers and 
the user just has to configure things ... up to now I didn't dare to look 
at the mfe.c.

Is there a more detailed documentation about slow control and drivers then 
the MIDAS manual? What is the data flow through the three layers system for 
drivers? What is the role of the framework and what is left to the user 
choice?

Thanks

14 Jan 2004

Stefan Ritt

Access to hardware in the MIDAS framework

There is some information at

http://midas.triumf.ca/doc/html/Internal.html#Slow_Control_system

and at

http://midas/download/course/course_rt03.zip , file "part1.ppt", expecially 
page 59 and page 62 "writing your own device driver".

So what you are missing for your application is a "device driver" for your 
multimeter. The only function it has to implement is the function CMD_INIT 
where you initialize the RS232 port, and the funciton CMD_GET, which sends 
a "R" and reads the value. Now you have two options:

1) You implement RS232 calls directly in your device driver

You link against rs232.c and directly call rs232_init() at the inizialization, 
then call rs232_write() and rs232_read() where you read your 14 ASCII 
characters.

2) You call a "bus driver" in your device driver

This method makes the device driver independent of the underlying transport 
interface. So if your next multimeter accepts the same "R" command over 
Ethernet, you can just replace the RS232 bus driver by the TCPIP bus driver 
without having to change your device driver. But I guess that method 2) is not 
worth for such a simple device like your multimeter.

So take nulldev.c or dastemp.c as your starting point, put some RS232 
initialization into the init routine and the communication via "R" into 
the "get" routine. The slow control frontend, driven by mfe.c, should then 
regularly read your multimeter and the value should appear in the ODB. Take 
the examples/slowcont/frontend.c as an example, and adjust the multi_driver[] 
list to use your new device driver (instead of the nulldev).

I would like to mention that the usage of midas only makes sense for some 
experiemnts which require event based readout, using VME or CAMAC crates. If 
your only task is to read out some devices which are called "slow control 
equipment" in the midas language, then you might be better of with labview or 
something.

16 Jan 2004

Razvan Stefan Gornea

Access to hardware in the MIDAS framework

The multimeter device is indeed to simple to use MIDAS but I am just trying 
it as a learning experience. The DAQ system to develop involves VME crates 
and general purpose I/O boards. The slow control part, especially accessing 
the I/O boards seem to me more complex then the VME access. I want to 
understand very well the "correct" way of using the MIDAS slow control 
framework before starting the project.

I chose the second method and created a meterdev.c driver (essentially a 
copy of the nulldev.c) where I changed the init. function and the get 
function. I am not sending a "INIT ..." string because for this device it 
is useless. In the get function I send a "D" and read my string. I changed 
the frontend of the example to have a new driver list (in the first try I 
eliminated the Output device but the ODB got corrupted, I guess the class 
multi needs to have defined output channels). The output channel is linked 
with nulldev and null (I guess this is like if they would not be present).

The result is strange because the get function is called all the time very 
fast (much faster then the 9 seconds as set in the equipment) and even 
before starting the run (I just put the flag RO_RUNNING).

Thanks for any help

Attachment 1: frontend.c

//********************************************************************************************
//
// Name:		frontend.c
// Created by:	Razvan Stefan Gornea
//
// Contents:	Slow Control frontend for a portable multimeter
//
// Log:			2004-01-15 14:22
//				Writing down initial code.
//
//********************************************************************************************

#include <stdio.h>
#include "midas.h"
#include "class/multi.h"
#include "device/nulldev.h"
#include "meterdev.h"
#include "bus/null.h"
#include "bus/rs232.h"

// globals variables

// frontend name
char *frontend_name = "Slow Control";
// frontend file name
char *frontend_file_name = __FILE__;
// frontend loop
BOOL frontend_call_loop = FALSE;
// frontend display refresh
INT display_period = 1000;
// maximum event size in bytes
INT max_event_size = 10000;
// maximum event size for fragments in bytes
INT max_event_size_frag = 5*1024*1024;
// buffer size in bytes
INT event_buffer_size = 10*10000;

// equipment list

// device drivers
DEVICE_DRIVER multi_driver[] = {
	{"Input", meterdev, 1, rs232, DF_INPUT},
	{"Output", nulldev, 1, null, DF_OUTPUT},
	{""}
};
// equipment list
EQUIPMENT equipment[] = {
  { "Multimeter",         /* equipment name */
    11, 0,                /* event ID, trigger mask */
    "SYSTEM",             /* event buffer */
    EQ_SLOW,              /* equipment type */
    0,                    /* event source */
    "FIXED",              /* format */
    TRUE,                 /* enabled */
    RO_RUNNING,           /* read when running */
    9000,                 /* read every 9 sec */
    0,                    /* stop run after this event limit */
    0,                    /* number of sub events */
    1,                    /* log history every event */
    "", "", "",
    cd_multi_read,        /* readout routine */
    cd_multi,             /* class driver main routine */
    multi_driver,         /* device driver list */
    NULL,                 /* init string */
  },
  { "" }
};

// routines

INT  poll_event(INT source[], INT count, BOOL test) {return 1;};
INT  interrupt_configure(INT cmd, INT source[], PTYPE adr) {return 1;};
// frontend initialization
INT frontend_init()
{
  return CM_SUCCESS;
}
// frontend exit
INT frontend_exit()
{
  return CM_SUCCESS;
}
// frontend loop
INT frontend_loop()
{
  return CM_SUCCESS;
}
// begin of run
INT begin_of_run(INT run_number, char *error)
{
  return CM_SUCCESS;
}
// end of run
INT end_of_run(INT run_number, char *error)
{
  return CM_SUCCESS;
}
// pause run
INT pause_run(INT run_number, char *error)
{
  return CM_SUCCESS;
}
// resume run
INT resume_run(INT run_number, char *error)
{
  return CM_SUCCESS;
}

Attachment 2: meterdev.c

//********************************************************************************************
//
// Name:		meter.c
// Created by:	Razvan Stefan Gornea
//
// Contents:	Device driver for a portable multimeter
//
// Log:			2004-01-15 14:22
//				Writing down initial code.
//
//********************************************************************************************

#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include "midas.h"

// globals variables

#define DEFAULT_TIMEOUT 10000 // 10 secondes

typedef struct {
  int  address;
} METERDEV_SETTINGS;

#define METERDEV_SETTINGS_STR "\
Address = INT : 1\n\
"
typedef struct {
  METERDEV_SETTINGS meterdev_settings; // device settings
  float         *array; // data array
  INT           num_channels; // number of channels associated with this device
  INT (*bd)(INT cmd, ...); // bus driver entry function
  void *bd_info; // private settings and data related to the bus driver
  HNDLE hkey; // ODB key for bd_info structure
} METERDEV_INFO;

// routines

// initialization function: access the ODB to creates the settings, initializes the variables
// and calls the initialization function of the bus driver
INT meterdev_init(HNDLE hkey, void **pinfo, INT channels, INT (*bd)(INT cmd, ...))
{
	int status, size;
	HNDLE hDB, hkeydd;
	METERDEV_INFO *info;

	// allocate info structure
	info = calloc(1, sizeof(METERDEV_INFO));
	*pinfo = info;
	// get handle on current experiment ODB
	cm_get_experiment_database(&hDB, NULL);
	// create METERDEV settings record
	status = db_create_record(hDB, hkey, "DD", METERDEV_SETTINGS_STR); // force the ODB structure to match the METERDEV_SETTINGS C structure
	if (status != DB_SUCCESS) {
		return FE_ERR_ODB;
	}
	db_find_key(hDB, hkey, "DD", &hkeydd); // get handle on the DD key in the ODB associated with a certain equipment and device as pointed by the "hkey" handle
	size = sizeof(info->meterdev_settings); // get the size of the device settings structure
	db_get_record(hDB, hkeydd, &info->meterdev_settings, &size, 0); // load the device settings for the ODB, i.e. ontent of DD key
	// initialize the driver
	info->num_channels = channels; // define the nmber of channels
	info->array = calloc(channels, sizeof(float)); // allocate space for data
	info->bd = bd; // set handle on bus driver
	info->hkey = hkey; // set handle on the ODB key for the evice driver
	if (!bd) { // if handle invalid return error
		return FE_ERR_ODB;
	}
	// call the bus driver initialization routine
	status = info->bd(CMD_INIT, info->hkey, &info->bd_info);
	if (status != SUCCESS) {
		return status;
	}
	// initialization of device, something like ... 
	//BD_PUTS("init");
	// for this device no initialization string is needed ...

  return FE_SUCCESS;
}

// decomission function: free memory allocation(s) and close device(s)
INT meterdev_exit(METERDEV_INFO *info)
{
	// call EXIT function of bus driver, usually closes device
	info->bd(CMD_EXIT, info->bd_info);
	// free local variables
	if (info->array) {
		free(info->array);
	}
	free(info);
	
	return FE_SUCCESS;
}

// set channel value
INT meterdev_set(METERDEV_INFO *info, INT channel, float value)
{
	char str[80];
	
	// set channel to a specific value, something like ...
	sprintf(str, "SET %d %lf", channel, value);
	BD_PUTS(str);
	BD_GETS(str, sizeof(str), ">", DEFAULT_TIMEOUT);
	// simulate writing by storing value in local array, has to be removed in a real driver
	if (channel < info->num_channels) {
		info->array[channel] = value;
	}
	
	return FE_SUCCESS;
}

// set all channels values
INT meterdev_set_all(METERDEV_INFO *info, INT channels, float *value)
{
	int  i;
	char str[1000];

	// put here some optimized form of setting all channels simultaneously like ...
	strcpy(str, "SETALL ");
	for (i=0 ; i<min(info->num_channels, channels) ; i++) {
		sprintf(str+strlen(str), "%lf ", value[i]);
	}
	BD_PUTS(str);
	BD_GETS(str, sizeof(str), ">", DEFAULT_TIMEOUT);
	// simulate writing by storing values in local array
	for (i=0 ; i<min(info->num_channels, channels) ; i++) {
		info->array[i] = value[i];
	}
	
	return FE_SUCCESS;
}

// get channel value
INT meterdev_get(METERDEV_INFO *info, INT channel, float *pvalue)
{
	int  status, i;
	char str[80];
	char ascii_number[5];

	// read value from channel, something like ...
	//sprintf(str, "GET %d", channel);
	sprintf(str, "D"); // request data
	BD_PUTS(str);
	status = BD_GETS(str, sizeof(str), "\n", DEFAULT_TIMEOUT); // read until getting a cariage return or exit on timeout
	for (i = 0; i < 4; i++) { // transfer the number
		ascii_number[i] = str[i+4];
	}
	ascii_number[4] = '\0'; // end of string
	*pvalue = (float) atof(ascii_number); // convert from ASCII to float
	// simulate reading by copying set data from local array
	//if (channel < info->num_channels) {
	//	*pvalue = info->array[channel];
	//}
	//else {
	//	*pvalue = 0.f;
	//}
	
	return FE_SUCCESS;
}

// get all channels values
INT meterdev_get_all(METERDEV_INFO *info, INT channels, float *pvalue)
{
	// int  i;
	
	/* put here some optimized form of reading all channels. If the deviced
	does not support such a function, one can call nulldev_get() in a loop 
	strcpy(str, "GETALL");
	BD_PUTS(str);
	BD_GETS(str, sizeof(str), ">", DEFAULT_TIMEOUT);
	for (i=0 ; i<min(info->num_channels, channels) ; i++)
	pvalue[i] = atof(str+i*5); // extract individual values from reply
	*/
	/* simulate reading by copying set data from local array */
	//for (i=0 ; i<min(info->num_channels, channels) ; i++)
	//	pvalue[i] = info->array[i];
	
	return FE_SUCCESS;
}

// device driver entry point
INT meterdev(INT cmd, ...)
{
	va_list argptr;
	HNDLE   hKey;
	INT     channel, status;
	DWORD   flags;
	float   value, *pvalue;
	void    *info, *bd;
	va_start(argptr, cmd);
	status = FE_SUCCESS;
	
	switch (cmd) {
	case CMD_INIT:
		hKey = va_arg(argptr, HNDLE);
		info = va_arg(argptr, void *);
		channel = va_arg(argptr, INT);
		flags = va_arg(argptr, DWORD);
		bd = va_arg(argptr, void *);
		status = meterdev_init(hKey, info, channel, bd);
		break;
	case CMD_EXIT:
		info = va_arg(argptr, void *);
		status = meterdev_exit(info);
		break;
	case CMD_SET:
		info = va_arg(argptr, void *);
		channel = va_arg(argptr, INT);
		value = (float) va_arg(argptr, double); // floats are passed as double
		status = meterdev_set(info, channel, value);
		break;
	case CMD_SET_ALL:
		info = va_arg(argptr, void *);
		channel = va_arg(argptr, INT);
		pvalue   = (float *) va_arg(argptr, float *);
		status = meterdev_set_all(info, channel, pvalue);
		break;
	case CMD_GET:
		info = va_arg(argptr, void *);
		channel = va_arg(argptr, INT);
		pvalue  = va_arg(argptr, float*);
		status = meterdev_get(info, channel, pvalue);
		break;
	case CMD_GET_ALL:
		info = va_arg(argptr, void *);
		channel = va_arg(argptr, INT);
		pvalue  = va_arg(argptr, float*);
		status = meterdev_get_all(info, channel, pvalue);
		break;
	default:
		break;
	}
	va_end(argptr);
	
	return status;
}

17 Jan 2004

Stefan Ritt

Access to hardware in the MIDAS framework

> The result is strange because the get function is called all the time very 
> fast (much faster then the 9 seconds as set in the equipment) and even 
> before starting the run (I just put the flag RO_RUNNING).

This is on purpose. When the frontend is idle, it loops over the slow control 
equipment as fast as possible. This way, you see changes in your hardware very 
quickly. I see no reason to waste CPU cycles in the frontend when there are 
better things to do like reading slow control equipment. Presume you have the 
alarm system running, which turns off some equipment in case of an over 
current. You better do this as quickly as possible, not wasting up to 9 
seconds each time.

The 9 seconds you mention are for reading *EVENTS*. You have double 
functionality: First, reading the slow control system, writing updated values 
to the ODB, where someone else can display or evaluate them (in the alarm 
system for example). Second, assemble events and sending them with the other 
data to disk or tape. Only the second one gets controlled by RO_RUNNING and 
the 9 seconds. You can see this by the updating event statists on your 
frontend display, which increments only when running and then every 9 seconds.

About remote control of front end part of MIDAS on chip

Hello!

My name is Soichiro Kuribayashi and I am a Ph.D. student at Kyoto University. 
I'm a T2K collaborator and working for Super FGD which is new detector in ND280.

I'm a beginner of MIDAS and I've just started to develop the DAQ software with 
MIDAS for Super FGD.
For the DAQ of Super FGD, we will run remotely front end part of MIDAS on ZYNQ 
which is system on chip.

For this remote control of front end part with mserver, we have to mount home 
directory of DAQ PC(Cent OS8) on that of Linux on ZYNQ.
So I wonder if we should use NFS(Network file system) + NIS(Network information 
service) + autofs for the mounting. Is it correct?

If you have any information or any suggestion for the remote control on chip, 
please let me know.

Best regards,
Soichiro

About remote control of front end part of MIDAS on chip

> My name is Soichiro Kuribayashi and I am a Ph.D. student at Kyoto University. 
> I'm a T2K collaborator and working for Super FGD which is new detector in ND280.

Hi! I did much of the DAQ software for the original FGD. I hope I can help.

> For the DAQ of Super FGD, we will run remotely front end part of MIDAS on ZYNQ 
> which is system on chip.

This would be the same as the existing FGD. Inside the FGD DCC is a Virtex4 FPGA
with a 300MHz PPC CPU running Linux from a CompactFlash card (Kentaro-san did this 
part). On this linux system runs the FGD DCC midas frontend. It connects
to the FGD midas instance using the mserver. This frontend executable is
copied to the DCC using "scp", there is no common nfs mounted home directory.

> For this remote control of front end part with mserver, we have to mount home 
> directory of DAQ PC(Cent OS8) on that of Linux on ZYNQ.
> So I wonder if we should use NFS(Network file system) + NIS(Network information 
> service) + autofs for the mounting. Is it correct?

Since you have a bigger SOC and you can run pretty much a complete linux,
I do recommend that you go this route. During development it is very convenient
to have common home directories on the main machine and on the frontend fpga
machines.

But this is not necessary. the midas mserver connection does not require
common (nfs-mounted) home directory, you can copy the files to the frontend
fpga using scp and rsync and you can use the gdb "remote debugger" function.

I can also suggest that on your frontend SOC/FPGA machine, you boot linux
using the "nfs-root" method. This way, the local flash memory only
contains a boot loader (and maybe the linux kernel image, depending on
bootloader limitations). The rest of the linux rootfs can be on your
central development machine. This way management of flash cards,
confusion with different contents of local flash and need to make backups
of frontend machines is much reduced.

If you use a fast SSD and ZFS with deduplication, you will also have good
performance gain (NFS over 1gige network to server with fast SSD works
so much better compared to the very slow SD/MMC/NAND flash).

I can point you to some of my documentation how we do this.

>
> If you have any information or any suggestion for the remote control on chip, 
> please let me know.
> 

I would say you are on a good track. For early development on just one board,
pretty much any way you do it will work, but once you start scaling up
beyound 3-4-5 frontends, you will start seeing benefits from common NFS-mounted
home directories, NFS-root booted linux, etc.

And of course you may want to study the existing ND280/FGD DAQ. I hope you
have access to the running system at Jparc. If not, I have a copy of
pretty much everything (except for running hardware, it is stored in the basement, 
dead) and I can give you access.

P.S. This reminds me that the cascade software from ND280 (they key part
for connecting the FGD, the TPC, the slow controls & etc into one experiment)
was never merged into the midas repository. I opened a ticket for this,
now we will not forget again:

https://bitbucket.org/tmidas/midas/issues/291/import-cascase-frontend-from-t2k-
nd280-fgd

K.O.

About remote control of front end part of MIDAS on chip

Dear Konstantin,

Thank you very much for your reply and detailed information.
I would appreciate if you could help us.

> I can also suggest that on your frontend SOC/FPGA machine, you boot linux
> using the "nfs-root" method. This way, the local flash memory only
> contains a boot loader (and maybe the linux kernel image, depending on
> bootloader limitations). The rest of the linux rootfs can be on your
> central development machine. This way management of flash cards,
> confusion with different contents of local flash and need to make backups
> of frontend machines is much reduced.

As you said, we can run complete Linux (Ubuntu 16) on ZYNQ and I'm using common NFS 
system now. However, I didn't know "nfs-root" method which you mentioned and this method 
seems to be reasonable way to just share linux rootfs.
First of all, I will try this method for simpler system.

> If you use a fast SSD and ZFS with deduplication, you will also have good
> performance gain (NFS over 1gige network to server with fast SSD works
> so much better compared to the very slow SD/MMC/NAND flash).
>
> I can point you to some of my documentation how we do this.

I'm concerned about such performance and I have checked the performance with common NFS 
over gige network and my DAQ PC roughly(data transfer rate ~ O(10) MByte/sec). However, I 
didn't know the ZFS and also how we can have performance gain with a fast SSD and ZFS.
Please let me know your documentation how to do it if possible.

> I would say you are on a good track. For early development on just one board,
> pretty much any way you do it will work, but once you start scaling up
> beyound 3-4-5 frontends, you will start seeing benefits from common NFS-mounted
> home directories, NFS-root booted linux, etc.

I'm developing with just one board and common NFS-mounted now. I'm looking forward to 
seeing such benefits when I will use multiple frontends.
 
> And of course you may want to study the existing ND280/FGD DAQ. I hope you
> have access to the running system at Jparc. If not, I have a copy of
> pretty much everything (except for running hardware, it is stored in the basement, 
> dead) and I can give you access.

I don't have access to the system at Jparc, but Nick has told us where FGD DAQ code is.
Is bellow URL everything of code of FGD DAQ?
https://git.t2k.org/hastings/fgddaq/-/tree/master

Best regards,
Soichiro

About remote control of front end part of MIDAS on chip

We also use a Zynq chip and boot in the following order:

1. SD Card
   a. First Stage Bootloader
   b. PL Firmware
   c. UBOOT
2. NFS over Ethernet
   a. Linux kernel
   b. RootFS
   c. Mounting home directories


If you need details I can bring you in contact with the person who actually implemented that.

Best,
Stefan

About remote control of front end part of MIDAS on chip

Dear Stefan,

Thank you very much for your help.

I have already contacted someone who has used ZYNQ in that order and It's working fine for now.
But, I'll let you know if something goes wrong.

Best regards,
Soichiro

About python interface

Coding in Python is faster than in C (but running slower). So, some python interfaces are useful for testing purpose. I hope you may like the PyMVME module for VME bus testing.

Abolished SYNC and ASYNC defines

A few months ago, definitions of SYNC and ASYNC in midas.h have been changed away from "0" and "1", 
and this caused problems with some event buffer management functions bm_xxx().

For example, when event buffers are getting full, bm_send_event(SYNC) unexpectedly started returning 
BM_ASYNC_RETURN instead of waiting for free space, causing unexpected crashes of frontend programs.

Part of the problem was confusion between SYNC/ASYNC used by buffer management (bm_xxx) and by run 
transition (cm_transition()) functions. Adding to confusion, documentation of bm_send_event() & co used 
FALSE/TRUE while most actual calls used SYNC/ASYNC.

To sort this out, an executive decision was made to abolish the SYNC/ASYNC defines:

For buffer management calls bm_send_event(), bm_receive_event(), etc, please use:
SYNC -> BM_WAIT
ASYNC -> BM_NO_WAIT

For run transitions, please use:
SYNC -> TR_SYNC
ASYNC -> TR_ASYNC
MTHREAD -> TR_MTHREAD
DETACH -> TR_DETACH

K.O.

API to read MIDAS format file

Dear MIDAS people,
I need to borrow your wisdom for a bit.
I am developing a piece of software that should read the history data stored in a
.midas file (MIDAS format) and integrate it into the WAGASCI data quality output.
In other words, I need to read some temperature values stored in a .midas file and
compare them with the MPPC gains and check for temperature/gain dependence.
I see three possibilities:

write a custom parser in C++ using the instructions contained in the Mhformat page;
call the mhist program from within my application;
call the mhdump program from within my application;

Which solution do you think is the best?
Because there is no need for raw performance, if possible, I would like to write my application in Python3 but C++ is also an option.

API to read MIDAS format file

API to read MIDAS format file

I guess all three options would work. I just tried mhist and it still works with the "FILE" history

mhist -e <equipment name> -v <variable name> -h 10

for dumping a variable for the last 10 hours.

I could not get mhdump to work with current history files, maybe it only works with "MIDAS" history and not "FILE" history (see https://midas.triumf.ca/MidasWiki/index.php/History_System#History_drivers). Maybe Konstantin who wrote mhdump has some idea.

Writing your own parser is certainly possible (even in Python), but of course more work.

Stefan

API to read MIDAS format file

Stefan Ritt wrote:

I guess all three options would work. I just tried mhist and it still works with the "FILE" historymhist -e <equipment name> -v <variable name> -h 10for dumping a variable for the last 10 hours.I could not get mhdump to work with current history files, maybe it only works with "MIDAS" history and not "FILE" history (see https://midas.triumf.ca/MidasWiki/index.php/History_System#History_drivers). Maybe Konstantin who wrote mhdump has some idea.Writing your own parser is certainly possible (even in Python), but of course more work.Stefan

Thank you for the quick reply. Do notice that we have "MIDAS" history files and not "FILE", so both mhist and mhdump should be fine (however I have only tested mhist). Hipotetically, which one between mhist and mhdump do you think is better suited to be "batched"? I mean to be controlled and read by a routine?

Goto page Previous 1, 2, 3 ... 139, 140, 141, 142, 143, 144 Next

ELOG V3.1.4-2e1708b5