Back Midas Rome Roody Rootana
  Midas DAQ System, Page 38 of 121  Not logged in ELOG logo
ID Date Author Topic Subjectup
  287   03 Aug 2006 Shawn BishopBug ReportMIDAS packaged examples: compilation bug?

Stefan Ritt wrote:

Shawn Bishop wrote:
Anyone have an idea what's going on here?


The Makefile contained the outdated target fal, which is a combined frontend/analyzer/logger. You don't need that, so I removed it from the makefile. Now it should compile fine.


Hi All,

I think I've solved this second problem. Konstantin sent me an email pointing out that cernlib I have wants RFIO. To make a long story short, I ended up solving this by changing the following line in the bbookexpt Makefile:

CERNLIB_PACK = $(CERNLIB)/lib/libpacklib --> CERNLIB_PACK = $(CERNLIB)/lib/libpacklib_noshift.a
  945   15 Jan 2014 Konstantin OlchanskiBug ReportMIDAS password protection is broken
If you follow the MIDAS documentation for setting up password protection, you will get strange messages:

ladd00:midas$ ./linux/bin/odbedit
[local:testexpt:S]/>passwd                <---- setup a password
Password: 
Retype password: 
[local:testexpt:S]/> exit

ladd00:midas$ odbedit
Password:    <---- enter correct password here
ss_semaphore_wait_for: semop/semtimedop(21135376) returned -1, errno 22 (Invalid argument)
ss_semaphore_release: semop/semtimedop(21135376) returned -1, errno 22 (Invalid argument)
[local:testexpt:S]/>ss_semaphore_wait_for: semop/semtimedop(21037069) returned -1, errno 43 (Identifier removed)

The same messages will appear from all other programs - mhttpd, etc. They will be printed about every 1 second.

So what do they mean? They mean what they say - the semaphore is not there, it is easy to check using "ipcs" that semaphores with 
those ids do not exist. In fact all the semaphores are missing (the ODB semaphore is eventually recreated, so at least ODB works 
correctly).

In this situation, MIDAS will not work correctly.

What is happening?

- cm_connect_experiment1() creates all the semaphores and remembers them in cm_set_experiment_semaphore()
- calls cm_set_client_info()
- cm_set_client_info() finds ODB /expt/sec/password, and returns CM_WRONG_PASSWORD
- before returning, it calls db_close_all_databases() and bm_close_all_buffers(), which delete all semaphores (put a print statement in 
ss_semaphore_delete() to see this).
- (values saved by cm_set_experiment_semaphore() are stale now).
- (if by luck you have other midas programs still running, the semaphores will not be deleted)
- we are back to cm_connect_experiment1() which will ask for the password, call cm_set_client_info() again and continue as usual
- it will reopen ODB, recreating the ODB semaphore
- (but all the other semaphores are still deleted and values saved by cm_set_experiment_semaphore() are stale)

I through to improve this by fixing a bug in cm_msg_log() (where the messages are coming from) - it tries to lock the "MSG" 
semaphore, but even if it could not lock it, it continues as usual and even calls an unlock at the end. (very bad). For catastrophic 
locking failures like this (semaphore is deleted), we usually abort. But if I abort here, I get completely locked out from odb - odbedit 
crashes right away and there is no way to do any corrective action other than delete odb and reload it from an xml file.

I know that some experiments use this password protection - why/how does it work there?

I think they are okey because they put critical programs like odbedit, mserver, mlogger and mhttpd into "/expt/sec/allowed 
programs". In this case the pass the password check in cm_set_client_info() and the semaphores are not deleted. If any subsequent 
program asks for the password, the semaphores survive because mlogger or mhttpd is already running and keeps semaphores from 
being deleted.

What a mess.

K.O.
  947   15 Jan 2014 Konstantin OlchanskiBug ReportMIDAS password protection is broken
> I through to improve this by fixing a bug in cm_msg_log() (where the messages are coming from)

The periodic messages about broken semaphore actually come from al_check(). I put some whining there, too.

K.O.
  954   05 Feb 2014 Stefan RittBug ReportMIDAS password protection is broken
> If you follow the MIDAS documentation for setting up password protection, you will get strange messages:

This is interesting. When I used it last time (some years ago...) it worked fine. I did not touch this, and now it's broken. Must be related to some modifications of the semaphore system. 
Well, anyhow, the problem seems to me the db_close_all_databses() and the re-opening of the ODB. Apparently the db_close_database() call does not clean up the semaphores properly. 
Actually there is absolutely no need to close and re-open the ODB upon a wrong password, so I just removed that code and now it works again.

/Stefan
  278   27 Jul 2006 Shawn BishopBug ReportMIDAS revision 3184 bombs on FC5
Hi All,

I just did a fresh download of midas (revision 3184) onto a newly setup FC5 box. Compilation bombs. Printout of compiler output as follows:

Regards,
Shawn

[midas@daruma midas]$ make
cc -c -g -O2 -Wall -Wuninitialized -Iinclude -Idrivers -I../mxml -Llinux/lib -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -m32 -DOS_LINUX -fPIC -Wno-unused-function -o linux/lib/musbstd.o drivers/usb/musbstd.c
In file included from drivers/usb/musbstd.c:14:
include/musbstd.h:29:17: error: usb.h: No such file or directory
In file included from drivers/usb/musbstd.c:14:
include/musbstd.h:32: error: expected specifier-qualifier-list before ‘usb_dev_handle’
drivers/usb/musbstd.c:54:1: warning: "HAVE_LIBUSB" redefined
include/musbstd.h:27:1: warning: this is the location of the previous definition
drivers/usb/musbstd.c: In function ‘musb_open’:
drivers/usb/musbstd.c:157: warning: implicit declaration of function ‘usb_init’
drivers/usb/musbstd.c:158: warning: implicit declaration of function ‘usb_find_busses’
drivers/usb/musbstd.c:159: warning: implicit declaration of function ‘usb_find_devices’
drivers/usb/musbstd.c:161: warning: implicit declaration of function ‘usb_get_busses’
drivers/usb/musbstd.c:161: warning: assignment makes pointer from integer without a cast
drivers/usb/musbstd.c:161: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:162: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:162: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:163: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:163: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:166: error: ‘usb_dev_handle’ undeclared (first use in this function)
drivers/usb/musbstd.c:166: error: (Each undeclared identifier is reported only once
drivers/usb/musbstd.c:166: error: for each function it appears in.)
drivers/usb/musbstd.c:166: error: ‘udev’ undeclared (first use in this function)
drivers/usb/musbstd.c:168: warning: implicit declaration of function ‘usb_open’
drivers/usb/musbstd.c:174: warning: implicit declaration of function ‘usb_set_configuration’
drivers/usb/musbstd.c:181: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:181: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:187: warning: implicit declaration of function ‘usb_claim_interface’
drivers/usb/musbstd.c:194: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:194: error: dereferencing pointer to incomplete type
drivers/usb/musbstd.c:200: error: ‘MUSB_INTERFACE’ has no member named ‘dev’
drivers/usb/musbstd.c:201: error: ‘MUSB_INTERFACE’ has no member named ‘usbinterface’
drivers/usb/musbstd.c: In function ‘musb_close’:
drivers/usb/musbstd.c:317: warning: implicit declaration of function ‘usb_release_interface’
drivers/usb/musbstd.c:317: error: ‘MUSB_INTERFACE’ has no member named ‘dev’
drivers/usb/musbstd.c:317: error: ‘MUSB_INTERFACE’ has no member named ‘usbinterface’
drivers/usb/musbstd.c:320: warning: implicit declaration of function ‘usb_close’
drivers/usb/musbstd.c:320: error: ‘MUSB_INTERFACE’ has no member named ‘dev’
drivers/usb/musbstd.c: In function ‘musb_write’:
drivers/usb/musbstd.c:339: warning: implicit declaration of function ‘usb_bulk_write’
drivers/usb/musbstd.c:339: error: ‘MUSB_INTERFACE’ has no member named ‘dev’
drivers/usb/musbstd.c: In function ‘musb_read’:
drivers/usb/musbstd.c:385: warning: implicit declaration of function ‘usb_bulk_read’
drivers/usb/musbstd.c:385: error: ‘MUSB_INTERFACE’ has no member named ‘dev’
drivers/usb/musbstd.c: In function ‘musb_reset’:
drivers/usb/musbstd.c:435: warning: implicit declaration of function ‘usb_reset’
drivers/usb/musbstd.c:435: error: ‘MUSB_INTERFACE’ has no member named ‘dev’
make: *** [linux/lib/musbstd.o] Error 1
[midas@daruma midas]$
  279   27 Jul 2006 Stefan RittBug ReportMIDAS revision 3184 bombs on FC5

Shawn Bishop wrote:
include/musbstd.h:29:17: error: usb.h: No such file or directory


This indicates that you are missing libusb. If you can find a RPM for libusb, that will solve your problem. But anyhow we should modify the makefile such that it does not try to compile the USB drivers if libusb is missing on a system.
  892   22 Jul 2013 Konstantin OlchanskiInfoMIDAS source code converted from SVN to GIT
The MIDAS source code repository was converted from SVN to GIT, hosted as bitbucket: 
https://bitbucket.org/tmidas.

A clonable copy of the repository is located at TRIUMF: git clone 
http://daq.triumf.ca/~daqweb/git/midas.git (and mxml.git).

The documentation is being slowly updated with GIT instructions (git clone) instead of SVN (svn 
checkout).

The MIDAS code history goes all the way to CVS/SVN rev 1 dated Thu Oct 8 13:46:02 1998.

K.O.
  893   22 Jul 2013 Stefan RittInfoMIDAS source code converted from SVN to GIT
Konstantin forgot to tell people outside of TRIUMF how to get the newest version of MIDAS. Here it is:

$ git clone https://bitbucket.org/tmidas/midas.git

Not that you can also browse the repository at

https://bitbucket.org/tmidas/midas

On some (older) systems, you might have to install git (http://git-scm.com/downloads).

/Stefan
  1328   21 Nov 2017 Konstantin OlchanskiInfoMIDAS support on el5?
It has been reported that the current midas release candidate does not build on el5 linux (SL/RHEL/CentOS-5).

According to Red Hat, el5 is end-of-life, last SL 5 (SL5.11) was done in 2014, so this linux is very old. Also as it happens, I do not have access to any 
el5 machines to check if midas builds or runs (but this can be fixed).

https://www.scientificlinux.org/downloads/sl-versions/sl5/
https://access.redhat.com/support/policy/updates/errata

On the midas web page (https://midas.triumf.ca) we do not explicitly state which versions of which linux we definitely support. Most other open-
source projects only support current major linux distributions, hardly anybody supports end-of-life linuxes such as el5. Some projects do not even 
support recent linuxes still widely in use (ROOT6 does not build on stock el6 and there is no KDE5 for el7).

So back to midas. Support for different operating systems comes down to:

1) C/C++ language support. We still use el6 (GCC 4.4.7), so use of c++-11 language features should be avoided
2) operating system features support:
a) sysv semaphores (sysv shared memory no longer used, cannot be used on macos)
aa) (macos also is missing parts of the sysv semaphore api, such as "wait for lock, with timeout", we are using an ugly work-around)
b) posix shared memory with mprotect() & co
c) posix mutexes, including recursive-type mutexes (this seems to be the problem on el5)
d) bsd networking (need to migrate from select() to poll() and from gethostbyname() to getaddrinfo() & co (for IPv6 support))

Not all of these operating system functions are required for all of midas. Running mhttpd and mlogger requires
pretty much everything. Running just a frontend connected to midas through the mserver requires the least features,
just the networking is enough, I think.

Obviously we cannot support midas in perpetuity on all versions of all operating systems, once I do not have
access to a machine, I cannot even check that midas builds and that it runs the basic functions.

Instead, we could provide a "feature reduced" build of midas (makefile target) that includes "just enough" of midas
to (say) run a frontend, maybe even odbedit. We already have some provisions for this, but no obvious documented
way actually doing it.

So back to el5.

How important it is to support very old operating systems?
How many people still use el5?
How about old  versions of Ubuntu? Macos?

If you use anything older than el6, can you speak up,
(and if possible say why you cannot migrate to an up-to-date linux).

K.O.
  1540   05 Jun 2019 Konstantin OlchanskiInfoMIDAS switched to C++
The last bits of code to switch MIDAS to C++ have been committed, see tag midas-2019-05-cxx.

Since the cmake conversion is still in progress, for now, I recommend using the old "make" build for trying this update.

From the switch to C++, the biggest change is the requirement that frontend programs be build and linked
using the C++ compiler. Since mfe.o and the rest of MIDAS are built with C++, building frontends
with C is no longer possible.

To help with this, I will post a short guide for converting C frontends to C++.

K.O.
  2407   17 May 2022 Razvan Stefan GorneaInfoMIDAS switched to C++
Hi, I have three naive questions about this:
 - have you posted somewhere this guide about converting C frontends to C++?
 - it was mentioned previously that there will be a 'tag the last "C" midas', which version is it?
 - it means that even a simple example like odb_test.c cannot be compile anymore? Even when using g++?

Something like

g++ -I $HOME/daq/packages/midas/include/ -L $HOME/daq/packages/midas/lib/ odb_test.c -l midas

is expected to fail or is just me glitching? Is it because of thread library differences?

Thanks!


> The last bits of code to switch MIDAS to C++ have been committed, see tag midas-2019-05-cxx.
> 
> Since the cmake conversion is still in progress, for now, I recommend using the old "make" build for trying this update.
> 
> From the switch to C++, the biggest change is the requirement that frontend programs be build and linked
> using the C++ compiler. Since mfe.o and the rest of MIDAS are built with C++, building frontends
> with C is no longer possible.
> 
> To help with this, I will post a short guide for converting C frontends to C++.
> 
> K.O.
  Draft   17 May 2022 Ben SmithInfoMIDAS switched to C++
>  - have you posted somewhere this guide about converting C frontends to C++?

See the instructions at:
https://daq00.triumf.ca/MidasWiki/index.php/Changelog#2019-06

>  - it was mentioned previously that there will be a 'tag the last "C" midas', which version is it?

>  - it means that even a simple example like odb_test.c cannot be compile anymore? Even when using g++?
> g++ -I $HOME/daq/packages/midas/include/ -L $HOME/daq/packages/midas/lib/ odb_test.c -l midas

Correct. Midas is built with C++, so names get mangled 
  2409   17 May 2022 Konstantin OlchanskiInfoMIDAS switched to C++
> Hi, I have three naive questions about this:

all good questions, ask more of them.

>  - have you posted somewhere this guide about converting C frontends to C++?

yes, in this elog here I posted a guide for converting C mfe.c frontends to C++ and
a guide for converting mfe.c frontend to C++ TMFE frontend. please use the "find" function,
if you cannot find them, let me know, I will look for it for you.

>  - it was mentioned previously that there will be a 'tag the last "C" midas', which version is it?

correct. please run "git tag", tags before "midas-2019-05-cxx"is "C", after is "C++".

>  - it means that even a simple example like odb_test.c cannot be compile anymore? Even when using g++?
> g++ -I $HOME/daq/packages/midas/include/ -L $HOME/daq/packages/midas/lib/ odb_test.c -l midas
> is expected to fail or is just me glitching? Is it because of thread library differences?

yes, it is expected to fail, you have spaces after "-I", "-L" and "-l", incorrect g++ command syntax. after
correcting this, it may or may not work depending on what you have inside odb_test.c. I would be happy
to help you debug this, but please start a separate thread instead of necroposting into the C++ announcements.

K.O.
  2410   17 May 2022 Ben SmithInfoMIDAS switched to C++
>  - have you posted somewhere this guide about converting C frontends to C++?

There's documentation in the wiki at:
https://daq00.triumf.ca/MidasWiki/index.php/Changelog#2019-06

It includes a step-by-step guide of how to upgrade, what changes need to be made to frontends, and common issues that people had.
  1532   28 May 2019 Stefan RittInfoMIDAS switching to Cmake
Great news! I got convinced by some colleagues to switch midas to Cmake. After spending about one day, I wrote some initial CMakeLists.txt file and am so excited about the advantages that I regret 
not having done this step much earlier. Here is some information:

- The Cmake and old Makefile systems can co-exist. So the old "make" in the midas root still works as previously.

- To use Cmake, do

midas$ mkdir build
midas$ cd build
midas/build$ cmake ..
midas/build$ make

Depending on your installation, it might be necessary to call "cmake3" instead of "cmake". The configuration requires Cmake 3.0 or later.

- After successful compilation, all programs and libraries are in the "build" directory. We kind of concluded that a system-wide midas installation (like under /usr/local/bin) is not necessary these days, 
as long as you have your MIDASSYS and PATH environment variables defined correctly. Some examples move all files from "build" to "bin"/"lib" under midas, but I'm not sure if we need that.

- Interestingly enough, in my iMac(Late 2015), the old Makefile build takes 19.5s, which the new one take 12s. So apparently some clever dependency checking is done in Cmake.

- The compile options are now handled in the Cmake cache file which is important to remember. Changing option(USE_SSL ON) in CMakeLists.txt just modifies the default value on a fresh install. To 
change the flags between compilations, use the "ccmake .." interface instead. This lets you also switch from Debug to Release mode easily.

- I love how the library handling is done. The code

  find_package(OpenSSL REQUIRED)
  include_directories(${OPENSSL_INCLUDE_DIR})
  target_link_libraries(mhttpd midas ${OPENSSL_LIBRARIES})

is so much simpler than our clumsy conditional compiling we needed in the old Makefile. 

- Cmake is the basis of the CLion IDE which is my favourite development environment now (https://www.jetbrains.com/clion/). So I can work inside the IDE and see the full project, I can do interactive 
debugging etc. and still do a simple 'make' on systems where CLion is not installed. I can only recommend everybody to have a look at CLion. It is free for university teachers and open source 
developers (like I got my free license because of ELOG).

- The CMakeLists.txt is not yet complete. It does not contain cross compilation, since I don't have access to these compilers. 

- The next step will be to add a CMakeLists.txt into each "example" directory and build everything hierarchically. 

- I'm a novice in cmake. If someone of your has more experience (and I'm sure that there are plenty of people out there!), please have a look at my CMakeLists.txt and check if things can be made 
simpler or more elegantly.

- Any comment are as usual welcome.

Have fun,
Stefan
  1533   28 May 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
> Great news!

Some additional information.

1) cmake3 is available on all currently supported systems:

- SL6 (el6), CentOS7 (el7): yum install cmake3 (from EPEL) (invoke as "cmake3")
- Ubuntu 18.04 LTS: apt-get install cmake (invoke as "cmake").
- MacOS: install "mac ports", then "port install cmake"
- Windows - we hope to revive windows10 support this summer

> - To use Cmake, do
> 
> midas$ mkdir build
> midas$ cd build
> midas/build$ cmake ..
> midas/build$ make
>
> - After successful compilation, all programs and libraries are in the "build" directory
> 

The old "linux", "darwin", etc subdirectories go away. Makefiles for frontends and analyzers become simplified
and can refer to MIDAS in a standard way:

header files: -I$(MIDASSYS)/include
libraries and object files: -L$(MIDASSYS)/build/lib -lmidas
executables: PATH += $(MIDASSYS)/build/bin

>
> ... cross compilation ...
>

We will review the situation with cross-compilation once the dust settles a little bit on changes
with cmake and with the switch to C++.

Since cross-compilation environments are rarely standardized, I do not expect cmake to be of much help and most
likely we will have a simplified Makefile for cross-building feature-reduced versions of MIDAS - probably only
the pieces needed for running remotely-connected frontends (see "ifdef LOCAL_ROUTINES").

K.O.
  1534   28 May 2019 Stefan RittInfoMIDAS switching to Cmake
> > - After successful compilation, all programs and libraries are in the "build" directory
> > 
> 
> The old "linux", "darwin", etc subdirectories go away. Makefiles for frontends and analyzers become simplified
> and can refer to MIDAS in a standard way:
> 
> header files: -I$(MIDASSYS)/include
> libraries and object files: -L$(MIDASSYS)/build/lib -lmidas
> executables: PATH += $(MIDASSYS)/build/bin

Actually the library and executables go directly into the build directory (without "lib" and "bin"), so we need

header files: -I$(MIDASSYS)/include
libraries and object files: -L$(MIDASSYS)/build -lmidas
executables: PATH += $(MIDASSYS)/build

Or course that can be changed in the Cmake file, but not sure if that would be necessary/useful.
  1535   29 May 2019 Stefan RittInfoMIDAS switching to Cmake
> 
> > > - After successful compilation, all programs and libraries are in the "build" directory
> > > 
> > 
> > The old "linux", "darwin", etc subdirectories go away. Makefiles for frontends and analyzers become simplified
> > and can refer to MIDAS in a standard way:
> > 
> > header files: -I$(MIDASSYS)/include
> > libraries and object files: -L$(MIDASSYS)/build/lib -lmidas
> > executables: PATH += $(MIDASSYS)/build/bin
> 
> Actually the library and executables go directly into the build directory (without "lib" and "bin"), so we need
> 
> header files: -I$(MIDASSYS)/include
> libraries and object files: -L$(MIDASSYS)/build -lmidas
> executables: PATH += $(MIDASSYS)/build
> 
> Or course that can be changed in the Cmake file, but not sure if that would be necessary/useful.

Actually I like the proposed separation between the library and the binaries, so I reworked it again. Now we have

header files: -I$(MIDASSYS)/include
libraries and object files: -L$(MIDASSYS)/lib -lmidas
executables: PATH += $(MIDASSYS)/bin

When issuing a "cmake .." followed by a "make" in the build directory, everything ends up in the build directory. To 
move things to the lib and bin directories, do a "make install". Seems to me like this is the standard way for
many packages so we should follow it.

Furthermore, I followed a proposal from KO to separate the code in the "src" directory between library source code
and programs. I moved all programs now to a separate "progs" directory, and left only code for the midas library in
the "src" directory. New CMakeLists.txt have been written for the "progs" and "utils" directories.

Care has been taken so that even when source files were moved around, their revision history is kept to "git annotate"
still works.

This is quite a change so sorry if this breaks some existing installations, but it will make things much easier in the future.

Stefan
  1539   03 Jun 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
> 1) cmake3 is available on all currently supported systems:
> 
> - SL6 (el6), CentOS7 (el7): yum install cmake3 (from EPEL) (invoke as "cmake3")
> - Ubuntu 18.04 LTS: apt-get install cmake (invoke as "cmake").
> - MacOS: install "mac ports", then "port install cmake"
> - Windows - we hope to revive windows10 support this summer

- el8 (RHEL8): cmake 3.11.something is part of the base system (latest cmake), (invoke as "cmake", the best I can tell).

K.O.
  1541   05 Jun 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
Status update on the cmake conversion:

- we have cmake builds working on all supported systems (el6, el7, ubuntu 18.04 LTS, macos 10.13, 10.14)
- I am happy with the result - for example, include file dependancies work much better now
- we are still fixing a few problems where the cmake build is different from the old make build (mfe.o, mlogger/rmlogger, etc)
- until all of these problems are straightened out, we cannot finalize the instructions for writing experiment makefiles (do we have to use -lmfe or 
we can keep the old mfe.o)

After everything is finalized, I hope to post a short guide for converting experiment makefiles to the new system.

The next release of MIDAS (midas-2019-06 series) will be the first C++ midas and cmake will be the primary build system.

K.O.
ELOG V3.1.4-2e1708b5