Back Midas Rome Roody Rootana
  Midas DAQ System, Page 38 of 120  Not logged in ELOG logo
ID Date Author Topic Subjectup
  1328   21 Nov 2017 Konstantin OlchanskiInfoMIDAS support on el5?
It has been reported that the current midas release candidate does not build on el5 linux (SL/RHEL/CentOS-5).

According to Red Hat, el5 is end-of-life, last SL 5 (SL5.11) was done in 2014, so this linux is very old. Also as it happens, I do not have access to any 
el5 machines to check if midas builds or runs (but this can be fixed).

https://www.scientificlinux.org/downloads/sl-versions/sl5/
https://access.redhat.com/support/policy/updates/errata

On the midas web page (https://midas.triumf.ca) we do not explicitly state which versions of which linux we definitely support. Most other open-
source projects only support current major linux distributions, hardly anybody supports end-of-life linuxes such as el5. Some projects do not even 
support recent linuxes still widely in use (ROOT6 does not build on stock el6 and there is no KDE5 for el7).

So back to midas. Support for different operating systems comes down to:

1) C/C++ language support. We still use el6 (GCC 4.4.7), so use of c++-11 language features should be avoided
2) operating system features support:
a) sysv semaphores (sysv shared memory no longer used, cannot be used on macos)
aa) (macos also is missing parts of the sysv semaphore api, such as "wait for lock, with timeout", we are using an ugly work-around)
b) posix shared memory with mprotect() & co
c) posix mutexes, including recursive-type mutexes (this seems to be the problem on el5)
d) bsd networking (need to migrate from select() to poll() and from gethostbyname() to getaddrinfo() & co (for IPv6 support))

Not all of these operating system functions are required for all of midas. Running mhttpd and mlogger requires
pretty much everything. Running just a frontend connected to midas through the mserver requires the least features,
just the networking is enough, I think.

Obviously we cannot support midas in perpetuity on all versions of all operating systems, once I do not have
access to a machine, I cannot even check that midas builds and that it runs the basic functions.

Instead, we could provide a "feature reduced" build of midas (makefile target) that includes "just enough" of midas
to (say) run a frontend, maybe even odbedit. We already have some provisions for this, but no obvious documented
way actually doing it.

So back to el5.

How important it is to support very old operating systems?
How many people still use el5?
How about old  versions of Ubuntu? Macos?

If you use anything older than el6, can you speak up,
(and if possible say why you cannot migrate to an up-to-date linux).

K.O.
  1540   05 Jun 2019 Konstantin OlchanskiInfoMIDAS switched to C++
The last bits of code to switch MIDAS to C++ have been committed, see tag midas-2019-05-cxx.

Since the cmake conversion is still in progress, for now, I recommend using the old "make" build for trying this update.

From the switch to C++, the biggest change is the requirement that frontend programs be build and linked
using the C++ compiler. Since mfe.o and the rest of MIDAS are built with C++, building frontends
with C is no longer possible.

To help with this, I will post a short guide for converting C frontends to C++.

K.O.
  2407   17 May 2022 Razvan Stefan GorneaInfoMIDAS switched to C++
Hi, I have three naive questions about this:
 - have you posted somewhere this guide about converting C frontends to C++?
 - it was mentioned previously that there will be a 'tag the last "C" midas', which version is it?
 - it means that even a simple example like odb_test.c cannot be compile anymore? Even when using g++?

Something like

g++ -I $HOME/daq/packages/midas/include/ -L $HOME/daq/packages/midas/lib/ odb_test.c -l midas

is expected to fail or is just me glitching? Is it because of thread library differences?

Thanks!


> The last bits of code to switch MIDAS to C++ have been committed, see tag midas-2019-05-cxx.
> 
> Since the cmake conversion is still in progress, for now, I recommend using the old "make" build for trying this update.
> 
> From the switch to C++, the biggest change is the requirement that frontend programs be build and linked
> using the C++ compiler. Since mfe.o and the rest of MIDAS are built with C++, building frontends
> with C is no longer possible.
> 
> To help with this, I will post a short guide for converting C frontends to C++.
> 
> K.O.
  Draft   17 May 2022 Ben SmithInfoMIDAS switched to C++
>  - have you posted somewhere this guide about converting C frontends to C++?

See the instructions at:
https://daq00.triumf.ca/MidasWiki/index.php/Changelog#2019-06

>  - it was mentioned previously that there will be a 'tag the last "C" midas', which version is it?

>  - it means that even a simple example like odb_test.c cannot be compile anymore? Even when using g++?
> g++ -I $HOME/daq/packages/midas/include/ -L $HOME/daq/packages/midas/lib/ odb_test.c -l midas

Correct. Midas is built with C++, so names get mangled 
  2409   17 May 2022 Konstantin OlchanskiInfoMIDAS switched to C++
> Hi, I have three naive questions about this:

all good questions, ask more of them.

>  - have you posted somewhere this guide about converting C frontends to C++?

yes, in this elog here I posted a guide for converting C mfe.c frontends to C++ and
a guide for converting mfe.c frontend to C++ TMFE frontend. please use the "find" function,
if you cannot find them, let me know, I will look for it for you.

>  - it was mentioned previously that there will be a 'tag the last "C" midas', which version is it?

correct. please run "git tag", tags before "midas-2019-05-cxx"is "C", after is "C++".

>  - it means that even a simple example like odb_test.c cannot be compile anymore? Even when using g++?
> g++ -I $HOME/daq/packages/midas/include/ -L $HOME/daq/packages/midas/lib/ odb_test.c -l midas
> is expected to fail or is just me glitching? Is it because of thread library differences?

yes, it is expected to fail, you have spaces after "-I", "-L" and "-l", incorrect g++ command syntax. after
correcting this, it may or may not work depending on what you have inside odb_test.c. I would be happy
to help you debug this, but please start a separate thread instead of necroposting into the C++ announcements.

K.O.
  2410   17 May 2022 Ben SmithInfoMIDAS switched to C++
>  - have you posted somewhere this guide about converting C frontends to C++?

There's documentation in the wiki at:
https://daq00.triumf.ca/MidasWiki/index.php/Changelog#2019-06

It includes a step-by-step guide of how to upgrade, what changes need to be made to frontends, and common issues that people had.
  1532   28 May 2019 Stefan RittInfoMIDAS switching to Cmake
Great news! I got convinced by some colleagues to switch midas to Cmake. After spending about one day, I wrote some initial CMakeLists.txt file and am so excited about the advantages that I regret 
not having done this step much earlier. Here is some information:

- The Cmake and old Makefile systems can co-exist. So the old "make" in the midas root still works as previously.

- To use Cmake, do

midas$ mkdir build
midas$ cd build
midas/build$ cmake ..
midas/build$ make

Depending on your installation, it might be necessary to call "cmake3" instead of "cmake". The configuration requires Cmake 3.0 or later.

- After successful compilation, all programs and libraries are in the "build" directory. We kind of concluded that a system-wide midas installation (like under /usr/local/bin) is not necessary these days, 
as long as you have your MIDASSYS and PATH environment variables defined correctly. Some examples move all files from "build" to "bin"/"lib" under midas, but I'm not sure if we need that.

- Interestingly enough, in my iMac(Late 2015), the old Makefile build takes 19.5s, which the new one take 12s. So apparently some clever dependency checking is done in Cmake.

- The compile options are now handled in the Cmake cache file which is important to remember. Changing option(USE_SSL ON) in CMakeLists.txt just modifies the default value on a fresh install. To 
change the flags between compilations, use the "ccmake .." interface instead. This lets you also switch from Debug to Release mode easily.

- I love how the library handling is done. The code

  find_package(OpenSSL REQUIRED)
  include_directories(${OPENSSL_INCLUDE_DIR})
  target_link_libraries(mhttpd midas ${OPENSSL_LIBRARIES})

is so much simpler than our clumsy conditional compiling we needed in the old Makefile. 

- Cmake is the basis of the CLion IDE which is my favourite development environment now (https://www.jetbrains.com/clion/). So I can work inside the IDE and see the full project, I can do interactive 
debugging etc. and still do a simple 'make' on systems where CLion is not installed. I can only recommend everybody to have a look at CLion. It is free for university teachers and open source 
developers (like I got my free license because of ELOG).

- The CMakeLists.txt is not yet complete. It does not contain cross compilation, since I don't have access to these compilers. 

- The next step will be to add a CMakeLists.txt into each "example" directory and build everything hierarchically. 

- I'm a novice in cmake. If someone of your has more experience (and I'm sure that there are plenty of people out there!), please have a look at my CMakeLists.txt and check if things can be made 
simpler or more elegantly.

- Any comment are as usual welcome.

Have fun,
Stefan
  1533   28 May 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
> Great news!

Some additional information.

1) cmake3 is available on all currently supported systems:

- SL6 (el6), CentOS7 (el7): yum install cmake3 (from EPEL) (invoke as "cmake3")
- Ubuntu 18.04 LTS: apt-get install cmake (invoke as "cmake").
- MacOS: install "mac ports", then "port install cmake"
- Windows - we hope to revive windows10 support this summer

> - To use Cmake, do
> 
> midas$ mkdir build
> midas$ cd build
> midas/build$ cmake ..
> midas/build$ make
>
> - After successful compilation, all programs and libraries are in the "build" directory
> 

The old "linux", "darwin", etc subdirectories go away. Makefiles for frontends and analyzers become simplified
and can refer to MIDAS in a standard way:

header files: -I$(MIDASSYS)/include
libraries and object files: -L$(MIDASSYS)/build/lib -lmidas
executables: PATH += $(MIDASSYS)/build/bin

>
> ... cross compilation ...
>

We will review the situation with cross-compilation once the dust settles a little bit on changes
with cmake and with the switch to C++.

Since cross-compilation environments are rarely standardized, I do not expect cmake to be of much help and most
likely we will have a simplified Makefile for cross-building feature-reduced versions of MIDAS - probably only
the pieces needed for running remotely-connected frontends (see "ifdef LOCAL_ROUTINES").

K.O.
  1534   28 May 2019 Stefan RittInfoMIDAS switching to Cmake
> > - After successful compilation, all programs and libraries are in the "build" directory
> > 
> 
> The old "linux", "darwin", etc subdirectories go away. Makefiles for frontends and analyzers become simplified
> and can refer to MIDAS in a standard way:
> 
> header files: -I$(MIDASSYS)/include
> libraries and object files: -L$(MIDASSYS)/build/lib -lmidas
> executables: PATH += $(MIDASSYS)/build/bin

Actually the library and executables go directly into the build directory (without "lib" and "bin"), so we need

header files: -I$(MIDASSYS)/include
libraries and object files: -L$(MIDASSYS)/build -lmidas
executables: PATH += $(MIDASSYS)/build

Or course that can be changed in the Cmake file, but not sure if that would be necessary/useful.
  1535   29 May 2019 Stefan RittInfoMIDAS switching to Cmake
> 
> > > - After successful compilation, all programs and libraries are in the "build" directory
> > > 
> > 
> > The old "linux", "darwin", etc subdirectories go away. Makefiles for frontends and analyzers become simplified
> > and can refer to MIDAS in a standard way:
> > 
> > header files: -I$(MIDASSYS)/include
> > libraries and object files: -L$(MIDASSYS)/build/lib -lmidas
> > executables: PATH += $(MIDASSYS)/build/bin
> 
> Actually the library and executables go directly into the build directory (without "lib" and "bin"), so we need
> 
> header files: -I$(MIDASSYS)/include
> libraries and object files: -L$(MIDASSYS)/build -lmidas
> executables: PATH += $(MIDASSYS)/build
> 
> Or course that can be changed in the Cmake file, but not sure if that would be necessary/useful.

Actually I like the proposed separation between the library and the binaries, so I reworked it again. Now we have

header files: -I$(MIDASSYS)/include
libraries and object files: -L$(MIDASSYS)/lib -lmidas
executables: PATH += $(MIDASSYS)/bin

When issuing a "cmake .." followed by a "make" in the build directory, everything ends up in the build directory. To 
move things to the lib and bin directories, do a "make install". Seems to me like this is the standard way for
many packages so we should follow it.

Furthermore, I followed a proposal from KO to separate the code in the "src" directory between library source code
and programs. I moved all programs now to a separate "progs" directory, and left only code for the midas library in
the "src" directory. New CMakeLists.txt have been written for the "progs" and "utils" directories.

Care has been taken so that even when source files were moved around, their revision history is kept to "git annotate"
still works.

This is quite a change so sorry if this breaks some existing installations, but it will make things much easier in the future.

Stefan
  1539   03 Jun 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
> 1) cmake3 is available on all currently supported systems:
> 
> - SL6 (el6), CentOS7 (el7): yum install cmake3 (from EPEL) (invoke as "cmake3")
> - Ubuntu 18.04 LTS: apt-get install cmake (invoke as "cmake").
> - MacOS: install "mac ports", then "port install cmake"
> - Windows - we hope to revive windows10 support this summer

- el8 (RHEL8): cmake 3.11.something is part of the base system (latest cmake), (invoke as "cmake", the best I can tell).

K.O.
  1541   05 Jun 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
Status update on the cmake conversion:

- we have cmake builds working on all supported systems (el6, el7, ubuntu 18.04 LTS, macos 10.13, 10.14)
- I am happy with the result - for example, include file dependancies work much better now
- we are still fixing a few problems where the cmake build is different from the old make build (mfe.o, mlogger/rmlogger, etc)
- until all of these problems are straightened out, we cannot finalize the instructions for writing experiment makefiles (do we have to use -lmfe or 
we can keep the old mfe.o)

After everything is finalized, I hope to post a short guide for converting experiment makefiles to the new system.

The next release of MIDAS (midas-2019-06 series) will be the first C++ midas and cmake will be the primary build system.

K.O.
  1553   17 Jun 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
> Status update on the cmake conversion:

It looks like cmake cannot do several things we need for building midas.

- it looks like cmake does not support bare object files as 1st class build targets.

We build at least two bare object files: mfe.o and mana.o for use by frontends and analyzers (and I am about
to add one more for the manalyzer). They all contain main() functions and cannot be in libmidas.a.

Stefan & co managed to kludge cmake to build mfe.o but so far I have been unable to figure out how to tell cmake
to actually use it for linking. Replacing "mfe" for -llibmfe in target_include_libraries() with "mfe.o" yields -llibmfe.o, clearly
they do not support linking to bare library object files.

So to avoid fighting cmake, libmfe.a and libmana.a are here to stay.

- it looks like cmake does not like building variant executables and object files, i.e. "with ROOT" and "without ROOT".

I need to set "-DHAVE_ROOT" for building "with ROOT" and unset it via remove_definitions() for building "without ROOT",
but remove_definitions() and add_definitions() do not work on a per-target basis, instead they operate
per-directory and per-project.

In midas, we build mlogger without ROOT (to avoid tangling it with the ROOT RPATH and ROOT shared libraries),
but if ROOT is present, we build rmlogger "with ROOT support". Same for the analyzer (mana.o and rmana.o).

For now we have this:
- mana.o is built with ROOT if ROOT is detected
- rmana.o is not built
- rmlogger is not built (not clear why)

K.O.
  1557   17 Jun 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
> > Status update on the cmake conversion:

After the latest updates from Stefan & co, it looks like the cmake builds are working correctly,
there is only one bug remaining (rmlogger is not built). (rmana.o is also not built, but I think only 0 people use it).

I will test this in a couple of our test experiments, write the instructions for migrating from the old midas and tag a new release (midas-2019-06)

K.O.
  1558   17 Jun 2019 Stefan RittInfoMIDAS switching to Cmake
> - it looks like cmake does not like building variant executables and object files, i.e. "with ROOT" and "without ROOT".
> 
> I need to set "-DHAVE_ROOT" for building "with ROOT" and unset it via remove_definitions() for building "without ROOT",
> but remove_definitions() and add_definitions() do not work on a per-target basis, instead they operate
> per-directory and per-project.

You should not use per-directory and per-project definitions, but per-target definitions, such as

target_compile_options(mhttpd PRIVATE -DMG_ENABLE_SSL)

> In midas, we build mlogger without ROOT (to avoid tangling it with the ROOT RPATH and ROOT shared libraries),
> but if ROOT is present, we build rmlogger "with ROOT support". Same for the analyzer (mana.o and rmana.o).
> 
> For now we have this:
> - mana.o is built with ROOT if ROOT is detected
> - rmana.o is not built
> - rmlogger is not built (not clear why)

I added rmlogger to the install instructions. I believe it was always built, but just not installed into the /bin directory.

Stefan
  1562   18 Jun 2019 Konstantin OlchanskiInfoMIDAS switching to Cmake
> target_compile_options(rmlogger PRIVATE -DHAVE_ROOT)

Got it. Now I can build the duplets of mana.o and rmana.o (and .a) - mana always without ROOT, rmana with ROOT if available. This is the same as 
the old Makefile, the best I can tell.

With the fix to rmlogger, all known problems with the cmake build seem to be fixed.

K.O.
  1796   28 Jan 2020 Amy RobertsSuggestionMIDAS tested with MariaDB?
We're using the History Logger MIDAS feature and writing to mySQL tables, but 
in some cases have run into issues installing mySQL on centos7 systems.

Has anyone ever tried running this MIDAS feature with MariaDB rather than 
mySQL?
  1797   28 Jan 2020 Lukas GerritzenSuggestionMIDAS tested with MariaDB?
I have used the mySQL runlog with MariaDB before. I don't recall any problems on Fedora Core 29 or so (after MIDAS compiled).


Quote:
MariaDB intended to maintain high compatibility with MySQL, ensuring a drop-in replacement capability with library binary parity and exact matching with MySQL APIs and commands.
  1798   28 Jan 2020 Lee PoolSuggestionMIDAS tested with MariaDB?
> We're using the History Logger MIDAS feature and writing to mySQL tables, but 
> in some cases have run into issues installing mySQL on centos7 systems.
> 
> Has anyone ever tried running this MIDAS feature with MariaDB rather than 
> mySQL?

Hi

I've used MIDAS Logger feature with MariaDB and had no issue thus far .
  1803   02 Feb 2020 Konstantin OlchanskiSuggestionMIDAS tested with MariaDB?
> We're using the History Logger MIDAS feature and writing to mySQL tables, but 
> in some cases have run into issues installing mySQL on centos7 systems.
> 
> Has anyone ever tried running this MIDAS feature with MariaDB rather than 
> mySQL?

The best I can tell, MariaDB is *the* mysql. "the other thing" is an abandonware fork.

I personally at this moment do not run any daq stations with mysql logging, and I accidentally
removed all the 25 versions and flavours of mysql from my home laptop, so if there
is any problems with mysql, please holler, and I will reinstall mysql (ahem, mariadb) and restore
my ability to test midas against it.

As a connected question, is there any need to have postgres support in midas as well?

K.O.
ELOG V3.1.4-2e1708b5