> ODB name lengths (the name of a key) are limited to 256 characters, the length of strings in the ODB should NOT be limited.

Right, I was not ever aware of such limitation until I just now looked at the .odb and .xml writing code. Definitely string length
is truncated to MAX_STRING_LENGTH on writing, chokes or truncates on reading.

The new json reader/writer handles overlength strings correctly. I would say we should deprecate the old formats and go forward
with json. Most current software can work with json data much easier than xml or custom .odb.

> I see now that db_paste & co. is hopelessly broken. To fix it, everything should be changed to std::string which is in my opinion 
> the only 'clean' solution. That would also remove the cumbersome strlcpy and strlcat.

Yes, that's the code for reading .odb format.

>
> But looking at odb.c, replacing everything with std::string would probably take a brave programmer a couple of weeks. Not sure if we should dive into that adventure right now.
>

I agree. Too much of an adventure.

Simpler solution could be add a db_get_data(), db_get_value() that allocates a data buffer of correct size (user has to remember to free it).

> a) The strings "CSS File", "Sqlite dir" etc. reported below get reduced to 256 characters (MAX_STRING_LENGTH).

We should fix the inconsistency, my vote is it should be either MAX_STRING_LENGTH or PATH_MAX (from limits.h).

K.O.

> ... the xml reader, probably has same problem
> ... xml writer truncates long strings via truncation in db_sprintf()

Removed truncation of overlong strings in the xml writer and confirmed that xml reader handles them correctly (always loaded overlong strings correctly).

Both JSON and XML odb dumps now handle strings of unlimited size correctly.

K.O.

The define for MAX_EVENT_SIZE was removed from midas.h.

Replacing it is DEFAULT_MAX_EVENT_SIZE set to 4 MiBytes and DEFAULT_BUFFER_SIZE 
set to 32 MiBytes.

For a long time now MIDAS does not have hardcoded maximum event size and buffer size 
and this change merely renames the define to reflect it's current function.

The actual maximum event size is set by ODB /Experiment/MAX_EVENT_SIZE.

The actual event buffer sizes are set by ODB "/Experiment/Buffer sizes/SYSTEM" & co

K.O.

Hi all,

I've ran into a problem where my experiment gets interrupted with a message from 
the logger saying that my disk is nearly full. This does not make sense to me 
because I have deleted almost all the data files from my data directory. I'm 
guessing that somewhere the ODB perceives that the directory is full when in 
reality its not.

Here is the exact message:

[ODBEdit,INFO] Run #252 stopped

09:22:19 [Logger,TALK] disk nearly full, stopping the run

09:22:19 [Logger,ERROR] [mlogger.cxx:4475:log_write,ERROR] Disk '/home/caendaq/A       
NIS/data/run00252.mid.lz4' is almost full: 81 MiBytes free out of 922497 MiBytes       
, stopping the run

Does any body have a solution for this? Thanks so much.

Isaac

The logger simple requests the disk free space level from the operating system in the same 
way as the "df" command does. Can you do a "df" on your system? I have seen that some file 
systems free up space not immediately if you delete files, but some times later (like 24h).

Stefan

> The logger simple requests the disk free space level from the operating system in the same 
> way as the "df" command does. Can you do a "df" on your system? I have seen that some file 
> systems free up space not immediately if you delete files, but some times later (like 24h).
> 
> Stefan

Thanks Stefan. Yes the files were still held open by some processes. It's solved now.

Cheers.

Isaac

Hello,
     I'm experiencing crashes of the mlogger program on the time scale of a couple 
of days. The only messages from MIDAS are:

05:34:47.336 2021/10/24 [mhttpd,INFO] Client 'Logger' (PID 14281) on database 
'ODB' removed by db_cleanup called by cm_periodic_tasks (idle 10.2s,TO 10s)
05:34:47.335 2021/10/24 [mhttpd,INFO] Client 'Logger' on buffer 'SYSMSG' removed 
by cm_periodic_tasks (idle 10.2s, timeout 10s)

Any suggestion to further investigate this issue?

Thank you very much,
         Francesco

The short term solution would be to increase the logger timeout in the ODB under

/Programs/Logger/Watchdog timeout

and set it to 6000 (one minute). But that is curing just the symptoms. It would be 
interesting to understand the cause of this error. Probably the logger takes more than 10 
seconds to start or stop the run. The reason could be that the history grow too big (what 
we have right now in MEG II), or some disk problems. But that needs detailed debugging on 
the logger side.

Stefan

Hello,

We are going to implement a list of sequencer files to allow users to select one 
of them. The name of this file will be transferred to 
/ODB/Sequencer/State/Filename field of ODB. 

Is it possible to get a list of Sequencer files from MIDAS? Is there a jrpc 
command for this?

Thanks.

Best,
Ruslan

If you load a file into the sequencer from the web interface, you get a list of all files in that directory. 
This basically gives you a list of possible sequencer files. It's even more powerful, since you can 
create subdirectories and thus group the sequencer files. Attached an example from our 
experiment.

Stefan

> If you load a file into the sequencer from the web interface, you get a list of all files in that directory. 
> This basically gives you a list of possible sequencer files. It's even more powerful, since you can 
> create subdirectories and thus group the sequencer files. Attached an example from our 
> experiment.
> 
> Stefan


Dear Stefan,

Thank you for the explanation.

Ruslan

> If you load a file into the sequencer from the web interface, you get a list of all files in that directory. 
> This basically gives you a list of possible sequencer files. It's even more powerful, since you can 
> create subdirectories and thus group the sequencer files. Attached an example from our 
> experiment.
> 
> Stefan

Dear Stefan,

Could you please answer one more question:

We have a custom webpage and trying to get list of files from the custom webpage and need jrpc command to show it 
in custom page. Is there a jrpc command to get this file list?

Thanks,

> 
> We have a custom webpage and trying to get list of files from the custom webpage and need jrpc command to show it 
> in custom page. Is there a jrpc command to get this file list?
> 

The rpc method used by sequencer web pages is "seq_list_files". How to use it, see resources/load_script.html.

To see list of all rpc methods implemented by your mhttpd, see "help"->"json-rpc schema, text format".

As general explanation, so far we have successfully resisted the desire to turn mhttpd into a generic NFS file
server - if we automatically give all web pages access to all files accessible to the midas user account, it is easy
to lose control of system security (i.e. bad things will happen if web pages can read the ssh private keys ~/.ssh/id_rsa and
modify ~/.ssh/authorized_keys). Generally it is impossible to come up with a whitelist or blacklist of "secrets" that
need to be "hidden" from web pages. But we did implement methods to access files from specific subdirectory trees
defined in ODB which hopefully do not contain any "secrets".

K.O.

I cannot checkout from the cvs server. So I download each latest file from the WWW
interface of CVS. While compiling these files, I encountered the following problems:
-------------
...
g++ -DHAVE_ROOT -c -g -O2 -Wall -Iinclude -Idrivers -Ldarwin/lib -DINCLUDE_FTPLIB  
-DOS_LINUX -DOS_DARWIN -DHAVE_STRLCPY -fPIC -Wno-unused-function -D_REENTRANT
-I/sw/include -I/opt/root/current/include -Wl,-rpath,/opt/root/current/lib -o
darwin/lib/rmana.o src/mana.c
g++: -rpath: linker input file unused because linking not done
g++: /opt/root/current/lib: linker input file unused because linking not done
...
g++ -g -O2 -Wall -Iinclude -Idrivers -Ldarwin/lib -DINCLUDE_FTPLIB   -DOS_LINUX
-DOS_DARWIN -DHAVE_STRLCPY -fPIC -Wno-unused-function -DHAVE_ROOT -D_REENTRANT
-I/sw/include -I/opt/root/current/include -Wl,-rpath,/opt/root/current/lib -o
darwin/bin/mlogger src/mlogger.c darwin/lib/libmidas.a -L/opt/root/current/lib -u
_G__cpp_setupG__Hist -u _G__cpp_setupG__Graf1 -u _G__cpp_setupG__G3D -u
_G__cpp_setupG__GPad -u _G__cpp_setupG__Tree -u _G__cpp_setupG__Rint -u
_G__cpp_setupG__PostScript -u _G__cpp_setupG__Matrix -u _G__cpp_setupG__Quadp -u
_G__cpp_setupG__Physics -lCore -lCint -lHist -lGraf -lGraf3d -lGpad -lTree -lRint
-lPostscript -lMatrix -lQuadp -lPhysics -lpthread -lm -L/sw/lib -ldl -lpthread
ld: unknown flag: -rpath
gmake: *** [darwin/bin/mlogger] Error 1
---------------
What does '-rpath' mean? It is just a linking error. Thanks.

> ld: unknown flag: -rpath
> gmake: *** [darwin/bin/mlogger] Error 1

Fixed. Good catch.

> What does '-rpath' mean?

You will have to read the "ld" manual. In the nutshell, it tells the executable where to look for shared libraries. 
Aparently it is not supported by Mac OS X.

K.O.

180:         printf("camop\n");
181:         camop();

void cam_op()
{
}

mcnaf.c:(.text+0x3b1): undefined reference to `camop'
collect2: ld returned 1 exit status

> If you compile each source into an object, you may encounter a link error ...

Also camac rpc did not work at all last time we tried to use it at triumf, maybe 4 month ago in the Dragon 
experiment (upgrade from older version of midas). Never got around to trace down why. YMMV.

K.O.

drivers/camac/camaclx.c:760:INLINE void cam_op()
drivers/camac/camacnul.c:200:INLINE void cam_op()
drivers/camac/camacrpc.c:563:void cam_op()
drivers/camac/cc7700pci.c:744:INLINE void cam_op()
drivers/camac/ces2117.c:227:void cam_op()
drivers/camac/ces8210.c:553:void cam_op(void)
drivers/camac/ces8210.c:576:    cam_op();
drivers/camac/ces8210.c:625:    cam_op();
drivers/camac/dsp004.c:692:void cam_op()
drivers/camac/hyt1331.c:1125:INLINE void cam_op()
drivers/camac/jorway73a.c:563:INLINE void cam_op()
drivers/camac/kcs2926.c:618:INLINE void cam_op()
drivers/camac/kcs2927.c:677:INLINE void cam_op()
drivers/camac/wecc32.c:554:INLINE void cam_op()

cc -g -O3 -Wall -Wuninitialized -Iinclude -Idrivers -I../mxml -Llinux/lib -DINCLUDE_FTPLIB   -D_LARGEFILE64_SOURCE -DHAVE_MYSQL -I/usr/include/mysql -DHAVE_ODBC -DHAVE_ZLIB -DOS_LINUX -fPIC -Wno-unused-function -o linux/bin/mcnaf utils/mcnaf.c drivers/camac/camacrpc.c linux/lib/libmidas.a -lutil -lpthread -lodbc -lz

#ifndef _ANULLF_H_
#define _ANULLF_H_ 1

#define EXTERNAL extern

#if defined( _MSC_VER )
#define INLINE __inline
#elif defined(__GNUC__)
#define INLINE __inline__
#else
#define INLINE
#endif
/* make functions under WinNT dll exportable */
#if defined(_MSC_VER) && defined(MIDAS_DLL)
#define EXPRT __declspec(dllexport)
#else
#define EXPRT
#endif

EXTERNAL INLINE void EXPRT camop() {  };

#endif

#include "anullf.h"

void cam_op() {}

#include "anullf.h"
#include <stdio.h>

int main(int argc, char** argv)
{
  camop();
}

$ gcc -O -o ma ma_1.c ma.c

Hi, 
I am missing a few features. Do any of the following exist and I have just
overlooked them?

- Arithmetics:
    SET foo, 2
    SET bar, 2
    SET FOOBAR, $foo + $bar
    -> FOOBAR is 2, not 4

- Vectors/arrays
    As far as I understand, "SET" only allows for single variables and no 
    "scripting" in variable names, i. e. doing something like the following
    scripted: 
    SET var_1, 0
    SET var_2, 0
    ...
    LOOP n, 10
        ...
        IF something
            SET var_${n}, 1
        ENDIF 
    
    The only way I could think of doing something similar is via the ODB.
    I don't know if it's good practice to fill the ODB with variables like this.

- Loading scripts at run time (see other bug)
    That would allow for script manipulation, even though it's kind of dirty, it
    might be useful in some cases.

- Obtaining results from external scripts
    A way to pass things from external scripts would be the solution to all
    problems. Something that is not implemented could be done in a bash or
    python script instead. 

Cheers
Lukas

> I am missing a few features.

MSL did not start out as a fully featured programming language.

Rather than extending it until it becomes one, I think a better approach would be to take
one of the existing extensible scripting language libraries and extend it with midas functions.

For example, LUA (https://www.lua.org/about.html) seems to be popular.

There were also requests for a midas extension for PYTHON (actually this has been done before,
but never added to the midas repository. I still have that code and I suppose it could be resurrected).

K.O.


> Do any of the following exist and I have just
> overlooked them?
> 
> - Arithmetics:
>     SET foo, 2
>     SET bar, 2
>     SET FOOBAR, $foo + $bar
>     -> FOOBAR is 2, not 4
> 
> - Vectors/arrays
>     As far as I understand, "SET" only allows for single variables and no 
>     "scripting" in variable names, i. e. doing something like the following
>     scripted: 
>     SET var_1, 0
>     SET var_2, 0
>     ...
>     LOOP n, 10
>         ...
>         IF something
>             SET var_${n}, 1
>         ENDIF 
>     
>     The only way I could think of doing something similar is via the ODB.
>     I don't know if it's good practice to fill the ODB with variables like this.
> 
> - Loading scripts at run time (see other bug)
>     That would allow for script manipulation, even though it's kind of dirty, it
>     might be useful in some cases.
> 
> - Obtaining results from external scripts
>     A way to pass things from external scripts would be the solution to all
>     problems. Something that is not implemented could be done in a bash or
>     python script instead. 
> 
> Cheers
> Lukas