ID |
Date |
Author |
Topic |
Subject |
1284
|
02 May 2017 |
Konstantin Olchanski | Info | added db_resize_string() | > Since we have been regularly running into problems with db_get_xxx(TID_STRING) and string buffers of mismatched size,
> I now implemented db_get_value_string(hdb, hkey, key_name, index, &string, create).
I run into problems with string arrays - non-array strings have unlimited length, but string arrays have fixed string length, usually set at creation time.
This causes a problem with growing arrays using db_get_value_string(), when converting a non-array variable to an array, the wrong
string length gets used, and one gets an array with useless string length. There is no way to specify the correct array string length
without adding more parameters to db_get_value_string() and confusing and complicating it for the typical case where it is used
against simple (non-array) odb entries.
To clarify the situation, db_get_value_string() was changed to reject attempts to resize an array and
calls of db_get_value_string(index>0 and create==TRUE) now return an error.
To create and resize string arrays, I added a new function - db_resize_array(hdb, hkey, key_name, num_values, max_string_size).
Here,
num_values is the new array size, making it possible to grow or shrink an array
max_string_size is the new string size, making it possible to change the array string length after the array was created (there was no midas function to do this before now).
I added a json-rpc call for db_resize_string().
But it still needs to be added to odbedit and mhttpd.
K.O. |
1280
|
26 Apr 2017 |
Konstantin Olchanski | Info | added db_get_value_string() | Since we have been regularly running into problems with db_get_xxx(TID_STRING) and string buffers of mismatched size,
I now implemented db_get_value_string(hdb, hkey, key_name, index, &string, create).
It works the same as db_get_value(TID_STRING), except that the string value is returned into an std::string object,
memory allocation is handled by std::string and there is no string length limit (other than std::string limits).
Accessing string arrays is done explicitly via an "index" parameter, if index is bigger than odb array size DB_OUT_OF_RANGE is returned
without logging an error message (e.g. db_get_data_index() will log an error). This makes is safe to iterate over array entries with a simple
loop of index from 0 and up until db_get returns an error.
As before, if the odb entry does not exist, it will be created (if create==true) and initialized with the value of the string parameter (zero-terminated in odb).
There is also newly added db_set_value_string() and cm_get_path_string(). if you want more of these, please ask, or send patches.
K.O. |
1282
|
26 Apr 2017 |
Stefan Ritt | Info | added db_get_value_string() | Just some thought for discussion:
Rather than "spicing up" the MIDAS library here and there with C++ objects such as std::string, wouldn't it make more sense to "cleanly" wrap an ODB value in a C++ class? We could use then
both APIs in parallel, and encourage the C++ API for new developments. We could then write things like:
ODBKEY<std::string> name("/Experiment/Name"); // constructor calls automatically db_get_value
name = "New Name"; // overloading the "=" operator, will call db_set_value()
or even
ODBKEY<std::vector, std::string> nameArray("...");
for (auto &s : nameArray)
std::cout << s << std::endl; // print all elements of string array
so we treat ODB arrays as vectors, which fixes array boundary violations nicely.
If the key does not exist, we could properly throw exceptions and forget about tons of nested return parameters for error conditions.
Many nice things could be done, common errors could be prevented, and we can do a "smooth" migration: We don't have to change the whole library completely, just where we feel it's currently
needed. So over time the code would be "objectified". Would be nice if we could rely on C++11 (like the "auto" feature above). Not sure about VxWorks, but every other OS should be fine.
Stefan
> Since we have been regularly running into problems with db_get_xxx(TID_STRING) and string buffers of mismatched size,
> I now implemented db_get_value_string(hdb, hkey, key_name, index, &string, create).
>
> It works the same as db_get_value(TID_STRING), except that the string value is returned into an std::string object,
> memory allocation is handled by std::string and there is no string length limit (other than std::string limits).
>
> Accessing string arrays is done explicitly via an "index" parameter, if index is bigger than odb array size DB_OUT_OF_RANGE is returned
> without logging an error message (e.g. db_get_data_index() will log an error). This makes is safe to iterate over array entries with a simple
> loop of index from 0 and up until db_get returns an error.
>
> As before, if the odb entry does not exist, it will be created (if create==true) and initialized with the value of the string parameter (zero-terminated in odb).
>
> There is also newly added db_set_value_string() and cm_get_path_string(). if you want more of these, please ask, or send patches.
>
> K.O. |
1289
|
02 May 2017 |
Konstantin Olchanski | Info | added db_get_value_string() | > Just some thought for discussion:
Even more thoughts:
- c++ interface for odb. been there, done that. see VirtualODB in rootana. Can access live ODB, XML odb dump from midas file, even ODB through http/mhttpd (needs to be converted to json rpc api).
- c++11. the ROOT team made the decision for us, for all practical reasons. RH/SL/CentOS <= 6 are left for dead. (but we still have machines as old as SL4).
- odb interface via severe operator overloading. writing "let x=42;" to simulate the universe from the big band to thermal death is elegant (overload operator= of class "let")
but there is a surprise for naive programmer (long run time, large memory consumption)
- c++ exceptions. defective by design, as they do not carry enough debug information (i.e. java exceptions carry the full stack trace). in the typical case, it is impossible to tell
who and why is throwing exceptions. error handling is reduced to "main() { try { real_main } catch exception { printf("sorry!"); }}.
see http://stackoverflow.com/questions/1736146/why-is-exception-handling-bad
- converting midas to a new simplified odb api. typical use via db_get_value() is already one (or two) line of code that cannot be reduced (have to specify odb path, tid, etc),
so little is gained from using a different api. getting rid of db_find_key()/db_get_key() would be helpful, but with db_get_value(), they are hardly ever used in new code.
There are weaknesses in the current api, would be nice to fix them some day, and a c++ api seems like the right way to go:
- fix the race condition between db_enum_key() and db_delete_key(). (it is same as between "ls" and "rm" - with nfs, try to "rm" on one client while running "ls" on another, fun!)
- fix the race condition between odb handles (pointers into shared memory) and db_delete_key() (and whatever else moves the keys around). This means using full odb paths for
all odb api functions.
- make it all work nice multithreaded - the above race conditions would become only worse if we encourage heavy use of threads in midas.
And I do need a "no-odb" odb api for my "no-midas" midas frontend framework (where I can build and run the frontend without linking and connecting with a real midas),
in practice it means all api "get" calls have to take a "default" value that is returned right back to me when I am not connected (or linked) with a real odb.
Good fodder for this summer discussions.
K.O.
>
> Rather than "spicing up" the MIDAS library here and there with C++ objects such as std::string, wouldn't it make more sense to "cleanly" wrap an ODB value in a C++ class? We could use then
> both APIs in parallel, and encourage the C++ API for new developments. We could then write things like:
>
> ODBKEY<std::string> name("/Experiment/Name"); // constructor calls automatically db_get_value
> name = "New Name"; // overloading the "=" operator, will call db_set_value()
>
> or even
>
> ODBKEY<std::vector, std::string> nameArray("...");
> for (auto &s : nameArray)
> std::cout << s << std::endl; // print all elements of string array
>
> so we treat ODB arrays as vectors, which fixes array boundary violations nicely.
>
> If the key does not exist, we could properly throw exceptions and forget about tons of nested return parameters for error conditions.
>
> Many nice things could be done, common errors could be prevented, and we can do a "smooth" migration: We don't have to change the whole library completely, just where we feel it's currently
> needed. So over time the code would be "objectified". Would be nice if we could rely on C++11 (like the "auto" feature above). Not sure about VxWorks, but every other OS should be fine.
>
> Stefan
>
> > Since we have been regularly running into problems with db_get_xxx(TID_STRING) and string buffers of mismatched size,
> > I now implemented db_get_value_string(hdb, hkey, key_name, index, &string, create).
> >
> > It works the same as db_get_value(TID_STRING), except that the string value is returned into an std::string object,
> > memory allocation is handled by std::string and there is no string length limit (other than std::string limits).
> >
> > Accessing string arrays is done explicitly via an "index" parameter, if index is bigger than odb array size DB_OUT_OF_RANGE is returned
> > without logging an error message (e.g. db_get_data_index() will log an error). This makes is safe to iterate over array entries with a simple
> > loop of index from 0 and up until db_get returns an error.
> >
> > As before, if the odb entry does not exist, it will be created (if create==true) and initialized with the value of the string parameter (zero-terminated in odb).
> >
> > There is also newly added db_set_value_string() and cm_get_path_string(). if you want more of these, please ask, or send patches.
> >
> > K.O. |
2526
|
09 Jun 2023 |
Konstantin Olchanski | Info | added IPv6 support for mserver and MIDAS RPC | as of commit 71fb5e82f3e9a1501b4dc2430f9193ee5d176c82, MIDAS RPC and the mserver
listen for connections both on IPv4 and IPv6. mserver clients and MIDAS RPC
clients can connect to MIDAS using both IPv4 and IPv6. In the default
configuration ("/Expt/Security/Enable non-localhost RPC" set to "n"), IPv4
localhost is used, as before. Support for IPv6 is a by product from switching
from obsolete non-thread-safe gethostbyname() and getaddrbyname() to modern
getaddrinfo() and getnameinfo(). This fixes bug 357, observed crash of mhttpd
inside gethostbyname(). K.O. |
475
|
02 Apr 2008 |
Konstantin Olchanski | Info | add "const" attributes to db_xxx() functions | Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some
problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB
access functions db_xxx(), i.e.
INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
becomes
INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the
rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.
P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by
the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a
complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())
K.O. |
476
|
02 Apr 2008 |
Konstantin Olchanski | Info | add "const" attributes to db_xxx() functions | Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some
problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB
access functions db_xxx(), i.e.
INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
becomes
INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the
rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.
P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by
the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a
complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())
K.O. |
477
|
02 Apr 2008 |
Stefan Ritt | Info | add "const" attributes to db_xxx() functions | > Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some
> problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB
> access functions db_xxx(), i.e.
> INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
> becomes
> INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
>
> If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the
> rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.
>
> P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by
> the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a
> complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())
I fully approve your idea. You are absolutely right that it also will help to prevent errors such as modifying
fixed strings. I was just too lazy to do that, because it requires some additional code like:
func(const char *p)
{
char str[256];
strlcpy(str, p, sizeof(str));
strlcat(str, ...)
}
So if you do it, it's great! |
478
|
03 Apr 2008 |
Konstantin Olchanski | Info | add "const" attributes to db_xxx() functions | > > I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB
> > access functions db_xxx(), i.e.
> > INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
> > becomes
> > INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
>
> I fully approve your idea.
Committed revision 4172.
K.O. |
2555
|
18 Jul 2023 |
Gennaro Tortone | Bug Report | access to filesystem through mhttpd | Hi,
after some networks security scans I received some warnings because mhttpd expose
server filesystem through HTTP(S)...
in details a MIDAS user can access to /etc/passwd or download other files from
filesystem using a web browser:
(e.g. http://midas.host:8080/etc/passwd)
I know that /etc/passwd does not contain users password and mhttpd runs as an
unprivileged user but in principle this should be avoided in order to minimize
security risks: if I authorize a user to use MIDAS interface in order to handle
acquisition tasks this should not authorize the user to access the server filesystem...
but this access should be restricted to MIDAS web pages, custom pages etc.
What do you think about this ?
Cheers,
Gennaro |
2557
|
18 Jul 2023 |
Konstantin Olchanski | Bug Report | access to filesystem through mhttpd | > (e.g. http://midas.host:8080/etc/passwd)
not again! I complained about this before, and I added a fix, but it must be broken again.
getting a copy of /etc/passwd is reasonably benign, but getting a copy of
/home/$USER/.ssh/id_rsa, id_rsa.pub, knownhosts and authorized_keys is a disaster.
(running mhttpd behind a web proxy does not solve the problem, number of attackers is
reduced to only the people who know the proxy password and to local users).
K.O. |
2558
|
19 Jul 2023 |
Zaher Salman | Bug Report | access to filesystem through mhttpd | Have you actually been able to read /etc/passwd this way? I tested this on a few of our servers and it does not work. As far as I know, there is access to files in resources, custom pages etc.
Other possible ways to access the file system is via mjsonrpc calls, but again these are restricted to certain folders.
Can you please give us more details about this.
Zaher
> > (e.g. http://midas.host:8080/etc/passwd)
>
> not again! I complained about this before, and I added a fix, but it must be broken again.
>
> getting a copy of /etc/passwd is reasonably benign, but getting a copy of
> /home/$USER/.ssh/id_rsa, id_rsa.pub, knownhosts and authorized_keys is a disaster.
>
> (running mhttpd behind a web proxy does not solve the problem, number of attackers is
> reduced to only the people who know the proxy password and to local users).
>
> K.O. |
895
|
26 Jul 2013 |
Konstantin Olchanski | Bug Report | abort on buffer overflow in odb.c::merge_records() | The odb.c function merge_records() has a fixed size array of 10000 bytes to handle the data and it
aborts with an assert() if passed data bigger than that. It is called from db_create_record() which
already allocates a data buffer of correct size for it's operations.
K.O. |
1622
|
16 Jul 2019 |
Konstantin Olchanski | Bug Report | a3818 and signals, Frontend killed at stop of run | Message from John M O'Donnell <odonnell@lanl.gov>
Folks,
The following might be related, if so great, if not sorry for the spam.
We had problems with MIDAS/CAEN_A3818 until two things happened:
1) CAEN found the root cause of a problem, as the A3818 and MIDAS both
used unix alarm signals, resulting in clashes. CAEN modified the
A3818 driver to have a "no alarm" option.
2) after downloading the modified driver, edit src/a3818.c to #define
USE_MIDAS 1 somewhere near the top.
Hope this helps,
John. |
1623
|
16 Jul 2019 |
Konstantin Olchanski | Bug Report | a3818 and signals, Frontend killed at stop of run | > Message from John M O'Donnell <odonnell@lanl.gov>
>
> the A3818 and MIDAS both used unix alarm signals, resulting in clashes.
>
FWIW, current midas no longer uses alarm signals. See forum messages and git commits about
removal of cm_watchdog().
K.O. |
2672
|
16 Jan 2024 |
Pavel Murat | Forum | a scroll option for "add history variables" window? | Dear all,
I have a "slow control" frontend which reads out 100 slow control parameters.
When I'm interactively adding a parameter to a history plot,
a nice "Add history variable" pops up .. , but with 100 parameters in the list,
it doesn't fit within the screen...
The browser becomes passive, and I didn't find any easy way of scrolling.
In the attached example, adding a channel 32 variable becomes rather cumbersome,
not speaking about channel 99.
Two questions:
a) how do people get around this "no-scrolling" issue? - perhaps there is a workaround
b) how big of a deal is it to add a scroll bar to the "Add history variables" popup ?
- I do not know javascript myself, but could find help to contribute..
-- many thanks, regards, Pasha |
Attachment 1: adding_a_variable_to_the_history_plot.png
|
|
2673
|
16 Jan 2024 |
Stefan Ritt | Forum | a scroll option for "add history variables" window? | Have you updated to the current midas version? This issue has been fixed a while ago. Below
you see a screenshot of a long list scrolled all the way to the bottom.
Revision: Thu Dec 7 14:26:37 2023 +0100 - midas-2022-05-c-762-g1eb9f627-dirty on branch
develop
Chrome on MacOSX 14.2.1
The fix is actually in "controls.js", so make sure your browser does not cache an old
version of that file. I usually have to clear my browser history to get the new file from
mhttpd.
Best regards,
Stefan |
Attachment 1: Screenshot_2024-01-17_at_08.17.30.png
|
|
2677
|
17 Jan 2024 |
Pavel Murat | Forum | a scroll option for "add history variables" window? | > Have you updated to the current midas version? This issue has been fixed a while ago.
Hi Stefan, thanks a lot! I pulled from the head, and the scrolling works now. -- regards, Pasha |
2690
|
28 Jan 2024 |
Konstantin Olchanski | Forum | a scroll option for "add history variables" window? | > > Have you updated to the current midas version? This issue has been fixed a while ago.
>
> Hi Stefan, thanks a lot! I pulled from the head, and the scrolling works now. -- regards, Pasha
Right, I remember running into this problem, too.
If you have some ideas on how to better present 100500 history variables, please shout out!
K.O. |
2697
|
29 Jan 2024 |
Pavel Murat | Forum | a scroll option for "add history variables" window? | > If you have some ideas on how to better present 100500 history variables, please shout out!
let me share some thoughts. In a particular case which lead to the original posting,
I was using a multi-threaded driver and monitoring several pieces of equipment with different device drivers.
In fact, it was not even hardware, but processes running on different nodes of a distributed computer farm.
To reduce the number of frontends, I was combining together the output of what could've been implemented
as multiple slow control drivers and got 100+ variables in the list - hence the scrolling experience.
At the same time, a list of control variables per driver could've been kept relatively short.
So if a list of control variables of a slow control frontend were split in a History GUI not only by the
equipment piece, but within the equipment "folder", also by the driver, that might help improving
the scalability of the graphical interface.
May be that is already implemented and it is just a matter of me not finding the right base class / example
in the MIDAS code
-- regards, Pasha |
|