> Hi,
> 
> Attempting to copy a subtree to a new location in the ODB using odbedit with "copy <src> <dest>" is 
> occasionally not copying the entire <src> subtree.
> 
> I am experiencing this issue consistently when trying to copy subtrees from the "/Equipment" ODB tree to 
> a new location.  The first 2-3 variables/directories of the <src> subtree will be copied to <dest> but the 
> full subtree will not be copied over.

I just tried myself and could successfully copy of even large trees in the /Equipment subtree. Need to reproduce the problem to fix 
it. Maybe close-to-full ODB?

Stefan

Can the MIDAS Sequencer call an external script? It seems that it should be able to. I have a simple 
test script to do so. It claims to execute, but the bash script never appears to be executed. Any 
suggestions?

1 COMMENT "This is a MSL test file"
  2 RUNDESCRIPTION "Test run"
  3 
  4 LOOP setting, 1,2, 3
  5      SCRIPT test_wheel.sh ,$setting 
  6      TRANSITION START
  7      WAIT Seconds 10
  8      TRANSITION STOP
  9 ENDLOOP

I've also tried using an xml script with <Script params="1">test_wheel.sh</Script>, but with the same 
result.

Thanks!

Nevermind. I just had to give it a path to my script. Now it's fine. 

> Can the MIDAS Sequencer call an external script? It seems that it should be able to. I have a simple 
> test script to do so. It claims to execute, but the bash script never appears to be executed. Any 
> suggestions?
> 
> 1 COMMENT "This is a MSL test file"
>   2 RUNDESCRIPTION "Test run"
>   3 
>   4 LOOP setting, 1,2, 3
>   5      SCRIPT test_wheel.sh ,$setting 
>   6      TRANSITION START
>   7      WAIT Seconds 10
>   8      TRANSITION STOP
>   9 ENDLOOP
> 
> I've also tried using an xml script with <Script params="1">test_wheel.sh</Script>, but with the same 
> result.
> 
> Thanks!

I've recently run into issues when using JSON.parse on ODB keys containing 
8-bit data.

For JSON.parse to successfully parse a string, (A) the string must be valid 
UTF-8, (B) several whitespace characters, control characters, and the 
characters " and \ must be escaped, and (C) you've got to follow the key-
value rules laid out in http://www.json.org/.

The web browser takes care of (A), and I verified that for this key Midas 
handled (C) correctly.  In principle, the function json_write in odb.c 
handles (B) - but json_write does not escape control characters.

To manage this problem, I modified json_write (in odb.c) to replace any 
control character with the more-inocuous character, 'C'.  My default case 
now looks like:

default:
         {
           // if a char is a control character,
           // print 'C' in its place
           // note that this loses data:
           // a more-correct method would be to print
           // \uXXXX, where XXXX is the character in hex
           if(iscntrl(*s)){
             (*buffer)[(*buffer_end)++] = 'C';
             s++;
           } else {
             (*buffer)[(*buffer_end)++] = *s++;
           }
         }
      
Where the call to iscntrl(*s) requires the addition of the ctype.h header 
file.

I'm guessing a blanket replacement of control characters with 'C' isn't 
something all Midas users would want to do.  Replacing the control character 
with its hex value seems like a good choice - but not without adding bounds 
checking!

An alternative to changing odb.c could be to add a regex to Midas response 
text which removes all control characters (U+0000 - U+001F): 

var resp_lint = req.response.replace(/[\u{0000}-\u{001F}]/gmu, '');
var json_obj = JSON.parse(resp_lint);

Unfortunately, the 'u' regex flax doesn't work on the Firefox version 
included in Scientific Linux 6.8.  

> I've recently run into issues when using JSON.parse on ODB keys containing 
> 8-bit data.

I am tempted to take a hard line and say that in general MIDAS TID_STRING data should be valid 
UTF-8 encoded Unicode. In the modern mixed javascript/json/whatever environment I think
it is impractical to handle or permit invalid UTF-8 strings.

Certainly in the general case, replacing all control characters with something else or escaping them or 
otherwise changing the value if TID_STRING data would wreck *valid* UTF-8 strings, which I would 
assume to be the normal use.

In other words, non-UTF-8 strings are following non-IEEE-754 floating point values into oblivion - as 
we do not check the TID_FLOAT and TID_DOUBLE is valid IEEE-754 values, we should not check 
that TID_STRING is valid UTF-8.

But in your specific case, why do you have random control characters in your TID_STRING data? 
Maybe you are using TID_STRING as general storage instead of arrays of TID_CHAR or 
TID_DWORD?

K.O.



> 
> For JSON.parse to successfully parse a string, (A) the string must be valid 
> UTF-8, (B) several whitespace characters, control characters, and the 
> characters " and \ must be escaped, and (C) you've got to follow the key-
> value rules laid out in http://www.json.org/.
> 
> The web browser takes care of (A), and I verified that for this key Midas 
> handled (C) correctly.  In principle, the function json_write in odb.c 
> handles (B) - but json_write does not escape control characters.
> 
> To manage this problem, I modified json_write (in odb.c) to replace any 
> control character with the more-inocuous character, 'C'.  My default case 
> now looks like:
> 
> default:
>          {
>            // if a char is a control character,
>            // print 'C' in its place
>            // note that this loses data:
>            // a more-correct method would be to print
>            // \uXXXX, where XXXX is the character in hex
>            if(iscntrl(*s)){
>              (*buffer)[(*buffer_end)++] = 'C';
>              s++;
>            } else {
>              (*buffer)[(*buffer_end)++] = *s++;
>            }
>          }
>       
> Where the call to iscntrl(*s) requires the addition of the ctype.h header 
> file.
> 
> I'm guessing a blanket replacement of control characters with 'C' isn't 
> something all Midas users would want to do.  Replacing the control character 
> with its hex value seems like a good choice - but not without adding bounds 
> checking!
> 
> An alternative to changing odb.c could be to add a regex to Midas response 
> text which removes all control characters (U+0000 - U+001F): 
> 
> var resp_lint = req.response.replace(/[\u{0000}-\u{001F}]/gmu, '');
> var json_obj = JSON.parse(resp_lint);
> 
> Unfortunately, the 'u' regex flax doesn't work on the Firefox version 
> included in Scientific Linux 6.8.  

Hi Midas forum,

I'm having a problem with odb hotlinks after increasing sub-directories in an 
odb. I now get the error code DB_NO_MEMORY after some db_open_record() calls. I 
tried 

1) increasing the parameter DEFAULT_ODB_SIZE in midas.h and make clean, make
but got the same error

2) increasing the parameter  MAX_OPEN_RECORDS in midas.h and make clean, make
but got fatal errors from odbedit and my midas FE and couldnt run anything

3) deleting my expts SHM files and starting odbedit with "odbedit -e SLAC -s 
0x1000000" to increse the odb size but got the same error?

4) I tried a different computer and got the same error code DB_NO_MEMORY

Maybe I running into some system limit that restricts the humber of open records? 
Or maybe I've not increased the correct midas parameter?

Best ,Tim.

> > I've recently run into issues when using JSON.parse on ODB keys containing 
> > 8-bit data.
> 
> I am tempted to take a hard line and say that in general MIDAS TID_STRING data should be valid 
> UTF-8 encoded Unicode. In the modern mixed javascript/json/whatever environment I think
> it is impractical to handle or permit invalid UTF-8 strings.
> ....
> But in your specific case, why do you have random control characters in your TID_STRING data? 
> Maybe you are using TID_STRING as general storage instead of arrays of TID_CHAR or 
> TID_DWORD?

I'm a little confused by this report and want to make sure I understand the situation.  Konstantin points
out that the TID_STRING should be valid UTF-8.  But I think that Amy agreed that the string was valid UTF-8.
 My understanding was that Amy's contention was that the valid UTF-8 string didn't get returned as valid JSON.

But I am having trouble reproducing your behaviour Amy.  I created a ODB string variable with a tab control
control character

  sprintf(mystring,"first line \t second line");
  status = db_set_value(hDB, 0,"/test2/mystring", &mystring, size, 1, TID_STRING);

and what I tried to pull the ODB using jcopy

http://neut18:8081/?cmd=jcopy&odb=/test2/mystring&format=json

I got 

{
"mystring/key" : { "type" : 12, "item_size" : 32, "access_mode" : 7, "last_written" : 1477416322 },
"mystring" : "first line \t second line"
}

which seems to be valid JSON.  

I only tried this with tab.  Are there other control characters that you are having trouble with?  Or maybe
I misunderstand the question?





> 
> > 
> > For JSON.parse to successfully parse a string, (A) the string must be valid 
> > UTF-8, (B) several whitespace characters, control characters, and the 
> > characters " and \ must be escaped, and (C) you've got to follow the key-
> > value rules laid out in http://www.json.org/.
> > 
> > The web browser takes care of (A), and I verified that for this key Midas 
> > handled (C) correctly.  In principle, the function json_write in odb.c 
> > handles (B) - but json_write does not escape control characters.
> > 
> > To manage this problem, I modified json_write (in odb.c) to replace any 
> > control character with the more-inocuous character, 'C'.  My default case 
> > now looks like:
> > 
> > default:
> >          {
> >            // if a char is a control character,
> >            // print 'C' in its place
> >            // note that this loses data:
> >            // a more-correct method would be to print
> >            // \uXXXX, where XXXX is the character in hex
> >            if(iscntrl(*s)){
> >              (*buffer)[(*buffer_end)++] = 'C';
> >              s++;
> >            } else {
> >              (*buffer)[(*buffer_end)++] = *s++;
> >            }
> >          }
> >       
> > Where the call to iscntrl(*s) requires the addition of the ctype.h header 
> > file.
> > 
> > I'm guessing a blanket replacement of control characters with 'C' isn't 
> > something all Midas users would want to do.  Replacing the control character 
> > with its hex value seems like a good choice - but not without adding bounds 
> > checking!
> > 
> > An alternative to changing odb.c could be to add a regex to Midas response 
> > text which removes all control characters (U+0000 - U+001F): 
> > 
> > var resp_lint = req.response.replace(/[\u{0000}-\u{001F}]/gmu, '');
> > var json_obj = JSON.parse(resp_lint);
> > 
> > Unfortunately, the 'u' regex flax doesn't work on the Firefox version 
> > included in Scientific Linux 6.8.  

oOne additional comment. I was able to trace the setting of the error code DB_NO_MEMORY 
to a call to the db_add_open_record() by mserver that is initiated during the start-up 
of my frontend via an RPC call. I checked with a debug printout that I have indeed 
reached the number of MAX_OPEN_RECORDS

> Hi Midas forum,
> 
> I'm having a problem with odb hotlinks after increasing sub-directories in an 
> odb. I now get the error code DB_NO_MEMORY after some db_open_record() calls. I 
> tried 
> 
> 1) increasing the parameter DEFAULT_ODB_SIZE in midas.h and make clean, make
> but got the same error
> 
> 2) increasing the parameter  MAX_OPEN_RECORDS in midas.h and make clean, make
> but got fatal errors from odbedit and my midas FE and couldnt run anything
> 
> 3) deleting my expts SHM files and starting odbedit with "odbedit -e SLAC -s 
> 0x1000000" to increse the odb size but got the same error?
> 
> 4) I tried a different computer and got the same error code DB_NO_MEMORY
> 
> Maybe I running into some system limit that restricts the humber of open records? 
> Or maybe I've not increased the correct midas parameter?
> 
> Best ,Tim.

Hi Tim,

I reproduced your problem and then managed to go through a procedure to increase the number of allowable open records.  The following is the procedure that I used 

1) Use odbedit to save current ODB

odbedit
save current_odb.odb

2) Stop all the running MIDAS processes, including mlogger and mserver using the web interface. Then stop mhttpd as well.

3) Remove your old ODB (we will recreate it after modifying MIDAS, using the backup you just made).

mv .ODB.SHM .ODB.SHM.20161104
rm /dev/shm/thomas_ODB_SHM

4) Make the following modifications to midas.  In this particular case I have increased the max number of open records from 256 to 1024.  You would need to change the constants if you want to change to other values

diff --git a/include/midas.h b/include/midas.h
index 02b30dd..33be7be 100644
--- a/include/midas.h
+++ b/include/midas.h
@@ -254,7 +254,7 @@ typedef std::vector<std::string> STRING_LIST;
-#define MAX_OPEN_RECORDS       256           /**< number of open DB records   */
+#define MAX_OPEN_RECORDS       1024           /**< number of open DB records   */
diff --git a/src/odb.c b/src/odb.c
index 47ace8f..ac1bef3 100755
--- a/src/odb.c
+++ b/src/odb.c
@@ -699,8 +699,8 @@ static void db_validate_sizes()
-   assert(sizeof(DATABASE_CLIENT) == 2112);
-   assert(sizeof(DATABASE_HEADER) == 135232);
+   assert(sizeof(DATABASE_CLIENT) == 8256);
+   assert(sizeof(DATABASE_HEADER) == 528448);

The calculation is as follows (in case you want a different number of open records): 
DATABASE_CLIENT = 64 + 8*MAX_OPEN_ERCORDS = 64 + 8*1024 = 8256
DATABASE_HEADER = 64 + 64*DATABASE_CLIENT = 64 + 64*8256 = 528448

5) Rebuild MIDAS

make clean; make

6) Create new ODB

odbedit -s 1000000

Change the size of the ODB to whatever you want.

7) reload your original ODB 

load  current_odb.odb

8) Rebuild your frontend against new MIDAS; then it should work and you should be able to produce more open records.

8.5*) Actually, I had a weird error where I needed to remove my .SYSTEM.SHM file as well when I first restarted my front-end.  Not sure if that was some unrelated error, but I mention it here for completeness.

This was a procedure based on something that originally was used for T2K (procedure by Renee Poutissou).  It is possible that not all steps are necessary and that there is a better way.  But this worked for me.

Also, any objections from other developers to tweaking the assert checks in odb.c so that the values are calculated automatically and MIDAS only needs to be touched in one place to modify the number of open records?

Let me know if it worked for you and I'll add these instructions to the Wiki.

Thomas


> oOne additional comment. I was able to trace the setting of the error code DB_NO_MEMORY 
> to a call to the db_add_open_record() by mserver that is initiated during the start-up 
> of my frontend via an RPC call. I checked with a debug printout that I have indeed 
> reached the number of MAX_OPEN_RECORDS
> 
> > Hi Midas forum,
> > 
> > I'm having a problem with odb hotlinks after increasing sub-directories in an 
> > odb. I now get the error code DB_NO_MEMORY after some db_open_record() calls. I 
> > tried 
> > 
> > 1) increasing the parameter DEFAULT_ODB_SIZE in midas.h and make clean, make
> > but got the same error
> > 
> > 2) increasing the parameter  MAX_OPEN_RECORDS in midas.h and make clean, make
> > but got fatal errors from odbedit and my midas FE and couldnt run anything
> > 
> > 3) deleting my expts SHM files and starting odbedit with "odbedit -e SLAC -s 
> > 0x1000000" to increse the odb size but got the same error?
> > 
> > 4) I tried a different computer and got the same error code DB_NO_MEMORY
> > 
> > Maybe I running into some system limit that restricts the humber of open records? 
> > Or maybe I've not increased the correct midas parameter?
> > 
> > Best ,Tim.

Hi Tim,

I reproduced your problem and then managed to go through a procedure to increase the number
of allowable open records.  The following is the procedure that I used 

1) Use odbedit to save current ODB

odbedit
save current_odb.odb

2) Stop all the running MIDAS processes, including mlogger and mserver using the web
interface. Then stop mhttpd as well.


3) Remove your old ODB (we will recreate it after modifying MIDAS, using the backup you just
made).

mv .ODB.SHM .ODB.SHM.20161104
rm /dev/shm/thomas_ODB_SHM

4) Make the following modifications to midas.  In this particular case I have increased the
max number of open records from 256 to 1024.  You would need to change the constants if you
want to change to other values

diff --git a/include/midas.h b/include/midas.h
index 02b30dd..33be7be 100644
--- a/include/midas.h
+++ b/include/midas.h
@@ -254,7 +254,7 @@ typedef std::vector<std::string> STRING_LIST;
-#define MAX_OPEN_RECORDS       256           /**< number of open DB records   */
+#define MAX_OPEN_RECORDS       1024           /**< number of open DB records   */
diff --git a/src/odb.c b/src/odb.c
index 47ace8f..ac1bef3 100755
--- a/src/odb.c
+++ b/src/odb.c
@@ -699,8 +699,8 @@ static void db_validate_sizes()
-   assert(sizeof(DATABASE_CLIENT) == 2112);
-   assert(sizeof(DATABASE_HEADER) == 135232);
+   assert(sizeof(DATABASE_CLIENT) == 8256);
+   assert(sizeof(DATABASE_HEADER) == 528448);

The calculation is as follows (in case you want a different number of open records): 
DATABASE_CLIENT = 64 + 8*MAX_OPEN_ERCORDS = 64 + 8*1024 = 8256
DATABASE_HEADER = 64 + 64*DATABASE_CLIENT = 64 + 64*8256 = 528448

5) Rebuild MIDAS

make clean; make

6) Create new ODB

odbedit -s 1000000

Change the size of the ODB to whatever you want.

7) reload your original ODB 

load  current_odb.odb

8) Rebuild your frontend against new MIDAS; then it should work and you should be able to
produce more open records.

8.5*) Actually, I had a weird error where I needed to remove my .SYSTEM.SHM file as well
when I first restarted my front-end.  Not sure if that was some unrelated error, but I
mention it here for completeness.

This was a procedure based on something that originally was used for T2K (procedure by Renee
Poutissou).  It is possible that not all steps are necessary and that there is a better way.
 But this worked for me.

Also, any objections from other developers to tweaking the assert checks in odb.c so that
the values are calculated automatically and MIDAS only needs to be touched in one place to
modify the number of open records?

Let me know if it worked for you and I'll add these instructions to the Wiki.

Thomas



> oOne additional comment. I was able to trace the setting of the error code DB_NO_MEMORY 
> to a call to the db_add_open_record() by mserver that is initiated during the start-up 
> of my frontend via an RPC call. I checked with a debug printout that I have indeed 
> reached the number of MAX_OPEN_RECORDS
> 
> > Hi Midas forum,
> > 
> > I'm having a problem with odb hotlinks after increasing sub-directories in an 
> > odb. I now get the error code DB_NO_MEMORY after some db_open_record() calls. I 
> > tried 
> > 
> > 1) increasing the parameter DEFAULT_ODB_SIZE in midas.h and make clean, make
> > but got the same error
> > 
> > 2) increasing the parameter  MAX_OPEN_RECORDS in midas.h and make clean, make
> > but got fatal errors from odbedit and my midas FE and couldnt run anything
> > 
> > 3) deleting my expts SHM files and starting odbedit with "odbedit -e SLAC -s 
> > 0x1000000" to increse the odb size but got the same error?
> > 
> > 4) I tried a different computer and got the same error code DB_NO_MEMORY
> > 
> > Maybe I running into some system limit that restricts the humber of open records? 
> > Or maybe I've not increased the correct midas parameter?
> > 
> > Best ,Tim.

The procedure I wrote seemed to work for Tim too, so I added a page to the wiki about it here:

https://midas.triumf.ca/MidasWiki/index.php/FAQ


> Hi Tim,
> 
> I reproduced your problem and then managed to go through a procedure to increase the number
> of allowable open records.  The following is the procedure that I used 
> 
> 1) Use odbedit to save current ODB
> 
> odbedit
> save current_odb.odb
> 
> 2) Stop all the running MIDAS processes, including mlogger and mserver using the web
> interface. Then stop mhttpd as well.
> 
> 
> 3) Remove your old ODB (we will recreate it after modifying MIDAS, using the backup you just
> made).
> 
> mv .ODB.SHM .ODB.SHM.20161104
> rm /dev/shm/thomas_ODB_SHM
> 
> 4) Make the following modifications to midas.  In this particular case I have increased the
> max number of open records from 256 to 1024.  You would need to change the constants if you
> want to change to other values
> 
> diff --git a/include/midas.h b/include/midas.h
> index 02b30dd..33be7be 100644
> --- a/include/midas.h
> +++ b/include/midas.h
> @@ -254,7 +254,7 @@ typedef std::vector<std::string> STRING_LIST;
> -#define MAX_OPEN_RECORDS       256           /**< number of open DB records   */
> +#define MAX_OPEN_RECORDS       1024           /**< number of open DB records   */
> diff --git a/src/odb.c b/src/odb.c
> index 47ace8f..ac1bef3 100755
> --- a/src/odb.c
> +++ b/src/odb.c
> @@ -699,8 +699,8 @@ static void db_validate_sizes()
> -   assert(sizeof(DATABASE_CLIENT) == 2112);
> -   assert(sizeof(DATABASE_HEADER) == 135232);
> +   assert(sizeof(DATABASE_CLIENT) == 8256);
> +   assert(sizeof(DATABASE_HEADER) == 528448);
> 
> The calculation is as follows (in case you want a different number of open records): 
> DATABASE_CLIENT = 64 + 8*MAX_OPEN_ERCORDS = 64 + 8*1024 = 8256
> DATABASE_HEADER = 64 + 64*DATABASE_CLIENT = 64 + 64*8256 = 528448
> 
> 5) Rebuild MIDAS
> 
> make clean; make
> 
> 6) Create new ODB
> 
> odbedit -s 1000000
> 
> Change the size of the ODB to whatever you want.
> 
> 7) reload your original ODB 
> 
> load  current_odb.odb
> 
> 8) Rebuild your frontend against new MIDAS; then it should work and you should be able to
> produce more open records.
> 
> 8.5*) Actually, I had a weird error where I needed to remove my .SYSTEM.SHM file as well
> when I first restarted my front-end.  Not sure if that was some unrelated error, but I
> mention it here for completeness.
> 
> This was a procedure based on something that originally was used for T2K (procedure by Renee
> Poutissou).  It is possible that not all steps are necessary and that there is a better way.
>  But this worked for me.
> 
> Also, any objections from other developers to tweaking the assert checks in odb.c so that
> the values are calculated automatically and MIDAS only needs to be touched in one place to
> modify the number of open records?
> 
> Let me know if it worked for you and I'll add these instructions to the Wiki.
> 
> Thomas
> 
> 
> 
> > oOne additional comment. I was able to trace the setting of the error code DB_NO_MEMORY 
> > to a call to the db_add_open_record() by mserver that is initiated during the start-up 
> > of my frontend via an RPC call. I checked with a debug printout that I have indeed 
> > reached the number of MAX_OPEN_RECORDS
> > 
> > > Hi Midas forum,
> > > 
> > > I'm having a problem with odb hotlinks after increasing sub-directories in an 
> > > odb. I now get the error code DB_NO_MEMORY after some db_open_record() calls. I 
> > > tried 
> > > 
> > > 1) increasing the parameter DEFAULT_ODB_SIZE in midas.h and make clean, make
> > > but got the same error
> > > 
> > > 2) increasing the parameter  MAX_OPEN_RECORDS in midas.h and make clean, make
> > > but got fatal errors from odbedit and my midas FE and couldnt run anything
> > > 
> > > 3) deleting my expts SHM files and starting odbedit with "odbedit -e SLAC -s 
> > > 0x1000000" to increse the odb size but got the same error?
> > > 
> > > 4) I tried a different computer and got the same error code DB_NO_MEMORY
> > > 
> > > Maybe I running into some system limit that restricts the humber of open records? 
> > > Or maybe I've not increased the correct midas parameter?
> > > 
> > > Best ,Tim.

> > I've recently run into issues when using JSON.parse on ODB keys containing 
> > 8-bit data.
> 
> I am tempted to take a hard line and say that in general MIDAS TID_STRING data should be valid 
> UTF-8 encoded Unicode. In the modern mixed javascript/json/whatever environment I think
> it is impractical to handle or permit invalid UTF-8 strings.
> 
> Certainly in the general case, replacing all control characters with something else or escaping them or 
> otherwise changing the value if TID_STRING data would wreck *valid* UTF-8 strings, which I would 
> assume to be the normal use.
> 
> In other words, non-UTF-8 strings are following non-IEEE-754 floating point values into oblivion - as 
> we do not check the TID_FLOAT and TID_DOUBLE is valid IEEE-754 values, we should not check 
> that TID_STRING is valid UTF-8.

I agree that I think we should start requiring strings to be UTF-8 encoded unicode. 

I'd suggest that before worrying about the TID_STRING data, we should start by sanitizing the ODB key names.
 I've seen a couple cases where the ODB key name is a non-UTF-8 string.  It is very awkward to use odbedit
to delete these keys.

I attach a suggested modification to odb.c that rejects calls to db_create_key with non-UTF-8 key names.  It
uses some random function I found on the internet that is supposed to check if a string is valid UTF-8.  I
checked a couple of strings with invalid UTF-8 characters and it correctly identified them.  But I won't
claim to be certain that this is really identifying all UTF-8 vs non-UTF-8 cases.  Maybe others have a
better way of identifying this.

creating a frontend on MAC Sierra OSX 10
include the midas.h file and when compiling with XCode I get an error based on
this entry in the midas.h include

#if !defined(OS_IRIX) && !defined(OS_VMS) && !defined(OS_MSDOS) &&
!defined(OS_UNIX) && !defined(OS_VXWORKS) && !defined(OS_WINNT)
#error MIDAS cannot be used on this operating system
#endif


Perhaps I should not use Xcode?
Perhaps I won't need Midas.h?

The MIDAS system is running on my MAC but I need to add a very simple front end
for testing and I encounted this error.

> creating a frontend on MAC Sierra OSX 10
> include the midas.h file and when compiling with XCode I get an error based on
> this entry in the midas.h include
> 
> #if !defined(OS_IRIX) && !defined(OS_VMS) && !defined(OS_MSDOS) &&
> !defined(OS_UNIX) && !defined(OS_VXWORKS) && !defined(OS_WINNT)
> #error MIDAS cannot be used on this operating system
> #endif
> 
> 
> Perhaps I should not use Xcode?
> Perhaps I won't need Midas.h?
> 
> The MIDAS system is running on my MAC but I need to add a very simple front end
> for testing and I encounted this error.

If you compile with the included Makefile, you will see a 

-DOS_LINUX -DOS_DARWIN

flag which tells the compiler that we are on a mac. If you do this with XCode, you have to do it via "Build Settings" (see 
attached picture).

Stefan

> > In other words, non-UTF-8 strings are following non-IEEE-754 floating point values into oblivion - as 
> > we do not check the TID_FLOAT and TID_DOUBLE is valid IEEE-754 values, we should not check 
> > that TID_STRING is valid UTF-8.
> ...
> I attach a suggested modification to odb.c that rejects calls to db_create_key with non-UTF-8 key names.  It
> uses some random function I found on the internet that is supposed to check if a string is valid UTF-8.  I
> checked a couple of strings with invalid UTF-8 characters and it correctly identified them.  But I won't
> claim to be certain that this is really identifying all UTF-8 vs non-UTF-8 cases.  Maybe others have a
> better way of identifying this.

At Konstantin's suggestion, I committed the function I found for checking if a string was UTF-8 compatible to
odb.c.  The function is currently not used; I commented out a proposed use in db_create_key.  Experts can decide
if the code was good enough to use.

> At Konstantin's suggestion, I committed the function I found for checking if a string was UTF-8 compatible to
> odb.c.  The function is currently not used; I commented out a proposed use in db_create_key.  Experts can decide
> if the code was good enough to use.

After more discussion, I have enabled the parts of the ODB code that check that key names are UTF-8 compliant. 

This check will show up in (at least) two ways:

1) Attempts to create a new ODB variable if the ODB key is not UTF-8 compliant.  You will see error messages like

[fesimdaq,ERROR] [odb.c:572:db_validate_name,ERROR] Invalid name "Eur€" passed to db_create_key: UTF-8 incompatible
string

2) When a program first connects to the ODB, it runs a check to ensure that the ODB is valid.  This will now include
a check that all key names are UTF-8 compliant. Any non-UTF8 compliant key names will be replaced by a string of the
pointer to the key.  You will see error messages like:

[fesimdaq,ERROR] [odb.c:572:db_validate_name,ERROR] Invalid name "Eur€" passed to db_validate_key: UTF-8
incompatible string
[fesimdaq,ERROR] [odb.c:647:db_validate_key,ERROR] Warning: corrected key "/Equipment/SIMDAQ/Eur€": invalid name
"Eur€" replaced with "0x7f74be63f970"

This behaviour (checking UTF-8 compatibility and automatically fixing ODB names) can be disabled by setting an
environment variable

MIDAS_INVALID_STRING_IS_OK

It doesn't matter what the environment variable is set to; it just needs to be set.  Note also that this variable is
only checked once, when a program starts.

> 
> > At Konstantin's suggestion, I committed the function I found for checking if a string was UTF-8 compatible to
> > odb.c.  The function is currently not used; I commented out a proposed use in db_create_key.  Experts can decide
> > if the code was good enough to use.
> 
> After more discussion, I have enabled the parts of the ODB code that check that key names are UTF-8 compliant. 
> 
> This check will show up in (at least) two ways:
> 
> 1) Attempts to create a new ODB variable if the ODB key is not UTF-8 compliant.  You will see error messages like
> 
> [fesimdaq,ERROR] [odb.c:572:db_validate_name,ERROR] Invalid name "Eur€" passed to db_create_key: UTF-8 incompatible
> string
> 
> 2) When a program first connects to the ODB, it runs a check to ensure that the ODB is valid.  This will now include
> a check that all key names are UTF-8 compliant. Any non-UTF8 compliant key names will be replaced by a string of the
> pointer to the key.  You will see error messages like:
> 
> [fesimdaq,ERROR] [odb.c:572:db_validate_name,ERROR] Invalid name "Eur€" passed to db_validate_key: UTF-8
> incompatible string
> [fesimdaq,ERROR] [odb.c:647:db_validate_key,ERROR] Warning: corrected key "/Equipment/SIMDAQ/Eur€": invalid name
> "Eur€" replaced with "0x7f74be63f970"
> 
> This behaviour (checking UTF-8 compatibility and automatically fixing ODB names) can be disabled by setting an
> environment variable
> 
> MIDAS_INVALID_STRING_IS_OK
> 
> It doesn't matter what the environment variable is set to; it just needs to be set.  Note also that this variable is
> only checked once, when a program starts.



I see you put some switches into the environment ("MIDAS_INVALID_STRING_IS_OK"). Do you think this is a good idea? Most variables are 
sitting in the ODB (/experiment/xxx), except those which cannot be in the ODB because we need it before we open the ODB, like MIDAS_DIR. 
Having them in the ODB has the advantage that everything is in one place, and we see a "list" of things we can change. From an empty 
environment it is not clear that such a thing like "MIDAS_INVALID_STRING_IS_OK" does exist, while if it would be an ODB key it would be 
obvious. Can I convince you to move this flag into the ODB?

> 
> I see you put some switches into the environment ("MIDAS_INVALID_STRING_IS_OK"). Do you think this is a good idea? Most variables are 
> sitting in the ODB (/experiment/xxx), except those which cannot be in the ODB because we need it before we open the ODB, like MIDAS_DIR. 
> Having them in the ODB has the advantage that everything is in one place, and we see a "list" of things we can change. From an empty 
> environment it is not clear that such a thing like "MIDAS_INVALID_STRING_IS_OK" does exist, while if it would be an ODB key it would be 
> obvious. Can I convince you to move this flag into the ODB?
>


Some additional explanation.

Time passed, the world turned, and the current web-compatible standard for text strings is UTF-8 encoded Unicode, see 
https://en.wikipedia.org/wiki/UTF-8
(ObCanadianContent, UTF-8 was invented the Canadian Rob Pike https://en.wikipedia.org/wiki/Rob_Pike)
(and by some other guy https://en.wikipedia.org/wiki/Ken_Thompson).

It turns out that not every combination of 8-bit characters (char*) is valid UTF-8 Unicode.

In the MIDAS world we run into this when MIDAS ODB strings are exported to Javascript running inside web
browsers ("custom pages", etc). ODB strings (TID_STRING) and ODB key names that are not valid UTF-8
make such web pages malfunction and do not work right.

One solution to this is to declare that ODB strings (TID_STRING) and ODB key names *must* be valid UTF-8 Unicode.

The present commits implemented this solution. Invalid UTF-8 is rejected by db_create() & co and by the ODB integrity validator.

This means some existing running experiment may suddenly break because somehow they have "old-style" ODB entries
or they mistakenly use TID_STRING to store arbitrary binary data (use array of TID_CHAR instead).

To permit such experiments to use current releases of MIDAS, we include a "defeat" device - to disable UTF-8 checks
until they figure out where non-UTF-8 strings come from and correct the problem.

Why is this defeat device non an ODB entry? Because it is not a normal mode of operation - there is no use-case where
an experiment will continue to use non-UTF-8 compatible ODB indefinitely, in the long term. For example, as the MIDAS user
interface moves to more and more to HTML+Javascript+"AJAX", such experiments will see that non-UTF-8 compatible ODB entries
cause all sorts of problems and will have to convert.


K.O.

> 
> If you compile with the included Makefile, you will see a 
> 
> -DOS_LINUX -DOS_DARWIN
> 

Moving forward, it looks like I can define these variables in midas.h and remove the need to define them on the compiler command line.

This would be part of the Makefile and header files cleanup to get things working on Windows10.

K.O.

> > 
> > If you compile with the included Makefile, you will see a 
> > 
> > -DOS_LINUX -DOS_DARWIN
> > 
> 
> Moving forward, it looks like I can define these variables in midas.h and remove the need to define them on the compiler command line.
> 
> This would be part of the Makefile and header files cleanup to get things working on Windows10.
> 
> K.O.

Will you detect the underlying OS automatically in midas.h? Note that you have several compilers in MacOS (llvm and gcc), and they might use different 
predefined symbols. I appreciate however getting rid of these flags in the Makefile.

Stefan