The MIDAS RPC layer can pass std::string and std::vector data of arbitrary size.

I am adding corresponding ODB API functions to use this. Instead of taking a pointer to preallocated array 
(and we must know the size ahead of time, i.e. by reading the ODB key), we pass a pointer to an 
std::vector<char>, all memory allocation is taken care of by the RPC layer.

INT EXPRT db_get_data_vec(HNDLE hdb, HNDLE key_handle, std::vector<char> *data, DWORD type);
INT EXPRT db_get_data_index_vec(HNDLE hdb, HNDLE hKey, std::vector<char> *data, INT index, DWORD type);

This helps with programming MIDAS "the c++ way" and fixes subtle errors like the memory leak
in the odbxx code if an exception is thrown - free(buffer) is never reached.

         //int size = rpc_tid_size(m_tid) * m_num_values;
         //void *buffer = malloc(size);
         //void *p = buffer;
         std::vector<char> odbdata;
         status = db_get_data_vec(s_hDB, m_hKey, &odbdata, m_tid);
         void *p = odbdata.data();
         for (int i = 0; i < m_num_values; i++) {
            if (m_tid == TID_UINT8 || m_tid == TID_CHAR)
               m_data[i].set(*static_cast<uint8_t *>(p));
            else
               mthrow("Invalid type ID " + std::to_string(m_tid));
         }
         //free(buffer);

next to add is db_get_value_vec(). we already have a db_get_value_string().

K.O.

My update of db_get_data_vec() got delayed because I had to figure out
how the links to array elements work and fix a number of places
where they are not handled correctly or consistently.

Current status after the db_get_data_vec() merge:

1) db_get_data(), db_get_data_vec(): links to array elements work
2) db_get_value(), db_get_data_index(), db_get_data_index(): not implemented, will return a type mismatch
3) mhttpd obsolete "jcopy": works correctly now (previously produced invalid json)
4) odbedit save odb.json: works correctly now (prevously produced invalid json)

I think for consistency, I should implement links to array elements in db_get_value(), db_get_value_string() and db_get_value_vec().

K.O.

> this is a draft message, it will be updated with additional information. K.O.
> 
> Back in 2007, Stefan implemented the very useful feature, ODB links to array elements,
> https://daq00.triumf.ca/elog-midas/Midas/418
> 
> This is handy for "Edit On Start", for history links, for feepics and other places.
> 
> If you have an array in ODB: 
> 
> ia5                             INT32   5     4     12m  0  RWD   
>                                         [0]             10
>                                         [1]             11
>                                         [2]             12
>                                         [3]             13
>                                         [4]             14
> 
> a normal ODB link works as a UNIX filesystem symlink, whereas an ODB link that includes an array index refers to just one element of the link target array:
> 
> symlink_to_ia5_2 -> /test_odb/ia5[2]
>                                 INT32   1     4     12m  0  RWD   12
> 
> Because there was some confusion over how it works and how it is actually implements,
> some ODB functions do not implement this feature consistently.
> 
> Here, I explain how it actually works:
> 
> 0) expected syntax is: "link name" -> "/link/target/array[12345]":
> 
> - "/link/target/array" is the absolute ODB path to the link target (must start with "/", relative links are not permitted)
> - array index "12345" should be an integer (converted using atoi())
> - there should be nothing after final "]"
> - there should be nothing between the "[" and the array index decimal numeric value
> 
> 1) db_get_data():
> 
> - before calling db_get_data() we must call db_find_key()
> - db_find_key() will resolve ODB links and return the final destination (or an error if dangling link or circular link or too many nested links)
> - db_get_data() will return the data from this ODB key. this is the path for normal symlinks.
> 
> - if db_find_key() encounters an ODB link to an array element (target path contains "["), it returns this key (of type TID_LINK).
> - db_get_data() checks the key type (normally it would be TID_INT, TID_STRING, etc). if it is TID_LINK, it means we have a link to an array index (as identified by db_find_key())
> - in this case, ODB link is resolved using db_find_key(), array index is extracted from the link
> - and db_get_data() returns data for the corresponding array element
> 
> If db_get_data() is called using an hKey returned by db_find_link() (instead of db_find_key())
> and it happens to be an ODB link (TID_LINK), we have an inconsistent result:
> 
> - if it's a normal link, db_get_data() will return a type mismatch error
> - db_get_data(TID_LINK) will return the link target string (NUL-terminated)
> - if it's a link to an array element, db_get_data() will resolve it and return data from the link target array element.
> - db_get_data(TID_LINK) will return a type mismatch error because it will resolve the link, then fail the type check as link target cannot be TID_LINK
> 
> 2) db_get_value(), db_get_data_index()
> 
> - normal links work, resolved by db_find_key()
> - links to array elements not implemented, will return a type mismatch error (i.e. TID_LINK vs user requested TID_STRING, TID_INT, etc)
> 
> 3) mhttpd obsolete "jcopy"
> 
> - broken, returns malformed json (old code), returns the complete array, not just the linked array element (new code)
> 
> 4) odbedit save odb.json
> 
> - broken, output file odb.json is empty (new code)
> 
> K.O.

we have one computer that crashes regularly and produces a corrupted ODB. while 
debugging it, I discovered that for some ODB corruptions and for some unexpected 
ODB errors, the ODB JSON encoder will produce invalid JSON files that cannot be 
reloaded, and also produce zero-size JSON files without any explicit error 
message.

these problems are now corrected, odbedit save odb.json will now always
produce valid JSON (ODB errors are encoded as JSON "null"), and always
write the encoded JSON data to odb.json, even if there were encoding errors.

to remember, the JSON encoder I originally wrote in C (before MIDAS wholesale 
switch to C++). The JSON parser (mjson.h, mjson.cxx) and the JSON ODB decoder 
(json_paste.cxx) I originally wrote in pre-C++11 C++ (odb.c was still in C
at the time, this is why json_paste.cxx is a separate file). all of them
are now updated to C++11.

merge commit 8a4c52da22f67ba7cbfc7d98a0fc81eeccdc7111

K.O.

some cleanup of mhttpd:
- remove obsolete c-generated equipment view page (replaced by eqtable.html)
- remove support for obsolete versions of mongoose (HAVE_MONGOOSE4, HAVE_MONGOOSE6, mhttpd6)
- remove support for https, openssl and mbedtls

support for https may return after we upgrade to a current version of mongoose (latest is 7.23, we have 6.16).

support for user passwords in mhttpd I leave alone for now,
but their security is limited:
a) lacking https, passwords are transmitted in clear text, this is only safe
   for localhost connections) and
b) mhttpd always used the unusual http digest authentication,
   which is no longer recommended because of:
b1) weak/insecure server-side password storage and
b2) weak security of the digest authentication token itself

for secure use of MIDAS, we recommend placing it behind a password protected https proxy (i.e. apache httpd)

K.O.

A split of src/midas.cxx was discussed during Stefan & Thomas R. visit to TRIUMF.

Main reason for the split is to simplify analysis using AI tools, they work best on smaller files that contain closely related 
code.

Main difficulty with the split is "git" tooling, which does not have a concept of file copy or file split. (still this is better 
than older tools like cvd and svn, which had no concept of branching and merging).

My first step is to do a strawman split of midas.cxx on a branch, push it out and ask people to check that they git tooling 
(vscode, clion, bitbucket, github, gitea, etc) see the split correctly.

Ideally, git log and git blame should be able to trace the revision of each split file
back to the beginning of time, instead of to the point of split from midas.cxx.

According to google AI, I should do this:

step 1: midas.cxx removed, split files are exact copies of midas.cxx, this allows git to identify the split point

cp midas.cxx split1.cxx
cp midas.cxx split2.cxx
...
git rm midas.cxx
git commit

step 2: edit the split files to remove duplicated code, commit.

step 3: check

git blame split1.cxx -> shows correct revision history for me. maybe need to use "-C"
git log split1.cxx -> shows correct revision history to split commit
git log --follow split1.cxx -> shows correct revision history for me

TD;DR continue below.

Identify all function names in midas.cxx:

cat midas.cxx | grep -v -e "^ " -e "^{" -e "^}" -e "^$" -e "^/" -e '^\\' -e "^@" -e "^*" -e "^#" | sort

^void functions:

void bk_xxx()
void bm_xxx()
void cm_xxx()
void dbg_xxx()
void rpc_xxx()

^std::string functions:

cm_xxx()
msprintf()
rpc_xxx()

^static functions:

bm_xxx
rpc_xxx
cm_xxx

^int functions:

bk_xxx
bm_xxx
cm_xxx
cm_msg_xxx
rb_xxx
rpc_xxx

^const data

cm_xxx
rpc_xxx

^bool functions:

rpc_xxx

^INT functions

bk_xxx
bm_xxx
cm_xxx
cm_msg_xxx
rpc_xxx

^BOOL functions

bk_xxx
cm_xxx

LC_ALL=C sort midas.cxx | grep -v -e "^/" -e "^\s" -e "^W" -e "^w" -e ^} -e ^{ -e ^[a-v] -e '^\\' -e ^[A-U] -e ^@ -e ^* -e ^# -e ^$

Stawman split:

midas.cxx -> deleted

cm_msg.cxx - all the cm_msg functions
bm.cxx     - all the event buffer functions
bk.cxx     - all the event bank functions
rb.cxx     - ring buffer functions (obsolete?)
rpc.cxx    - all the RPC functions

cm.cxx     - everything else

K.O.

> A split of src/midas.cxx was discussed during Stefan & Thomas R. visit to TRIUMF.

A test of split src/midas.cxx is now pushed to branch test/split-midas-cxx

midas.cxx -> deleted, as required by git (see below)

cm_msg.cxx - will be all the cm_msg functions
bm.cxx     - all the event buffer functions
bk.cxx     - will be all the event bank functions
rb.cxx     - ring buffer functions (obsolete?)
rpc.cxx    - will be all the RPC functions
cm.cxx     - everything else

when I do the split for real, I think I will split run transitions to cm_transition.cxx

please take a look at this branch, run git log and git blame using your preferred tool of choice (vscode, clion, etc)
and let me know if tracking of revision history works through the split commit and if there
is any other oddities.

also, better names for the split files are most welcome.

only rb.cxx and bm.cxx are actually reduced in size, the rest of the files are still copies of midas.cxx (this is only a test of the 
split).

K.O.

P.S. More details below:

> 
> Main reason for the split is to simplify analysis using AI tools, they work best on smaller files that contain closely related 
> code.
> 
> Main difficulty with the split is "git" tooling, which does not have a concept of file copy or file split. (still this is better 
> than older tools like cvd and svn, which had no concept of branching and merging).
> 
> My first step is to do a strawman split of midas.cxx on a branch, push it out and ask people to check that they git tooling 
> (vscode, clion, bitbucket, github, gitea, etc) see the split correctly.
> 
> Ideally, git log and git blame should be able to trace the revision of each split file
> back to the beginning of time, instead of to the point of split from midas.cxx.
> 
> According to google AI, I should do this:
> 
> step 1: midas.cxx removed, split files are exact copies of midas.cxx, this allows git to identify the split point
> 
> cp midas.cxx split1.cxx
> cp midas.cxx split2.cxx
> ...
> git rm midas.cxx
> git commit
> 
> step 2: edit the split files to remove duplicated code, commit.
> 
> step 3: check
> 
> git blame split1.cxx -> shows correct revision history for me. maybe need to use "-C"
> git log split1.cxx -> shows correct revision history to split commit
> git log --follow split1.cxx -> shows correct revision history for me
> 
> TD;DR continue below.
> 
> Identify all function names in midas.cxx:
> 
> cat midas.cxx | grep -v -e "^ " -e "^{" -e "^}" -e "^$" -e "^/" -e '^\\' -e "^@" -e "^*" -e "^#" | sort
> 
> ^void functions:
> 
> void bk_xxx()
> void bm_xxx()
> void cm_xxx()
> void dbg_xxx()
> void rpc_xxx()
> 
> ^std::string functions:
> 
> cm_xxx()
> msprintf()
> rpc_xxx()
> 
> ^static functions:
> 
> bm_xxx
> rpc_xxx
> cm_xxx
> 
> ^int functions:
> 
> bk_xxx
> bm_xxx
> cm_xxx
> cm_msg_xxx
> rb_xxx
> rpc_xxx
> 
> ^const data
> 
> cm_xxx
> rpc_xxx
> 
> ^bool functions:
> 
> rpc_xxx
> 
> ^INT functions
> 
> bk_xxx
> bm_xxx
> cm_xxx
> cm_msg_xxx
> rpc_xxx
> 
> ^BOOL functions
> 
> bk_xxx
> cm_xxx
> 
> LC_ALL=C sort midas.cxx | grep -v -e "^/" -e "^\s" -e "^W" -e "^w" -e ^} -e ^{ -e ^[a-v] -e '^\\' -e ^[A-U] -e ^@ -e ^* -e ^# -e ^$
> 
> Stawman split:
> 
> midas.cxx -> deleted
> 
> cm_msg.cxx - all the cm_msg functions
> bm.cxx     - all the event buffer functions
> bk.cxx     - all the event bank functions
> rb.cxx     - ring buffer functions (obsolete?)
> rpc.cxx    - all the RPC functions
> 
> cm.cxx     - everything else
> 
> K.O.

> > A split of src/midas.cxx was discussed during Stefan & Thomas R. visit to TRIUMF.
> 
> A test of split src/midas.cxx is now pushed to branch test/split-midas-cxx
> 
> please take a look at this branch, run git log and git blame using your preferred tool of choice (vscode, clion, etc)
> and let me know if tracking of revision history works through the split commit and if there
> is any other oddities.
>

bitbucket web gui for src/rb.cxx
- show history - only to split commit (ouch!)
- show blame - all the way to the beginning of time (good)

K.O.

> The history of at least bm.cxx seems to be screwed up. Bitbucket shows this:
>   https://bitbucket.org/tmidas/midas/annotate/fa649e6e9bb7447b4da01e964c6dc14b01e64052/src/bm.cxx?at=test%2Fsplit-midas-cxx
> and CLion shows what I attached here. Most lines are tagged "Today Olchanski" which cannot be.

yup, I see the same nonsense for bm.cxx from gitea.

back to the drawing board, will try branch, rename, merge as described here:
https://devblogs.microsoft.com/oldnewthing/20190916-00/?p=102892/

the naming of split files looks ok or we prefer more descriptive names "event_buffer.cxx",
"transition.cxx", "message.cxx", etc?

K.O.

Hello Stefan,
is there a difference for the later data processing (after writing the ring buffer blocks)
if we write single events or multiple in one rb_get_wp - memcopy - rb_increment_wp cycle?
Both Marius and me have seen some inconsistencies in the number of events produced that is reported in the status page when writing multiple events in one go,
so I was wondering if this is due to us treating the buffer badly or the way midas handles the events after that.

Given that we produce the full event in our (FPGA) domain, an option would be to always copy one event from the dma to the midas-system buffer in a loop.
The question is if there is a difference (for midas) between
[pseudo code, much simplified]

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
    dma_read_index+=event_size;
    continue;   
  }
  copy_n(dma_buffer, pdata, event_size);
  rb_increment_wp(event_size);
  dma_read_index+=event_size;
} 

and

while(dma_read_index < last_dma_write_index){
  if(rb_get_wp(pdata)!=SUCCESS){
     ...
  };
  total_size=max_n_events_that_fit_in_rb_block();
  copy_n(dma_buffer, pdata, total_size);
  rb_increment_wp(total_size);
  dma_read_index+=total_size;
}

Cheers,
Konrad

> The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:
> 
> 1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection any more, so if you obtain pdata from rb_get_wp, you can of course write 4GB to pdata, overwriting everything in your memory, causing a total crash. It's your responsibility to not write more bytes into pdata then 
> what you specified as max event size in rb_create()
> 
> 2) Once you obtain a write pointer to the ring buffer via rb_get_wp, this function might fail when the receiving side reads data slower than the producing side, simply because the buffer is full. In that case the producing side has to wait until space is freed up in the buffer by the receiving side. If your call to rb_get_wp returns DB_TIMEOUT, it means that the 
> function did not obtain enough free space for the next event. In that case you have to wait (like ss_sleep(10)) and try again, until you succeed. Only when rb_get_wp() returns DB_SUCCESS, you are allowed to write into pdata, up to the maximum event size specified in rb_create of course. I don't see this behaviour in your code. You would need something 
> like
> 
> do {
>    status = rb_get_wp(rbh, (void **)&pdata, 10);
>    if (status == DB_TIMEOUT)
>       ss_sleep(10);
>    } while (status == DB_TIMEOUT);
> 
> Best,
> Stefan
> 
> 
> > Dear all,
> > 
> > we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer where it has written the last 4kB of data. We use this pointer for telling the ring buffer of midas where the new events are. The buffer looks something like:
> > 
> > // event 1
> > dma_buf[0] = 0x00000001; // Trigger and Event ID
> > dma_buf[1] = 0x00000001; // Serial number
> > dma_buf[2] = TIME; // time
> > dma_buf[3] = 18*4-4*4; // event size
> > dma_buf[4] = 18*4-6*4; // all bank size
> > dma_buf[5] = 0x11; // flags
> > // bank 0
> > dma_buf[6] = 0x46454230; // bank name
> > dma_buf[7] = 0x6; // bank type TID_DWORD
> > dma_buf[8] = 0x3*4; // data size
> > dma_buf[9] = 0xAFFEAFFE; // data
> > dma_buf[10] = 0xAFFEAFFE; // data
> > dma_buf[11] = 0xAFFEAFFE; // data
> > // bank 1
> > dma_buf[12] = 0x1; // bank name
> > dma_buf[12] = 0x46454231; // bank name
> > dma_buf[13] = 0x6; // bank type TID_DWORD
> > dma_buf[14] = 0x3*4; // data size
> > dma_buf[15] = 0xAFFEAFFE; // data
> > dma_buf[16] = 0xAFFEAFFE; // data
> > dma_buf[17] = 0xAFFEAFFE; // data
> > 
> > // event 2
> > .....
> > 
> > dma_buf[fpga_pointer] = 0xXXXXXXXX;
> > 
> > 
> > And we do something like:
> > 
> > while{true}
> >    // obtain buffer space
> >    status = rb_get_wp(rbh, (void **)&pdata, 10);
> >    fpga_pointer = fpga.read_last_data_add();
> > 
> >    wlen = last_fpga_pointer - fpga_pointer; \\ in 32 bit words
> >    copy_n(&dma_buf[last_fpga_pointer], wlen, pdata);
> >    rb_status = rb_increment_wp(rbh, wlen * 4); \\ in byte
> > 
> >    last_fpga_pointer = fpga_pointer;
> > 
> > Leaving the case out where the dma_buf wrap around this works fine for a small data rate. But if we increase the rate the fpga_pointer also increases really fast and wlen gets quite big. Actually it gets bigger then max_event_size which is checked in rb_increment_wp leading to an error. 
> > 
> > The problem now is that the event size is actually not to big but since we have multi events in the buffer which are read by midas in one step. So we think in this case the function rb_increment_wp is comparing actually the wrong thing. Also increasing the max_event_size does not help.
> > 
> > Remark: dma_buf is volatile so memcpy is not possible here.
> > 
> > Cheers,
> > Marius

> Hi there:
> I have a question if there's anybody out there running MIDAS with event builder
> that assembles events from more that just a few front ends (say on the order of
> 0x10 or more)?
> Any experiences with scalability?
> 
> Cheers
>  Piotr

Mulan (which you hopefully remember with great fondness :-) is currently running
around ten frontends, six of which produce data at any rate.  If I'm remembering
correctly, the event builder handles about 30-40MB/s.  You could probably ping Tim
Gorringe or his current postdoc Volodya Tishenko (tishenko@pa.uky.edu) if you want
more details.  Volodya solved a significant number of throughput related
bottlenecks in the year leading up to our 2006 run.

creating a frontend on MAC Sierra OSX 10
include the midas.h file and when compiling with XCode I get an error based on
this entry in the midas.h include

#if !defined(OS_IRIX) && !defined(OS_VMS) && !defined(OS_MSDOS) &&
!defined(OS_UNIX) && !defined(OS_VXWORKS) && !defined(OS_WINNT)
#error MIDAS cannot be used on this operating system
#endif


Perhaps I should not use Xcode?
Perhaps I won't need Midas.h?

The MIDAS system is running on my MAC but I need to add a very simple front end
for testing and I encounted this error.

Dear all,

please do not be surprised if your 2D plots start looking different: The default
colour scheme changed to the (now kind of standard one) 'viridis' from
matplotlib to have perceptually uniform sequential colour scheme. So if you
print your plots in black and white or have problems colour vision deficiency
the plots should not have artificial artifacts anymore.


Cheers,
Julian

One can collapse the side panel when looking at history pages with the button in
the top left, great! We want to see many pages so screen real estate is important

The issue we face is that when the page refreshes, the side panel expands. Can
we make the panel state more 'sticky'?

Many thanks
Joseph (ALPHA)

Version: 	2.1
Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
on branch feature/midas-2017-10

> > 
> > 
> > One can collapse the side panel when looking at history pages with the button in
> > the top left, great! We want to see many pages so screen real estate is important
> > 
> > The issue we face is that when the page refreshes, the side panel expands. Can
> > we make the panel state more 'sticky'?
> > 
> > Many thanks
> > Joseph (ALPHA)
> > 
> > Version: 	2.1
> > Revision: 	Mon Mar 19 18:15:51 2018 -0700 - midas-2017-07-c-197-g61fbcd43-dirty
> > on branch feature/midas-2017-10
> 
> Hi Joseph,
> 
> In principle a page refresh should now not be necessary, since pages should reload automatically 
> the contents which changes. If a custom page needs a reload, it is not well designed. If necessary, I 
> can explain the details. 
> 
> Anyhow I implemented your "stickyness" of the side panel in the last commit to the develop branch.
> 
> Best regards,
> Stefan

Hi Stefan,

I apologise for miss using the word refresh. The re-appearing sidebar was also seen with the automatic
reload, I have implemented your fix here and it now works great!

Thank you very much!
Joseph

Hi,

I have been looking at the 2019 workshop slides, I am interested in the C++ future of MIDAS. 

I am quite interested in using the object oriented 


ALPHA will start data taking in 2021

I have written a system monitor tool for MIDAS, that has been merged in the develop branch today: sysmon

https://bitbucket.org/tmidas/midas/pull-requests/8/system-monitoring-a-new-frontend-to-log/diff

To use it, simply run the new program
sysmon
on any host that you want to monitor, no configuring required.




The program is a frontend for MIDAS, there is no need for configuration, as upon initialisation it builds a history display for you. Simply run one instance per machine you want to monitor. By default, it only logs once per 10 seconds.

The equipment name is derived from the hostname, so multiple instances can be run across multiple machines without conflict. A new history display will be created for each host.

sysmon uses the /proc pseudo-filesystem, so unfortunately only linux is supported. It does however work with multiple architectures, so x86 and ARM processors are supported.

If the build machine has NVIDIA drivers installed, there is an additional version of sysmon that gets built: sysmon-nvidia. This will log the GPU temperature and usage, as well as CPU, memory and swap. A host should only run either sysmon or sysmon-nvidia

elog:1727/1 shows the History Display generated by sysmon-nvidia. sysmon would only generate the first two displays (sysmon/localhost and sysmon/localhost-CPU)

A little advertised feature of the modifications needed support the msysmon program is 
that MIDAS equipment names can support the injecting of the hostname of the system 
running the frontend at runtime (register_equipment(void)).

https://midas.triumf.ca/MidasWiki/index.php/Equipment_List_Parameters#Equipment_Name

A special string ${HOSTNAME} can be put in any position in the equipment name. It will 
be replaced with the hostname of the computer running the frontend at run-time. Note, 
the frontend_name string will be trimmed down to 32 characters.
Example usage: msysmon


EQUIPMENT equipment[] = {

  { "${HOSTNAME}_msysmon",   /* equipment name */    {
      EVID_MONITOR, 0,      /* event ID, trigger mask */
      "SYSTEM",             /* event buffer */
      EQ_PERIODIC,          /* equipment type */
      0,                    /* event source */
      "MIDAS",              /* format */
      TRUE,                 /* enabled */
      RO_ALWAYS,            /* Read when running */
      10000,                /* poll every so milliseconds */
      0,                    /* stop run after this event limit */
      0,                    /* number of sub events */
      1,                    /* history period */
      "", "", ""
    },
    read_system_load,/* readout routine */
  },
  { "" }
};

Hi all,

I have been experimenting with a frontend solution for my experiment 
(ALPHA). The intention to replace how we log data from PCs running LabVIEW. 

I am at the proof of concept stage. So far I have some promising 
performance, able to handle 10-100x more data in my test setup (current 
limitations now are just network bandwith, MIDAS is impressively efficient). 

==========================================================================

Our experiment has many PCs using LabVIEW which all log to MIDAS, the 
experiment has grown such that we need some sort of load balancing in our 
frontend.

The concept was to have a 'supervisor frontend' and an array of 'worker 
frontend' processes. 
-A LabVIEW client would connect to the supervisor, then be referred to a 
worker frontend for data logging. 
-The supervisor could start a 'worker frontend' process as the demand 
required.

To increase accountability within the experiment, I intend to have a 'worker 
frontend' per PC connecting. Then any rouge behavior would be clear from the 
MIDAS frontpage.

Presently there around 20-30 of these LabVIEW PCs, but given how the group 
is growing, I want to be sure that my data logging solution will be viable 
for the next 5-10 years. With the increased use of single board computers, I 
chose the target of benchmarking upto 1000 worker frontends... but I quickly 
hit the '64 MAX CLIENTS' and '64 RPC CONNECTION' limit. Ok...

branching and updating these limits:
https://bitbucket.org/tmidas/midas/branch/experimental-beyond_64_clients

I have two commits. 
1. update the memory layout assertions and use MAX_CLIENTS as a variable
https://bitbucket.org/tmidas/midas/commits/302ce33c77860825730ce48849cb810cf
366df96?at=experimental-beyond_64_clients
2. Change the MAX_CLIENTS and MAX_RPC_CONNECTION
https://bitbucket.org/tmidas/midas/commits/f15642eea16102636b4a15c8411330969
6ce3df1?at=experimental-beyond_64_clients

Unintended side effects:
I break compatibility of existing ODB files... the database layout has 
changed and I read my old ODB as corrupt. In my test setup I can start from 
scratch but this would be horrible for any existing experiment.

Edit: I noticed 'make testdiff' pipeline is failing... also fails locally... 
investigating

Early performance results:
In early tests, ~700 PCs logging 10 unique arrays of 10 doubles into 
Equipment variables in the ODB seems to perform well... All transactions 
from client PCs are finished within a couple of ms or less

==========================================================================

Questions:

Does the community here have strong opinions about increasing the 
MAX_CLIENTS and MAX_RPC_CONNECTION limits? 
Am I looking at this problem in a naive way?


Potential solutions other than increasing the MAX_CLIENTS limit:
-Make worker threads inside the supervisor (not a separate process), I am 
using TMFE, so I can dynamically create equipment. I have not yet taken a 
deep dive into how any multithreading is implemented
-One could have a round robin system to load balance between a limited pool 
of 'worker frontend' proccesses. I don't like this solution as I want to 
able to clearly see which client PCs have been setup to log too much data

==========================================================================

Thank you very much for feedback.

I am satisfied with not changing the 64 client limit. I will look at re-writing my frontend to spawn threads rather than 
processses. The load of my frontend is low, so I do not anticipate issues with a threaded implementation. 

In this threaded scenario, it will be a reasonable amount of time until ALPHA bumps into the 64 client limit.

If it avoids confusion, I am happy for my experimental branch 'experimental-beyond_64_clients' to be deleted.

Perhaps a item for future discussion would be for the odbinit program to be able to 'upgrade' the ODB and enable some backwards 
compatibility.

Thanks again
Joseph

A user reported an issue that if they were to plot some history data from 
2019 (a range of one day), the plot would spend ~4 minutes loading then 
crash the browser tab. This seems to effect chrome (under default settings) 
and not firefox

I can reproduce the issue, "Data Being Loaded" shows, then the page and 
canvas loads, then all variables get a correct "last data" timestamp, then 
the 'Updating data ...' status shows... then the tab crashes (chrome)


It seems that the browser is loading all data until the present day (maybe 4 
Gb of data in this case). In chrome the tab then crashes. In firefox, I do 
not suffer the same crash, but I can see the single tab is using ~3.5 Gb of 
RAM

Tested with midas-2020-08-a up until the HEAD of develop

I could propose the user use firefox, or increase the memory limit in 
chrome, however are there plans to limit the data loaded when specifically 
plotting between two dates?