Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  08 Sep 2016, Amy Roberts, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
    Reply  30 Sep 2016, Konstantin Olchanski, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
       Reply  25 Oct 2016, Thomas Lindner, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
       Reply  01 Dec 2016, Thomas Lindner, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail odb_modifications.txt
          Reply  15 Jan 2017, Thomas Lindner, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
             Reply  23 Jan 2017, Thomas Lindner, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
                Reply  30 Jan 2017, Stefan Ritt, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
                   Reply  01 Feb 2017, Konstantin Olchanski, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
                      Reply  01 Feb 2017, Stefan Ritt, Bug Report, control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
Message ID: 1196     Entry time: 08 Sep 2016     Reply to this: 1204
Author: Amy Roberts 
Topic: Bug Report 
Subject: control characters not sanitized by json_write - can cause JSON.parse of mhttpd result to fail 
I've recently run into issues when using JSON.parse on ODB keys containing 
8-bit data.

For JSON.parse to successfully parse a string, (A) the string must be valid 
UTF-8, (B) several whitespace characters, control characters, and the 
characters " and \ must be escaped, and (C) you've got to follow the key-
value rules laid out in http://www.json.org/.

The web browser takes care of (A), and I verified that for this key Midas 
handled (C) correctly.  In principle, the function json_write in odb.c 
handles (B) - but json_write does not escape control characters.

To manage this problem, I modified json_write (in odb.c) to replace any 
control character with the more-inocuous character, 'C'.  My default case 
now looks like:

default:
         {
           // if a char is a control character,
           // print 'C' in its place
           // note that this loses data:
           // a more-correct method would be to print
           // \uXXXX, where XXXX is the character in hex
           if(iscntrl(*s)){
             (*buffer)[(*buffer_end)++] = 'C';
             s++;
           } else {
             (*buffer)[(*buffer_end)++] = *s++;
           }
         }
      
Where the call to iscntrl(*s) requires the addition of the ctype.h header 
file.

I'm guessing a blanket replacement of control characters with 'C' isn't 
something all Midas users would want to do.  Replacing the control character 
with its hex value seems like a good choice - but not without adding bounds 
checking!

An alternative to changing odb.c could be to add a regex to Midas response 
text which removes all control characters (U+0000 - U+001F): 

var resp_lint = req.response.replace(/[\u{0000}-\u{001F}]/gmu, '');
var json_obj = JSON.parse(resp_lint);

Unfortunately, the 'u' regex flax doesn't work on the Firefox version 
included in Scientific Linux 6.8.  
ELOG V3.1.4-2e1708b5