Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  14 May 2015, Konstantin Olchanski, Suggestion, checksums for midas data files 
    Reply  14 May 2015, Stefan Ritt, Suggestion, checksums for midas data files 
       Reply  15 May 2015, Konstantin Olchanski, Suggestion, checksums for midas data files 
    Reply  05 Oct 2016, Lee Pool, Suggestion, checksums for midas data files 
       Reply  13 Oct 2016, Konstantin Olchanski, Suggestion, checksums for midas data files 
          Reply  13 Mar 2017, Konstantin Olchanski, Suggestion, checksums for midas data files 
Message ID: 1058     Entry time: 14 May 2015     In reply to: 1057     Reply to this: 1059
Author: Stefan Ritt 
Topic: Suggestion 
Subject: checksums for midas data files 
> Any thoughts on this?

We use binary midas files now for ~20 years and never felt the necessity to put any checksums or even encryption on these files. The reason for that is the following: Data on 
modern hard disks is already protected by CRC code or even ERC on the lower level, so it's very unlikely that single bytes change. If something happens, then it's a 
corruption of the file system, so a few sectors of a file are missing or wrong. In that case a CRC won't help you much, just tells you that the files are corrupt. But you see that 
also in the midas event structure. Each event has a header with the size of the event, so you can follow the file event by event. If something is missing, the next event header 
is no event header but something in the middle of the date, and you recognise this immediately since the header does not make any send (date is off by many years, event ID 
is arbitrary, event size is very different). So this redundancy in the midas event structure helps you to identify any corrupt files as good in my opinion as a CRC code will. I 
would not want to waste a single CPU cycle on lengthy CRC or even SHA algorithms, unless I see single bytes change inside events. But in this case this can even happen at 
the network level between frontend and backends. So we should add the CRC/SHA code at the frontend level. This could increase the dead time of the experiment which is 
bad. And what about VME transfer? While hard disks and Ethernet networks have already built-in CRC checks, VME transfer doesn't. So how can you be sure that no bits 
get corrupt between your ADC and your frontend computer?

If people insist of having CRC or SHA protection/encryption for some reason I do not understand yet, we should make this optional, so that I can turn it off, since I don't 
need it.

/Stefan
ELOG V3.1.4-2e1708b5