About identical check of new and existing file in builder.

Currently there is a limit of file size defined by "bufferLength".
When the file size exceed the size, the file will be always overwritten by builder.
And if the file is existing task cpp file, user will lose a part of his code.

Actually, LPConfig.cpp is larger than that.

So I wrote a simple function. it will be called recursively when the file is larger than buffer.
bufferLength is not necessary to be so large.

bool checkIdentical(const int fileHandle,const char* pattern,bool flag=true)
{
   static int position
   int  i;
   char fileBuffer[bufferLength];

   if(flag)
      position = 0;

   int nb = read(fileHandle,&fileBuffer, sizeof(fileBuffer));
  
   if(nb == bufferLength){
      if((int)strlen(pattern)-position < bufferLength)
         return false;
     
      for (i=0;i<nb;i++)
         if (pattern[position+i] != fileBuffer[i])
            return false;     
      position += bufferLength;
      return checkIdentical(fileHandle,pattern,false);
   }  
   else{
      if (nb != (int)strlen(pattern) - position)
         return false;
     
      for (i=0;i<nb;i++)
         if (pattern[position+i] != fileBuffer[i])
            return false;
   }
   return true;
}


You may use this function in builder like,
   int fileHandle = open("test.txt",O_RDONLY);
   bool identical = checkIdentical(fileHandle,buffer);
   close(fileHandle);

And there is another way. TString can read file. this featrue can be used when you read existing Task cpp file.

   ifstream ifile("test.txt");
   TString fileBuffer;
   str.ReadFile(ifile);
   bool identical = ( fileBuffer == buffer );

In ROMEBuilder.cpp, there are some mismatchs of formart and argument of ROMEString::Set or 
AppendFormatted.

1000       format.SetFormatted("   void Set%%s%%%ds(%%-%ds %%s%%%ds) { f%%s%%%ds = %%s; };
\n",lb,typeLen,lb,lb,lb); 

3980             format.SetFormatted("   %%s%%s*%%%ds  Get%%sAt(int index)%%%ds\n",typeLen-
folderName[i].Length()-scl,0+nameLen-folderName[i].Length(),lt);

5875    buffer.AppendFormatted("   bool CheckConfiguration(int runNumber);\n",shortCut.Data()); 

6487    buffer.AppendFormatted("#include <TBranchElement.h>\n",shortCut.Data()); 

7231       buffer.AppendFormatted("   return buffer;\n",stringBuffer.Data());

7271                buffer.AppendFormatted("   if (!this->GetDataBase(name.Data())->Read
(values,path,gAnalyzer->GetCurrentRunNumber())) {\n",folderName[i].Data(),valueName[i][j].Data());

7358          buffer.AppendFormatted("%s                  xml->WriteElement(\"HistArrayStartIndex\",\"0\");
\n",blank.Data(),pointerI.Data(),histoName[taskHierarchyClassIndex[i]][j].Data()); 

8771    buffer.AppendFormatted(" obj/%sAnalyzer.obj obj/%sEventLoop.obj obj/%sConfig.obj obj/
main.obj",shortCut.Data(),shortCut.Data(),shortCut.Data(),shortCut.Data(),shortCut.Data());

8815    buffer.AppendFormatted("     g++ $(Flags) -o $@ $(objects) $(Libraries)\n\n",shortCut.Data
(),mainProgName.Data());

8872    buffer.AppendFormatted("obj/%sFAnalyzer.obj: src/framework/%sFAnalyzer.f src/framework/%
sAnalyzer.cpp include/framework/%sAnalyzer.h\n",shortCut.Data(),shortCut.Data(),shortCut.Data());

9341    htmlFile.SetFormatted("%s%sUserHTML.html",outDir.Data(),shortCut.Data(),mainProgName.Data
());

done.

> About identical check of new and existing file in builder.
> 
> Currently there is a limit of file size defined by "bufferLength".
> When the file size exceed the size, the file will be always overwritten by builder.
> And if the file is existing task cpp file, user will lose a part of his code.
> 
> Actually, LPConfig.cpp is larger than that.
> 
> So I wrote a simple function. it will be called recursively when the file is larger than buffer.
> bufferLength is not necessary to be so large.
> 
> bool checkIdentical(const int fileHandle,const char* pattern,bool flag=true)
> {
>    static int position
>    int  i;
>    char fileBuffer[bufferLength];
> 
>    if(flag)
>       position = 0;
> 
>    int nb = read(fileHandle,&fileBuffer, sizeof(fileBuffer));
>   
>    if(nb == bufferLength){
>       if((int)strlen(pattern)-position < bufferLength)
>          return false;
>      
>       for (i=0;i<nb;i++)
>          if (pattern[position+i] != fileBuffer[i])
>             return false;     
>       position += bufferLength;
>       return checkIdentical(fileHandle,pattern,false);
>    }  
>    else{
>       if (nb != (int)strlen(pattern) - position)
>          return false;
>      
>       for (i=0;i<nb;i++)
>          if (pattern[position+i] != fileBuffer[i])
>             return false;
>    }
>    return true;
> }
> 
> 
> You may use this function in builder like,
>    int fileHandle = open("test.txt",O_RDONLY);
>    bool identical = checkIdentical(fileHandle,buffer);
>    close(fileHandle);
> 
> And there is another way. TString can read file. this featrue can be used when you read existing Task cpp file.
> 
>    ifstream ifile("test.txt");
>    TString fileBuffer;
>    str.ReadFile(ifile);
>    bool identical = ( fileBuffer == buffer );

done.
> In ROMEBuilder.cpp, there are some mismatchs of formart and argument of ROMEString::Set or 
> AppendFormatted.
> 
> 1000       format.SetFormatted("   void Set%%s%%%ds(%%-%ds %%s%%%ds) { f%%s%%%ds = %%s; };
> \n",lb,typeLen,lb,lb,lb); 
> 
> 3980             format.SetFormatted("   %%s%%s*%%%ds  Get%%sAt(int index)%%%ds\n",typeLen-
> folderName[i].Length()-scl,0+nameLen-folderName[i].Length(),lt);
> 
> 5875    buffer.AppendFormatted("   bool CheckConfiguration(int runNumber);\n",shortCut.Data()); 
> 
> 6487    buffer.AppendFormatted("#include <TBranchElement.h>\n",shortCut.Data()); 
> 
> 7231       buffer.AppendFormatted("   return buffer;\n",stringBuffer.Data());
> 
> 7271                buffer.AppendFormatted("   if (!this->GetDataBase(name.Data())->Read
> (values,path,gAnalyzer->GetCurrentRunNumber())) {\n",folderName[i].Data(),valueName[i][j].Data());
> 
> 7358          buffer.AppendFormatted("%s                  xml->WriteElement(\"HistArrayStartIndex\",\"0\");
> \n",blank.Data(),pointerI.Data(),histoName[taskHierarchyClassIndex[i]][j].Data()); 
> 
> 8771    buffer.AppendFormatted(" obj/%sAnalyzer.obj obj/%sEventLoop.obj obj/%sConfig.obj obj/
> main.obj",shortCut.Data(),shortCut.Data(),shortCut.Data(),shortCut.Data(),shortCut.Data());
> 
> 8815    buffer.AppendFormatted("     g++ $(Flags) -o $@ $(objects) $(Libraries)\n\n",shortCut.Data
> (),mainProgName.Data());
> 
> 8872    buffer.AppendFormatted("obj/%sFAnalyzer.obj: src/framework/%sFAnalyzer.f src/framework/%
> sAnalyzer.cpp include/framework/%sAnalyzer.h\n",shortCut.Data(),shortCut.Data(),shortCut.Data());
> 
> 9341    htmlFile.SetFormatted("%s%sUserHTML.html",outDir.Data(),shortCut.Data(),mainProgName.Data
> ());

This is a proposal on specifying relation of tasks.

Currently, ROME has task hierarchy system. It may be enough.
But a missing thing is that we can not use "||".

So a possibility is like this.

For example, there are two calibration tasks B and D, and a calculation task A requires one of B or D.
B and D dependes on data filling task C.

This case, one writes down the relation in definition file.
<TaskDependenses>
  <TaskDependense taskname="A"/> B || D </TaskDependense>
  <TaskDependense taskname="B"/> C </TaskDependense>
  <TaskDependense taskname="D"/> C && !B </TaskDependense>
</TaskDependeses>

Then, builder analyzes the relation and re-arranges the execution order like,
C->B->D->A

At run time, ROMEEventLoop checks if depending tasks were executed before calling a task. And when 
B or C is executed, it calls A.

This way users can specify complicated relation like ((A && B) || C ) || !D )....

A problem is that you can not express the relation with indent in configuration file and documentation.


This is just a proposal, I'm not sure which is better, TaskHierarcy or TaskDependense.
Personaly, I am not using task controll.

I think that fUniqueID and fBits in TObject is not useful to store in TTree.
I found in ROOT users guide that we can disable that with
MyClass::Class()->IgnoreTObjectStreamer(); 
.

If you also think these are useless information, please implement it in ROME.



second topic.
This is just a matter of favor.
We sometimes use TTree interactively. At that time, I feel fRunNumber, fEventNumber, fTimeStamp, 
fSequentialNumber are long to type.
I prefer shorter without upper case letter.
If many people agree it, I would like to ask to change them run,event,timestamp,number like MIDAS analyzer.

Hello,

It seems some compliers have a problem with making ROME, sending error message
saying,

 In file included from builder/src/ROMEBuilder.cpp:574:
/home/pekochan/rome/20050626/rome/include/ROMEString.h:41: type spec ifier
   omitted for parameter `va_list'
 ...etc


This problem can be fixed in file "ROMEString.h" by adding the following 
either line;

#include <stdarg.h>

or 

#include <Varargs.h>

> Hello,
> 
> It seems some compliers have a problem with making ROME, sending error message
> saying,
> 
>  In file included from builder/src/ROMEBuilder.cpp:574:
> /home/pekochan/rome/20050626/rome/include/ROMEString.h:41: type spec ifier
>    omitted for parameter `va_list'
>  ...etc
> 
> 
> This problem can be fixed in file "ROMEString.h" by adding the following 
> either line;
> 
> #include <stdarg.h>
> 
> or 
> 
> #include <Varargs.h>


Thanks for reporting that problem. 

#include <stdarg.h>
is added now in the cvs.

In Reset method of folders. Support folder members are ignored currently.
Reset method of support folder member had better to be called.

In addition, isModified() should check if the member support folders were modified.

For example

class A : public TObject
{
protected:
   TypeB* B;
   TClonesArray* C;
.
.
public:
   A() {
      B = 0;
      C = 0;
   };
.
.
.
   void Reset() {
      int i;
      fModified = false;
      if(B) B->Reset();
      if(C) for(i=0;i<C->GetEntries();i++) { ((TypeC*)C->At(i))->Reset(); }
   };

   ClassDef(A,1)
};

> I think that fUniqueID and fBits in TObject is not useful to store in TTree.
> I found in ROOT users guide that we can disable that with
> MyClass::Class()->IgnoreTObjectStreamer(); 
> .
> 
> If you also think these are useless information, please implement it in ROME.
> 
> 
> 
> second topic.
> This is just a matter of favor.
> We sometimes use TTree interactively. At that time, I feel fRunNumber, fEventNumber, fTimeStamp, 
> fSequentialNumber are long to type.
> I prefer shorter without upper case letter.
> If many people agree it, I would like to ask to change them run,event,timestamp,number like MIDAS analyzer.

MyClass::Class()->IgnoreTObjectStreamer(); is implemented.

> > Hello,
> > 
> > It seems some compliers have a problem with making ROME, sending error message
> > saying,
> > 
> >  In file included from builder/src/ROMEBuilder.cpp:574:
> > /home/pekochan/rome/20050626/rome/include/ROMEString.h:41: type spec ifier
> >    omitted for parameter `va_list'
> >  ...etc
> > 
> > 
> > This problem can be fixed in file "ROMEString.h" by adding the following 
> > either line;
> > 
> > #include <stdarg.h>
> > 
> > or 
> > 
> > #include <Varargs.h>
> 
> 
> Thanks for reporting that problem. 
> 
> #include <stdarg.h>
> is added now in the cvs.
replaced with #include <Varargs.h>

> This is just a matter of favor.
> We sometimes use TTree interactively. At that time, I feel fRunNumber, fEventNumber, fTimeStamp, 
> fSequentialNumber are long to type.
> I prefer shorter without upper case letter.
> If many people agree it, I would like to ask to change them run,event,timestamp,number like MIDAS analyzer.
changed to run, event, time and number.

When one makes a support folder type field in folder, pointer to the support folder or pointer of 
TClonesArray is in the folder.
But There is no "new" statement to create the object.

ResetFolders are called in DAQEvent for every events.

I have some folders who's information have to be kept during the run.

One possibility is to add new tag like <ResetBoforeEvents> in folder definition.
The other is give up reset in ROMEEventLoop and give users resposibility of reset.

> ResetFolders are called in DAQEvent for every events.
> 
> I have some folders who's information have to be kept during the run.
> 
> One possibility is to add new tag like <ResetBoforeEvents> in folder definition.
> The other is give up reset in ROMEEventLoop and give users resposibility of reset.

done.

The name of the tag is <NoResetByFramework>

Current file handling of ROME is based on run number. This way is simple and easy to understand. But 
there are some arguments on it.

(i) In case of .mid file. ROME assume the file name is like run00001.mid. But this is not always true. 
MIDAS users can have different name.

(ii) ROME saves one TTree in one file. But there are cases that user want to save several TTrees in a file

(iii) Especially rare event search experiments like MEG, one file may contain signal candidate events 
from several runs.

To solve this issue one idea is to move from run base I/O to filename base I/O.

* Input
Users may specify filename instead of run number with command line option like.
XXXanalyzer.exe -f run00001.mid,run00002.mid

Then ROMEAnalyzer provides information of filename to DAQ class.

How files are handled is depending on DAQ class. Normally each files will be read one by one.
But in case of bartender program, they will be read in parallel to mix.

In case of .mid file, run number is written in run header of files.
ROMEMidas should change current run number at the BeginOfRun.

* Output
For the output filename we can employ same technique as database path.
This way users can use also run number and some other parameters.
And they can store several TTrees in a file.

  <Tree>
      <TreeName>tree</TreeName>
      <TreeOutputFileName>"%s#.root",gAnalyzer->GetGSP()->GetFilePrefix()<TreeOutputFileName>
      <Branch>
        <BranchName>kine</BranchName>
        <RelatedFolder>Kinematics</RelatedFolder>
      </Branch>
  </Tree>
 

This change is quite big, and it affects all ROME users.
We have to discuss enough on it before changing.

(One choice is to remain run number base I/O.)

When the filename base IO was added, it seems that TFile creation was moved from 
ROMEEventLoop::DAQInit to ROMEEventLoop:DAQEndOfRun.

So ROME makes TTrees without outputfile.

This is dangerous when a TTree becomes huge.
It can consume large amount of memory.

> When the filename base IO was added, it seems that TFile creation was moved from 
> ROMEEventLoop::DAQInit to ROMEEventLoop:DAQEndOfRun.
> 
> So ROME makes TTrees without outputfile.
> 
> This is dangerous when a TTree becomes huge.
> It can consume large amount of memory.

Your right. I have changed it back.

I am thinking to divide my definition xml file into several files.

I do not know if there is a general way, but I found some web pages mentioning about it. According to 
these pages we can include an xml document to  other one like.

---------- booklist.xml ------------
<?xml version="1.0"?>
<!DOCTYPE books [
<!ENTITY book1 SYSTEM "book1.xml">
]>

<books>
  &book1;
</books>
------------------------------------
------------ book1.xml -------------
<?xml version="1.0"?>
<book>
  <title>Title of the book</title>
  <author>Author of the book</author>
</book>
------------------------------------

Is it possible to do it with mxml ?


I have two reasons.
1.
 I have two ROME programs.  I call them as "writer" and "reader" in this message. "reader" reads output 
file from "writer". In this case, "reader" needs to know the structure of branch folder defined in "writer"'s 
xml file.
What I am doing is adding headers of "writer" to DictionaryHeaders in Makefile.user of "reader". At least, 
it works, but this way I have to write reading function of trees by hand. On the other hand, If it is 
possible to include a part of xml file of "writer" in that of "reader", romebuilder can create proper 
functions automatically.

2.
 My project definition file is already big. It has more than 2000 lines. If one has good XML editor, it may 
not be a problem. But unfortunately I do not know good editor on Linux. What I am thinking is making a 
directory structure like this.
.
|-- main_definition.xml
`-- xml
    |-- folders
    |   |-- folder_1.xml
    |   |-- folder_2.xml
    |   `-- folder_3.xml
    `-- tasks
        |-- task_1.xml
        |-- task_2.xml
        `-- task_3.xml

folder_[1-3].xml and task_[1-3].xml will be included in main_definition.xml.

> I am thinking to divide my definition xml file into several files.
> 
> I do not know if there is a general way, but I found some web pages mentioning about it. According to 
> these pages we can include an xml document to  other one like.
> 
> ---------- booklist.xml ------------
> <?xml version="1.0"?>
> <!DOCTYPE books [
> <!ENTITY book1 SYSTEM "book1.xml">
> ]>
> 
> <books>
>   &book1;
> </books>
> ------------------------------------
> ------------ book1.xml -------------
> <?xml version="1.0"?>
> <book>
>   <title>Title of the book</title>
>   <author>Author of the book</author>
> </book>
> ------------------------------------
> 
> Is it possible to do it with mxml ?

I made a patch to enable it with mxml.
Please test it.

If it is reliable, I will commit it.

----
cd $ROMESYS
patch -p0 < mxml.diff  
----