About configuration XML files.

Now we merged ROME and ARGUS.
So I expect that users often have several configuration files for one project.

But I realized many people don't know that they can specify configuration file.

So I propose following scheme.

* When a program starts, it searches all XML files located in current directory which has <Configuration> tag at the top.
* If it does not find any XML file, it asks user to generate new one.
   * ask configuration type [R/A/M/N]
   * ask filename. Default is romeConfig.xml
* If it find only one XML file, it reads the XML file.
* If it find several XML files, it asks user which file should be used with a prompt like,
     Which configuration file do you use ?
      [1] filename1.xml
      [2] romeConfig.xml
      [3] filename2.xml
      [q] quit program
* When user starts program with option -c, it asks creates new configuration anyway.
* When user starts program with option -i, it uses specified file.

done.
Program searches configuration XML file in ./ and ./config

|-- include
|   |-- daqs       : include files of user defined DAQs.
|   |-- databases  : include files of user defined databases.
|   |-- folders    : include files of folders come when 'ChangeableClassFile" is true.
|   |-- generated  : include files which can be overwritten by builder.
|   `-- tasks      : include files of tasks come when 'ChangeableClassFile" is true.
`-- src
    |-- daqs       : source files of user defined DAQs.
    |-- databases  : source files of user defined databases.
    |-- folders    : source files of folders come when 'ChangeableClassFile" is true.
    |-- generated  : source files which can be overwritten by builder.
    `-- tasks      : source files of tasks.         

I am thinking to divide my definition xml file into several files.

I do not know if there is a general way, but I found some web pages mentioning about it. According to 
these pages we can include an xml document to  other one like.

---------- booklist.xml ------------
<?xml version="1.0"?>
<!DOCTYPE books [
<!ENTITY book1 SYSTEM "book1.xml">
]>

<books>
  &book1;
</books>
------------------------------------
------------ book1.xml -------------
<?xml version="1.0"?>
<book>
  <title>Title of the book</title>
  <author>Author of the book</author>
</book>
------------------------------------

Is it possible to do it with mxml ?


I have two reasons.
1.
 I have two ROME programs.  I call them as "writer" and "reader" in this message. "reader" reads output 
file from "writer". In this case, "reader" needs to know the structure of branch folder defined in "writer"'s 
xml file.
What I am doing is adding headers of "writer" to DictionaryHeaders in Makefile.user of "reader". At least, 
it works, but this way I have to write reading function of trees by hand. On the other hand, If it is 
possible to include a part of xml file of "writer" in that of "reader", romebuilder can create proper 
functions automatically.

2.
 My project definition file is already big. It has more than 2000 lines. If one has good XML editor, it may 
not be a problem. But unfortunately I do not know good editor on Linux. What I am thinking is making a 
directory structure like this.
.
|-- main_definition.xml
`-- xml
    |-- folders
    |   |-- folder_1.xml
    |   |-- folder_2.xml
    |   `-- folder_3.xml
    `-- tasks
        |-- task_1.xml
        |-- task_2.xml
        `-- task_3.xml

folder_[1-3].xml and task_[1-3].xml will be included in main_definition.xml.

> I am thinking to divide my definition xml file into several files.
> 
> I do not know if there is a general way, but I found some web pages mentioning about it. According to 
> these pages we can include an xml document to  other one like.
> 
> ---------- booklist.xml ------------
> <?xml version="1.0"?>
> <!DOCTYPE books [
> <!ENTITY book1 SYSTEM "book1.xml">
> ]>
> 
> <books>
>   &book1;
> </books>
> ------------------------------------
> ------------ book1.xml -------------
> <?xml version="1.0"?>
> <book>
>   <title>Title of the book</title>
>   <author>Author of the book</author>
> </book>
> ------------------------------------
> 
> Is it possible to do it with mxml ?

I made a patch to enable it with mxml.
Please test it.

If it is reliable, I will commit it.

----
cd $ROMESYS
patch -p0 < mxml.diff  
----

> I made a patch to enable it with mxml.
> Please test it.
> 
> If it is reliable, I will commit it.
> 
> ----
> cd $ROMESYS
> patch -p0 < mxml.diff  
> ----

I modified bugs, and commited.

When the filename base IO was added, it seems that TFile creation was moved from 
ROMEEventLoop::DAQInit to ROMEEventLoop:DAQEndOfRun.

So ROME makes TTrees without outputfile.

This is dangerous when a TTree becomes huge.
It can consume large amount of memory.

> When the filename base IO was added, it seems that TFile creation was moved from 
> ROMEEventLoop::DAQInit to ROMEEventLoop:DAQEndOfRun.
> 
> So ROME makes TTrees without outputfile.
> 
> This is dangerous when a TTree becomes huge.
> It can consume large amount of memory.

Your right. I have changed it back.

Current file handling of ROME is based on run number. This way is simple and easy to understand. But 
there are some arguments on it.

(i) In case of .mid file. ROME assume the file name is like run00001.mid. But this is not always true. 
MIDAS users can have different name.

(ii) ROME saves one TTree in one file. But there are cases that user want to save several TTrees in a file

(iii) Especially rare event search experiments like MEG, one file may contain signal candidate events 
from several runs.

To solve this issue one idea is to move from run base I/O to filename base I/O.

* Input
Users may specify filename instead of run number with command line option like.
XXXanalyzer.exe -f run00001.mid,run00002.mid

Then ROMEAnalyzer provides information of filename to DAQ class.

How files are handled is depending on DAQ class. Normally each files will be read one by one.
But in case of bartender program, they will be read in parallel to mix.

In case of .mid file, run number is written in run header of files.
ROMEMidas should change current run number at the BeginOfRun.

* Output
For the output filename we can employ same technique as database path.
This way users can use also run number and some other parameters.
And they can store several TTrees in a file.

  <Tree>
      <TreeName>tree</TreeName>
      <TreeOutputFileName>"%s#.root",gAnalyzer->GetGSP()->GetFilePrefix()<TreeOutputFileName>
      <Branch>
        <BranchName>kine</BranchName>
        <RelatedFolder>Kinematics</RelatedFolder>
      </Branch>
  </Tree>
 

This change is quite big, and it affects all ROME users.
We have to discuss enough on it before changing.

(One choice is to remain run number base I/O.)

ResetFolders are called in DAQEvent for every events.

I have some folders who's information have to be kept during the run.

One possibility is to add new tag like <ResetBoforeEvents> in folder definition.
The other is give up reset in ROMEEventLoop and give users resposibility of reset.

> ResetFolders are called in DAQEvent for every events.
> 
> I have some folders who's information have to be kept during the run.
> 
> One possibility is to add new tag like <ResetBoforeEvents> in folder definition.
> The other is give up reset in ROMEEventLoop and give users resposibility of reset.

done.

The name of the tag is <NoResetByFramework>

When one makes a support folder type field in folder, pointer to the support folder or pointer of 
TClonesArray is in the folder.
But There is no "new" statement to create the object.

I think that fUniqueID and fBits in TObject is not useful to store in TTree.
I found in ROOT users guide that we can disable that with
MyClass::Class()->IgnoreTObjectStreamer(); 
.

If you also think these are useless information, please implement it in ROME.



second topic.
This is just a matter of favor.
We sometimes use TTree interactively. At that time, I feel fRunNumber, fEventNumber, fTimeStamp, 
fSequentialNumber are long to type.
I prefer shorter without upper case letter.
If many people agree it, I would like to ask to change them run,event,timestamp,number like MIDAS analyzer.

> I think that fUniqueID and fBits in TObject is not useful to store in TTree.
> I found in ROOT users guide that we can disable that with
> MyClass::Class()->IgnoreTObjectStreamer(); 
> .
> 
> If you also think these are useless information, please implement it in ROME.
> 
> 
> 
> second topic.
> This is just a matter of favor.
> We sometimes use TTree interactively. At that time, I feel fRunNumber, fEventNumber, fTimeStamp, 
> fSequentialNumber are long to type.
> I prefer shorter without upper case letter.
> If many people agree it, I would like to ask to change them run,event,timestamp,number like MIDAS analyzer.

MyClass::Class()->IgnoreTObjectStreamer(); is implemented.

> This is just a matter of favor.
> We sometimes use TTree interactively. At that time, I feel fRunNumber, fEventNumber, fTimeStamp, 
> fSequentialNumber are long to type.
> I prefer shorter without upper case letter.
> If many people agree it, I would like to ask to change them run,event,timestamp,number like MIDAS analyzer.
changed to run, event, time and number.

Hello,

It seems some compliers have a problem with making ROME, sending error message
saying,

 In file included from builder/src/ROMEBuilder.cpp:574:
/home/pekochan/rome/20050626/rome/include/ROMEString.h:41: type spec ifier
   omitted for parameter `va_list'
 ...etc


This problem can be fixed in file "ROMEString.h" by adding the following 
either line;

#include <stdarg.h>

or 

#include <Varargs.h>

> Hello,
> 
> It seems some compliers have a problem with making ROME, sending error message
> saying,
> 
>  In file included from builder/src/ROMEBuilder.cpp:574:
> /home/pekochan/rome/20050626/rome/include/ROMEString.h:41: type spec ifier
>    omitted for parameter `va_list'
>  ...etc
> 
> 
> This problem can be fixed in file "ROMEString.h" by adding the following 
> either line;
> 
> #include <stdarg.h>
> 
> or 
> 
> #include <Varargs.h>


Thanks for reporting that problem. 

#include <stdarg.h>
is added now in the cvs.

> > Hello,
> > 
> > It seems some compliers have a problem with making ROME, sending error message
> > saying,
> > 
> >  In file included from builder/src/ROMEBuilder.cpp:574:
> > /home/pekochan/rome/20050626/rome/include/ROMEString.h:41: type spec ifier
> >    omitted for parameter `va_list'
> >  ...etc
> > 
> > 
> > This problem can be fixed in file "ROMEString.h" by adding the following 
> > either line;
> > 
> > #include <stdarg.h>
> > 
> > or 
> > 
> > #include <Varargs.h>
> 
> 
> Thanks for reporting that problem. 
> 
> #include <stdarg.h>
> is added now in the cvs.
replaced with #include <Varargs.h>

In Reset method of folders. Support folder members are ignored currently.
Reset method of support folder member had better to be called.

In addition, isModified() should check if the member support folders were modified.

For example

class A : public TObject
{
protected:
   TypeB* B;
   TClonesArray* C;
.
.
public:
   A() {
      B = 0;
      C = 0;
   };
.
.
.
   void Reset() {
      int i;
      fModified = false;
      if(B) B->Reset();
      if(C) for(i=0;i<C->GetEntries();i++) { ((TypeC*)C->At(i))->Reset(); }
   };

   ClassDef(A,1)
};

This is a proposal on specifying relation of tasks.

Currently, ROME has task hierarchy system. It may be enough.
But a missing thing is that we can not use "||".

So a possibility is like this.

For example, there are two calibration tasks B and D, and a calculation task A requires one of B or D.
B and D dependes on data filling task C.

This case, one writes down the relation in definition file.
<TaskDependenses>
  <TaskDependense taskname="A"/> B || D </TaskDependense>
  <TaskDependense taskname="B"/> C </TaskDependense>
  <TaskDependense taskname="D"/> C && !B </TaskDependense>
</TaskDependeses>

Then, builder analyzes the relation and re-arranges the execution order like,
C->B->D->A

At run time, ROMEEventLoop checks if depending tasks were executed before calling a task. And when 
B or C is executed, it calls A.

This way users can specify complicated relation like ((A && B) || C ) || !D )....

A problem is that you can not express the relation with indent in configuration file and documentation.


This is just a proposal, I'm not sure which is better, TaskHierarcy or TaskDependense.
Personaly, I am not using task controll.

In ROMEBuilder.cpp, there are some mismatchs of formart and argument of ROMEString::Set or 
AppendFormatted.

1000       format.SetFormatted("   void Set%%s%%%ds(%%-%ds %%s%%%ds) { f%%s%%%ds = %%s; };
\n",lb,typeLen,lb,lb,lb); 

3980             format.SetFormatted("   %%s%%s*%%%ds  Get%%sAt(int index)%%%ds\n",typeLen-
folderName[i].Length()-scl,0+nameLen-folderName[i].Length(),lt);

5875    buffer.AppendFormatted("   bool CheckConfiguration(int runNumber);\n",shortCut.Data()); 

6487    buffer.AppendFormatted("#include <TBranchElement.h>\n",shortCut.Data()); 

7231       buffer.AppendFormatted("   return buffer;\n",stringBuffer.Data());

7271                buffer.AppendFormatted("   if (!this->GetDataBase(name.Data())->Read
(values,path,gAnalyzer->GetCurrentRunNumber())) {\n",folderName[i].Data(),valueName[i][j].Data());

7358          buffer.AppendFormatted("%s                  xml->WriteElement(\"HistArrayStartIndex\",\"0\");
\n",blank.Data(),pointerI.Data(),histoName[taskHierarchyClassIndex[i]][j].Data()); 

8771    buffer.AppendFormatted(" obj/%sAnalyzer.obj obj/%sEventLoop.obj obj/%sConfig.obj obj/
main.obj",shortCut.Data(),shortCut.Data(),shortCut.Data(),shortCut.Data(),shortCut.Data());

8815    buffer.AppendFormatted("     g++ $(Flags) -o $@ $(objects) $(Libraries)\n\n",shortCut.Data
(),mainProgName.Data());

8872    buffer.AppendFormatted("obj/%sFAnalyzer.obj: src/framework/%sFAnalyzer.f src/framework/%
sAnalyzer.cpp include/framework/%sAnalyzer.h\n",shortCut.Data(),shortCut.Data(),shortCut.Data());

9341    htmlFile.SetFormatted("%s%sUserHTML.html",outDir.Data(),shortCut.Data(),mainProgName.Data
());

done.
> In ROMEBuilder.cpp, there are some mismatchs of formart and argument of ROMEString::Set or 
> AppendFormatted.
> 
> 1000       format.SetFormatted("   void Set%%s%%%ds(%%-%ds %%s%%%ds) { f%%s%%%ds = %%s; };
> \n",lb,typeLen,lb,lb,lb); 
> 
> 3980             format.SetFormatted("   %%s%%s*%%%ds  Get%%sAt(int index)%%%ds\n",typeLen-
> folderName[i].Length()-scl,0+nameLen-folderName[i].Length(),lt);
> 
> 5875    buffer.AppendFormatted("   bool CheckConfiguration(int runNumber);\n",shortCut.Data()); 
> 
> 6487    buffer.AppendFormatted("#include <TBranchElement.h>\n",shortCut.Data()); 
> 
> 7231       buffer.AppendFormatted("   return buffer;\n",stringBuffer.Data());
> 
> 7271                buffer.AppendFormatted("   if (!this->GetDataBase(name.Data())->Read
> (values,path,gAnalyzer->GetCurrentRunNumber())) {\n",folderName[i].Data(),valueName[i][j].Data());
> 
> 7358          buffer.AppendFormatted("%s                  xml->WriteElement(\"HistArrayStartIndex\",\"0\");
> \n",blank.Data(),pointerI.Data(),histoName[taskHierarchyClassIndex[i]][j].Data()); 
> 
> 8771    buffer.AppendFormatted(" obj/%sAnalyzer.obj obj/%sEventLoop.obj obj/%sConfig.obj obj/
> main.obj",shortCut.Data(),shortCut.Data(),shortCut.Data(),shortCut.Data(),shortCut.Data());
> 
> 8815    buffer.AppendFormatted("     g++ $(Flags) -o $@ $(objects) $(Libraries)\n\n",shortCut.Data
> (),mainProgName.Data());
> 
> 8872    buffer.AppendFormatted("obj/%sFAnalyzer.obj: src/framework/%sFAnalyzer.f src/framework/%
> sAnalyzer.cpp include/framework/%sAnalyzer.h\n",shortCut.Data(),shortCut.Data(),shortCut.Data());
> 
> 9341    htmlFile.SetFormatted("%s%sUserHTML.html",outDir.Data(),shortCut.Data(),mainProgName.Data
> ());

About identical check of new and existing file in builder.

Currently there is a limit of file size defined by "bufferLength".
When the file size exceed the size, the file will be always overwritten by builder.
And if the file is existing task cpp file, user will lose a part of his code.

Actually, LPConfig.cpp is larger than that.

So I wrote a simple function. it will be called recursively when the file is larger than buffer.
bufferLength is not necessary to be so large.

bool checkIdentical(const int fileHandle,const char* pattern,bool flag=true)
{
   static int position
   int  i;
   char fileBuffer[bufferLength];

   if(flag)
      position = 0;

   int nb = read(fileHandle,&fileBuffer, sizeof(fileBuffer));
  
   if(nb == bufferLength){
      if((int)strlen(pattern)-position < bufferLength)
         return false;
     
      for (i=0;i<nb;i++)
         if (pattern[position+i] != fileBuffer[i])
            return false;     
      position += bufferLength;
      return checkIdentical(fileHandle,pattern,false);
   }  
   else{
      if (nb != (int)strlen(pattern) - position)
         return false;
     
      for (i=0;i<nb;i++)
         if (pattern[position+i] != fileBuffer[i])
            return false;
   }
   return true;
}


You may use this function in builder like,
   int fileHandle = open("test.txt",O_RDONLY);
   bool identical = checkIdentical(fileHandle,buffer);
   close(fileHandle);

And there is another way. TString can read file. this featrue can be used when you read existing Task cpp file.

   ifstream ifile("test.txt");
   TString fileBuffer;
   str.ReadFile(ifile);
   bool identical = ( fileBuffer == buffer );

done.

> About identical check of new and existing file in builder.
> 
> Currently there is a limit of file size defined by "bufferLength".
> When the file size exceed the size, the file will be always overwritten by builder.
> And if the file is existing task cpp file, user will lose a part of his code.
> 
> Actually, LPConfig.cpp is larger than that.
> 
> So I wrote a simple function. it will be called recursively when the file is larger than buffer.
> bufferLength is not necessary to be so large.
> 
> bool checkIdentical(const int fileHandle,const char* pattern,bool flag=true)
> {
>    static int position
>    int  i;
>    char fileBuffer[bufferLength];
> 
>    if(flag)
>       position = 0;
> 
>    int nb = read(fileHandle,&fileBuffer, sizeof(fileBuffer));
>   
>    if(nb == bufferLength){
>       if((int)strlen(pattern)-position < bufferLength)
>          return false;
>      
>       for (i=0;i<nb;i++)
>          if (pattern[position+i] != fileBuffer[i])
>             return false;     
>       position += bufferLength;
>       return checkIdentical(fileHandle,pattern,false);
>    }  
>    else{
>       if (nb != (int)strlen(pattern) - position)
>          return false;
>      
>       for (i=0;i<nb;i++)
>          if (pattern[position+i] != fileBuffer[i])
>             return false;
>    }
>    return true;
> }
> 
> 
> You may use this function in builder like,
>    int fileHandle = open("test.txt",O_RDONLY);
>    bool identical = checkIdentical(fileHandle,buffer);
>    close(fileHandle);
> 
> And there is another way. TString can read file. this featrue can be used when you read existing Task cpp file.
> 
>    ifstream ifile("test.txt");
>    TString fileBuffer;
>    str.ReadFile(ifile);
>    bool identical = ( fileBuffer == buffer );

As ROOTSYS is required for the ROME build, I would suggest to
include the code below in the builder for the application Makefile.
This will make Rome less dependent on the ROOT path.

Replace in ROMEBuilder.cpp:

buffer.AppendFormatted("rootcint -f %sDict.cpp -c -p ",shortCut.Data());

by

buffer.AppendFormatted("LD_LIBRARY_PATH=$(ROOTSYS)/lib $(ROOTSYS)/bin/rootcint
-f %sDict.cpp -c -p ",shortCut.Data());

> As ROOTSYS is required for the ROME build, I would suggest to
> include the code below in the builder for the application Makefile.
> This will make Rome less dependent on the ROOT path.
> 
> Replace in ROMEBuilder.cpp:
> 
> buffer.AppendFormatted("rootcint -f %sDict.cpp -c -p ",shortCut.Data());
> 
> by
> 
> buffer.AppendFormatted("LD_LIBRARY_PATH=$(ROOTSYS)/lib $(ROOTSYS)/bin/rootcint
> -f %sDict.cpp -c -p ",shortCut.Data());

That's in the cvs now, thanks.

> > As ROOTSYS is required for the ROME build, I would suggest to
> > include the code below in the builder for the application Makefile.
> > This will make Rome less dependent on the ROOT path.
> > 
> > Replace in ROMEBuilder.cpp:
> > 
> > buffer.AppendFormatted("rootcint -f %sDict.cpp -c -p ",shortCut.Data());
> > 
> > by
> > 
> > buffer.AppendFormatted("LD_LIBRARY_PATH=$(ROOTSYS)/lib $(ROOTSYS)/bin/rootcint
> > -f %sDict.cpp -c -p ",shortCut.Data());
> 
> That's in the cvs now, thanks.

LD_LIBRARY_PATH is not dedecated for ROOT. One can already have it, and overwriting it may cause error.
For instance, if he installed libstdc++ in non-standard place, he will set LD_LIBRARY_PATH there.

In case of GNU Makefile, we can use "if" statemente like,
ifdef LD_LIBRARY_PATH
LD_LIBRARY_PATH += $(ROOTSYS)/lib
else
LD_LIBRARY_PATH := $(ROOTSYS)/lib
endif

This is not strong request, just a proposal because the same thing can be done by configuration file.

It may be useful if users can have his command line options.

For instance.
<CommandOptions>
  <CommandOption>
    <OptionFlag>m<OptionFlag>
    <OptionName>isMC<OptionName>
    <OptionType>Bool_t<OptionType>
    <OptionInitialization>false<OptionInitialization>
  </CommandOption>
  <CommandOption>
    <OptionFlag>t<OptionFlag>
    <OptionName>TriggerType<OptionName>
    <OptionType>Int_t<OptionType>
    <OptionInitialization>0<OptionInitialization>
  </COmmandOption>
</CommandOptions>

Then user can change the variable with command line option like
./megframework -m -t 2;

Then builder creates new method.
(Bool_t) MEGAnalyzer::GetisMC()
(Int_t) MEGAnalyzer::GetTriggerType()

> Then user can change the variable with command line option like
> ./megframework -m -t 2;
> 
> Then builder creates new method.
> (Bool_t) MEGAnalyzer::GetisMC()
> (Int_t) MEGAnalyzer::GetTriggerType()

I realized better way. In this way users can add or remove command line options without changing their
task files. And it is not necessary to add new object in ROME.

What I propose is to make it in steering parameter. That is <SPCOmmandOption> in steering parameter's
properties.

  <SteeringParameterField>
    <SPFieldName>isMC</SPFieldName>
    <SPFieldType>Bool_t</SPFieldType>
    <SPCommandOption>m</SPCommandOption>
  </SteeringParameterField>
  <SteeringParameterField>
    <SPFieldName>TriggerType</SPFieldName>
    <SPFieldType>Int_t</SPFieldType>
    <SPCommandOption>t</SPCommandOption>
  </SteeringParameterField>

Then user can change the variable with command line option like
./megframework -m -t 2;
when -m is typed, isMC becomes true.
Trigger type will be 2.

They can specify the value both with config file and command line option just same as run numbers.
Values in config file will be overwritten by command line option.

 

> 
> > Then user can change the variable with command line option like
> > ./megframework -m -t 2;
> > 
> > Then builder creates new method.
> > (Bool_t) MEGAnalyzer::GetisMC()
> > (Int_t) MEGAnalyzer::GetTriggerType()
> 
> I realized better way. In this way users can add or remove command line options without changing their
> task files. And it is not necessary to add new object in ROME.
> 
> What I propose is to make it in steering parameter. That is <SPCOmmandOption> in steering parameter's
> properties.
> 
>   <SteeringParameterField>
>     <SPFieldName>isMC</SPFieldName>
>     <SPFieldType>Bool_t</SPFieldType>
>     <SPCommandOption>m</SPCommandOption>
>   </SteeringParameterField>
>   <SteeringParameterField>
>     <SPFieldName>TriggerType</SPFieldName>
>     <SPFieldType>Int_t</SPFieldType>
>     <SPCommandOption>t</SPCommandOption>
>   </SteeringParameterField>
> 
> Then user can change the variable with command line option like
> ./megframework -m -t 2;
> when -m is typed, isMC becomes true.
> Trigger type will be 2.
> 
> They can specify the value both with config file and command line option just same as run numbers.
> Values in config file will be overwritten by command line option.
> 
>  

done.