The mhttpd ODB displays and mhist truncate values of "float" and "double"
floating point variables to 6 digits. In reality, "float" has 7 significant
digits and "double" has 16. I recommend that db_sprintf() in odb.c be changed to
read this:

      case TID_FLOAT:
         sprintf(string, "%.7g", *(((float *) data) + index));
         break;
      case TID_DOUBLE:
         sprintf(string, "%.16g", *(((double *) data) + index));
         break;

K.O.

> The mhttpd ODB displays and mhist truncate values of "float" and "double"
> floating point variables to 6 digits. In reality, "float" has 7 significant
> digits and "double" has 16. I recommend that db_sprintf() in odb.c be changed to
> read this:
> 
>       case TID_FLOAT:
>          sprintf(string, "%.7g", *(((float *) data) + index));
>          break;
>       case TID_DOUBLE:
>          sprintf(string, "%.16g", *(((double *) data) + index));
>          break;
> 
> K.O.

I had there

      case TID_FLOAT:
         if (ss_isnan(*(((float *) data) + index)))
            sprintf(string, "NAN");
         else
            sprintf(string, "%g", *(((float *) data) + index));
         break;
      case TID_DOUBLE:
         if (ss_isnan(*(((double *) data) + index)))
            sprintf(string, "NAN");
         else
            sprintf(string, "%lg", *(((double *) data) + index));
         break;

so I assumed that "%g" takes care of the maximal resolution. But apparently it does
not. So I changed it as you proposed.

Aparently the mhttpd elog will corrupt the elog files if two (or more?) elog entries are being edited at the 
same time. K.O.

[quote="Stefan Ritt"][Quote="K.O.]Aparently the mhttpd elog will corrupt the
elog files if two (or more\?) elog entries are being edited at the same time.
K.O.[/quote]

The corruption is very simple. mhttpd elog indexes the elog entries by the elog
file and offset inside the file, i.e. "http://ladd00:8088/EL/060927.318",
"060927" corresponds to log file "060927.log", "318" is the offset inside the
file where the message is located.

During "edit", the code "remembers" the offset of the original message and in
el_submit() blindly writes the edited message into the file at the remembered
offset.

If another message was edited before the edit of the first message is submitted,
the remembered offset becomes invalid (messages have shifted inside the file)
and el_submit() writes the edited text into the wrong place in the file,
corrupting it.

I have now added a check for this and we crash instead of corrupting the elog
file (midas.c rev 3340).

I do not know how to "properly" fix this bug without changing the indexing
scheme to something similar to what is used by elogd- message numbers instead of
file indices. In the existing scheme, message editing also breaks URLs shown in
the email notifications (they contain file indices that point to the wrong
places after messages are moved around by editing) and "reply threading" links.

Here is how I reproduce this bug:

1) start with an empty elog
2) create two messages
3) "edit" the second message, but do not submit it yet.
4) "edit" the first message, change the text to make sure the message size
becomes different; submit this change.
5) submit the "edit" of the first message. !!BOOM!!

K.O.

> I do not know how to "properly" fix this bug without changing the indexing
> scheme to something similar to what is used by elogd- message numbers instead of
> file indices. In the existing scheme, message editing also breaks URLs shown in
> the email notifications (they contain file indices that point to the wrong
> places after messages are moved around by editing) and "reply threading" links.

Well, the development of elogd with it's message numbers was actually stimulated by
the problem you mentioned. After that all those problems went away. Another
incarnation of that problem is if you edit an mhttpd log file manually. Afterwards
the file offsets are different and the system gets corrupted. To fix this properly,
one would have to backport the el_xxx functions from elogd to mhttpd, or, even
simpler, remove the elog functionality in mhttpd and "force" everybody to use elogd
(after doing elconv to convert the files into the new format).

first-console$ python evprint.py receive
second-console$ python evsend.py 123
[first console]
id=2007 mask=2007 serial=2007 time=1169372833 len=3 '123'

$ python evprint.py --log --path 127.0.0.1/online receive"
cm_connect_experiment('127.0.0.1', 'online', 'evprint.py', None)
bm_open_buffer('SYSTEM', 1048576, &c_long(2)) -> BM_CREATED
bm_request_event(2, -1, -1, 2, &c_long(0), None)
... wait for a couple of seconds ...
[midas.c:9348:rpc_call] rpc timeout, routine = "bm_receive_event"
[system.c:3570:send_tcp] send(socket=0,size=8) returned -1, errno: 88 (Socket 
operation on non-socket)
[midas.c:9326:rpc_call] send_tcp() failed

bm_receive_event(2, ...) -> RPC_TIMEOUT

bm_remove_event_request(2, 0) -> BM_INVALID_HANDLE
bm_close_buffer(2) -> BM_INVALID_HANDLE
cm_disconnect_experiment()

   1: python evprint.py receive
   2: python evsend.py 123
   1: id=2007 mask=2007 serial=2007 time=1169372833 len=3 '123'

   1: python evprint.py --path 127.0.0.1/ dispatch
   2: python evsend.py --path 127.0.0.1/ 123
   1: id=2007 mask=2007 serial=2007 time=1169373366 len=3 '123'

   1: python evprint.py dispatch
   2: python evsend.py --path 127.0.0.1/ 123
   1: (nothing printed)

   1: python evprint.py --path 127.0.0.1/ dispatch
   2: python evsend.py 123
   1: (nothing printed)

D:\denis\cmd\midas\current\06jan21-export\midas\NT\bin>produce.exe
ID of event to produce: 1
Host to connect: 127.0.0.1
Event size: 10
Level:   0.0 %, Rate: 0.64 MB/sec
flush
Level:   0.0 %, Rate: 0.64 MB/sec
Level:   0.0 %, Rate: 0.63 MB/sec
Level:   0.0 %, Rate: 0.64 MB/sec
Level:   0.0 %, Rate: 0.61 MB/sec
Level:   0.0 %, Rate: 0.62 MB/sec
Level:   0.0 %, Rate: 0.62 MB/sec
Level:   0.0 %, Rate: 0.64 MB/sec
Level:   0.0 %, Rate: 0.63 MB/sec
Level:   0.0 %, Rate: 0.63 MB/sec
Level:   0.0 %, Rate: 0.64 MB/sec
flush
Level:   0.0 %, Rate: 0.62 MB/sec

## Now I've started consume in the other console ##

[system.c:3570:send_tcp] send(socket=1900,size=8136) returned -1, errno: 0 (No error)
send_tcp() returned -1
[midas.c:9669:rpc_send_event] send_tcp() failed
rpc_send_event returned error 503, event_size 10

D:\denis\cmd\midas\current\06jan21-export\midas\NT\bin>consume.exe
ID of event to request: 1
Host to connect:
Get all events (0/1): 1
Receive via callback ([y]/n):
Level:   0.0 %, Rate: 0.00 MB/sec, ser mismatches: 0
Level:   0.0 %, Rate: 0.00 MB/sec, ser mismatches: 0
Level:   0.0 %, Rate: 0.00 MB/sec, ser mismatches: 0
Received break. Aborting...

D:\denis\cmd\midas\current\06jan21-export\midas\NT\bin\mserver.exe started interactively
[midas.c:2315:bm_validate_client_index] Invalid client index 0 in buffer 'SYSTEM'.
Client name 'Power Consumer', pid 1964 should be 3216

D:\denis\cmd\midas\current\06jan21-export\midas\NT\bin>produce.exe
ID of event to produce: 1
Host to connect:
Event size: 10
Client 'Producer' (PID 2584) on 'ODB' removed by cm_watchdog (idle 144.1s,TO 10s)
Level:   0.0 %, Rate: 3.20 MB/sec
flush
Level:   0.0 %, Rate: 3.20 MB/sec
Level:   0.0 %, Rate: 3.11 MB/sec
Level:   0.0 %, Rate: 3.13 MB/sec
Level:   0.0 %, Rate: 3.06 MB/sec
Level:   0.0 %, Rate: 3.20 MB/sec
Level:   0.0 %, Rate: 2.96 MB/sec
Level:   0.0 %, Rate: 3.11 MB/sec
Level:   0.0 %, Rate: 3.18 MB/sec
Level:   0.0 %, Rate: 3.13 MB/sec
Level:   0.0 %, Rate: 3.17 MB/sec
flush
Level:   0.0 %, Rate: 3.19 MB/sec
Level:   0.0 %, Rate: 3.08 MB/sec
Level:   0.0 %, Rate: 3.06 MB/sec

## Now I've started consume ##

[midas.c:2315:bm_validate_client_index] Invalid client index 0 in buffer 'SYSTEM'. Client name '', pid 0 should be 760

D:\denis\cmd\midas\current\06jan21-export\midas\NT\bin>consume.exe
ID of event to request: 1
Host to connect: 127.0.0.1
Get all events (0/1): 1
Receive via callback ([y]/n):
Level:   0.0 %, Rate: 0.00 MB/sec, ser mismatches: 0
Level:   0.0 %, Rate: 0.00 MB/sec, ser mismatches: 0
Received break. Aborting...
Level:   0.0 %, Rate: 0.00 MB/sec, ser mismatches: 0

D:\denis\cmd\midas\current\06jan21-export\midas\NT\bin>produce.exe
ID of event to produce: 1
Host to connect: 127.0.0.1
Event size: 10
Level:   0.0 %, Rate: 0.65 MB/sec
flush
Level:   0.0 %, Rate: 0.66 MB/sec
Level:   0.0 %, Rate: 0.65 MB/sec
Level:   0.0 %, Rate: 0.60 MB/sec
Level:   0.0 %, Rate: 0.64 MB/sec
Level:   0.0 %, Rate: 0.63 MB/sec
Level:   0.0 %, Rate: 0.61 MB/sec
Level:   0.0 %, Rate: 0.63 MB/sec
Level:   0.0 %, Rate: 0.65 MB/sec
Level:   0.0 %, Rate: 0.65 MB/sec
Level:   0.0 %, Rate: 0.67 MB/sec
flush
Level:   0.0 %, Rate: 0.66 MB/sec
Level:   0.0 %, Rate: 0.65 MB/sec
Level:   0.0 %, Rate: 0.65 MB/sec
Level:   0.0 %, Rate: 0.66 MB/sec
Level:   0.0 %, Rate: 0.66 MB/sec
Level:   0.0 %, Rate: 0.65 MB/sec
Level:   0.0 %, Rate: 0.66 MB/sec
Level:   0.0 %, Rate: 0.66 MB/sec
Level:   0.0 %, Rate: 0.66 MB/sec
Level:  66.8 %, Rate: 0.66 MB/sec
flush
Level:   0.0 %, Rate: 0.00 MB/sec
Level:  66.8 %, Rate: 0.31 MB/sec
Level:  57.2 %, Rate: 0.15 MB/sec
Level:  57.3 %, Rate: 0.14 MB/sec
Level:  57.3 %, Rate: 0.15 MB/sec
Level:  57.3 %, Rate: 0.14 MB/sec
Level:  57.3 %, Rate: 0.14 MB/sec
Level:  57.3 %, Rate: 0.14 MB/sec
Received break. Aborting...
Received 2nd break. Hard abort.
[midas.c:1581:] cm_disconnect_experiment not called at end of program

D:\denis\cmd\midas\current\06jan21-export\midas\NT\bin>consume.exe
ID of event to request: 1
Host to connect: 127.0.0.1
Get all events (0/1): 1
Receive via callback ([y]/n):
[consume.c:73:process_event] Serial number mismatch: Ser: 1397076, OldSer: 0, ID: 1, size: 10
Level:  37.1 %, Rate: 0.00 MB/sec, ser mismatches: 1
Level:   0.0 %, Rate: 0.15 MB/sec, ser mismatches: 1
Level:  95.4 %, Rate: 0.08 MB/sec, ser mismatches: 1
Level:  66.8 %, Rate: 0.14 MB/sec, ser mismatches: 1
Level:  66.8 %, Rate: 0.12 MB/sec, ser mismatches: 1
Level:  76.3 %, Rate: 0.12 MB/sec, ser mismatches: 1
Level:  95.4 %, Rate: 0.11 MB/sec, ser mismatches: 1
Level:  57.3 %, Rate: 0.15 MB/sec, ser mismatches: 1
Level:  66.8 %, Rate: 0.11 MB/sec, ser mismatches: 1
Level:  85.9 %, Rate: 0.11 MB/sec, ser mismatches: 1
Level:  95.5 %, Rate: 0.12 MB/sec, ser mismatches: 1
Level:  57.4 %, Rate: 0.15 MB/sec, ser mismatches: 1
Level:   9.7 %, Rate: 0.15 MB/sec, ser mismatches: 1
[Producer] [midas.c:1581:] cm_disconnect_experiment not called at end of program
Level:   0.0 %, Rate: 0.03 MB/sec, ser mismatches: 1
Level:   0.0 %, Rate: 0.00 MB/sec, ser mismatches: 1
Received break. Aborting...

Hello,

We have problems analyzing large files under Windows XP. For small file sizes,
everything is ok. We have events of 2.8 MB each, and we can read ~30 events per
second. But if the file gets larger than typically 600-800 MB, then access
becomes very slow, about 1 event per second. This is not the case under Linux,
where it stays at 30 Hz (~90 MB/sec). 

Looking at the low level file access, it is obvious that this has nothing to do
with midas, this problem can be reproduced with a simple program reading chunks
of 3MB from a 1GB file. The Windows XP file system is NTFS, default formatting.
Does anyone else have observed a similar problem or maybe even have some
suggestions? Unfortunately many people here want to analyze midas data under
Windows...

Stefan Ritt

cc -c -g -O2 -Wall -Wuninitialized -Iinclude -Idrivers -I../mxml -Llinux/lib -DINCLUDE_FTPLIB   -D_LARGEFILE64_SOURCE -DHAVE_MYSQL -DHAVE_ROOT -pthread -I/opt/root/current/include -DOS_LINUX -fPIC -Wno-unused-function -o linux/lib/system.o src/system.c
src/system.c:958: error: expected declaration specifiers or ‘...’ before ‘gettid’
src/system.c:958: warning: data definition has no type or storage class
src/system.c:958: warning: type defaults to ‘int’ in declaration of ‘_syscall0’
src/system.c: In function ‘ss_gettid’:
src/system.c:1005: warning: implicit declaration of function ‘gettid’
src/system.c: In function ‘ss_suspend_init_ipc’:
src/system.c:2948: warning: pointer targets in passing argument 3 of ‘getsockname’ differ in signedness
src/system.c: In function ‘ss_suspend’:
src/system.c:3414: warning: pointer targets in passing argument 6 of ‘recvfrom’ differ in signedness
src/system.c:3441: warning: pointer targets in passing argument 6 of ‘recvfrom’ differ in signedness
make: *** [linux/lib/system.o] 错误 1

void ss_force_single_thread()
{
   _single_thread = TRUE;
}

#if defined(OS_DARWIN)
// blank
#elif defined(OS_LINUX)
_syscall0(pid_t,gettid);
#endif

INT ss_gettid(void)