MisterX Tech

OS: CentOS 4.8
Apache : Custom RPM from source with only a single change to the location of the suexec directory

strace -t -f -v  -p $process -o /path/to/outputfile (note the $process is the primary Apache Process)

To find the primary Apache Process you do a :

ps -ef | grep httpd

and it returns something like this :

apache   26898 22378  8 13:50 ?        00:00:01 /usr/sbin/httpd -k start

the second number 22378 is the PID of the Apache parent process. I then waited for a :

Dec 11 10:02:20 web02 kernel: httpd[7121]: segfault at 0000007fbf3fff0c rip 0000002a9567344a rsp 0000007fbf3ffe90 error 6

in my /var/log/messages. Once that came I did a:

grep SIGSEGV /path/to/file_generated_w/strace

and noted times and PIDs. Here is a example output :

19730 12:07:35 — SIGSEGV (Segmentation fault) @ 0 (0) —
19784 12:08:56 — SIGSEGV (Segmentation fault) @ 0 (0) —

I then grepped out the PIDs (19784 and 19730 in the above example) with a segfault to different files and began reading. To grep this out I did :

grep 19730 /path/to/file_generated_w/strace > /tmp/out.19730

It was in these files I found my problem. Your mileage may vary but I found this method much easier than using the Apache config setting of CoreDumpDirectory which requires several changes that have to be undone. The CoreDumpDirectory setting also requires a few restarts of Apache which in a production environment can be undesired.

The main caveat to using strace is that , on a busy server, you can generate 100-300M of logs per minute so make sure you have the diskspace on the partition you are sending out strace output.

To throw my .02$ into this fray I say:

Linux is for work and Windows is for play :)

But seriously I have been amazing at how many free tools Linux includes with the OS (or via the net/repos) for free.

Then there is the issue of the constantly evolving Windows “shell” which keeps changing commands almost every major release of the OS. With Linux I can run basically the same command,with little or no changes, that I ran almost 15 years ago when I started playing around the OS. If Windows did this I would have much more respect for them and their OS.

I got the chance to finally do some benchmarking on the new X5550 Xeons. Here is what I came up with using sysbench.

System:
CPU:2x X5550 Xeons (8 cores)
RAM: 16G
Hardware Vendor (Model): Dell (R510)
OS: CentOS release 5.4 (Final)
Kernel : 2.6.18-164.6.1.el5 x86_64
HardDrive(s): OS - Raid 1  / /var/lib/mysql Raid10
Harddrive Controller: Perc6

my.cnf
innodb_log_group_home_dir=/var/log/innodb_logs
innodb_log_file_size=256M
innodb_log_files_in_group=2
innodb_buffer_pool_size=6G
innodb_additional_mem_pool_size=60M
innodb_log_buffer_size=4M
innodb_thread_concurrency=0 #As of MYSQL 5.0.19 0 makes this unlimited
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=2 #Risky but not a worry for this customer due to mainly static data. 

Sysbench Command :
sysbench --test=oltp --db-driver=mysql --num-threads=16
--mysql-user=root --max-time=60 --max-requests=0 --oltp-read-only=on

Test Result (subsequent tests were withing a small percentage of this resulte)
OLTP test statistics:
    queries performed:
        read:                            4457866
        write:                           0
        other:                           636838
        total:                           5094704
    transactions:                        318419 (5306.75 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 4457866 (74294.56 per sec.)
    other operations:                    636838 (10613.51 per sec.)
Test execution summary:
 total time:                          60.0026s
 total number of events:              318419
 total time taken by event execution: 958.0739
 per-request statistics:
 min:                                  0.98ms
 avg:                                  3.01ms
 max:                                334.80ms
 approx.  95 percentile:              10.86ms

Threads fairness:
 events (avg/stddev):           19901.1875/1010.54
 execution time (avg/stddev):   59.8796/0.03
sysbench --test=fileio --max-time=60 --max-requests=1000000  --file-num=1 --file-extra-flags=direct --file-fsync-freq=0 
--file-total-size=128M --file-test-mode=rndrd run
 
sysbench 0.4.10:  multi-threaded system evaluation benchmark
 
 Running the test with following options:
 Number of threads: 1
 
 Extra file open flags: 16384
 1 files, 128Mb each
 128Mb total file size
 Block size 16Kb
 Number of random requests for random IO: 1000000
 Read/Write ratio for combined random IO test: 1.50
 Calling fsync() at the end of test, Enabled.
 Using synchronous I/O mode
 Doing random read test
 Threads started!
 Time limit exceeded, exiting...
 Done.
Operations performed:  910652 Read, 0 Write, 0 Other = 910652 Total
 Read 13.895Gb  Written 0b  Total transferred 13.895Gb  (237.15Mb/sec)
 15177.50 Requests/sec executed