MisterX Tech

Was troubleshooting a coding issue for a customer the other day when I noticed the number of requests currently being processed. I am going to write about this in the coming days. Basically there was a code issue (we do not support code for this customer) that caused a large number of “sleep” states in MySQL as well as a huge number of ‘W Sending Reply’ states in Apache.

Current Time: Thursday, 11-Feb-2010 11:47:54 CST

Restart Time: Thursday, 11-Feb-2010 09:17:12 CST

Parent Server Generation: 0

Server uptime: 2 hours 30 minutes 41 seconds

Total accesses: 226733 – Total Traffic: 10.8 GB

CPU Usage: u1812.56 s80.08 cu.04 cs0 – 20.9% CPU load

25.1 requests/sec – 1.2 MB/second – 49.8 kB/request

1337 requests currently being processed, 113 idle workers

OS: CentOS 4.8
Apache : Custom RPM from source with only a single change to the location of the suexec directory

strace -t -f -v  -p $process -o /path/to/outputfile (note the $process is the primary Apache Process)

To find the primary Apache Process you do a :

ps -ef | grep httpd

and it returns something like this :

apache   26898 22378  8 13:50 ?        00:00:01 /usr/sbin/httpd -k start

the second number 22378 is the PID of the Apache parent process. I then waited for a :

Dec 11 10:02:20 web02 kernel: httpd[7121]: segfault at 0000007fbf3fff0c rip 0000002a9567344a rsp 0000007fbf3ffe90 error 6

in my /var/log/messages. Once that came I did a:

grep SIGSEGV /path/to/file_generated_w/strace

and noted times and PIDs. Here is a example output :

19730 12:07:35 — SIGSEGV (Segmentation fault) @ 0 (0) —
19784 12:08:56 — SIGSEGV (Segmentation fault) @ 0 (0) —

I then grepped out the PIDs (19784 and 19730 in the above example) with a segfault to different files and began reading. To grep this out I did :

grep 19730 /path/to/file_generated_w/strace > /tmp/out.19730

It was in these files I found my problem. Your mileage may vary but I found this method much easier than using the Apache config setting of CoreDumpDirectory which requires several changes that have to be undone. The CoreDumpDirectory setting also requires a few restarts of Apache which in a production environment can be undesired.

The main caveat to using strace is that , on a busy server, you can generate 100-300M of logs per minute so make sure you have the diskspace on the partition you are sending out strace output.