MisterX Tech

# uptime
15:05:59 up 271 days, 22:45,  2 users,  load average: 1789.01, 2333.21, 3025.28

This was a managed system where qmail ran away with a high remote concurrency set (5000). System was very responsive despite the load .

I am usually not one to get involved with politics and this matter is no different. Whichever way you lean on this DNS matter I feel it is a good chance to learn more about what options are out there. If you need to catchup here is some reading:

http://torrentfreak.com/bittorrent-based-dns-to-counter-us-domain-seizures-101130/
http://hotair.com/archives/2010/11/27/doj-seizes-domain-names-of-more-than-70-websites-suspected-of-piracy/
http://da.feedsportal.com/c/270/f/470440/s/1023e6ae/l/0Lnews0Btechworld0N0Csme0C32510A10A0Ctorrent0Efinder0Eshut0Edown0Eby0Eus0Eagencies0C0Dolo0Frss/ia1.htm

These show how government is getting more involved in DNS which is the core of how we do things on the net. This opens the door for further understanding/learning of the other options that are available  on the net.

One of these is a great project for a open DNS system:

http://www.opennicproject.org

What OpenNIC are doing already and the push to a torrent driven DNS system with the .p2p extension are making the DNS world get interesting!

So what you have here is a change that is causing “ripples” all over the ‘net that could lead to major/minor changes into how DNS is done. Whatever you think will become of this it is going to be interesting to watch play out.

PRC | sys    0.22s  | user   0.01s  |               |               | #proc    158  | #zombie    0  | clones     0  |               |               | #exit      0  |
CPU | sys       1%  | user      0%  | irq     131%  |               | idle    668%  |               | wait      0%  |               | steal     0%  | guest     0%  |
cpu | sys       0%  | user      0%  | irq      80%  |               | idle     20%  |               | cpu007 w  0%  |               | steal     0%  | guest     0%  |
cpu | sys       0%  | user      0%  | irq      52%  |               | idle     48%  |               | cpu006 w  0%  |               | steal     0%  | guest     0%  |
cpu | sys       1%  | user      0%  | irq       0%  |               | idle     99%  |               | cpu004 w  0%  |               | steal     0%  | guest     0%  |
cpu | sys       0%  | user      0%  | irq       0%  |               | idle    100%  |               | cpu000 w  0%  |               | steal     0%  | guest     0%  |
CPL | avg1    0.07  | avg5    0.10  |               | avg15   0.07  |               | csw     3209  | intr   65623  |               |               | numcpu     8  |
MEM | tot     5.8G  | free    5.4G  | cache 185.4M  | dirty   0.0M  | buff   91.2M  | slab   33.8M  |               |               |               |               |
SWP | tot     2.2G  | free    2.2G  |               |               |               |               |               |               | vmcom 136.4M  | vmlim   5.1G  |
NET | transport     | tcpi       3  | tcpo       3  | udpi       0  | udpo       0  | tcpao      0  | tcppo      0  | tcprs      0  | tcpie      0  | udpip      0  |
NET | network       | ipi   438568  | ipo   438568  | ipfrw      0  | deliv 438569  |               |               |               | icmpi 438565  | icmpo 438565  |
NET | eth0     70%  | pcki  438531  | pcko  438566  | si   70 Mbps  | so   50 Mbps  | coll       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |

PID    RUID         EUID          THR      SYSCPU      USRCPU      VGROW      RGROW     RDDSK      WRDSK     ST     EXC     S     CPUNR      CPU    CMD         1/1
24    root         root            1       0.18s       0.00s         0K         0K        0K         0K     —       –     S         7       6%    ksoftirqd/7
4314    root         root            1       0.01s       0.01s         0K         0K        0K         0K     —       –     S         0       1%    atop
3727    root         root            1       0.02s       0.00s         0K         0K        0K         0K     —       –     S         7       1%    kondemand/7
7825    root         root            1       0.01s       0.00s         0K         0K        0K         0K     —       –     R         4       0%    atop
7135    root         root            1       0.00s       0.00s         0K         0K        0K         0K     —       –     S         7       0%    sshd

This was created with several `ping -s 1 -q -f $IP` commands across several systems. The CPU is :

Intel(R) Xeon(R) CPU W3530  @ 2.80GHz

What can I say , I am impressed ! I am going to try and max it out and see what it can do all out! This is a HUGE jump from our old P4 based firewall that barfed at 80-100K pps inbound!

Had to do a quick test of a Magento server getting ready for production recently. Being a big fan of Apaches Worker MPM (even when using PHP with ZTS) I thought i would test their recommended settings for Prefork vs one of my homebrew Worker configs. The results can be found on my wiki at  http://misterx.org/wiki/index.php/Worker_vs_prefork_MPM.

Note the packets in/out :
pcki  115264 – pcko  100013

I feel I have a minor ethernet issue as the IRQ load should not be quite that high but that is for another post. This box is a single core P4 so its not too far off.

ATOP – firewall02                                                   2010/10/04  09:34:23                                                   –x—                                                    3s elapsed
PRC | sys    3.02s  |              | user   0.01s  |              |               | #proc     96 |               | #zombie    0 |               | clones     0 |               |              |  #exit      0 |
CPU | sys       1%  | user      1% |               | irq     100% |               |              | idle     99%  | wait      0% |               |              |  steal     0% |              |  guest     0% |
cpu | sys       0%  | user      0% |               | irq     100% |               |              | idle      0%  | cpu000 w  0% |               |              |  steal     0% |              |  guest     0% |
cpu | sys       1%  | user      0% |               | irq       0% |               |              | idle     98%  | cpu001 w  0% |               |              |  steal     0% |              |  guest     0% |
CPL | avg1    1.01  |              | avg5    1.05  | avg15   1.11 |               |              |               | csw      187 |               | intr    4963 |               |              |  numcpu     2 |
MEM | tot     2.0G  | free  617.6M |               | cache 904.1M | dirty   0.0M  | buff  127.0M |               | slab  304.8M |               |              |               |              |               |
SWP | tot     4.0G  | free    4.0G |               |              |               |              |               |              |               |              |               | vmcom 123.9M |  vmlim   5.0G |
MDD |          md1  | busy      0% |               | read       0 | write     47  | KiB/r      0 |               | KiB/w      4 |  MBr/s   0.00 | MBw/s   0.06 |               | avq     0.00 |  avio 0.00 ms |
MDD |          md3  | busy      0% |               | read       0 | write     20  | KiB/r      0 |               | KiB/w      4 |  MBr/s   0.00 | MBw/s   0.03 |               | avq     0.00 |  avio 0.00 ms |
DSK |          sdb  | busy      5% |               | read       0 | write     57  | KiB/r      0 |               | KiB/w      5 |  MBr/s   0.00 | MBw/s   0.10 |               | avq     4.87 |  avio 2.84 ms |
DSK |          sda  | busy      4% |               | read       0 | write     57  | KiB/r      0 |               | KiB/w      5 |  MBr/s   0.00 | MBw/s   0.10 |               | avq     6.40 |  avio 2.09 ms |
NET | transport     | tcpi       7 | tcpo       4  | udpi       0 | udpo       0  | tcpao      0 |               | tcppo      0 |  tcprs      0 | tcpie      0 |  tcpor      2 | udpnp      3 |  udpip      0 |
NET | network       | ipi   118305 |               | ipo   102606 | ipfrw   3072  | deliv     27 |               |              |               |              |               | icmpi     16 |  icmpo  99524 |
NET | eth2      2%  | pcki    3030 | pcko  100013  |              | si  881 Kbps  | so   24 Mbps | coll       0  | mlti       0 |  erri       0 |              |  erro       0 | drpi       0 |  drpo       0 |
NET | eth3      1%  | pcki  115264 | pcko    2578  |              | si   19 Mbps  | so  787 Kbps | coll       0  | mlti       2 |  erri       0 |              |  erro       0 | drpi  208056 |  drpo       0 |

PID       RUID            EUID             THR        SYSCPU         USRCPU         VGROW        RGROW         RDDSK         WRDSK       ST        EXC        S       CPUNR         CPU        CMD        1/1
3       root            root               1         2.89s          0.00s            0K           0K            0K            0K       —          –        R           0         96%        ksoftirqd/0
Found the offender via tcpdump:

10:01:51.488936 00:0b:cd:3e:c6:93 > 00:30:48:94:94:5f, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 104, id 4711, offset 0, flags [DF], proto: TCP (6), length: 48) 118.110.xx.xx.6697 > 173.201.xx.xx.http: P, cksum 0x6196 (correct), 2735265098:2735265098(0) ack 4261832542 win 63809 <mss 1460,nop,nop,sackOK>

10:01:51.488998 00:0b:cd:3e:c6:93 > 00:30:48:94:94:5f, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl  45, id 24124, offset 0, flags [DF], proto: TCP (6), length: 48) 97.17.xx.xx.11383 > 173.201.xx.xx.h
ttp: P, cksum 0x34dd (correct), 487590775:487590775(0) ack 1325631541 win 61462 <mss 1460,nop,nop,sackOK>

None of the IPs listed above (even though they have been edited to protect all parties) are ours.  I did a `ip route add blackhole` till I could turn off the switch port on the offender.

Useful sysctl commands if you want to stop this (spoofed attacks):

net.ipv4.conf.all.rp_filter = 1

Stops spoofed packets dead in their tracks ! Then you can focus on re-balancing your interrupts or better yet turning off their switch port!

[Fri Oct 01 12:37:39 2010] [error] [client *.*.*.*] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 46912496530832 bytes) in Unknown on line 0

Personally I think thats a little greedy with the RAM but thats just me ;)

Did a quick hack to see whats going on with a Linux IPTABLES firewall connection wise.

iptstate -s | awk ‘{print $3,$2,$1}’| cut -d”:” -f1-2 | uniq -c | sort -g

This shows you

# number of connections | Protocol |  Dest_IP:PORT | Source IP

I will add this to my bash wiki section in case it can help anyone.

I was working with a managed customer who handles their own DBA tasks on their servers when I noticed a recent change had created a major slow down in the speed of their front page. Here is the Apache Bench of their main page (actual URL removed to protect the guilty).

Server Software:        Apache/2.2.0
Server Hostname:        <REMOVED>
Document Path:          /index.php
Document Length:        103866 bytes
Concurrency Level:      10
Time taken for tests:   46.619197 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      10454000 bytes
HTML transferred:       10386600 bytes
Requests per second:    2.15 [#/sec] (mean)
Time per request:       4661.920 [ms] (mean)
Time per request:       466.192 [ms] (mean, across all concurrent requests)
Transfer rate:          218.97 [Kbytes/sec] received

The power of a simple index:
CREATE INDEX phpsession_speed on session_data(phpsessionid);

Server Software:        Apache/2.2.0
Server Hostname:        <REMOVED>
Server Port:            80
Document Path:          /index.php
Document Length:        103866 bytes
Concurrency Level:      10
Time taken for tests:   3.396604 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      10454000 bytes
HTML transferred:       10386600 bytes
Requests per second:    29.44 [#/sec] (mean)
Time per request:       339.660 [ms] (mean)
Time per request:       33.966 [ms] (mean, across all concurrent requests)
Transfer rate:          3005.35 [Kbytes/sec] received

There are MANY more optimizations I need to work with their DBA/Webdev because now their SQL server is saturating a gigabit connection with the webserver at the 30-40 RPS mark but this fixed their immediate problem.

I did a test for a SSD as a external journal for a ext3 based single SCSI drive with some mildly interesting results ! I was going to rerun with a larger test size but ran out of time. If anyone re-runs this test and can send me the results I would like to see them.

(remove old journal)
tune2fs -O ^has_journal /dev/sdc1
(add new journal a 400M parition on a SSD)
mke2fs -b 4096 -O journal_dev -J device=/dev/sdk1,size=400 /dev/sdk1
tune2fs -j -J device=/dev/sdk1,size=400 /dev/sdc1

All tests done on a Dual Xeon 2.8Ghz (HT enabled) 800FSB 1M cache
Benchmarks used :

tiobench (from RPMForge) run from CLI with defaults `tiobench`
dd : ` dd if=/dev/zero of=/mnt/test.tmp bs=256k count=10000`

Results