MisterX Tech

Note the packets in/out :
pcki  115264 – pcko  100013

I feel I have a minor ethernet issue as the IRQ load should not be quite that high but that is for another post. This box is a single core P4 so its not too far off.

ATOP – firewall02                                                   2010/10/04  09:34:23                                                   –x—                                                    3s elapsed
PRC | sys    3.02s  |              | user   0.01s  |              |               | #proc     96 |               | #zombie    0 |               | clones     0 |               |              |  #exit      0 |
CPU | sys       1%  | user      1% |               | irq     100% |               |              | idle     99%  | wait      0% |               |              |  steal     0% |              |  guest     0% |
cpu | sys       0%  | user      0% |               | irq     100% |               |              | idle      0%  | cpu000 w  0% |               |              |  steal     0% |              |  guest     0% |
cpu | sys       1%  | user      0% |               | irq       0% |               |              | idle     98%  | cpu001 w  0% |               |              |  steal     0% |              |  guest     0% |
CPL | avg1    1.01  |              | avg5    1.05  | avg15   1.11 |               |              |               | csw      187 |               | intr    4963 |               |              |  numcpu     2 |
MEM | tot     2.0G  | free  617.6M |               | cache 904.1M | dirty   0.0M  | buff  127.0M |               | slab  304.8M |               |              |               |              |               |
SWP | tot     4.0G  | free    4.0G |               |              |               |              |               |              |               |              |               | vmcom 123.9M |  vmlim   5.0G |
MDD |          md1  | busy      0% |               | read       0 | write     47  | KiB/r      0 |               | KiB/w      4 |  MBr/s   0.00 | MBw/s   0.06 |               | avq     0.00 |  avio 0.00 ms |
MDD |          md3  | busy      0% |               | read       0 | write     20  | KiB/r      0 |               | KiB/w      4 |  MBr/s   0.00 | MBw/s   0.03 |               | avq     0.00 |  avio 0.00 ms |
DSK |          sdb  | busy      5% |               | read       0 | write     57  | KiB/r      0 |               | KiB/w      5 |  MBr/s   0.00 | MBw/s   0.10 |               | avq     4.87 |  avio 2.84 ms |
DSK |          sda  | busy      4% |               | read       0 | write     57  | KiB/r      0 |               | KiB/w      5 |  MBr/s   0.00 | MBw/s   0.10 |               | avq     6.40 |  avio 2.09 ms |
NET | transport     | tcpi       7 | tcpo       4  | udpi       0 | udpo       0  | tcpao      0 |               | tcppo      0 |  tcprs      0 | tcpie      0 |  tcpor      2 | udpnp      3 |  udpip      0 |
NET | network       | ipi   118305 |               | ipo   102606 | ipfrw   3072  | deliv     27 |               |              |               |              |               | icmpi     16 |  icmpo  99524 |
NET | eth2      2%  | pcki    3030 | pcko  100013  |              | si  881 Kbps  | so   24 Mbps | coll       0  | mlti       0 |  erri       0 |              |  erro       0 | drpi       0 |  drpo       0 |
NET | eth3      1%  | pcki  115264 | pcko    2578  |              | si   19 Mbps  | so  787 Kbps | coll       0  | mlti       2 |  erri       0 |              |  erro       0 | drpi  208056 |  drpo       0 |

PID       RUID            EUID             THR        SYSCPU         USRCPU         VGROW        RGROW         RDDSK         WRDSK       ST        EXC        S       CPUNR         CPU        CMD        1/1
3       root            root               1         2.89s          0.00s            0K           0K            0K            0K       –          -        R           0         96%        ksoftirqd/0
Found the offender via tcpdump:

10:01:51.488936 00:0b:cd:3e:c6:93 > 00:30:48:94:94:5f, ethertype IPv4 (0×0800), length 62: (tos 0×0, ttl 104, id 4711, offset 0, flags [DF], proto: TCP (6), length: 48) 118.110.xx.xx.6697 > 173.201.xx.xx.http: P, cksum 0×6196 (correct), 2735265098:2735265098(0) ack 4261832542 win 63809 <mss 1460,nop,nop,sackOK>

10:01:51.488998 00:0b:cd:3e:c6:93 > 00:30:48:94:94:5f, ethertype IPv4 (0×0800), length 62: (tos 0×0, ttl  45, id 24124, offset 0, flags [DF], proto: TCP (6), length: 48) 97.17.xx.xx.11383 > 173.201.xx.xx.h
ttp: P, cksum 0x34dd (correct), 487590775:487590775(0) ack 1325631541 win 61462 <mss 1460,nop,nop,sackOK>

None of the IPs listed above (even though they have been edited to protect all parties) are ours.  I did a `ip route add blackhole` till I could turn off the switch port on the offender.

Useful sysctl commands if you want to stop this (spoofed attacks):

net.ipv4.conf.all.rp_filter = 1

Stops spoofed packets dead in their tracks ! Then you can focus on re-balancing your interrupts or better yet turning off their switch port!

[Fri Oct 01 12:37:39 2010] [error] [client *.*.*.*] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 46912496530832 bytes) in Unknown on line 0

Personally I think thats a little greedy with the RAM but thats just me ;)

Did a quick hack to see whats going on with a Linux IPTABLES firewall connection wise.

iptstate -s | awk ‘{print $3,$2,$1}’| cut -d”:” -f1-2 | uniq -c | sort -g

This shows you

# number of connections | Protocol |  Dest_IP:PORT | Source IP

I will add this to my bash wiki section in case it can help anyone.

I was working with a managed customer who handles their own DBA tasks on their servers when I noticed a recent change had created a major slow down in the speed of their front page. Here is the Apache Bench of their main page (actual URL removed to protect the guilty).

Server Software:        Apache/2.2.0
Server Hostname:        <REMOVED>
Document Path:          /index.php
Document Length:        103866 bytes
Concurrency Level:      10
Time taken for tests:   46.619197 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      10454000 bytes
HTML transferred:       10386600 bytes
Requests per second:    2.15 [#/sec] (mean)
Time per request:       4661.920 [ms] (mean)
Time per request:       466.192 [ms] (mean, across all concurrent requests)
Transfer rate:          218.97 [Kbytes/sec] received

The power of a simple index:
CREATE INDEX phpsession_speed on session_data(phpsessionid);

Server Software:        Apache/2.2.0
Server Hostname:        <REMOVED>
Server Port:            80
Document Path:          /index.php
Document Length:        103866 bytes
Concurrency Level:      10
Time taken for tests:   3.396604 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      10454000 bytes
HTML transferred:       10386600 bytes
Requests per second:    29.44 [#/sec] (mean)
Time per request:       339.660 [ms] (mean)
Time per request:       33.966 [ms] (mean, across all concurrent requests)
Transfer rate:          3005.35 [Kbytes/sec] received

There are MANY more optimizations I need to work with their DBA/Webdev because now their SQL server is saturating a gigabit connection with the webserver at the 30-40 RPS mark but this fixed their immediate problem.

I did a test for a SSD as a external journal for a ext3 based single SCSI drive with some mildly interesting results ! I was going to rerun with a larger test size but ran out of time. If anyone re-runs this test and can send me the results I would like to see them.

(remove old journal)
tune2fs -O ^has_journal /dev/sdc1
(add new journal a 400M parition on a SSD)
mke2fs -b 4096 -O journal_dev -J device=/dev/sdk1,size=400 /dev/sdk1
tune2fs -j -J device=/dev/sdk1,size=400 /dev/sdc1

All tests done on a Dual Xeon 2.8Ghz (HT enabled) 800FSB 1M cache
Benchmarks used :

tiobench (from RPMForge) run from CLI with defaults `tiobench`
dd : ` dd if=/dev/zero of=/mnt/test.tmp bs=256k count=10000`

Results

I saw these yet again today on a spam that found its way into a spam trap I have.

X-Priority: 3
X-Mailer: ZuckMail [version 1.00]
X-Facebook-Notify: password_reset; mailid=
Errors-To: terrace45@rotortug.com
X-FACEBOOK-PRIORITY: 1
MIME-Version: 1.0

The offending sender is (91.90.12.239) which , surprise surprise, isn’t a Facebook IP. I am working on a Spamassassin rule for this if anyone wants a “beta” copy of the meta rule let me know.

Ok I tested it and it appears to be working. The last 24 hours have seen over 100 hits all obvious spam (total volume during that time was 564K). The meta rule I am using is:

header CS_881                   X-Mailer =~ /\bZuckMail\b/i
header CS_882                   Received !~ /\bfacebook.com\b/i
meta FAKEFACEBOOK_01            (CS_881 && CS_882)
score FAKEFACEBOOK_01           3.9

Change the header names,meta names, and score to reflect what you feel is best for your system.

Love it or hate it google has around 600GB or captured wifi data  ! I do not want to get into war of opinions about this but I do think this should be a wake up call for everyone using wireless who has not secured their network to do so.

If you dont know how to do this find a family member, neighbor, Nerd Herd person, Google search,  or whoever to do it for you (even if it costs money). Aside from keeping your neighbors “out of your bushes” it will help keep your data safe from random war  Google driving.

If you do a :

telnet misterx.org 1701

You can get several MoTDs and some info on Star Trek online. Why did I do this..who knows…I was bored one night waiting for a FS benchmark to finish. Heck I dont even play Star Trek online I just was playing with a way to parse content in a bash script.

This totally geeky waste of time is brought to you thanks to : Steam, lynx, DJB (TCPSERVER+DAEMONTOOLS), and IMDB.com!

[root@localhost ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.0 Beta (Santiago)

LET THE FLOGGINGS (benchmarks) BEGIN!

Update: Appears the project iotop that I have been eyeing for sometime is now a option in the standard install. This is very nice I hope they choose some of the other system tools (like updated dstat, atop, and htop) to add as well.

Update: Ran a ps finally (I know I am slow getting to these kind of important things these days) and I noticed sendmail is gone ! Oh my now thats a change for the better. After 12 years of living with it as the default mailer I am excited to get to know postfix ! I have run it before in the past but 12 years running Qmail doesnt leave one much time to play around with other mailers. I am not biased I think the top two MTAs are Qmail and Postfix so I cant complain about the choice.

Well I am in the process of testing clusterFS solutions for work and figured it would also be a great chance to put our Atom product (SuperMicro’s 5015A-H) through the paces. During the testing phase, as seen here, I found that tests would need more horsepower to really be worthwhile.

I did find out that the Atoms can do about 36MBs a second using Replicated GlusterFS with caching tweaks before the processor max out. On my newer MSI-X systems the Ethernet load would have been pushed around several (or all) cores but since the Atoms only have MSI capabilities on their NICs I watched as one CPU pegged and other other one sat back and drank a Mojito. I was almost tempted into firing up a few high CPU tasks and using ole taskset to make the other CPU suffer but I refrained.

I will be updating once I can get a more powerful test setup going.