Re: NFS under load?

Chris Friesen wrote:
> Unless you've got newer E1000 hardware (82571 or later) with fairly
> recent drivers, you'll likely eventually run into problems with 9K
> frames once you've been up and running for a while.
>

Looks like the hardware is older:
04:07.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet
Controller (rev 05)
     Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter
     Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 50
     Memory at f7fc0000 (32-bit, non-prefetchable) [size=128K]
     Memory at f7fa0000 (32-bit, non-prefetchable) [size=128K]
     I/O ports at e880 [size=64]
     Expansion ROM at 88000000 [disabled] [size=128K]
     Capabilities: [dc] Power Management version 2
     Capabilities: [e4] PCI-X non-bridge device

> The hardware only handles power-of-two buffers, so any MTU over 8192
> gets bumped up to 16K. Then you add a bit of skb overhead and round up
> to the nearest power-of-two and you end up trying to do an atomic
> allocation of 32KB of physically contiguous memory for every incoming
> packet. After a while the system can get fragmented enough that this
> just isn't available.

Well, that explains that. I haven't spelunked the driver code to very;
the one machine is using a newer 2.6.20-something, while the other uses
2.6.17 (Ubuntu 6.10 kernel under 7.04).

> We spent quite a while on a solution for this issue...

Those sorts of problems always seem to suck :)

>> The issue I still have is that NFS is kicking around 60-120Mbps, not
>> the 300 I can get using iperf over TCP. I'll have to do some more
>> tweaking (Tony Arkles sent me some great resources to work from).
>
> If you haven't already, mount using 8K for "rsize" and "wsize". If
> you're using NFS4, it can handle up to 32K.

They are set appropriately in my fstab (8k and 16k).
I dropped the 16k one to 8k, and I got a longer window of higher-speeds
(15 megabytes/second) until the number of nfsd processes on the server
reaches its maximum threshold. Once this occurs, the server kernel ends
up doing a lot of waiting to write stuff (this is XFS via NFS 3; the HD
is a SATA 7200 rpm drive w/ 16mb of cache). Based on some simple tests,
it seems like this happens once it fills up the ram marked in top as
buffers.

The average speed seems to be settling down to the maximum sustained
write speed of the disk, despite my async option in exportfs.

/proc/net/rpc/nfsd seems to indicate that I spend most of my time at an
nfsd-thread-load of 1 or 8, depending. Small writes/reads that take not
too much, or the full-on-write-load which spike the user load #s to
2.xx-3.xx (from their usual < 0.50ish).

I'm not sure how to solve this to reduce the disk from being the
bottleneck, except via more RAM and more spindles. NFS 4 might help
(from reading various docs), so I may yet try that.
Received on Sat Jul 7 00:01:01 2007