Re: NFS under load?

Tony Arkles wrote:
>
> On 7-Jul-07, at 12:00 AM, Dylan Griffiths wrote:
>> The average speed seems to be settling down to the maximum sustained
>> write speed of the disk, despite my async option in exportfs.
>
> You'll eventually run out of RAM to buffer in I think...

Yea. RAM seems to have been a big part of the problem. It's a good
thing that DDR2 is priced such that 4gb of DDR2 800 is $190! (Vs. the
$110 I paid for an 8mb SIMM 10 years ago...)

Here is what happened:

Before:
  Server -> Athlon XP 2500+, 1gb RAM 1500 MTU
  Client -> Athlon 64 X2 3800+ 939, 2gb RAM 1500 MTU

  Server to client was 12-15mb/s. Sustained 12mb/s.
  Client to server was 12-15mb/s. Sustained 12mb/s.

Now:
  Server -> Athlon 64 X2 3800+ 939, 2gb RAM 1500 MTU
  Client -> Athlon 64 X2 3600+ AM2, 4gb RAM 8192 MTU

  Server to client is 18-32mb/s. Sustained 26mb/s.
  Client to server is 18-31mb/s. Sustained 23mb/s.

More RAM certainly helped. It also helps that the AM2 processor has an
INSANE amount of memory bandwidth. Memtest86+ puts the L2 cache and
DDR2 800 4gb RAM at the same bandwidth (!!!) of around 2.5Gbps, vs. the
S939 DDR1 system with DDR 400 sustaining 1.4-1.8Gbps (Athlon XP to A64).

I also upgraded the switch from the Airlink to the Netgear, and have
increased the MTU settings on the cards.

Setting the MTU to 8k on both machines marginally improves throughput,
but only while it can remain within memory cache.

EG: Copying a 1.1gb file yielded a burst of 56mb/s (458Mbps) with both
machines using 8192 MTU. However, the average speed was closer to
17mb/s due to the CPU/HD involved in writing through NFS to XFS ontop of
LVM. Doing the same thing to an XFS partition not-on-top of LVM yielded
a burst of 68mb/s (557Mbps), but a still low average. SCPing to an ext3
partition yielded a higher 19.5mb/s (159Mbps) sustained. Obviously the
best way to tune out the HD latency is more RAM and, when that runs out,
putting smarter filesystems in.

In summary, if you have a good NIC (eePro 1000), a good router (NetGear
GS108 or D-Link that supports jumbo frames -- Google them up), and a
decent amount of RAM, the gains are pretty good over fast ethernet
(roughly 2-6x). Now the hard drives are the bottleneck :) For things
like streaming video and audio over the LAN (which happens a lot thanks
to MythTV), it's a very nice experience. With the HDs being the
bottlenecks, NFS pretty much performs like my machine has that HD
directly connected to its SATA headers.

I'll be upgrading the server to 3gb of RAM once my next pair of DIMMs is
tested as OK. Hopefully that'll further improve my speeds :)
Received on Thu Jul 12 23:03:36 2007