Netapp: High CPU/NFS Ops – low network input

Troubleshooting note to self






High CPU and NFS operations, however network input is low
Symptoms
High CPU and NFS operations, however network input is low
Cause of this problem
Numerous NFS clients continuously send the filer NFS operation requests, but the clients/users may not have permission to access the filer’s data.
Solution

If the output of the filer’s sysstat command displays higher than normal NFS operations per second and higher than normal CPU utilization, and sysstat displays low network input, determine if NFS clients are sending invalid requests to the filer that are being denied. To determine which NFS clients are sending numerous packets to the filer, complete the following steps:

Note: Enabling NFS per client statistics may cause a slight performance impact.

  1. Enable NFS per client statistics:

    options nfs.per_client_stats.enable on

  2. To zero the statistics, enter:

    nfsstat -z

  3. To display the usage of all NFS clients that mount the filer, enter

    nfsstat -l

  4. Investigate the client(s) with the highest usage. Sample nfsstat -l output:

    filer> nfsstat -l
    10.10.10.1 client1 NFSOPS = 49 (1%)
    10.10.10.2 client2 NFSOPS = 47 (1%)
    10.10.10.3 client3 NFSOPS = 745 (96%)
    10.10.10.4 client4 NFSOPS = 48 (1%)
    10.10.10.2 client5 NFSOPS = 50 (1%)

    In the sample output above, client3 has the highest usage and is most likely causing the packet storm.

  5. Turn off NFS per client statistics options nfs.per_client_stats.enable off
  6. Investigate NFS clients that show high usage in nfsstat -l output for processes that may be causing the packet storm.


Share and Enjoy:
  • Facebook
  • Twitter
  • Posterous

27.Jan.09 geek


You can follow any responses to this entry through the RSS 2.0 feed.
Both comments and pings are currently closed.