ZFS、NFS、Nexenta

Question

このメッセージは誰かに馴染みがありますか？

zfs-clu-01a rpcmod: [ID 851375 kern.warning] WARNING: svc_cots_kdup no slots free zfs-clu-01a last message repeated 75220 times

ZFSでNFSを使用する際に多くの問題が発生します。実際には、NFSを介して大きなファイル（> 500mo）をコピーすることはできず、パフォーマンスの問題があります。

NFS client using NFSv3 over TCP **--NFS Server Configuration :** Concurrent NFSD Servers: 4096 NFSD queue length: 512 Concurrent LOCKD Servers: 4096 LOCKD queue length 512 **-- System configuration** **1/ Disable ZIL** Settings -> Preferences -> System Sys_zfs_nocacheflush Yes (default: No) Warning: it is unsafe without UPS **2/Disable Nagel algoritm** Settings -> Preferences -> Network Net_tcp_naglim_def 1 (default: 4095) **4/ disable sync** Data Management -> Shares -> Folder xxx Sync disable (default: Standard)

必要に応じて、より多くの情報を提供できます。

-------- UPDATE 01 -------------

私は新しい興味深いものを持っています：

この問題は、「centos」システム（リリース6.5）でのみまたはほとんど見られます。

Debian（最後のバージョン）と同じことが、同じ期間と同じようにこのファイルのコピーでうまく機能しています

詳細はこちら：

 OS: Nexenta Ver. 31-5-0 Load Average 1.25, 1.26, 1.21 NFSv3 Server (configuration in my last email)

1）NFSクライアント

 Clients Number: 15 with very high access demand and then we have more than 20 others low access Centos 6 and Debian 7 NFS Client v3 Mout options : nfs (rw,noatime,nodiratime,vers=3,rsize=16384,wsize=16384,proto=tcp,mountproto=tcp,nolock) Nics: 1x 1g Dedicated NFS vlan

2）zpoolリスト

 NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT syspool 464G 181G 283G 39% 1.00x ONLINE - tank1 6.52T 2.38T 4.13T 36% 1.00x ONLINE -

）zpoolステータス

 pool: syspool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM syspool ONLINE 0 0 0 c0t0d0s0 ONLINE 0 0 0 errors: No known data errors pool: tank1 state: ONLINE scan: resilvered 190G in 81h49m with 0 errors on Sun Jan 26 15:35:10 2014 config: NAME STATE READ WRITE CKSUM tank1 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c6t5000C5005F0C34A7d0 ONLINE 0 0 0 c6t5000C5005F1261CBd0 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 c6t5000C5005F531E2Fd0 ONLINE 0 0 0 c6t5000C5005FD1677Fd0 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 c6t5000C5005FD18BF7d0 ONLINE 0 0 0 c6t5000C5005FD1907Bd0 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 c6t50000394B820EA30d0 ONLINE 0 0 0 c6t5000C5005FD1D133d0 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 c6t5000C5005FD73F97d0 ONLINE 0 0 0 c6t5000C5005FD7518Bd0 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 c6t5000C5005FD7C2A3d0 ONLINE 0 0 0 c6t5000C5005FDC123Bd0 ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 c6t5000C5005FDE1D13d0 ONLINE 0 0 0 c6t5000C5005FDEAFDFd0 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 c6t5000C5005FDEB127d0 ONLINE 0 0 0 c6t5000C5005FDEB513d0 ONLINE 0 0 0 mirror-8 ONLINE 0 0 0 c6t5000C5005FDF88CBd0 ONLINE 0 0 0 c6t5000C5005FDF9C4Fd0 ONLINE 0 0 0 mirror-9 ONLINE 0 0 0 c6t5000C50067802A17d0 ONLINE 0 0 0 c6t5000C500678037A7d0 ONLINE 0 0 0 mirror-10 ONLINE 0 0 0 c6t5000C5006780D6F7d0 ONLINE 0 0 0 c6t5000C5006780DEB3d0 ONLINE 0 0 0 mirror-11 ONLINE 0 0 0 c6t5000C5006780E90Fd0 ONLINE 0 0 0 c6t5000C5006780E99Bd0 ONLINE 0 0 0 cache c0t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 spares c6t5000C5006BF8C91Fd0 AVAIL errors: No known data errors

4）hddisco

 =c6t5000C5006BF8C91Fd0 driver sd driver_instance 37 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS08 serial S0M19QGA guid 5000c5006bf8c91f device_id id1,sd@n5000c5006bf8c91f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006bf8c91d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006bf8c91e P end =c6t5000C5005F531E2Fd0 driver sd driver_instance 26 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M04DCN guid 5000c5005f531e2f device_id id1,sd@n5000c5005f531e2f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005f531e2d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005f531e2e P end =c6t5000C5005FDEAFDFd0 driver sd driver_instance 27 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M061KF guid 5000c5005fdeafdf device_id id1,sd@n5000c5005fdeafdf is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdeafdd P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdeafde P end =c6t5000C5006780E90Fd0 driver sd driver_instance 29 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06GV9 guid 5000c5006780e90f device_id id1,sd@n5000c5006780e90f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006780e90d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006780e90e P end =c6t5000C5005FD1677Fd0 driver sd driver_instance 28 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M05WCR guid 5000c5005fd1677f device_id id1,sd@n5000c5005fd1677f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd1677d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd1677e P end =c6t5000C5005FDF9C4Fd0 driver sd driver_instance 30 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M03SZR guid 5000c5005fdf9c4f device_id id1,sd@n5000c5005fdf9c4f is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdf9c4d P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdf9c4e P end =c6t5000C5005FD7518Bd0 driver sd driver_instance 25 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06JCM guid 5000c5005fd7518b device_id id1,sd@n5000c5005fd7518b is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd75189 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd7518a P end =c6t5000C5005F1261CBd0 driver sd driver_instance 24 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M03LZY guid 5000c5005f1261cb device_id id1,sd@n5000c5005f1261cb is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005f1261c9 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005f1261ca P end =c6t5000C5006780E99Bd0 driver sd driver_instance 20 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06GV8 guid 5000c5006780e99b device_id id1,sd@n5000c5006780e99b is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006780e999 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006780e99a P end =c6t5000C5005FD1907Bd0 driver sd driver_instance 23 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M0657D guid 5000c5005fd1907b device_id id1,sd@n5000c5005fd1907b is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd19079 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd1907a P end =c6t5000C5005FDF88CBd0 driver sd driver_instance 22 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M04CQ7 guid 5000c5005fdf88cb device_id id1,sd@n5000c5005fdf88cb is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdf88c9 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdf88ca P end =c6t5000C5005FDC123Bd0 driver sd driver_instance 21 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M05WMD guid 5000c5005fdc123b device_id id1,sd@n5000c5005fdc123b is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdc1239 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdc123a P end =c6t5000C5006780D6F7d0 driver sd driver_instance 19 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M05TVK guid 5000c5006780d6f7 device_id id1,sd@n5000c5006780d6f7 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006780d6f5 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006780d6f6 P end =c6t5000C5005FDEB127d0 driver sd driver_instance 13 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M061JX guid 5000c5005fdeb127 device_id id1,sd@n5000c5005fdeb127 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdeb125 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdeb126 P end =c6t5000C5005FD73F97d0 driver sd driver_instance 16 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06J86 guid 5000c5005fd73f97 device_id id1,sd@n5000c5005fd73f97 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd73f95 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd73f96 P end =c6t5000C5005F0C34A7d0 driver sd driver_instance 14 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M0405D guid 5000c5005f0c34a7 device_id id1,sd@n5000c5005f0c34a7 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005f0c34a5 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005f0c34a6 P end =c6t5000C500678037A7d0 driver sd driver_instance 15 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M03FNP guid 5000c500678037a7 device_id id1,sd@n5000c500678037a7 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c500678037a5 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c500678037a6 P end =c6t5000C5005FD18BF7d0 driver sd driver_instance 17 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M0659G guid 5000c5005fd18bf7 device_id id1,sd@n5000c5005fd18bf7 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd18bf5 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd18bf6 P end =c6t5000C50067802A17d0 driver sd driver_instance 18 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M03WJV guid 5000c50067802a17 device_id id1,sd@n5000c50067802a17 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c50067802a15 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c50067802a16 P end =c6t50000394B820EA30d0 driver sd driver_instance 32 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor Toshiba product AL13SEB600 revision DE06 serial 43L0A02HFRD3 guid 50000394b820ea30 device_id id1,sd@n50000394b820ea30 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w50000394b820ea32 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w50000394b820ea33 P end =c6t5000C5005FDE1D13d0 driver sd driver_instance 11 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M060LE guid 5000c5005fde1d13 device_id id1,sd@n5000c5005fde1d13 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fde1d11 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fde1d12 P end =c6t5000C5005FDEB513d0 driver sd driver_instance 6 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M061HL guid 5000c5005fdeb513 device_id id1,sd@n5000c5005fdeb513 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fdeb511 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fdeb512 P end =c6t5000C5005FD7C2A3d0 driver sd driver_instance 9 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M06HG0 guid 5000c5005fd7c2a3 device_id id1,sd@n5000c5005fd7c2a3 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd7c2a1 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd7c2a2 P end =c6t5000C5005FD1D133d0 driver sd driver_instance 7 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M064EN guid 5000c5005fd1d133 device_id id1,sd@n5000c5005fd1d133 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5005fd1d131 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5005fd1d132 P end =c6t5000C5006780DEB3d0 driver sd driver_instance 10 device_type disk removable no media_present yes size 600127266816 size_str 558.9GB caching_supported no vendor SEAGATE product ST600MM0006 revision LS06 serial S0M060TL guid 5000c5006780deb3 device_id id1,sd@n5000c5006780deb3 is_ssd no mpxio_enabled yes parent mpxio parent_instance 0 path_count 2 P start P parent mpt_sas P parent_instance 5 P path_state ONLINE P target_port w5000c5006780deb1 P end P start P parent mpt_sas P parent_instance 4 P path_state ONLINE P target_port w5000c5006780deb2 P end =c0t2d0 driver sd driver_instance 34 device_type disk removable no media_present yes size 511503761408 size_str 476.4GB caching_supported no vendor Dell product PERC H710 revision 3.13 serial 0022d14c0cc2ee9e1900ac67e7a0a38c guid 6b8ca3a0e767ac00199eeec20c4cd122 device_id id1,sd@n6b8ca3a0e767ac00199eeec20c4cd122 is_ssd no mpxio_enabled no lun_no 0 parent mr_sas parent_instance 0 =c0t1d0 driver sd driver_instance 33 device_type disk removable no media_present yes size 511503761408 size_str 476.4GB caching_supported no vendor Dell product PERC H710 revision 3.13 serial 0027a4060badee9e1900ac67e7a0a38c guid 6b8ca3a0e767ac00199eeead0b06a427 device_id id1,sd@n6b8ca3a0e767ac00199eeead0b06a427 is_ssd no mpxio_enabled no lun_no 0 parent mr_sas parent_instance 0 =c0t0d0 driver sd driver_instance 5 device_type disk removable no media_present yes size 499558383616 size_str 465.3GB caching_supported no vendor Dell product PERC H710 revision 3.13 serial 003f0f9308bb2c521900ac67e7a0a38c guid 6b8ca3a0e767ac0019522cbb08930f3f device_id id1,sd@n6b8ca3a0e767ac0019522cbb08930f3f is_ssd no mpxio_enabled no lun_no 0 parent mr_sas parent_instance 0

Gijs · Answer

うーん、システム（Nexenta）が取得しているNFS接続が多すぎるか、バグが発生しているようです。 maxdupreqsを上げてみると、問題が隠されている可能性がありますが、最初に試してみましょう。

echo 'rpcmod`maxdupreqs/W0t4096' | mdb -kw echo 'rpcmod`cotsmaxdupreqs/W0t4096' | mdb -kw

現在の値を確認するには、次を使用します。

echo 'rpcmod`maxdupreqs::print' | mdb -k echo 'rpcmod`cotsmaxdupreqs::print' | mdb -k

しかし、多分それは助けになります。