Bull GNU/Linux NFSv4 project

2.6.22-rc5-CITI_NFS4_ALL-1
Robustness regression tests

Introduction

Here are the last results for the Robustness and Performances testing of the kernel and CITI patch: linux-2.6.22-rc5-CITI_NFS4_ALL-1

Tests performed

The following tools have been used:

Test #1 : connectathon 04 (update 2003/12) (ltp-full-20060822) LTP

run basic (-b) , general (-g) , special (-s), and lock tests (-l)
mode extended (-t) is used

Test #2 : nfs_fsstress (ltp-full-20060822) LTP

Tests are done with various number of subprocessess : 2, 4, 8, 16, 32
Tests are done with various length of instructions lists: 1, 2, 4, 8, 16, 32, 64, 128

Test #3 : fsx (ltp-full-20060822) LTP

fsx is run with 50000 iterations

Test #4 : Iozone (version 3.263) HERE

Run iozone -+q 30 -ace -r 64 -i 0 -i 1 -i 2 -g 2G -U mountpoint

Test #5 : ffsb (version 5.2.1) FFSB

Run with default profile_stress_test file as profile

Test #6 : locks testing HERE

Run mode local done with 500 processes
Run mode network done with two clients , 500 processes each

Test #7 : dbench (3.04) HERE

Run with 10, 100, 1000 clients

Test #8 : ACL testing (ltp-full-20060822) LTP

Not Done

Performances Results

We run iozone (Test#4) to measure:
- read and write performance throughputs for the asynchronous and synchronous modes of the command mount (async exports used in both cases)
- The different security flavours are measured: sys, krb5, krb5i, krb5p
The following parameters are used:
iozone -+q 5 -ace -r 64 -i 0 -i 1 -g 1G -f /mnt/dir/file -U /mnt/dir
mount options rsize,wsize are set to 262144 (due to 2G ram memory for both client and server)

Kernel .config file impact is very important on performances. Check it when bad performances.
For example, CONFIG_DEBUG_SLAB set will drastically decrease the performances.
The results on reads for asynchronous mount are HERE

Remarks:

We retrieve logically a degradation following the security flavor used. There is no longer the performance bug for krb5p (BUGZILLA#117) we got with 2.6.19-rc6-CITI_NFS4_ALL-1.
Nevertheless,

The results on reads for synchronous mount are HERE

Remarks:

Same as asynchronous mode.

The results on writes for asynchronous mount are HERE

For any flavors but krb5p, we get a drastic degradation performance for 512 Mbytes files and over even having 2G RAM memory. We didn't see that in the previous 2.6.19-rc6-CITI_NFS4_ALL-1

The results on writes for synchronous mount are HERE

Robustness Results

The objective was to run:

Tests #1 #2 #3 #4 #5 #6 #7 alone for: 2 hours, 15 hours, 60 hours.

Tests #1 #2 #3 #4 #5 were also run all together simultaneously for: 2 hours , 15 hours , 60 hours (option -U not used for iozone for this all together longrun)

Asynchronous mode is used for both: mount command and export option

The results are HERE

Remarks:

Mount error. See BUGZILLA#145

*When doing the following NFSV4 mount and umount sequence ,
*the first "umount" doesn't give error messages  even when we are under the
mounting point
*but the second umount tells well unmounting is not possible


mount /mnt/nosec
cd /mnt/nosec
umount /mnt/nosec

*No messages here telling unmounting not possible
*The commands "mount" tells /mnt/nosec is no longer mounted
*but /proc/mounts still contains the mount.
*If we do a second:

umount /mnt/nosec

*we get normally

umount: /mnt/nosec: device is busy

*By no longer being under the mounted directory

cd /
umount /mnt/nosec

*We effectively unmount

Test #3:

Get an issue when running fsx.
Sometimes results are not good. See BUGZILLA#143
We get:

truncating to largest ever: 0x3ffff
Size error: expected 0x1fe6 stat 0x2b000 seek 0x2b000
LOG DUMP (38414 total operations):
38415(15 mod 256): WRITE 0x134c7 thru 0x20c13 (0xd74d bytes)
38416(16 mod 256): WRITE 0x22379 thru 0x2f895 (0xd51d bytes) EXTEND
38417(17 mod 256): TRUNCATE UP from 0x2f896 to 0x3b913
38418(18 mod 256): WRITE 0xbba3 thru 0x1b0ca (0xf528 bytes)
38419(19 mod 256): MAPWRITE 0xd8cd thru 0xe939 (0x106d bytes)
38420(20 mod 256): READ 0x1203c thru 0x2064c (0xe611 bytes)
38421(21 mod 256): MAPREAD 0x194de thru 0x24595 (0xb0b8 bytes)
38422(22 mod 256): MAPWRITE 0x36ed7 thru 0x39bca (0x2cf4 bytes)
38423(23 mod 256): READ 0x2611b thru 0x33e13 (0xdcf9 bytes)


You can reproduce it by only running in a while true loop fsx-linux and iozone.

Kerberos Robustness Results

The objective was to run :

Tests #1 #2 #3 #4 #5 alone for: 2 hours, 15 hours, 60 hours using the different flavors krb5, krb5i , krb5p for 2 hours, 15 hours, 60 hours
Tests #1 #2 #3 #4 #5 were also run all together simultaneously for: 2 hours , 15 hours , 60 hours

Asynchronous mode is used for both: mount command and export option

The results are HERE

Remarks:

Test #3

With any flavor krb5,krb5i,krb5p we got the following issue (described at BUGZILLA#143).

When running ./fsx-linux -N 50000 /mnt/krb5p/file

gets sometimes the following error:


You can reproduce it by only running in a while true loop fsx-linux and iozone.

Tests #1 #2 #3 #4 #5 were also run all together

OOps when running the 60Hrs tests. See BUGZILLA#144.

Mount error.

With any flavor krb5,krb5i,krb5p we got the following issue (described at BUGZILLA#145).

*When doing the following NFSV4 mount and umount sequence ,
*the first "umount" doesn't give error messages  even when we are under the
mounting point
*but the second umount tells well unmounting is not possible

mount /mnt/nosec
cd /mnt/nosec
umount /mnt/nosec

*No messages here telling unmounting not possible
*The commands "mount" tells /mnt/nosec is no longer mounted
*but /proc/mounts still contains the mount.
*If we do a second:

umount /mnt/nosec

*we get normally

umount: /mnt/nosec: device is busy

*By no longer being under the mounted directory

cd /
umount /mnt/nosec
*We effectively unmount

Software versions

Linux	linux-2.6.22-rc5-CITI_NFS4_ALL-1
Client userland package	util-linux-2.12 + util-linux-2.12-CITI_NFS4_ALL-3.dif
Linux nfs-utils version	nfs-utils-1.1.0 +nfs-utils-1.1.0-CITI_NFS4_ALL-1.dif
gssapi library	libgssapi-0.11
rpcsecgss library	librpcsecgss-0.14
nfsidmap library	libnfsidmap-0.19
acl library	acl_2.2.29-1 +acl-2.2.29-CITI_NFS4_ALL-3
Linux TI-RPC	0.1.7

Hardware configuration

Client

Xeon(TM) CPU 2.80GHz, cache 512 KB, 2 processors

Total memory: 2GBytes

Ethernet: 1Gbits/s link

Distribution : Fedora Core 6 64bits

Kerberos V5 MIT1.4.3-4.

Server

Xeon(TM) CPU 2.80GHz, cache 512 KB, 4 processors

Total memory: 2GBytes

Ethernet: 1Gbits/s link

Distribution : Fedora Core 6 64bits

Kerberos V5 MIT1.4.3-4.

Conclusion

Core linux-2.6.22-rc5-CITI_NFS4_ALL-1 functions should be considered as stable for the security flavors sys, krb5 ,krb5i and krb5p.
There is a good improvement about krb5p.
There is a good improvement about write performance with nevertheless a big performance degradation when using iozone with file size from 512Mbytes.
Some unexplained hangs can still happen when the machine is client and server in the same time.
There is also the mount issue BUGZILLA#145 which can be troubling in many scripts.