Rethinking Storage Protocols

Here’s a doc I found presented at vmworld where a vender did a white paper to compare the performance of the 3 storage solutions. The test lab was 2 ESX hosts running 8 guests each. Fibre channel won on speed (and is not vunerable to the network like the other two), but the ip based protocols were not far behind. In this test, iscsi and nfs are roughly equivalent, but I think the test setup had an impact on that (the small number of ESX hosts and guests used). Here are a few quotes I’ve found from other sites as well:

http://www.vi411.org/2006/10/10/nasnfs-vs-iscsi-for-esx.html

“With a single VM and/or connection, iSCSI outperforms NFS anywhere from 10-50%. However, as the number of VMs per server increase, NFS gradually catches up then exceeds iSCSI performance at about 15 virtual machines per server. There are a few reasons, mainly in how NFS locks files compared to iSCSI and also that from the client-side of things, NFS uses much less CPU than iSCSI. There are other advantages of NFS as well - by default VMDK files on NFS are formatted as sparse volumes allowing for thin provisioning. Also, being normal file shares, they are much easier to manage and backup.”

http://blog.scottlowe.org/2007/09/21/nfs-for-vmware-storage/

“FC is always faster with 1 or 2 ESX hosts…however the more ESX host you add the faster NFS performs. This is because of FC SCSI reservations in ESX. Ideally only one host can read or write to a LUN at a time with FCP. With NFS this is not a limitation. Hence the more hosts, the better performance on NFS then presenting a number of LUNs to several hosts on FC….”

So looks like in everybody’s opinion, scsi reservations are the major factor in the performance debate. With TOE and/or hardware initiators the CPU usage of ISCSI can be brought down to a NFS comparable level probably. And finally NFS is just more familiar and manageable to people than ISCSI.

NetApp_Whitepaper