Performance perspective on virtual disk controllers in VMware environments.

There are different types/standards of virtual storage controllers (i.e. SATA, SCSI and NVMe) that we can give to VMs. It is obvious for everyone that disk performance of VM is depended on physical storage performance but what about “virtual world” ? 

Will type of virtual disk controller have any impact on VM disk performance ?

Sometimes during vSphere courses I get this question from my students so I decided to do few quick tests and write this short article.

There are many layers when we deal with storage I/O in virtual-physical environments but in tests below I will focus only on differences between currently most popular virtual disk controllers – SATA, LSI SAS, VMware Paravirtual SCSI (pvSCSI) and NVMe. 

LSI Logic SAS is usually default controller for disks and SATA controller for CD-ROM that we can see during VM creation . LSI SAS and VMware Paravirtual SCSI are available for virtual machines since ESXi 4.x. AHCI SATA is available since ESXi 5.5 and NVMe since ESXi 6.5 – so which one we should choose ?

To find an answer we can do some tests and the scenario is simple – one VM with 4 virtual disks located on the same datastore. All four disks are thick provisioned, eagerly zeroed to exclude impact of expanding/zeroing operation. All four disk have the same size and all of them are working on the same physical device “underground” (NVMe disk in esxi 8.0 host). The only difference is just type of virtual controller so we will have almost unbiased starting point (esxi run in nested nviroment).

Here is picture of test architecture :

 ESXi is VM (nested virtualization) - 8 vCPU, 24GB RAM, virtual disks on SSD (datastore for tests is located on physical Samsung NVMe 1TB Pro PCIe 4.0 as thick provisioned and eager-zeroed)

Test VM is Windows 10 Pro - 4 sockets/4vCPU, 8GB RAM, all disks thick provisioned and eager-zeroed.

Here are esxi host client views :

No other virtual disks were on datastore during tests :

 View from guest OS :

All test were done with open-source disk drive benchmark tool for Microsoft Windows called CrystalDiskMark with 1GiB and 8 GiB test file size. However, defaults are not for testing enterprise-level storage but rather desktop workstations disk testing. I recommend to do tests with a realistic working set size you have in production application. Higher size usually simulates “more real” behaviour (I have some limits in my LAB – that’s why I used 1 and 8 GiB).

And here are tests results :

Here are detailed results (in text) of test above :

Test 1

------------------------------------------------------------------------------

CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo

                 Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

 SEQ  1MiB (Q= 8, T= 1):  983.059 MB/s [  937.5 IOPS] < 8497.49 us>

 SEQ  1MiB (Q= 1, T= 1):  772.812 MB/s [  737.0 IOPS] < 1340.10 us>

 RND  4KiB (Q= 32, T= 1):  27.165 MB/s [  6632.1 IOPS] < 4772.07 us>

 RND  4KiB (Q= 1, T= 1):  12.556 MB/s [  3065.4 IOPS] <  323.72 us>

[Write]

 SEQ  1MiB (Q= 8, T= 1):  623.469 MB/s [  863.6 IOPS] < 13355.42 us>

 SEQ  1MiB (Q= 1, T= 1):  587.249 MB/s [  560.0 IOPS] < 1779.27 us>

 RND  4KiB (Q= 32, T= 1):  26.754 MB/s [  6531.7 IOPS] < 4873.26 us>

 RND  4KiB (Q= 1, T= 1):  12.238 MB/s [  2987.8 IOPS] <  332.70 us>

Profile: Default

  Test: 1 GiB (x5) [E: 0% (0/10GiB)]

  Mode: [Admin]

  Time: Measure 5 sec / Interval 5 sec 

  Date: 2022/11/12 9:49:28

   OS: Windows 10 [10.0 Build 19044] (x64)

Test 2

------------------------------------------------------------------------------

CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo

                 Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

 SEQ  1MiB (Q= 8, T= 1): 1187.575 MB/s [  1132.6 IOPS] < 7045.74 us>

 SEQ  1MiB (Q= 1, T= 1):  979.512 MB/s [  934.1 IOPS] < 1068.04 us>

 RND  4KiB (Q= 32, T= 1):  43.130 MB/s [ 10529.8 IOPS] < 3026.76 us>

 RND  4KiB (Q= 1, T= 1):  15.131 MB/s [  3694.1 IOPS] <  269.47 us>

[Write]

 SEQ  1MiB (Q= 8, T= 1):  948.589 MB/s [  904.6 IOPS] < 8795.39 us>

 SEQ  1MiB (Q= 1, T= 1):  741.115 MB/s [  706.8 IOPS] < 1410.36 us>

 RND  4KiB (Q= 32, T= 1):  41.559 MB/s [ 10146.2 IOPS] < 3147.13 us>

 RND  4KiB (Q= 1, T= 1):  14.049 MB/s [  3429.9 IOPS] <  290.25 us>

Profile: Default

  Test: 1 GiB (x5) [F: 0% (0/10GiB)]

  Mode: [Admin]

  Time: Measure 5 sec / Interval 5 sec 

  Date: 2022/11/12 9:56:28

   OS: Windows 10 [10.0 Build 19044] (x64)

Test 3

------------------------------------------------------------------------------

CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo

                 Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

 SEQ  1MiB (Q= 8, T= 1): 1173.777 MB/s [  1119.4 IOPS] < 7127.42 us>

 SEQ  1MiB (Q= 1, T= 1):  943.628 MB/s [  899.9 IOPS] < 1109.75 us>

 RND  4KiB (Q= 32, T= 1):  91.363 MB/s [ 22305.4 IOPS] < 1432.27 us>

 RND  4KiB (Q= 1, T= 1):  15.026 MB/s [  3668.5 IOPS] <  271.33 us>

[Write]

 SEQ  1MiB (Q= 8, T= 1):  871.642 MB/s [  831.3 IOPS] < 9541.46 us>

 SEQ  1MiB (Q= 1, T= 1):  706.031 MB/s [  673.3 IOPS] < 1481.82 us>

 RND  4KiB (Q= 32, T= 1):  76.721 MB/s [ 18730.7 IOPS] < 1706.21 us>

 RND  4KiB (Q= 1, T= 1):  13.458 MB/s [  3285.6 IOPS] <  302.84 us>

Profile: Default

  Test: 1 GiB (x5) [G: 0% (0/10GiB)]

  Mode: [Admin]

  Time: Measure 5 sec / Interval 5 sec 

  Date: 2022/11/12 10:02:15

   OS: Windows 10 [10.0 Build 19044] (x64)

Test 4

------------------------------------------------------------------------------

CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo

                 Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

 SEQ  1MiB (Q= 8, T= 1): 1389.187 MB/s [  1324.8 IOPS] < 6023.65 us>

 SEQ  1MiB (Q= 1, T= 1): 1025.644 MB/s [  978.1 IOPS] < 1019.83 us>

 RND  4KiB (Q= 32, T= 1):  57.067 MB/s [ 13932.4 IOPS] < 2293.40 us>

 RND  4KiB (Q= 1, T= 1):  14.757 MB/s [  3602.8 IOPS] <  276.27 us>

[Write]

 SEQ  1MiB (Q= 8, T= 1):  972.378 MB/s [  927.3 IOPS] < 8576.60 us>

 SEQ  1MiB (Q= 1, T= 1):  739.312 MB/s [  705.1 IOPS] < 1404.63 us>

 RND  4KiB (Q= 32, T= 1):  51.024 MB/s [ 12457.0 IOPS] < 2562.60 us>

 RND  4KiB (Q= 1, T= 1):  13.405 MB/s [  3272.7 IOPS] <  304.23 us>

Profile: Default

  Test: 1 GiB (x5) [H: 0% (0/10GiB)]

  Mode: [Admin]

  Time: Measure 5 sec / Interval 5 sec 

  Date: 2022/11/12 10:07:46

   OS: Windows 10 [10.0 Build 19044] (x64)

Test 5

------------------------------------------------------------------------------

CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo

                 Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

 SEQ  1MiB (Q= 8, T= 1):  828.972 MB/s [  790.6 IOPS] < 10087.18 us>

 SEQ  1MiB (Q= 1, T= 1):  604.462 MB/s [  576.5 IOPS] < 1731.47 us>

 RND  4KiB (Q= 32, T= 1):  24.823 MB/s [  6060.3 IOPS] < 5071.45 us>

 RND  4KiB (Q= 1, T= 1):  15.008 MB/s [  3664.1 IOPS] <  271.71 us>

[Write]

 SEQ  1MiB (Q= 8, T= 1):  621.090 MB/s [  592.3 IOPS] < 13404.82 us>

 SEQ  1MiB (Q= 1, T= 1):  527.068 MB/s [  502.7 IOPS] < 1985.00 us>

 RND  4KiB (Q= 32, T= 1):  28.394 MB/s [  6932.1 IOPS] < 4582.36 us>

 RND  4KiB (Q= 1, T= 1):  13.924 MB/s [  3399.4 IOPS] <  292.80 us>

Profile: Default

  Test: 8 GiB (x5) [E: 0% (0/10GiB)]

  Mode: [Admin]

  Time: Measure 5 sec / Interval 5 sec 

  Date: 2022/11/12 10:13:38

   OS: Windows 10 [10.0 Build 19044] (x64)

Test 6

------------------------------------------------------------------------------

CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo

                 Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

 SEQ  1MiB (Q= 8, T= 1): 1196.382 MB/s [  1141.0 IOPS] < 6996.01 us>

 SEQ  1MiB (Q= 1, T= 1):  980.639 MB/s [  935.2 IOPS] < 1066.31 us>

 RND  4KiB (Q= 32, T= 1):  47.117 MB/s [ 11503.2 IOPS] < 2778.45 us>

 RND  4KiB (Q= 1, T= 1):  14.748 MB/s [  3600.6 IOPS] <  276.46 us>

[Write]

 SEQ  1MiB (Q= 8, T= 1):  950.533 MB/s [  906.5 IOPS] < 8762.75 us>

 SEQ  1MiB (Q= 1, T= 1):  765.892 MB/s [  730.4 IOPS] < 1365.64 us>

 RND  4KiB (Q= 32, T= 1):  44.457 MB/s [ 10853.8 IOPS] < 2941.94 us>

 RND  4KiB (Q= 1, T= 1):  13.227 MB/s [  3229.2 IOPS] <  308.33 us>

Profile: Default

  Test: 8 GiB (x5) [F: 0% (0/10GiB)]

  Mode: [Admin]

  Time: Measure 5 sec / Interval 5 sec 

  Date: 2022/11/12 10:20:06

   OS: Windows 10 [10.0 Build 19044] (x64)

Test 7

------------------------------------------------------------------------------

CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo

                 Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

 SEQ  1MiB (Q= 8, T= 1): 1128.005 MB/s [  1075.7 IOPS] < 7416.45 us>

 SEQ  1MiB (Q= 1, T= 1):  936.863 MB/s [  893.2 IOPS] < 1116.44 us>

 RND  4KiB (Q= 32, T= 1):  90.878 MB/s [ 22187.0 IOPS] < 1439.75 us>

 RND  4KiB (Q= 1, T= 1):  14.935 MB/s [  3646.2 IOPS] <  272.96 us>

[Write]

 SEQ  1MiB (Q= 8, T= 1):  864.428 MB/s [  824.4 IOPS] < 9638.63 us>

 SEQ  1MiB (Q= 1, T= 1):  725.691 MB/s [  692.1 IOPS] < 1441.28 us>

 RND  4KiB (Q= 32, T= 1):  74.691 MB/s [ 18235.1 IOPS] < 1747.29 us>

 RND  4KiB (Q= 1, T= 1):  14.157 MB/s [  3456.3 IOPS] <  288.04 us>

Profile: Default

  Test: 8 GiB (x5) [G: 0% (0/10GiB)]

  Mode: [Admin]

  Time: Measure 5 sec / Interval 5 sec 

  Date: 2022/11/12 10:27:54

   OS: Windows 10 [10.0 Build 19044] (x64)

Test 8

------------------------------------------------------------------------------

CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo

                 Crystal Dew World: https://crystalmark.info/

------------------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes

[Read]

 SEQ  1MiB (Q= 8, T= 1): 1436.921 MB/s [  1370.4 IOPS] < 5824.36 us>

 SEQ  1MiB (Q= 1, T= 1): 1036.204 MB/s [  988.2 IOPS] < 1009.45 us>

 RND  4KiB (Q= 32, T= 1):  57.503 MB/s [ 14038.8 IOPS] < 2275.57 us>

 RND  4KiB (Q= 1, T= 1):  14.413 MB/s [  3518.8 IOPS] <  280.81 us>

[Write]

 SEQ  1MiB (Q= 8, T= 1): 1041.296 MB/s [  993.1 IOPS] < 8016.43 us>

 SEQ  1MiB (Q= 1, T= 1):  808.892 MB/s [  771.4 IOPS] < 1292.75 us>

 RND  4KiB (Q= 32, T= 1):  45.991 MB/s [ 11228.3 IOPS] < 2845.20 us>

 RND  4KiB (Q= 1, T= 1):  13.164 MB/s [  3213.9 IOPS] <  309.88 us>

Profile: Default

  Test: 8 GiB (x5) [H: 80% (8/10GiB)]

  Mode: [Admin]

  Time: Measure 5 sec / Interval 5 sec 

  Date: 2022/11/12 10:42:08

   OS: Windows 10 [10.0 Build 19044] (x64)

Conclusion

The results presented in this article are by no means the upper limit for the I/O operations achievable in any other environment. The intent is to show how people can test their vSphere infrastructure to discover settings for achieving best performance in their datacenters/production. Different types of virtual controllers will fit to different types of workload (file sharing/streaming/databases/…)

Additionally many other aspects of the VM, esxi and hardware can impact such tests (for example numbers of vCPU, vRAM in VM as well as things like queue parameters etc.)

Be aware my results are still a little bit biased because esxi was running in nested virtualization. Be aware that esxi version is 8.0 (first official release are for brave ones 😉). 

However we can see that SATA interface is not good choice for disk controller. It can be used for CD-ROM – in other scenarios using it will be just a waste. We can also see that NVMe and pvSCSI are “main two players” (NVMe for throughput and pvSCSI for iops). 

From my experience (from productions at customers sites) I can say that in general pvSCSI and NVMe adapter offers a reduction in CPU utilization and in most cases increased throughput compared to the default virtual storage adapters.

Much more reliable test would be with physical esxi servers (I don’t have such right now) like here :

Other interesting references you can look at :

Performance Best Practices for VMware vSphere 7.0 :

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi-vcenter-server-70-performance-best-practices.pdf