A Nutanix cluster can persist a replicated write across two nodes in around 250 uSec which is critical for single-threaded DB write workloads. The performance compares very well with hosted cloud database instances using the same class of processor (db.r5.4xlarge in the figure below). The metrics below are for SQL insert transactions not the underlying IO.
TL;DR Comparison of Paravirtual SCSI Vs Emulated SCSI in with measurements. PVSCSI gives measurably better response times at high load.
During a performance debugging session, I noticed that the response time on two of the SCSI devices was much higher than the others (Linux host under vmware ESX). The difference was unexpected since all the devices were part of the same stripe doing a uniform synthetic workload.
The immediate observation is that queue length is higher, as is wait time. All these devices reside on the same back-end storage so I am looking for something else. When I traced back the devices it turned out that the “slow devices” were attached to LSI emulated controllers in ESX. Whereas the “fast devices” are attached to para-virtual controllers.
I was surprised to see how much difference using para virtual (PV) SCSI drivers made to the guest response time once IOPS started to ramp up. In these plots the y-axis is iostat “await” time. The x-axis is time (each point is a 3 second average).
PVSCSI = Gey Dots
LSI Emulated SCSI = Red Dots
Lower is better.
Each plot is from a workload which uses a different offered IO rate. The offered rates are 8000,9000 and 10,000 the storage is able to meet the rates even though latency increases because there is a lot of outstanding IO. The workload is mixed read/write with bursts.
After converting sdh and sdi to PV SCSI the response time is again uniform across all devices.
