Windows Server 2019 VMware Snapshot Bug
I wanted to capture a few thoughts about an issue that has quickly received visibility today due to a late breaking article posted by The Register found here. This describes an issue that users are experiencing trying to take quiesced snapshots of Windows Server 2019 which makes use of VMware Tools. What are Quiesced Snapshots? Why are they important? How is the error produced in VMware vSphere with a Windows Server 2019 VM? Let’s take a look at Windows Server 2019 VMware Snapshot bug and look at the details of this issue.
What are Quiesced Snapshots and Why Should You Care?
When you create a snapshot, there are options that can chosen for the snapshot you are about to create. Two of the options that are especially important are found on the Take Snapshot dialog box. You can place a check by one or the other. The options are:
- Snapshot the virtual machine’s memory
- Quiesce guest file system (Needs VMware Tools installed)
The latter option is not the option that is selected by default. When you create a snapshot by default, the Snapshot the virtual machine’s memory is selected.
However, if you do not want to capture memory information which is more helpful in the case of a snapshot you want to use more for backup purposes (you want to make sure you know what you are doing with this as snapshots are NOT backups in themselves), you can use the Quiesce guest file system (Needs VMware Tools installed).
When the Quiesce guest file system option is used, VMware Tools is invoked to properly flush all the data that may exist in memory or pending I/O operations to disk so that when the snapshot of the disk is taken without the memory information, the data is in a consistent state.
This is critically important when you are dealing with applications that rely on consistency of transactions. A great example of this would be Microsoft SQL Server. If all the data that exists in memory or pending I/O operations is not flushed correctly to disk, then you will have issues with the consistency of your data which in basic terms means data corruption.
Windows Server 2019 VMware Snapshot Bug
To be honest, for me, before this article caught my attention today, I have not noticed the bug with Windows Server 2019. I rely on and use Windows Server 2019 in my home lab and use it frequently. I use snapshots on many of the Windows Server 2019 VMs as well.
However, I very rarely use the Quiesced snapshot that is referenced in this article. I decided to run through a few basic tests of the supposed Windows Server 2019 VMware snapshot bug and see what results I came up with.
First of all, I am running:
- ESXi hosts 6.7.0 14320388
- vCenter Server 6.7.0 14368073
- Using vSAN as primary datastore
One thing I noticed with vSAN snapshots that were taken as Quiesced snapshots, they succeeded, however, are showing upon looking at the snapshot as Quiesced No. Since it has been a while since I have used this option I need to go back and revisit this topic with vSAN. VMware vSAN does use a bit different technology with capturing snapshots.
I decided to test this with a Windows Server 2019 VM on a standard non-vSAN datastore and interestingly found that I received different results and those that support the article and recent issues noted in an official VMware KB found here.
After seeing the error in the vSphere Client, I navigated to the Application Log in the Event Viewer on the Windows Server 2019 VM and sure enough I see the An expected hidden volume arrival did not complete because this LUN was not detected error which is noted in the KB.
This is certainly an issue for those who may use the Quiesced snapshot with VMware vSphere and Windows Server 2019 as noted in the article.
If you are using a modern backup solution that interacts with their own technology with VSS this apparently is not an issue. Products like Veeam and others are noted to not be affected by this potential bug in the current releases of VMware vSphere.
The buzz right now over this issue is the workaround is simply turning off App Quiescence, however, the repercussions of this if you don’t understand what you are doing could potentially be corrupted data.
Final Thoughts for Now
The Windows Server 2019 VMware Snapshot Bug apparently is a “thing” as I wanted to verify in my own lab testing. It will be interesting to see what VMware will do to resolve the issue. Maybe it will be a new release of VMware Tools or something else completely. I will keep this post updated as I know more or do more testing with this issue in particular.
