Configure vSAN Error Disk with VSAN uuid failed to appear in CMMDS

0

In configuring a 2-node VMware vSAN cluster with a Witness host I ran into a strange issue that took a bit of head scratching and Googling to figure out. However, the solution was actually fairly simple. In attempting to configure vSAN on the 2-node cluster, adding disks to the disk group on one of the hosts continued to fail with a peculiar error. Let’s take a look at the Configure VSAN Error Disk uuid failed to appear in CMMDS and see the simple resolution in my case.

Configure vSAN Error Disk with VSAN uuid failed to appear in CMMDS

As mentioned I was configuring a 2-node VMware vSAN cluster with a Witness host for home lab purposes.  One of the host’s would add its (2) local disks without issue (cache tier and capacity tier).  However, the other host would take quite a long while and finally error out with the error: Disk with VSAN uuid failed to appear in CMMDS as shown below:

Disk-with-VSAN-uuid-failed-to-appear-in-CMMDS Configure vSAN Error Disk with VSAN uuid failed to appear in CMMDS

Disk with VSAN uuid failed to appear in CMMDS

A little bit of background on the “just prior” events performed on the host in question.  The host had a 1 TB NVMe drive that would serve as a capacity tier for the local disk group for that server.  However, this drive had previously been used as a local DAS datastore.  I had unmounted and deleted the datastore prior to running the vSAN configuration on the cluster.

The disk group continued to fail with the error message shown in the screenshot.  Additionally in running a few command line queries to gather some more information, this is what was shown.  A helpful command in checking out the status of the individual disks in the vSAN disk group is the esxcli vsan storage list command.  The disk in question showed the following in the output of the command – In CMMDS: false which is not what we want to see.  Also, in the disk group window, the vSAN Health Status did not show anything except a line.

Disk-showing-is-used-by-CMMDS-false Configure vSAN Error Disk with VSAN uuid failed to appear in CMMDS

Disk showing In CMMDS false

I then found the following VMware KB article:  https://kb.vmware.com/s/article/2143296 which covers the topic, “After removing and re-inserting a disk associated with vSAN, the disk is not re-added to the vSAN disk group (2143296)”.  The part that grabbed my attention:


“When a disk used in a vSAN disk group is removed, open handles to the device are closed and device is unregistered from the ESXi SCSI subsystem. When the disk is reinserted, the ESXi SCSI subsystem re-enumerates the device and makes it available, but vSAN does not automatically reclaim the disk.”
This seemed to fit my situation of having removed the disk since it was unmounted and deleted.  The workaround as listed in the KB was fairly simple:
  1. Re-insert the vSAN disk.
  2. Refresh the host storage system in the vSphere Web Client.
  3. The vSAN host reclaims the device.

I went ahead and ran both the refresh and rescan options on the storage.

Refreshing-and-rescanning-the-host-storage-subsystem Configure vSAN Error Disk with VSAN uuid failed to appear in CMMDS

Refreshing and rescanning the host storage subsystem

After refreshing and rescanning, I reconfigured vSAN again to see if this made a difference with the disk group creation.

Adding-disks-to-vSAN-after-rescanning-the-host-storage Configure vSAN Error Disk with VSAN uuid failed to appear in CMMDS

Adding disks to vSAN after rescanning the host storage

After refreshing and rescanning, the vSAN disk group was created successfully.

The-disk-group-was-created-successfully-and-vSAN-health-status-was-healthy Configure vSAN Error Disk with VSAN uuid failed to appear in CMMDS

The disk group was created successfully and vSAN health status was healthy

Concluding Thoughts

The vSAN disk group creation process relies on the host having successfully enumerated the SCSI device in the storage subsystem.  I had removed the disk and simply attempted the vSAN configuration immediately afterwards and apparently the host had lost its association with the disk.  This resulted in Configure vSAN Error Disk with VSAN uuid failed to appear in CMMDS.  As the VMware KB noted, a quick refresh should resolve the issue.  To cover the bases, I did the refresh and a full rescan of storage.  Afterwards, the vSAN disk group created successfully.