I’m evaluating backup solutions for our environment, and honestly, I’m hitting walls with most of the tools we’ve tried so far.

The setup is fairly complex and diverse:

• Around 120 VMs split between VMware and Hyper-V, spread across multiple sites
  
  
• A handful of physical Linux and Windows servers still in production
  
  
• 70TB+ of Microsoft 365 data across Exchange, SharePoint, OneDrive, and Teams
  
  
• Hybrid storage: some local disk, NAS, cloud (S3-compatible), and SAN
  
  
• Proxmox and Nutanix used in R&D, with potential to expand
  
  
• Strong compliance requirements and need for immutable backup and granular recovery
  
  
• Limited resources for managing heavy infrastructure or additional proxies
  
  

Every solution I’ve tested either lacks proper platform coverage, becomes too resource-intensive, or fails to deliver flexible storage and full backup/recovery for everything we need. I’m trying to avoid stitching together 2–3 products just to make this work.

Has anyone here faced similar challenges and found one solution that can truly cover it all, across physical, virtual, cloud, and Microsoft 365, with reasonable resource demands?

Appreciate any real-world input!

Generate the download token

First, we need to generate the download token on the Broadcom support site. To do that, follow the steps below. To do that, after you login, navigate to the quick links section in your Dashboard and click the Generate Download Token button.

Next, pick your SiteID from the dropdown, and click the button Generate Token.

You will be asked Are you sure you want to Generate Token?

Once you confirm, you will see your download token generated.

Download the Depot Changer Script

Now, we need to download the Depot changer script that is provided by VMware. You can find that at this KB:

Authenticated Download Configuration Update Script

Running the script

Now that we have the token and the script, just extract the .zip file and you will have the folder containing the script and relevant sample JSON files that we will need to modify.

First, set your PowerCLI configuration to ignore certificate validation:

In the folder you downloaded, you will see several "sample" JSON files.

I am using a vcenter configuration and not VCF, so if you open the "sample-vcenters.json" file, you will see the example of what you need to configure/modify with your own vCenter Server name, username and password:

I modified the file name to just "vcenter.json" and only a single vCenter connection. You will use the command to feed in your JSON file:

VCF: ./VMwareDepotChange.ps1 -Connect -Endpoint Vcf -JsonInput <path to JSON file>

vCenter: ./VMwareDepotChange.ps1 -Connect -Endpoint vCenter -JsonInput <path to JSON file>

Now that we have connectivity established using the JSON file, we can start running commands like checking where our repository is pointed currently:

Check Depot
./VMwareDepotChange.ps1 -Check

Adding the Download Token

Now that we have a feel for the script, we can add the download token and have it reconfigure the URLs along with the token:

Update Depot
./VMwareDepotChange.ps1 -Update -DownloadToken <32 character token>

After running the update parameter, we can run the check parameter again to check and make sure it is now pointed to the dl.broadcom.com link:

Hopefully, this forum post helps any who are struggling with introducing the new method of authenticating the downloads for the new Broadcom download URL. You can read the official KB from Broadcom here:

VCF Authenticated downloads Configuration Update Instructions

What are vSphere Permissions?

What exactly are vSphere Permissions, and what do they allow you to do? They detail what a user can do in the vSphere environment. When you create permissions, these get assigned to roles. You can then assign these to users or groups.

VMware PowerCLI install

To work with permissions, roles, etc, you will need to install VMware.PowerCLI module. It is easy to install with the command:

Install-Module VMware.PowerCLI

We will use this module and associated cmdlets in the rest of the walkthrough below.

Connect to vCenter Server

First let’s connect to vCenter Server.

# Connect to vCenter Server
Connect-VIServer -Server vcenter-server-address -User username -Password password

List roles and vSphere permissions PowerCLI

Let’s list the available roles in vSphere using the cmdlet:

Get-VIRole

PowerCLI provides a command to list all available privileges in vCenter. This can be a helpful command for understanding privileges.

# List all available privileges
Get-VIPrivilege

Checking Specific User Permissions

To check the permissions assigned to a specific user, use:

# Check permissions for a specific user
Get-VIPermission -Principal "User1"

Assigning Privileges to Roles

Assigning privileges to roles is important. You can create a new role and assign specific privileges using PowerCLI. As always you don’t want to assign more privileges than needed. Below, we are assigning the permissions to power on VMs to a new role called “CustomRole“.

# Create a new role with specific privileges 
New-VIRole -Name "CustomRole" -Privilege (Get-VIPrivilege -Id "VirtualMachine.Interact.PowerOn")

Changing Existing Roles

To modify an existing role, adding or removing privileges, use the commands that follow:

# Add a privilege to an existing role
Set-VIRole -Role "CustomRole" -AddPrivilege (Get-VIPrivilege -Id "VirtualMachine.Interact.PowerOff")

# Remove a privilege from an existing role
Set-VIRole -Role "CustomRole" -RemovePrivilege (Get-VIPrivilege -Id "VirtualMachine.Interact.PowerOn")

Assigning and Propagating Permissions

Permissions link roles with users or groups. To create a permission and assign it in your environment:

# Assign a role to a user at a specific level
New-VIPermission -Entity (Get-VM -Name "VM1") -Principal "User1" -Role "CustomRole" -Propagate:$true

When assigning permissions, you will likely want to propagate them to child objects in the environment. Here’s how you do that:

# Assign and propagate permissions to child objects
New-VIPermission -Entity (Get-Cluster -Name "Cluster1") -Principal "User1" -Role "CustomRole" -Propagate:$true

Managing User and Group Permissions

To manage local users, groups, and policies in vSphere using PowerCLI, you need to leverage an open-source module called VMware.vSphere.SsoAdmin. Check out the official VMware blog here: New Open Source PowerCLI Module for managing vCenter Single Sign-On (SSO) – VMware PowerCLI Blog.

By default, if you just install the VMware.PowerCLI module, you can’t manage local SSO users and groups and policies. This SSOAdmin module includes cmdlets to manage Single Sign-On (SSO) users and groups within the vSphere environment.

To install the module, you can use the cmdlet:

Install-Module VMware.vSphere.SsoAdmin

Connecting to the SSOAdmin Server

This module has its own connect command that needs to be used in the form of:

connect-ssoadminserver -server vcsa.cloud.local -user administrator@vsphere.local -skipcertificatecheck

Creating a new SSO user

With the new module installed, we can create a new user in the SSO domain, using the cmdlet New-SsoPersonUser:

New-SsoPersonUser -User mytestadmin -Password 'MyStrongPa$$w0rd' -EmailAddress 'mytestadmin@vsphere.local' -FirstName 'My' -LastName 'Admin'

Creating a New Group

Also, you can create a new group using the New-SsoGroup cmdlet:

# Create a new group in the local SSO domain 
New-SsoGroup -Name 'myGroup' -Description 'My Group Description'

Assigning Users to Groups

Once you have created users and groups, you can assign users to groups using the Add-SsoGroupMember cmdlet:

#Get the group details
$administratorsGroup = Get-SsoGroup -Name 'Administrators' -Domain 'vsphere.local'

#Add the user to the group
Get-SsoPersonUser -Name 'TestUser' -Domain 'MyDomain' | Add-UserToSsoGroup -TargetGroup $administratorsGroup

Assigning Roles to Users and Groups

# Assign a role to a user 
Set-VIPermission -Entity (Get-Datacenter -Name "Datacenter1") -Principal "vsphere.local\newuser" -Role "CustomRole" 

# Assign a role to a group 
Set-VIPermission -Entity (Get-Datacenter -Name "Datacenter1") -Principal "vsphere.local\newgroup" -Role "CustomRole"

Roles with PowerCLI

Creating custom roles for your permissions needs:

# Create a new role with specific privileges
New-VIRole -Name "BackupAdmin" -Privilege (Get-VIPrivilege -Id "Datastore.AllocateSpace", "Datastore.Browse")

Deleting Roles

You can also delete roles this way:

# Delete an existing role
Remove-VIRole -Role "BackupAdmin"

Importing and Exporting Permissions

You can export permissions as well for backup, as an example:

# Export permissions to a file
Get-VIPermission | Export-Csv -Path "permissions.csv"

Importing Permissions

To import permissions from a file:

# Import permissions from a file
Import-Csv -Path "permissions.csv" | ForEach-Object {
    New-VIPermission -Entity (Get-View -Id $_.Entity) -Principal $_.Principal -Role $_.Role -Propagate:$_.Propagate
}

Install Dell System Update Utility

The steps to install the utility in VMware ESXi are fairly straightforward. It is a simple .ZIP file that you download from Dell and then use the command line esxcli software vib install command to install the package.

First, download the utility from Dell from the link here: DELL System Update (DSU) | Dell US.

Choose the package that matches your ESXi version. Once you download the .ZIP file, you do not have to extract it. The esxcli software vib install command will take the ZIP file as it is to install.

Once you have downloaded the .ZIP file, copy it to the /tmp directory on your target ESXi server.

The command you need to run to install the Dell system update utility in VMware ESXi is the following. Be sure to replace the file name with the filename you download from Dell.

esxcli software vib install -d /tmp/DEL-dell-system-update_2.0.2.0.820-DEL.800.0.0_21376842.zip

Finally, after installing it in VMware ESXi, just reboot your host:

reboot

Running the Dell System Update utility in VMware ESXi

Running the Dell System Update utility in VMware ESXi is very easy. You simply establish an SSH session and type the command:

dsu

When you type the dsu command it will automatically start scanning your system and running an inventory of hardware and possible updates.

Once the initial scan is complete, you just select the updates you want to apply. There are a few things to note:

• Typing “a” means you are accepting all available updates
• Typing the “number” at the front of the listed update means you are selecting that update specifically
• Updates that appear with a “-” in the box at the front means you have the latest version
• When you select an update, it will place a “*” in the box at the front

After you use one of the means to select updates, you need to confirm the updates you have selected by simply typing:

c

Here another look at confirming the updates.

Below, you can see the asterisk that is place in front of selected updates.

Error in method invocation Errno 1 Unknown host details

The error comes at the beginning of the Stage 2 process:

This is an error that ones see at the beginning of Stage 2 of the Upgrade process. When you click Continue on the screen shown above, it begins its first prerequisite check after the new VCSA appliance has been deployed, you will see the error pop up, that has the messaging: Error in method invocation Errno 1 Unknown host.

The cause of the issue

If you note closely below, instead of a hostname being used for the ESXi host or vCenter Server that manages the source vCenter Server, we see an IP address.

The upgrade process evidently has an issue with the use of an IP address here, even if the vCenter Server Installer and all appliances involved can connect to the ESXi or vCenter Server by IP address.

The resolution

Instead of using an IP address for the ESXi host or vCenter Server that manages the source vCenter Server, you need to use a hostname instead. If you are targeting an ESXi host that you typically don’t have in DNS, just create a temporary DNS record for the upgrade process and use that DNS name in the wizard instead of the IP address.

After entering in the FQDN/hostname of the ESXi host or vCenter Server that manages the appliance, you will be able to get past the error message. I am not sure what the requirement is here for FQDN vs IP address, especially if targeting the ESXi host directly.

However, VMware vCenter Server upgrades have historically been sensitive to DNS being correct and making sure you had the DNS records in place before running an install, etc. So, I am guessing this is more of the same underlying pickiness on DNS being setup correctly, even if connectivity is successful to an IP address.

**Note** Also, you do not have to start all the way back over. Instead, close out of the vCenter Server Installer wizard and then connect to the temporary IP address of the new appliance port 5480. When you do this, it will let you change the values of what you want to target to recontinue the wizard.

Hopefully this little quick tidbit will help any who may be struggling with the Error in method invocation Errno 1 Unknown host error message when upgrading their vCenter Server VCSA appliance to VCSA 8. The workaround is fairly simple in using the FQDN vs IP address. I just wish VMware would make this part of the preflight checks instead of just throwing a generic error message.

When I saw this issue

Just a quick background of when I saw this issue. I had a Kingston 1 TB NVMe drive that I have used on my home lab bench for quite some time for various testing, etc. I noticed that when I put it in my mini PC I wanted to do some testing with, I saw an issue where when I went to create a datastore in VMware vSphere, it saw the drive ok, but the VMFS datastore size was showing as 0, and there was no way to increase the size.

Here is a screenshot of what I saw in my environment when selecting the drive.

After selecting the drive that I wanted to use, the next screen, I chose VMFS 6.

And for the issue. Under the partition configuration there was nothing available in the dropdown. Also, the slider bar, even though it looks active, you can’t move it over to increase the size of the datastore.

What caused the issue?

I am not 100% sure, but I had recently tested with the NVMe drive in a RAID configuration on the MS-A1 that I have been using in a lot of testing recently over the past week or so. After creating the array with the NVMe disk, and then destroying the array created. I assumed the NVMe drive was ok to use immediately following the RAID array creation.

Let’s look at the resolution to the NVMe drive showing the 0 size. It is actually pretty simple actually, but not necessarily intuitive or obvious.

Resolution for the NVMe drive showing 0 size

As is the case with many issues that we may need to solve in the datacenter or in the home lab, I had to resort to the command line to resolve the issue.

To find your NVMe disk identifier you need to run the command:

esxcli storage core path list

Copy the identifier that is listed for the Device.

Now that you have the device identifier for the NVMe drive that is showing up as a 0 size, we need to initialize the disk with a GPT (GUID Partition Table) partition table.

partedUtil mklabel /vmfs/devices/disks/t10.NVMe____KINGSTON_O
M8PGP41024Q2DA0________________8595183038B72600 gpt

Now, after we run the command to initialize the disk with the GPT partition table, if we go back through the wizard, the NVMe disk will show the proper size:

You can find the Broadcom KB here: Privileges Required for Configuring a vCenter Adapter Instance.

Creating a custom role with these permissions by hand can be cumbersome. Below is a script to create the role with the permissions needed for VMware Aria Operations.

Hopefully, this will allow you to shortcut the process of creating permissions manually:

# Variables
$vcenterServer = "vcenter.yourdomain.com" # Replace with your vCenter server
$roleName = "vROps Role" # Replace with the desired role name

# Permissions list (from VMware documentation)
$permissions = @(
    "Global.Diagnostics",
    "Global.Licenses",
    "Global.Settings",
    "Host.Config.Network",
    "Host.Config.Storage",
    "Host.Inventory.AddHostToCluster",
    "Host.Inventory.AddStandaloneHost",
    "Host.Inventory.CreateCluster",
    "Host.Inventory.DeleteHost",
    "Host.Inventory.EditCluster",
    "Network.Assign",
    "Resource.AssignVMToPool",
    "Resource.HotMigrate",
    "Resource.ModifyQuota",
    "Resource.Move",
    "Resource.Rename",
    "System.Anonymous",
    "System.Read",
    "System.View",
    "System.Write",
    "Task.Create",
    "Task.Update",
    "VirtualMachine.Interact.PowerOff",
    "VirtualMachine.Interact.PowerOn",
    "VirtualMachine.Inventory.Create",
    "VirtualMachine.Inventory.CreateFromExisting",
    "VirtualMachine.Inventory.Delete",
    "VirtualMachine.Inventory.Register",
    "VirtualMachine.Inventory.Unregister",
    "VirtualMachine.Provisioning.Clone",
    "VirtualMachine.Provisioning.CloneTemplate",
    "VirtualMachine.Provisioning.Customize",
    "VirtualMachine.State.Reset",
    "VirtualMachine.State.Suspend"
)

# Connect to vCenter
Connect-VIServer -Server $vcenterServer

# Check if the role already exists
$role = Get-VIRole | Where-Object { $_.Name -eq $roleName }
if (-not $role) {
    # Create the role with the specified privileges
    Write-Host "Creating role: $roleName"
    $role = New-VIRole -Name $roleName -Privilege (Get-VIPrivilege | Where-Object { $_.Id -in $permissions })
} else {
    Write-Host "Role $roleName already exists"
}

Write-Host "Script execution completed! The role '$roleName' has been created or verified. Assign the role manually to the desired user."

Once you have created the role and name it what you want, you can assign this role to your user in vCenter Server permissions.

This can be an irritating byproduct of updating VMware Workstation Pro. However, there is a simple and quick fix for this. If you have updated VMware Workstation and it is now opening all your console sessions in VMware Workstation instead of the VMRC, run the following two commands in PowerShell:

Set-Item HKLM:\SOFTWARE\Classes\vmrc\DefaultIcon -Value '"C:\Program Files (x86)\VMware\VMware Remote Console\vmrc.exe",0'

Set-Item HKLM:\SOFTWARE\Classes\vmrc\shell\open\command -Value '"C:\Program Files (x86)\VMware\VMware Remote Console\vmrc.exe" "%1"'

No reboot is required after running the commands. Simply run the commands in Powershell and your consoles will immediately be returned back to opening in VMRC.

Hopefully this quick tip will help!

Stop your vCenter UI service:

service-control --stop vsphere-ui

Then open the file located at /etc/vmware/vsphere-ui/webclient.properties and look for this line. Uncomment it and place your alias after the equals sign. Then save it.

sso.serviceprovider.alias.whitelist=<your fqdn or shor hostname>

After saving the file, start your vsphere-ui service back up:

service-control --start vsphere-ui

Make sure you have your DNS configured to point this new hostname to your vCenter IP address or a proxy that is sitting in front of it that is handling SSL termination.

Troubleshooting

If the service doesn't start, make sure vsphere-ui is the owner of the webclient.properties file. As a side note, I was able to put vCenter Server behind a Traefik proxy for SSL certificates in front of vCenter using this method.

chage -l root

Then we can set it to not expire using the following command:

chage -I -1 -m 0 -M 99999 -E -1 root

That's it, now you can enjoy not having to reset your root password for your VCSA appliance again.