Hot Add Troubleshooting (VMware)

Test to see if the disk of the virtual machine to be backed up can be manually HotAdded to the proxy

Performing the following steps will help determine if the infrastructure is present to allow the proxy to attach the disks of the virtual machine to be processed in HotAdd.

1.  Create a snapshot on the virtual machine to be processed by backup proxy (Source virtual machine to be backed up).

2.  Within a vSphere client, attach the base disk from the virtual machine in Step 1 to the backup proxy.

1. Edit the Druva backup proxy that will be processing the virtual machine from Step 1.
2. Click Add.
3. Choose “Hard Disk”.
4. Chose “Use an existing virtual disk”.
5. Navigate to the location of the base disk from the virtual machine in step 1.
6. Specify that the disk should be attached Independent - Nonpersistent.
7. Click Finish.

3.  If this task completes this means the infrastructure is present to allow the Proxy to HotAdd the virtual machines base disk.

4.  Detach the disk that was attached to the backup proxy.

1. Edit the Druva backup proxy from Step 2.
2. Highlight the attached disk.
3. Click Remove (DO NOT change the radio option).
4. Click Finish.

 5. Remove the snapshot created in Step 1.

Removing Stuck VMDK’s from the backup proxy

Sometimes the VMware API call to remove the VMDK is not received or did not complete properly. So the VMDKs are stuck to the backup proxy. In order to remove those disks perform the following steps.

Note: Kindly perform these steps only when there is no backup going on in the backup proxy.

1. Through vSphere, you should be able to right-click the Druva proxy virtual machine and select edit settings. Look for the Hard disks attached to the virtual machine. Normally the stuck disks will be listed at the bottom. You should also be able to identify the disks by the name and the datastore they are in under the disk file box (top right).

2. To remove them, select the disk by clicking on it and select remove at the top right. Under removal options choose “remove from virtual machine”. DO NOT choose the option that deletes it from disk (the second option).

Original restore fails

• Restore fails, if the vRDM disk is converted to independent vRDM or pRDM.
  
  Resolution

Revert from the independent vRDM disk/pRDM to vRDM disk. Then try original restore.

• Restore fails, if the new uuid does not match to old uuid
  
  Resolution

If the uuid of the new created vmdk could match to the old uuid restore will fail. 

Backup fails of virtual machine having vRDM disks

If the same lun is mapped to two different disks, the backup fails.

Resolution

Unmap the lun from both disks and then map to only one of the disks.

Restore fails on a standalone disk

Restore fails on the standalone disk with the following error message:

[2015-08-21 16:15:54,708] [ERROR] (vim.fault.FileAlreadyExists) {
  dynamicType = <unset>,
  dynamicProperty = (vmodl.DynamicProperty) [],
  msg = 'Cannot complete the operation because the file or folder /vmfs/volumes/5575befd-d1e64fa5-f919-00224db1d2b5/Test-rhel-vm-saranya_5/Test-rhel-vm-saranya_5.vmx already exists',
  faultCause = <unset>,
  faultMessage = (vmodl.LocalizableMessage) [],
  file = '/vmfs/volumes/5575befd-d1e64fa5-f919-00224db1d2b5/Test-rhel-vm-saranya_5/Test-rhel-vm-saranya_5.vmx'
}

Resolution

Rename the folder or vmx file that creates the conflict. Restore work fine after this.

Virtual machine with EFI firmware cannot boot after restore

Perform the following steps if a virtual machine is configured with EFI firmware, and cannot boot after restore.  For more information on EFI, see Using EFI/UEFI firmware in a VMware Virtual Machine.

Note: The following steps are applicable if the backup proxy version 4.6.4 or earlier.

Resolution

1. Log into vSphere. 
2. Select the virtual machine that is unable to boot. 
3. Right-click the virtual machine and select Power> Power Off.
4. On the Confirm Power Off screen, click Yes.
5. Right-click the virtual machine and select Edit Settings, the Edit Settings window is displayed.
6. Under the Options tab, select EFI option.
   
7. Click OK.
8. Right-click the virtual machine and select Power> Power On.
   The virtual machine should boot automatically.

If the virtual machine does not boot, perform the following steps:

1. Right-click the virtual machine and select Edit Settings, the Edit Settings window is displayed.
2. Under the Options tab, select The next time the virtual machine boots, force entry into the EFI setup screen option.
   
3. Click OK.
4. Right-click the virtual machine and select Power> Reset.
5. Right-click the virtual machine and select Open Console. The Boot Maintenance Manager is displayed.
6. Select Enter setup.
7. Select Boot from file and then on the next screen highlight the boot partition and press Enter.
8. Select the EFI > <Operating system> > <.efi file>.
   The virtual machine will boot automatically.

Virtual machine not visible in Phoenix UI after configuring backup proxy. 

The issue occurs if the descriptor file for the virtual machine gets corrupted. The following error is received in backup proxy logs.

One of the virtual machine's VMDK have none as UUID. Skipping VM:ARX_FR_DC01 from listing.  

Resolution

Contact VMWare to fix the issue.

See Backup software shows error: Disk does not have a UUID (1033370).

Volume mounting failed.

This error occurs when some volumes are not mounted during File Level Restore.

Resolution

• Ensure that the Partition ID of a volume is consistent with the actual partition type of the volume. If it is not consistent, then the volume will not get mounted.
• If the error still persists, contact Druva Support.

Backup proxy upgrade has failed.

This error can occur due to the following reasons:

1. If you have installed a hotfix on your existing Phoenix version, you cannot automatically upgrade to the next version from the Phoenix user interface.
2. If the server that hosts the backup proxy restarts during the upgrade.
3. If the old version is not present on client machine then upgrade rollback will fail.

Resolution

Manually upgrade the backup proxy.

Backup proxy is unable to connect to the Phoenix cloud.

This can happen if both network interface cards are enabled, and:

• You use one network interface card for public network and the other network interface card for private network
• The network interface card configured for private network uses DHCP

The backup proxy fails to connect to Phoenix cloud because the DHCP changes routing table sequence. 

Resolution

You can use the following command to manually add a static route entry for the public network in the backup proxy settings. 

nmcli connection modify <connection_name> +ipv4.routes "<ip_range>/<subnet_prefix> <gateway>"

For example:
nmcli connection modify eth0 +ipv4.routes "192.168.122.0/24 10.10.10.1"

Virtual machine does not boot after restore

This can happen if:

• A floppy disk was inserted in the virtual machine at the time of backup
• User triggers a restore to original location
• The virtual machine on the original location has the floppy inserted

If the restored virtual machine has the floppy disk inserted, the restored virtual machine attempts to boot from the floppy disk and fails. 

Resolution

After restoring the virtual machine:

• Remove the floppy disk before starting the virtual machine
• Change the virtual machine boot order

VMware backup proxy deployment fails for standalone ESXi 6.5 using web client.

This happens due to a VMware issue. For more information, see Unable to deploy OVF from ESXi 6.x Host UI Client (51497).

Resolution

Perform the following steps to resolve the problem.

1. Download the OVF tool From the VMware utility page [External Link].
2. Install the tool on a computer that you can use to deploy the OVA.
   For detailed steps to install the OVF tool, see the VMware doc.
3. To deploy the OVA using the OVF tool:
   • For Windows
     On the terminal, enter the following command: 

     1	ovatool.exe  -ds=datastore_name --diskmode=thin/thick --name=name_of_new_backup_proxy path_to_ova_template vi://user@IP_of_ESX_host

     
     For example:

     1	"C:\Program Files\VMware\VMware OVF Tool\ovftool.exe" -ds="datastore1" --diskMode=thin --name="EXAMPLE_BACKUPRPOXY_WINDOWS_OVA_TOOL"  \\nas01\builds\CertQA\CloudPushBuilds\CP74\Druva_Phoenix_BackupProxy_standalone_4.6.12.ova  vi://root@192.0.2.9

   • For Linux
     On the terminal, enter the following command: 

     1	ovatool  -ds=datastore_name --diskmode=thin/thick --name=name_of_new_backup_proxy path_to_ova_template vi://user@IP_of_ESX_host

     
     For example:

     1	ovftool --acceptAllEulas -ds="datastore1" --diskMode=thin --name="EXAMPLE_BACKUPROXY_LINUX_OVA_TOOL_2" Druva_Phoenix_BackupProxy_standalone_4.6.12.ova  vi://root@192.0.2.9

4. After the OVA is deployed, accept Druva end-user license agreement. In the command line window:
   • Enter Yes to accept SSL fingerprint for ESX host.
   • Enter the password for the ESX user.
   • Enter Yes to Accept Druva end-user license agreement..

Note: Ensure that you use OVF tool version 4.2.0.

Restore is blocked on Phoenix management console

At the time of restore, backup proxy checks if the administrator can access the destination datastore, folder, network, or resource pool.  If the administrator cannot access the resources, restore is disabled on Phoenix management console.

Resolution

Phoenix recommends that the backup proxy is activated by user credentials having required vCenter/ESXi permissions. The same user credentials should be used to activate all backup proxies.

vMotion is disabled on the vCenter/Standalone server 

Phoenix disables virtual machine vMotion at the time of backup and restore and re-enables it after backup is complete. If, due to some reason, a user restarts Phoenix service, the backup proxy fails to re-enable vMotion for that virtual machine, and it stays disabled.

Resolution

To enable vMotion if it is disabled on your server, you can use the vmcontrol utility. 

To use the vmcontrol utility 

1. From the vSphere Client console, click VMs and Templates, and start the backup proxy virtual machine. 
2. Open a terminal on the backup proxy. 

Note: Alternatively, you can use a terminal client such as PuTTY to access the backup proxy. 

3. Log on to the backup proxy. 

Note: The default username is root, and the default password is druvaphoenix. If you changed the password, use the password that you set. 

   4. To enable the vmotion, use the following command:

vmcontrol enable_vmotion <vm_name> [ -u instance_uuid ] [ -f esx_fqdn ]

Linux virtual machine boots into maintenance mode after restore 

If the disk sequence changes after restore, the boot order changes for the Linux virtual machine and the virtual machine boots into maintenance mode or some virtual disks are not visible. 

Resolution

For the Linux virtual machine, change the disk sequence in /etc/fstab.  The virtual machine boots without any problem after you enter the correct disk sequence. For more information, see RedHat documentation. [External link]

Network proxy settings on the backup proxy are unable to use the Kerberos authentication protocol

This issue occurs if the DNS server, KDC server, and the web proxy host fully qualified domain names (FQDNs) are not reachable.

Resolution

• It is possible that the backup proxy is unable to resolve the FQDNs. To resolve this issue, contact your IT team and update the network manager so that the DNS server, KDC server, and web proxy host FQDNs you provide resolves to the appropriate servers. If the issue persists, update the /etc/hosts file to provide the FQDNs and IP addresses of the DNS server, KDC server, and the web proxy host. 
• Ensure that the DNS credentials you have entered at the time of specifying the network proxy settings on the backup proxy are correct.

To verify the issue is resolved, after providing all the network proxy settings on the backup proxy, run the following command on the backup proxy virtual machine terminal to see if it is connected to the right DNS server.

1	# realm list

The KDC ticket is not generated even if correct credentials are provided

This issue can occur if there is a configuration problem with the web proxy. 

Resolution

If you are using WinGate, ensure that WinGate is properly configured. In addition, ensure that the KDC user is getting authenticated on the DNS server. 

If the issue persists, check with your IT administrators and ensure that the proxy server is up and running. 

Network proxy settings fail at the web proxy

This issue can occur if:

• The Kerberos user needs permissions on the web proxy. 
• You have selected socks4/socks5 proxy type at the time of registering the Phoenix backup proxy however the web proxy is not configured use it. 

Resolution

• Ensure that the Kerberos user you provide has the required permissions on the web proxy. 
• Ensure that the web proxy is configured to use the proxy type you select at the time of registering the Phoenix backup proxy.

Backup fails after a few backups were already successfully

This issue can occur if the web proxy authentication method is Kerberos and the Kerberos ticket service on the backup proxy is not running. 

Resolution

Check if the PhoenixSetKrbTicket service is running. If the service is not running, manually start the service. If it is running.

Web proxy setting fails for a user who is not an administrator on the DNS server 

This issue can occur if you choose Kerberos authentication method and provide a DNS server user who is not an administrator. 

Resolution

Ensure that the non administrator DNS user whose credentials are provided at the time of configuring the web proxy has the required permissions in the Active Directory. 

Log backups fail on a virtual machine with Microsoft SQL Server if its time is reset

This issue can occur if VMware Tools reset the time on a virtual machine.

Resolution 

To resolve this error, disable time synchronization. For more information on how to disable time-synchronization, see this article on the VMware knowledge base.