VirtualBox Nested Virtualization with VMware ESXi

Oracle VirtualBox is an extremely popular enthusiast hypervisor that many use to learn various technologies and get their feet wet using virtualization technologies. VirtualBox is a freely available hypervisor that provides many great options, including a broad range of compatible host operating systems and the ability to run many different guest operating systems. Let’s take a look at VirtualBox nested virtualization with VMware ESXi. Nested virtualization with VirtualBox can go either way. You can use either nested VirtualBox inside of ESXi or ESXi inside VirtualBox. Or, what about both? First, let’s take a look at how to enable nested virtualization in VirtualBox.

Enabling Nested Virtualization in VirtualBox

If you are not as familiar with Oracle VirtualBox, especially if you have mainly had your hands on VMware vSphere or Microsoft Hyper-V, you may not be as familiar with or have ever configured nested virtualization inside of VirtualBox.

With VirtualBox, you can enable nested virtualization using the VirtualBox GUI or using the command-line. For the majority who may be using Intel processors, you will need to enable nested virtualization settings from the VirtualBox command-line.

There is a special tool included with VirtualBox that allows enabling the nested virtualization flag. This is the vboxmanage utility. Using vboxmanage, you can easily enable nested virtualization on your VirtualBox VM.

Below, we are using the vboxmanage list vm command to list out the VMs that are running inside VirtualBox. Then, you use the command:

vboxmanage modifyvm "<your vm name>" --nested-hw-virt on

Using the virtualbox command line to enable nested virtualization

As mentioned earlier, you can’t enable nested virtualization for Intel CPUs using the GUI. As you can see below, the option is greyed out if you are running an Intel CPU.

Using the virtualbox gui to enable nested virtualization

Now that nested virtualization is enabled for your VirtualBox VM, you can run a nested operating system that is virtualizing workloads underneath.

VirtualBox Nested VMware ESXi host

Since we have nested virtualization in VirtualBox fresh on the mind, let’s look at nesting VMware ESXi inside a VirtualBox VM. The first thing we need to do is create a new VirtualBox virtual machine for the ESXi host. Use the version of Other Linux (64-bit) for your ESXi host.

Create a new esxi virtual machine in virtualbox

Using the same process as mentioned above for enabling nested virtualization, you will want to set the nested-hw-virt flag on the newly created ESXi host.

Enabling nested virtualization for the esxi host in virtualbox

Upload your VMware vSphere ESXi media and load up the ESXi host in VirtualBox.

Nesting VirtualBox inside a VMware ESXi Virtual Machine

Let’s now consider the other way around. How do you nest your VirtualBox installation inside a VMware ESXi virtual machine? This is fairly straightforward and requires the same thing from the VMware ESXi side – hardware virtualization for the CPU needs to be exposed to the guest operating system.

So, the virtual machine in VMware ESXi that you are using to house your VirtualBox installation will need to have the nested virtualization setting configured for the CPU. Under the Virtual Hardware configuration for the VirtualBox VM, CPU configuration, place a check in the box:

• Expose hardware assisted virtualization to the guest OS

Configuring the hardware virtualization setting for virtualbox nested virtualization inside vmware esxi

What happens if you don’t configure the hardware virtualization setting on the properties of the VirtualBox virtual machine? Unlike installing Hyper-V or nested ESXi inside VMware ESXi, you won’t receive an error during the installation of VirtualBox. The installation will behave the same as a virtual machine that has the setting enabled.

However, you will see a couple of odd things when you forget to enable hardware virtualization. When you go to create a virtual machine on a nested ESXi VirtualBox installation without the hardware virtualization flag configured, you will only see the options for 32-bit operating systems, and no Windows Server operating systems in the Create Virtual Machine.

Only 32 bit operating systems are displayed and no server operating systems

Also, you will see an Invalid settings detected message on the System Settings screen for the CPU in VirtualBox. The note above the message details the hardware virtualization is enabled in the cceleration section of the System page although it is not supported by the host system. This is the tell tale sign that hardware virtualization was not enabled in VMware vSphere for the VirtualBox VM.

Invalid system settings error displayed in the virtualbox processor settings

Needless to say, you will want to have hardware virtualization enabled. Let’s look at creating an ESXi VM in VirtualBox nested inside a VMware vSphere ESXi Windows 10 VM. Create a Other Linux (64-bit) VM. Under Storage Click the little Plus sign to add an optical drive. If you already have an optical drive listed, you can select to add optical media.

Adding an optical drive to the virtualbox vm

Select Optical Drive.

Choose optical drive for adding the esxi iso

Browse your computer and select the ESXi ISO file you downloaded from VMware.

Choose and select your esxi iso file you downloaded

After selecting the ESXi ISO file, you should be ready to power on and start loading ESXi in your VirtualBox VM.

Powering on and booting your esxi vm in virtualbox

Why would you nest VirtualBox inside ESXi?

The obvious question may be why would you do this? Well, the short answer is to play around with doing cool things! However, there are some real world use cases I can think of, especially if you already have an ESXi lab. You may want to play around with VirtualBox and want to do this in a nested environment that you can easily setup in your homelab. There are probably others such as setting up networking labs and other types of interesting uses.

• Check out Virtual Box here: Oracle VM VirtualBox
• Check out VMware ESXi free here: Download VMware vSphere Hypervisor for Free