Install a VMWare ESXi 6.0 Hypervisor in a Hyper-V VM

Recently I’ve been playing around with the new Hyper-V Nested Virtualization feature within Windows 10 (build 10565 and greater) and Windows Server 2016. It is pretty cool to be able to create virtualized lab environments running that contain Hyper-V clusters. But what if we want a lab that contains VMWare ESXi Hypervisors running on Hyper-V host. I couldn’t find the process documented anywhere and I couldn’t even confirm if it should be possible. But after lots of asking a lot of annoying questions – thanks Adam Burns – Googling and hair pulling I managed to get it going:

ss_vmwareinhv_proof

So this seems like a good topic for a blog post.

What You’ll Need

You are going to need a few things to get this working:

  • A Hyper-V host running on Windows 10 (built 10565 or greater) or Windows Server 2016 TP4.
  • Enable-NestedVM.ps1 – A PowerShell script for enabling Nested Virtualization in a Hyper-V VM.
  • A VMWare account – just sign up for one here if you don’t already have one.
  • VMWare PowerShell CLI installed – I used 6.3 release 1 that I downloaded from here.
  • ESXi-Customizer-PS.ps1 – A PowerShell script for injecting network drivers into an ESXi 5.x/6.x ISO. I downloaded it from here.

I suggest you download all of the above items to a working folder – I called mine d:\ESX-In-Hyper-V, so these instructions will reflect that, but you can call your folder what ever you like.

You should end up with a folder containing these files:

ss_vmwareinhv_neededfiles

And before you ask: No, you don’t need an VMWare ESXi 6.0 ISO – this will get downloaded and produced for us.

The Process

Part 1 – Prepare an ESXi 6.0 ISO with Network Drivers

The biggest problem I ran into when trying to install ESXi onto Hyper-V was that the ESXi kernel doesn’t come with drivers for the Microsoft Virtual Network Adapter or the Microsoft Legacy Network Adapter (emulates a DECchip 21140). So you’ll need to inject these drivers into the VMWare ESXi 6.0 ISO. Luckily there is a script available and the appropriate drivers DECchip 21140 (called “net-tulip” for some reason) that makes this process a breeze:

  1. Install WMWare PowerCLI.
  2. Open a PowerShell console.
  3. Enter the following commands:
    CD D:\ESX-In-Hyper-V\
    .\ESXi-Customizer-PS-v2.4.ps1 -v60 -vft -load net-tulip
    
  4. After a few minutes the VMWare ESXi 6.0 ISO will be downloaded and the “net-tulip” drivers merged with it:

ss_vmwareinhv_createesxiiso

The ISO will now be available in the D:\ESX-In-Hyper-V folder:

ss_vmwareinhv_neededfilesandiso

Part 2 – Create the Hyper-V VM

  1. In Hyper-V Manager create a new Virtual Machine:ss_vmwareinhv_newvmpath
  2. Click Next.
  3. Select Generation 1 and click Next.
  4. Set the Startup Memory to at least 4096MB.
  5. Uncheck Use Dynamic Memory for this Virtual Machine:ss_vmwareinhv_newvmmemory
  6. Click Next.
  7. Don’t bother to Configure Networking on the next step – just click Next.
  8. Select Create a new virtual hard disk and set the Size to 10GB (this is just going to be the boot disk for the ESXi Hypervisor):ss_vmwareinhv_newvmdisk
  9. Click Next.
  10. Select Install an operating system from a bootable CD/DVD-ROM.
  11. Select Image file (.iso) and browse to the ISO created in Part 1.ss_vmwareinhv_newvminstallation
  12. Click Next then click Finish to create the Virtual Machine:ss_vmwareinhv_newvm
  13. Right click the new Virtual Machine and select Settings.
  14. Select the Processor node and increase the Number of Virtual Processors to at least 2:ss_vmwareinhv_vmsettings_processor
  15. Select the existing Network Adapter node and click Remove:ss_vmwareinhv_vmsettings_removenetwork
  16. Select the Add Hardware node and select Legacy Network Adapter:ss_vmwareinhv_vmsettings_addnetwork
  17. Click Add.ss_vmwareinhv_vmsettings_addlegacy
  18. Select a Virtual Switch to connect the ESXi Host to.
  19. Click OK.

The Virtual Machine is almost ready to start up, but there is one more thing to do.

Part 3 – Enable Nested Virtualization

Before the starting up the Virtual Machine we need to enable Nested Virtualization Extensions on it. This is done by running a PowerShell script.

  1. Open a PowerShell console.
  2. Enter the following commands (adjusting the vmName to match the name of your Virtual Machine):
    CD D:\ESX-In-Hyper-V\
    .\Enable-NestedVm.ps1 -vmName 'VMWARE ESXi Host 1'
    
  3. Enter Y when asked to confirm any of the changes:ss_vmwareinhv_enablenestedvirtualization
  4. The Virtual Machine is now ready to have ESXi installed into it.

If you run into any problems with enabling nested virtualization, I’d recommend reviewing the documentation. Covering all the possible ways Nested Virtualization might not be configured correctly is beyond the scope of this post. Also, this is still a preview feature and so may still have issues.

Part 4 – Boot ESXi Virtual Machine

  1. Start up the ESXi Virtual Machine and make sure you’re connected to it so you can see the ESXi boot screen:ss_vmwareinhv_bootfirst
  2. Quickly press Tab.
  3. Add the ignoreHeadless=TRUE to the Boot Options: ss_vmwareinhv_bootoptions
  4. Press Enter.
  5. The ESXi Installation system will start up.ss_vmwareinhv_bootscreenfirst
  6. After a couple of minutes the VMWare ESXi 6.0.0 Installer will start up:ss_vmwareinhv_esxiinstaller
  7. You can now go through the ESXi installation process.
  8. You will receive this warning during the installation process but you can ignore it:ss_vmwareinhv_esxiinstallerwaring
  9. The installation process will begin:ss_vmwareinhv_esxiinstallerinstall
  10. Once the ESXi installation has completed you will see this message:ss_vmwareinhv_esxiinstallercomplete
  11. Eject the ESXi Installation ISO before rebooting the Virtual Machine:ss_vmwareinhv_ejectiso
  12. Press Enter to reboot the VM.

Part 5 – Configure the ESXi Boot Options

The final thing we have to do is permanently set the boot options for the ESXi host so that the ignoreHeadless setting is always set to TRUE.

  1. When the ESXi machine reboots, quickly press SHIFT-O to set the boot options.
  2. Add the ignoreHeadless=TRUE to the Boot Options:ss_vmwareinhv_bootsecondoptions
  3. Press Enter to boot up the ESXi host:ss_vmwareinhv_bootsecond_started
  4. Once the ESXi has booted up, press F2.
  5. Enter the root login credentials that were set during the ESXi installation process.
  6. Select Troubleshooting Options and press Enter.
  7. Select Enable ESXi Shell and press Enter:ss_vmwareinhv_bootsecond_enableshell
  8. Press ALT+F1 to bring up the console:ss_vmwareinhv_bootsecond_console
  9. Enter your root credentials.
  10. Enter the following command:
    esxcfg-advcfg --set-kernel "TRUE" ignoreHeadless

    ss_vmwareinhv_bootsecond_command

  11. Press ALT+F2 to return to the main ESXi screen.

The ESXi host can now be restarted without having to worry about the ignoreHeadless=TRUE setting.

You now have a fully running ESXi Host running inside a Hyper-V Virtual Machine. I shouldn’t have to point out that this is a completely unsupported way of installing an ESXi Host and should never be used for production workloads. But at least we now have a way of running ESXi Hosts in a Hyper-V Lab environment.

Here’s hoping that someone finds this useful!

125 thoughts on “Install a VMWare ESXi 6.0 Hypervisor in a Hyper-V VM

    • Thankyou🙂

      I did have vCSA running in a Windows Server 2012 R2 guest in a Hyper-V VM. Because vCSA doesn’t contain a hypervisor itself it installs just fine in a Hyper-V host – no special steps are required. Just boot the guest Windows Server OS, mount the vCSA ISO and install it.

      Like

      • Prabakaran says:

        Hi Dan… Excellent document you have provided us. This was so helpful in creating ESXi host running on Hyper-V VM. Please provide us a same document for running vCSA on Windows Server 2012 R2 guest in a Hyper-V VM. This will be a great help for us. Thanks🙂

        Liked by 1 person

      • Hi Prabakaran,

        Thanks for reading! I don’t think it is possible to do this on Windows Server 2012 R2 because Microsoft hasn’t enabled Nested Virtualization Extensions. This is a requirement for running another Hypervisor (e.g vCSA) inside another Hypervisor. But then I haven’t got much experience with vCSA – is it actually a hypervisor? If it isn’t then it might still be possible to run inside a Hyper-V VM on Windows Server 2012 R2 host. HTH

        Like

      • Prabakaran says:

        Hi Dan, Thanks for your response. I’m trying to do the same as you mentioned above vCSA running in a Windows Server 2012 R2 guest in a Hyper-V VM, but I get an error message (VMware ESX and Hyper-V are not compatible remove the Hyper-V role) when vCevter Server Appliance is deployed during the vCSA installation process.

        My Setup
        Physical Server – Windows Server 2012 R2 (Hyper-V Manager feature installed)
        Hyper-V VM – ESXi host (as per your document, Thanks)
        Hyper-V VM – Windows Server 2012 R2 (Trying to install vCSA)

        I’m just doing this for VMware vSphere 6.0 (ESXi & vCSA) lab setup.

        Like

      • Hi Daniel. Thank you for your great work on this. I am running into a issue with the ISO download. Everything goes well and seems to hang when downloading. “Exporting the Imageprofile to ‘C:\***\ESXi-6.0.0-20161004001-standard-customized.iso’. Please be patient ..

        No progress is shown in the folder. Any thoughts?

        Like

    • Hi Joel,
      Quick questions:
      1. The ISO that gets created by Part 1 should be named ESXi-6.0.0-20160302001-standard-customized.iso and should have a SHA1 hash of 2d 45 7f 98 b7 c8 e9 d1 df 07 8a ee 80 d5 cc 98 d3 8d 9d 9e (although I can’t confirm this is the same for everyone or not).
      2. You are adding a Legacy Network Adapter, not the standard Network Adapter to the VM?

      If you can post a screenshot of your VM settings I’ll see if I can see any particular problems with it.

      Thanks
      Dan

      Like

    • Everything you’re doing looks 100% correct to me.

      I don’t like that the Hash is different so I’ll try generating the image again on my machine and see if I get a different hash. This might be expected, but I’d rather know for sure.

      I wonder if there is a difference to the legacy adapter in different builds of Windows 10. Do you know which Windows 10 build you’re using? I am using 10586: http://imgur.com/aNGUXvT

      During the installation process you can actually break into the ESXi bash shell and examine the Virtual Machine hardware (as Linux sees it), which is how I found out what hardware Linux detects. I’ll dig this process out and you could check what hardware is being reported by your system. Give me a couple of hours to document this and get some screenshots.

      I’ll comment again soon.
      Cheers!

      Like

    • Ok, here’s what we can try:

      When you get to the No Network Adapters screen, press ALT+F1. This will bring up the Linux shell. Login as root with no password.

      Then enter the command:
      lspci | egrep -i ‘network|ethernet’

      This should return something like this:

      Hopefully your Legacy adapter is being reported as a DECchip 21140.

      Like

    • I see. So although ESXi is seeing the adapter it isn’t getting bound to the name ‘vmnic0’.

      This might be a red herring, but it seems to be the only difference that I can see – but what causes it?

      I’ve got a Windows 10 build 14342 machine here – I’ll try installing on that and see what happens.

      Like

      • Just select your Host computer name in Hyper-V manager and click “Virtual Switch Manager” on the right hand side. Then select “Internal” and click “Create Virtual Switch”. http://imgur.com/qPJ5aRF

        Once you’ve done that you can edit your VM and change the switch that your Legacy adapter is attached to. This still seems like a long shot to me. I’ll see if I can find out more about where this vmnic0 comes from. Google shall help!

        Like

    • Hehe I hope not. One other thing to look at though:

      Can you execute the following command in an Administrator PowerShell console on your Windows 10 Host – it will output all the details of the Virtual Legacy Adapter:

      Get-VMNetworkAdapter -VMName ‘VMWARE ESXi Host 1’ | fl *

      Change the VMName to the name of the Hyper-V VM you’re installing ESX into.

      Mine looks like this – your one should look similar. If you can post a screenshot I’ll see if I can see anything that might help;

      Like

    • Hmm. The link isn’t working – getting a 404. You don’t need an ImgUr account to post to it – I don’t have one🙂. Just go there, click upload, drag-drop the image and then click upload. It’ll give you a link you can then paste into here which then gets put inline.

      Like

    • The only difference I can see is that you’ve got the “ClusterMonitored” switch on – this is the “Protected Network” checkbox on the Advanced tab on Legacy Adapter in the VM settings. Other than that I can’t see any other difference. I doubt the “Protected network” checkbox would make any difference though. But you could try turning it off. Otherwise I’m all out of ideas😦😦😦

      Like

      • If you could do that, it would be awesome!🙂 It would definitely be a huge deal if that could be done, because I reckon it would solve a lot of problems (and also be a lot faster – the Legacy adapters are SLOW).

        Like

      • Michael says:

        Thats the point here: I’m not an expert in this, so i also would need help for that to do so😉

        Like

      • Haha – ok right – I think it’s also above my skill level (I’m a Windows stack guy mainly). My Linux kung fu is very weak. I wonder if the person who wrote the script to inject the VIB’s into the VMWare image could do it? I mean these VIB’s must come from somewhere right?

        Like

      • Michael says:

        No i dont think that these VIBs with Hyper-V Network drivers exist already..
        The tool to create these VIBs is for example this here:
        http://www.v-front.de/p/esxi-community-packaging-tools.html
        In the LIS Iso there were some .KO Files (these must be the drivers) and these i think you have to package then with this VIB Tool. But there are some things not clear for me:
        1. What Linux Disro (because there are different ones in that ISO) should we take – I would suggest RedHat
        2. Can we take this .KO Files out of the Box because some blogs explain that you need to compile them (and thats so far as i read a very complex process on linux)
        3. What should be the folder structure in the TGZ Archive metioned on the first step of the Package Tools Site…

        So these are then Expert Questions, i think😉

        Liked by 1 person

      • That is some good investigation and info – I think with a bit of detective work it might be possible to get this going.

        I’m not at all certain which Linux distro would be the best to use, but I’m sure some digging will turn it up. From what I’ve read the ESXi kernel is not Linux, but hopefully the drivers are still compatible for at least one distro.

        I might do some more digging in the info you’ve dug up and see if I can’t do something with it. I can probably manage compiling the code (or find a Linux programmer who can) if we need to do that.

        I’ll put aside some time this week to dig about too – let me know if you come up with anything more as well!

        Like

      • Michael says:

        I have some news here:
        To integrate the native Hyper-V drivers in ESXi we need to have the linux source code for these drivers. Therefore we cannot use this ISO, we need something like that on GitHub:
        https://github.com/LIS/lis-next
        There is some source code that a Linux/ESX guru maybe can use to go through that Build Process, which is described here in more details:
        http://www.vm-help.com/forum/viewforum.php?f=34&sid=70f0a89652bb690d05848591a85f9773
        There are four Posts called “ESXI 5.x Drivers – Part 1-4”
        And here starts the hard hacking😉

        Like

      • Michael says:

        Some news here:
        To get that Hyper-V native drivers in the ESXi ISO we cannot use that LIS ISO from MS because we need to compile the source code for the drivers. So a start point for the source code could be that GitHub page:
        https://github.com/LIS/lis-next
        A little guidance for compiling ESXi Drivers is this four part series from trickstarter:
        http://www.vm-help.com/forum/viewforum.php?f=34&sid=2f2bcd8dd86c1cde0812aa405269ed71
        “ESXI 5.x Drivers” Part 1-4
        And here the Linux/ESXi Haking knowledge is required😉

        Liked by 1 person

      • Again, that is some awesome info and researching. I haven’t had much time to work on this lately (too many code reviews). But I will try and make some time this weekend to investigate further. Thanks again for posting this info here too- hopefully there might be other people out there who are know more about compiling Linux drivers.

        Like

  1. Michael says:

    It would be also nice if ESXi Installation could start on a Gen2 VM. But as i tried it stucks on:
    “Relocating modules and starting up the kernel…”
    The ignoreHeadless=TRUE Option does not help, unfortunatly😦

    Liked by 1 person

      • In theory it should be possible, but because the native Adapter shows up as a Microsoft Virtual Adapter (rather than an emulated H/W nic) someone would need to figure out if any drivers are compatible. Microsoft provides it’s own drivers for Microsoft Virtual Adapters in the Guest OS (Win2K12R2 etc). I can’t see VMWare doing this themselves either unfortunately. But it might be possible that Microsoft would do it for the community.

        Like

  2. Michael says:

    I realized now that you cannot start any VM in that VMware VM that is running on Hyper-V. The message says that “VMware ESX and Hyper-V are not compatible”.

    Like

  3. Mani says:

    Thank you for these instructions.

    For the anyone else trying these, here are my observations. If anyone is able to successfully boot a 64bit linux, please post a response here.

    I was able to install ESXi in a VM on Hyper-V on Windows 10. I tried to boot a x86_64 ubuntu ISO and failed. I’m only able to boot x86 (32-bit) linux iso, on a VM in ESXi.

    This check returns false for ESXi running as a VM on Hyper-V.
    http://www.vladan.fr/how-to-check-if-your-host-is-capable-to-run-nested-esxi/

    To access the URL mentioned in the above article https://IP_Of_Your_Host/mob/?moid=ha-host I followed the instructions in this article.
    http://www.virtuallyghetto.com/2015/02/quick-tip-vsphere-mob-is-disabled-by-default-in-esxi-6-0.html

    Like

  4. Patrick says:

    Awesome article thank you! Unfortunately I have the same issue with the ethernet connection. No ping to the host but the esxi vm can ping the router…

    Like

    • Hmmm. I haven’t been working on this lately so haven’t managed to solve it either😦 The problem doesn’t seem to happen on my Windows 10 Preview 14367 build machine though… It might be a problem with older Windows 10 builds…

      Like

  5. The installation is good, but impossible for me of ping ESXI since the physical machine. Nevertheless ESXI gets back well an IP address since the server DHCP of my box internet, but on ESXI i ping all my network except my physical machine.

    A solution ?

    Windows 10 build 1511
    Hyper-V v10.0.10586.0
    network address : 192.168.1.0/24
    ip address esxi : 192.168.1.247/24
    ip address physical machine : 192.168.1.90/24
    gateway : 192.168.1.254/24

    Like

  6. James says:

    Hi All,

    Just curious if any of you encounter similar issue that Hyper-V unable to ping ESXi host, vice versa. Firewall has been turned off.

    Care to shade some light?

    Thanks in advance.

    Like

    • I have had some odd behaviour like that on the ESXi hosts on build 10586 not being able to ping anything (but being able to get an IP Address). It seemed to be fixed on the later Windows 10 preview builds though. I have a feeling this might be a problem with the MAC spoogfing in Nested VMs. But I’m not certain. The newer Windows 10 is released August 2, so I’m hoping that these odd network issues will be resolved with that release.

      Like

      • James says:

        Same issue by enabling the MAC spoofing.

        I’ve created a Windows 10 VM and able to ping Windows Server 2016 vice versa. But unable to ping ESXi from both Microsoft Windows.

        ESXi should be able to access internet since I’m able to ping 8.8.8.8.

        Thanks.

        P/s: Thanks for creating this guide too!

        Like

      • My pleasure! Are you running the ESXi host from a Windows 10 host or from a WS 2016 TP? host? It does seem like a few people are having a similar issue with this – e.g. Limited connectivity from within the ESXi host itself. Although there does seem to be at least some connectivity because an IP address is usually able to be assigned via DHCP. If I can figure out how to replicate the problem I could connect up Wireshark or even hook up MS Message Analyzer directly to the Virtual Switch to see what is going on.

        Like

      • James says:

        Physical server: Windows Server 2016 TP4 Version 1511 Build 10586

        Virtual machine:
        – ESXi 6.0
        – Windows 10 Build 10240

        Like

    • Malcolm says:

      I ran into the same pinging problem. Set up wireshark on the virtual switch and all looked fine when talking to the DHCP server. Messed around with routing, but no luck.

      I don’t have a true fix, but if you’re looking for a workaround to manage your ESXI guest, you can set up a second hyper-v vm with Windows/Linux. Create an internal vswitch and give it a static ip address. Connect both vms to the internal switch through legacy adapters and assign them static ips as well. Set the gateway for both to the ip of the vswitch and make sure the subnet masks are all the same. This should allow you to ping/manage the ESXI guest.

      Of course, to download ISOs, you’ll have to switch your second vm back to an external vswitch, then switch back to internal to use them for vm setup.

      Like

    • Malcolm says:

      I ran into the same pinging problem. Set up wireshark on the virtual switch and all looked fine when talking to the DHCP server. Messed around with routing, but no luck.

      I don’t have a true fix, but if you’re looking for a workaround to manage your ESXI guest, you can set up a second hyper-v vm with Windows/Linux. Create an internal vswitch and give it a static ip address. Connect both vms to the internal switch through legacy adapters and assign them static ips as well. Set the gateway for both to the ip of the vswitch and make sure the subnet masks are all the same. This should allow you to ping/manage the ESXI guest.

      Of course, to download ISOs or vcenter, you’ll have to switch your second vm back to an external vswitch, then switch back to internal to use them.

      Like

  7. amur says:

    Hi and thanks for this. Everything is working fine but when i try to add disks, they wont mount. What i’m trying to do is an vsan witness server. Have you tried to add disks into hyper-v esxi?

    Like

    • TBH it has been a while since I’ve looked into this as I’ve been putting all my time into work recently. I haven’t done much with the guests in EXSi – just spun up basic Windows Servers in them. But I did have to add disks I think.

      Like

  8. Hello, thanks for the info here. I have ESXi 6.0 running, with network properly recognized, but I have no connection to the external network.
    ESXi can get an IP address from a DHCP server, but I can’t ping that address or the default gw.

    Is there some specific network configuration to do in Hyper-V ?

    thanks,
    fausto

    Like

  9. markus. says:

    thx for the great howto… the install basically worked out as described. It would be great if somebody could post information about the reachable-smb-real-life-transfer-speed via cifs into a vm on the esxi vm …. what is the practical limit of the tulip driver?? the information says half duplex. Can someone confirm this? It would be awesome if the bandwith of the tulip interface could be tweaked…..

    Like

  10. Daniel says:

    First, thank you for the great howto. I installed it without problems. But after the configuration of ip addresse and dns settings, i can not ping the router. The esxi vm is a part of my lab02. I’ve created for every lab a separate hyper-v-switch from type “internal”. I have a vm with server 2012R2 in the same lab (vm type gen2) and from there i can ping the router successfully.

    After that i do another test. I have another hyper-v-switch, witch is connected external to my physical network card. I change the network settings of esxi to dhcp and in settings of the vm i configured the external switch. The esxi vm get’s a ip adress from my isp router and i can ping from the shell any host in my private network.

    My host ist windows 10 professional (build 14393.10).
    I used for the guest vmware ESXi-6.0.0-20160504001.

    Does somebody knows a solution for my problem?

    Like

    • Some people have reported this problem. I’ve also seen this on one of my machines (not all however). I haven’t been able to find a solution yet. I had through it would be solved in Win 10 Anniversary Edition but it does not seem to be. It might be worth using a NAT switch (introduced in one of the more recent Win10 builds), but I’m very short on time at the moment.

      Like

    • Corinne says:

      I also had the problem. The ping between MGMT Server and ESX Server was not possible. In the legacy network adapter advanced settings you have to set the port mirroring settings do SOURCE Mode

      Like

    • Hi Salvador,

      Have you tried the suggestion that Corinne mentioned in the comments for the ping problem?

      I’m not sure about the ignoreHeadless being saved. I haven’t run into that. Sorry I can’t suggest much to fix that one.

      Like

  11. Marc Willemsen says:

    @Salvador: ignoreHeadless wasn’t saved at my host because I changed more then one setting at once. After I set the ignoreHeadless and then rebooted (before changing anything else) it worked.

    The ping works between VMs that use a Legacy NIC.
    So just build your Vms with a Legacy NIC and you’ll be good to go.

    Question:
    I have a WIndows 10 machine with Hyper-V installed.
    On this Hyper-V I build a ESXi VM. This works.
    Now I am trying to start a Windows 2012R2 x64 VM on the ESXi VM but this won’t work. I get the message:
    “Failed to start the virtual machine.
    This virtual machine is configured for 64-bit guest operating systems. However, 64-bit operation is not possible. This host supports Intel VT-x, but the Intel VT-x implementation is incompatible with VMware ESX.For more detailed information, see http://vmware.com/info?id=152. ”

    Doas someone know the solution for this?

    Like

    • Good work on the ignoreHeadless and Legacy NIC problems.

      As for the “Failed to start the VM issue” – this is because Windows Server 2012 R2 doesn’t support Nested Virtualization. This is a requirement of running ESXi inside a VM running on Windows. Only Windows 10 build 10586 and above support this feature. Windows Server 2016 TP5 (and soon the RTM Windows Server 2016) do support Nested Virtualization. So the solution is to wait till WS 2016 is out (in 15 days time).

      Like

  12. Hi Daniel, I am Gilson Banin, Premier Field Engineer at Microsoft. I have good news for you. I have done a lot of troubleshooting about this.

    By using the Tulip driver and the EMULATED DEC/Intel 21140 Ethernet NIC, PLEASE keep in mind that you’re going down a completely EMULATED (READ: VERY, VERY, SLOW) I/O path. This is the nature of emulation and is why we provide synthetic (high-speed, optimized) devices. For ESX to use a synthetic device, VMware would need to provide a synthetic driver for ESX. They certainly could and are welcome to do so. Our Linux storage and NIC drivers are open sourced for Linux and considering how liberally ESX borrows from Linux, it seems quite doable if they choose to do so, but that is a VMware decision.

    I’ll pile onto this a little bit. I read the datasheet (manual, specification) for the Digital Equipment Corporation’s chip codenamed “Tulip” in 1994 or maybe ‘95. It was, effectively, the world’s third PCI device.It’s so old at this point that it has literally none of the features that a hypervisor expects to find.

    I have created two nested VM at Windows 10 Hyper-V, ESXi 6 e Xen Server 7 both Linux Generation 1 with Legacy Network Adapter, they were able to communicate each other. So, I have decided to create a new Windows VM Gen 1 with Legacy Network Adapter too.
    This new VM was able to communicate to Citrix and VMWare VM.

    Tulip Driver has a lot of restrictions, so nested VM VMWare can communicate only to other VM with Legacy Network Adapter too, it can’t communicate to physical host or other VMs with synthetic NICs (Generation 2 for example).

    Just for demos and tests, create a Private Virtual Switch on Hyper-V, create all vms Generation 1, remove synthetic adapter, include Legacy Network Adapters and have fun. You will be able to isolate and ping each other.

    Citrix Xen Server doesn’t have this issue. It can communicate to synthetic nics vms.

    Regards

    Gilson Banin
    blogs.technet.com/gbanin

    Like

    • That is some awesome information on this Gilson! Thank you very much for sharing it here.

      That does explain alot about the NIC issues that people have been having. It is really helpful information.

      I mainly looked into this ESXi in Hyper-V “to see if I could do it” – I actually didn’t expect it to actually work. I can’t imagine it would be used for anything other than learning about ESXi. Being a Microsoft/PowerShell person I am much more comfortable with Hyper-V, but it is occasionally useful to have an ESXi host handy when I’m out and about🙂

      Anyway, again, thank you very much for sharing this really great info.

      Cheers
      Dan

      Like

  13. Rui Bastos says:

    Very interesting information, thank you. I managed follow this guide and I have a similar but slightly different “ping issue”.

    I have a lab with a host running Windows 2016 RTM with Hyper-V and II was able to create two ESXi hosts following the instructions above but got the following variations of the “ping issue”:

    – Both ESXi hosts grab the DHCP from the router and are able to ping other hosts on the LAN
    – The Hyper-v is able to ping both ESXi, but none os the ESXi is able to ping the host
    – Other VMs running Windows 2012 on the Host aren’t able to ping the ESXi, and neither ESXi is able to ping any VMs on the host (all those Windows 2012 VMs were running with synthetic networkd card)
    – Another VM running Linux got the same issue as all the other Windows 2012 VMs.

    I haven’t yet tried other machines with legacy network cards because all the VMs are gen 2….

    Any other suggestion apart the one from Gilson above?

    Cheers,
    Rui

    Like

    • Rui Bastos says:

      As a side note, I ended up creating additional Gen 1 VMs on Hyper-V and I can confirm that they’re able to ping both ways the ESX and, as expected, they’re able to talk to the other Gen 2 machines.

      It feels weird to have on the same lan machine A talking to B, B talking to C but A can’t talk to C…

      Like

      • Tell me about it. Great having the info from Gilson, but definitely would be great to figure out if something can be done. I’m going to migrate my Window Server 2012 R2 over to WS 2016 soon and once that is done I’ll try the ESXi process on it and see if anything has changed.

        Like

  14. Rui Bastos says:

    Some further information: my early tests (with the Hyper-V version of 2016) revealed that the machine ESXi on the Gen 1 VM were able to talk to the Hyper-V host. Well, that started failing in a random way (I realised that because my Hyper-V was also an NFS server for the ESX datastores).

    Further problems using another NFS server (WD NAS): up to now I wasn’t able to install any VM inside the ESX.

    – With a Windows 10 64 bits, even with the “nested=true”, the OS boots but fails to start the installation (Windows logo but then a blank screen).

    – With a 32 bit version of Windows 10, somewhere in the middle of installation, the ESX looses the network connection and can’t ping anywhere and no one can ping it, and one has to go to the ESX console and shutdown/reboot it. It seems to happen at a specific stage of the Windows 10 installation fase (just a feeling due the duration it took to fail the 3 time I tried).

    Liked by 1 person

  15. Hallo says:

    hallo
    thanks for your guide, i managed to install nested esxi on hyper v. But i encounter a problem.

    So i cannot ping to GW if i using vswitch to virtual router ( pfsense ). But if i change to my physical router, it can ping, even to internet.

    So the question, is there something wrong with my setup?
    Is there any equivalent “promocious mode” at hyper v?

    Like

  16. Hallo says:

    hallo

    thanks for this guide, i manage to successfully install esxi 6 on hyper, but i got connectivity problem. I cannot ping to virtual router ( pfsense ) inside hyper v but can ping to physical router, even to internet. I suspect because promisious mode.

    i found explanation abaout promicous mode > http://blog.myvirtualvision.com/2013/04/03/vmware-vsphere-nested/
    but i cannot found any xml configuration inside my folder

    i even do this command

    $VM = Get-VM “MyVM”
    Write-Host $VM.ConfigurationLocation
    Write-Host $VM.VMId

    but cannot found at xml. I am using win10 anniversary edition.

    please help

    Like

  17. Sa3k says:

    hallo

    thanks for this guide, i manage to successfully install esxi 6 on hyper, but i got connectivity problem. I cannot ping to virtual router ( pfsense ) inside hyper v but can ping to physical router, even to internet. I suspect because promisious mode.

    i found explanation abaout promicous mode > http://blog.myvirtualvision.com/2013/04/03/vmware-vsphere-nested/
    but i cannot found any xml configuration inside my folder

    i even do this command

    $VM = Get-VM “MyVM”
    Write-Host $VM.ConfigurationLocation
    Write-Host $VM.VMId

    but cannot found at xml. I am using win10 anniversary edition.

    please help

    Like

  18. Shaft says:

    Hello can you help me to resolve the ping issue?
    My hyper-V OS is WIndows 10 enterprise Version 1607 update jul 2016, I follow the tuto and setup my ESXi6. I made a private virtual switch and connect my ESXi on this switch with the correct ip address. I connected 2 others Servers on this Private switch in the same network as ESXI. 2 servers can ping together but no server can ping the ESXi and all test network failed.

    Thank your for your help

    Like

    • Hi There,

      I’m not sure there is actually a working solution for this. Have a look at the comments from Gilson Banin (a PFE from MSFT) that cover the reasons why this occurs and the limitations. I’ve not actually managed to work around the issue myself unfortunately. Sorry I can’t be more helpful – but I’m in the same boat😦

      Like

  19. Arwin says:

    Hi Daniel,

    Things work for me as perfect as i follow your procedure step by step.I’m using Windows Server 2016 evaluation copy as my OS on laptop with 16GB ram 320GB hd SSD.

    I can now run EXSi 6.0 under Hyper V with no issue but the problem for me is i can’t ping ESXi ip address for the the Host Hyper V

    My Hyper V vlan is assign as internal as as follows:

    ip address: 192.168.1.2
    subnet mask: 255.255.255.0
    default gatreway: 192.168.1.3
    preferred dns: 192.168.1.2

    My EXSi vlan is assign as internal as follows:
    ip address: 192.168.1.3
    subnet mask: 255.255.255.0
    default gateway:192.168.1.3
    preferred dns: 192.168.1.3

    Testing Management Network test is ok
    Pinging Address 1 = ok
    Pinging Address 2 = ok
    Resolve Hostname = ok

    The problem is, i can’t ping EXSi ip address fro the HyperV Host, cannot login to PuTTY and vSphere Client

    Thanks

    Like

    • Hi Arwin,

      There are some networking issues with this configuration. If you look further up in the comments you’ll find a comment by a Microsoft PFE describing the problem and what causes it. There are a few work arounds though, but they’re far from perfect. So far no one has found a complete solution to this problem unfortunately😦

      Like

    • Hi Brijesh – there is actually a comment further up from a Microsoft PFE giving information as to why there are networking issues accessing the “outside” world from the ESXi host. No one has yet found any way to resolve these networking issues.😦

      Like

  20. Chris Martinez says:

    Hello Daniel. Thank you for your article. It has been extremely helpful for me. I successfully installed esxi 6.0. I unfortunately cannot connect to it from the vSphere client. If I ping the server it pings fine back and forward from the server to the client and client to server. Firewall is off, antivirus, and anything that I can think would prevent communication. Thank you for your help in advance.

    Like

    • Hi Chris, there are indeed some networking anomalies with this method. If you look further up you’ll see a comment from a Microsoft PFE giving further depth into this issue. No one has yet managed to completely solve this issue, however you can still set up some networking communication with the ESXi hosts by using an internal virtual switch.

      Like

  21. CyberNanny says:

    Thank you for these instructions Daniel. I am getting an error message upon trying to start up a VM inside of my nested ESXI host. For some reason it is detecting hyper-v as being present. Here is my error message:
    Failed to start the virtual machine.
    VMware ESX and Hyper-V are not compatible. Remove the Hyper-V role from the system before running VMware ESX.

    Does anyone else have this problem and know of a solution?

    Like

    • Hi There,

      Hmm. I’m not at all sure about this. I’ve not seen anything like that. It sounds like you’re attempting to install ESXi onto the host rather than into a VM on the host – but I could be completely and totally wrong🙂.

      Like

  22. CyberNanny says:

    Thank you again for these instructions! I am having an issue with getting any VMs in ESXI to start up. Upon trying to start a VM on my nested ESXI host i get the following error message:
    Failed to start the virtual machine.
    VMware ESX and Hyper-V are not compatible. Remove the Hyper-V role from the system before running VMware ESX.

    Has anyone else had this issue or know of a fix?

    Like

    • Prabakaran says:

      I too have the same problem. Unable to bring up the VMs hosted on ESX host.

      Failed to start the virtual machine.
      VMware ESX and Hyper-V are not compatible. Remove the Hyper-V role from the system before running VMware ESX.

      Anyone has fix for this issue ?

      Like

  23. I too am getting a message from the vSphere Client when trying to startup a VM in ESXi 6.0 Update 2 virtualized under Windows Server 2016 Hyper-V “VMware ESX and Hyper-V are not compatible”. I didn’t have any problem installing the ESXi (following your excellent instructions) nor with using the vSphere Client connecting to the ESXi VM. The ESXi VM could also connect to the Internet (I downloaded Update 2 from an ssh session). It does appear like the ESXi recognizes that it’s running under Hyper-V which is why it is producing the error.
    I did notice that when I ran Get-NestedVirtStatus.ps1 my Host Server said “Virtualization Based Security is running” and “HostNestedSupport : False”. This script appears to run “Get-CimInstance -classname Win32_DeviceGuard -namespace root\Microsoft\Windows\DeviceGuard” with the result of “VirtualizationBasedSecurityStatus : 2” creating the above errors.
    I have set the local Group Policy for “Computer Configuration\Administrative Templates\System\Device Guard\Turn On Virtualization Based Security” to Disabled and rebooted but I still get the “HostNestedSupport : False”.
    Could the ESXi error and the “HostNestedSupport : False” be tied together? I’d certainly appreciate any insight that anyone could offer. Thanks!

    Like

    • Tom Watson says:

      Did you edit the /etc/vmware/config file mentioned above?

      I managed to get networking to work, as long as I had a Windows VM running under Hyper-V connected to the same virtual switch. The Windows VM needed to have a Legacy Network Adapter.

      As far as running a VM within the virtualized ESXi host, I managed to get a small 32 bit Linux VM running.

      My Virtualized ESXi host (IP: 192.168.2.102):-

      My Win7 VM, connecting to 192.168.2.102 via vSphere, and firing up a small Linux VM :-

      Like

  24. Tom Watson says:

    Have your tried editing /etc/vmware/config as mentioned above?

    I managed to get the networking working, as long as I was connecting from a Windows VM running on the same Hyper-V switch, and it was connected via a Legacy Network Adapter.

    I also managed to import a small 32 bit Linux VM into the ESXi, and start it up. I’m not sure if 32 bit VMs is a limitation of the nesting. I haven’t tried a 64 bit one yet.

    Virtualized ESXi host running on 192.168.2.102 :-

    Win7 VM connecting to 192.168.2.102, and starting a Linux VM using vSphere console.

    Like

    • Tom,
      Thanks for the suggestions. I was able to ESXi to start a VM by editing the /etc/vmware/config. And I was able to create a new Hyper-V Switch that wasn’t used for management by the host OS. This allowed me to be able to access the VMWare management using the vSphere client.
      But when I tried to create VM’s I kept getting stuck. I tried a CentOS 6 VM which I imported – that started to go through the bootup process but then kept rebooting. I tried to install Windows 2012 R2 but that hung before I could get to any of the prompts to start the install. And I tried to install Cacti (CentOS 5) from an ISO and that failed a few seconds after hitting Enter at the boot: prompt.
      So the networking is ok, and ESXi will attempt to start VM’s. But then they just fail.
      Any ideas? I was hoping to use this to run an application which only runs on ESXi on my customer’s Windows 2016. Doesn’t look too promising at this point.
      Thanks.. dave

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s