AWS Adds support for nested virtualization

OCI supports it with Intel. I know it works with AMD, but we don't officially support that so far as I'm aware. The performance hit on AMD is bigger than Intel, last I looked.

Azure has had nested virt available for a while too. I used to run HyperV in cloud

You can use an expensive AWS VM instead of an expensive AWS bare–metal image. Does anyone realise how expensive AWS is, even in the best case?

whats the ~ perf hit of something like this?

Was hoping this comment would be here. Firecracker and microVMs are good use-case. Also, being able to simply test and develop is a nice to have.

Nested virtualization can mean a lot of things. Not just full VMs.

> Bare-metal on Hetzner now has a setup fee of up to 350 EUR.. you can find some stuff with 0 setup fee, but it's usually quite old kit.

I don't understand what you are paying for here, nested virtualization doesn't need any extra setup for hardware compared to normal one

... or you are saying Hetzner wants 350 EUR for turning on normal virtualization option in BIOS ?

Fair concern, but this has been quietly production-stable on GCP and Azure since 2017 — that's 8+ years at cloud scale. The LKML debates you're referencing are mostly about edge cases in exotic VMX features (nested APIC virtualization, SGX passthrough), not the core nesting path that workloads like Firecracker and Kata actually exercise.

The more interesting signal is that AWS is restricting this to 8th-gen Intel instances only (c8i/m8i/r8i). They're likely leveraging specific microarchitectural improvements in those chips for VMCS shadowing — picking the hardware generation where they can guarantee their reliability bar rather than enabling it broadly and dealing with errata on older silicon. That's actually the careful engineering approach you'd want from a cloud provider.

It's been around for almost 15 years and stable enough for several providers to roll it out in production the past 10 years (GCP and Azure in 2017).

AWS is just late to the game because they've rolled so much of their own stack instead of adapting open source solutions and contributing back to them.

Azure has had nested virt for a while - maybe it’s related to OpenShift but you could run OpenShift on Azure for some time. I used to run HyperV in Azure on certain SKUs

Azure? OpenShift? "I don't think about you at all." — Matt Garman probably

It is great for isolation. There are so many VM based containerization solutions at this point, like Kata Containers, gvisor, and Firecracker. With kata, your kubernetes pods run in isolated VMs. It also opens the door for live migration of apps between ec2 instances, making some kinds of maintenance easier when you have persistent workloads. Even if not for security, there are so many ways a workload can break a machine such that you need to reboot or replace (like detaching an ebs volume with a mounted xfs filesystem at the wrong moment).

The place I've probably wanted it the most though is in CI/CD systems: it's always been annoying to build and test system images in EC2 in a generic way.

It also allows for running other third party appliances unmodified in EC2.

But also, almost every other execution environment offers this: GCP, VMWare, KVM, etc, so it's frustrating that EC2 has only offered it on their bare metal instance types. When ec2 was using xen 10+ years ago, it made sense, but they've been on kvm since the inception of nitro.

It's when you want to do stuff with your own VMs and don't want to pay extra for bare metal machine, basically.

There is no real reason to use it on hardware you own; but in case of cloud you just not always have enough to do to excuse paying for whole entire server

When you run nested virt, you can do multicast in Cloud between the nested VMs. You can’t do multicast across VMs inside the Cloud.

Basically you setup a small LAN with HyperV or something similar (I have only done it with HyperV)

Hopefully it mean that you can finally run a network simulator like GNS3 https://www.gns3.com/ in an AWS instance.

You can now run VMs inside a cheaper AWS instance instead of having to pay for an entire bare-metal instance. This is useful for things like network simulation where you use QEMU to emulate network hardware.

If you have some workload that creates VMs, now you can run that workload on EC2 rather than having to use bare metal or some other provider that allows nested virtualization. There are many many such workloads. Just to give one example: testing a build system that spins up VMs to host CI jobs.

Yep. It's pretty boring. I've been using it at home for years and years with libvirt on very not-special consumer hardware. I guess the AWS clown is finally catching up on this one little not-new-at-all thing.

It says VSM is automatically disabled... so I would assume not.

From memory, the virtualisation operations themselves aren't nested. The VM instructions interact with the external virtualisation hardware, so it's more of a cooperative situation, e.g. a guest can create & manage virtualisation structures that are run alongside it.

I don't know if this applies to the specific nested virtualisation AWS are providing though.

depends on the workload and how they've done it

pure CPU should be essentially unaffected, if they're not emulating the MMU/page tables in software

the difference in IO ranges from barely measurable to absolutely horrible, depending on their implementation

traps/vmexits have another layer to pass through (and back)

As a practical matter, anywhere from 5-15%.

I haven't watched The Thirteenth Floor in a while. The kids today don't even know about it.

Yeah, though honestly if I'm deploying anything I'd just build an image with nix rather than use nested virtualization.

DO has Nested Virtualization enabled for years.

Highly doubt that