Show HN: Sub-millisecond VM sandboxes using CoW memory forking

Good callout. We seed entropy before snapshot to unblock getrandom(), but forks still share CSPRNG state. The proper fix per Firecracker’s docs is RNDADDENTROPY + RNDRESEEDCRNG after each fork, plus reseeding userspace PRNGs like numpy separately. On the roadmap. https://github.com/firecracker-microvm/firecracker/blob/main...

I suppose it'd be easy enough to re-seed RNGs, but re-relocating ASLR sounds like a pain. (Although I suppose for Python that doesn't matter)

I don't mean to turn this into a religious war, but honestly, I sometimes wonder what would be the net benefit for humanity if Windows slowly disappeared. And I'm saying this as someone who appreciates the good stuff done by Microsoft in the past (windows 9* UI, decades-long support for Win32 APIs etc.).

Web browsers don't even work properly in Windows Sandbox. There is a bug that hasn't been patched in over a year whereby web browsers can't use the GPU to render a page so all it displays is a white page. Users have to create a configuration file that turns off vGPU and launch Windows Sandbox from that.

Feel you. That's why we're actively working on Windows and macOS sandbox support at Daytona - with proper isolation, agents tools, dynamic resizing etc; not just "networking on/off" level controls.

If you're building agents on Windows and want to give it a spin, reach out for early access.

Can give you access to win sandboxes on Daytona, just fill this in here and lmk! https://www.daytona.io/docs/en/computer-use/

I guess get busy contacting Microsoft or get busy using Open Source software instead.

Fair question. The fork engine itself is general purpose -- you could use it for anything that needs fast isolated execution. We say 'AI agents' because that's where the demand is right now. Every agent framework (LangChain, CrewAI, OpenAI Assistants) needs sandboxed code execution as a tool call, and the existing options (E2B, Daytona, Modal) all boot or restore a VM/container per execution. At sub-millisecond fork times, you can do things that aren't practical with 100-200ms startup: speculative parallel execution (fork 10 VMs, try 10 approaches, keep the best), treating code execution like a function call instead of an infrastructure decision, etc.

The thing people tend to gloss over is how CoW shines until you need to update the base image, then you start playing whack-a-mole with stale memory and hotpatching. Snapshots give you a magic boot, but god help you when you need to roll out a security fix to hundreds of forks with divergent state.

Fast startup is nice. If the workload is "run plain Python on a trusted codebase" you win, but once it gets hairier the maintenance overhead sends you straight back to yak shaving.

simonw seems like he's always wanting what you describe, maybe more for wasm though

1 vCPU per fork currently. Multi-vCPU is doable (per-vCPU state restore in a loop) but would multiply fork time.

On Firecracker version: tested with v1.12, but the vmstate parser auto-detects offsets rather than hardcoding them, so it should work across versions.

I am not sure exactly what you are asking but firecracker does need access to /dev/kvm so nesting needs to be enabled on the VM.

If each node has its own warmed-up VM awaiting from startup, there's no need to clone across nodes.

Agreed, cross-node is the hard next step. For now single-node density gets you surprisingly far. 1000 concurrent sandboxes on one $50 box. When we need multi-node, userfaultfd with remote page fetch is the likely path.

Is this relevant?

https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...

[dead]

The API in the readme is live right now -- you can curl it. Plan is multi-region, custom templates with your own dependencies, and usage-based pricing. Email in my profile if you want early access.

Thanks! Yes, there's going to be a managed version.

You could throw this on a VPS or server and it could help in that regard: (disclaimer, my thing)

https://GitHub.com/jgbrwn/vibebin

CRIU is great for save/restore. The nice thing about CoW forking is it's cheap branching, not just checkpointing. You can clone a running state thousands of times at a few hundred KB each.

Nested page tables / nested virtualization made it to consumer CPUs about a decade ago, so yes :)

It's pretty common to run VMs within containers so an attacker has to escape twice. You can probably disable 99% of system calls.

Done, thanks!

On tail latency: KVM VM creation is 99.5% of the fork cost - create_vm, create_irq_chip, create_vcpu, and restoring CPU state. The CoW mmap is ~4 microseconds regardless of load. P99 at 1000 concurrent is 1.3ms. The mmap CoW page faults during execution are handled transparently by the host kernel and don't contribute to fork latency.

On snapshot staleness: yes, forks inherit all internal state including RNG seeds. For dependency updates you rebuild the template (~15s). No incremental update - full re-snapshot, similar to rebuilding a Docker image.

On the memory number: 265KB is the fork overhead before any code runs. Under real workloads we measured 3.5MB for a trivial print(), ~27MB for numpy operations. But 93% of pages stay shared across forks via CoW. We measured 100 VMs each running numpy sharing 2.4GB of read-only pages with only 1.75MB private per VM. So the real comparison to E2B's ~128MB is more like 3-27MB depending on workload, with most of the runtime memory shared.