MicroVMs

Posted: Feb 12, 2025

MicroVMs: an introduction

This article aims to explain the concept behind “MicroVMs”. MicroVMs combine container-like speed with VM-level isolation to securely manage high-density workloads. By leveraging minimalist hypervisors and hardware virtualization, they reduce the attack surface while providing near-instant startup times. AWS Firecracker exemplifies this approach, enabling the launch of hundreds of microVMs per second in serverless environments.

Introduction

A micro VM is a lightweight virtual machine that isolates an untrusted computing operation from a computer’s host operating system. This design allows multiple high-performance, secure workloads to run on one machine. It merges the security and isolation of traditional VMs with container-like resource usage.

Bromium introduced one of the earliest micro VM products in 2012 where Micro VMs act as an alternative to software sandboxing. They rely on hardware separation to stop infections from user-driven tasks such as web browsing or file downloads. AWS released the open source Firecracker micro VM in 2018, followed by QEMU’s microvm project in 2019 with a similar feature set. Firecracker is powering all of AWS Lambda serverless architecture.

A quick comparison of VMs vs. Containers vs. MicroVMs

A micro VM offers VM isolation for each instance, incorporating container-like architecture. This approach allows administrators to deploy many micro VMs quickly and use resources efficiently, and it maintains logical isolation and security for each instance. Standard VMs and containers both have drawbacks. When organizations use these technologies together, they reduce those drawbacks and increase the overall benefits.

VMs

Virtual machines (VMs) provide high isolation but demand significant resources. They run on a hypervisor, keep memory spaces separate, and do not detect other VMs on the same host. Each VM includes its own guest OS, allowing multiple OS types and versions on one server.

This level of separation can make VMs large and slow to deploy. Every VM needs an operating system, creating repeated OS components across multiple VMs and increasing resource use.

Containers

Containers occupy fewer resources than virtual machines and share a single operating system. They run on engines such as Docker or Podman, and they often start in seconds. One server can support many containers, though it might only handle a few VMs.

This efficiency stems from a common OS kernel, typically a Linux variant, along with any required dependencies packaged inside each container. This design supports flexible deployment across diverse hosts. A shared OS can introduce potential security or functional risks, since a single issue in the OS may affect every container.

Micro VMs

Micro VMs combine strong separation with security features. A typical setup uses standard server hardware and a host OS (such as Linux) that supplies kernel-based virtualization (KVM). A micro VM engine, for example Firecracker, runs on the OS and acts as a hypervisor. It provides an API, along with network, storage, and administration components to operate each micro VM.

Once active, the micro VM engine creates fully isolated virtual instances that can host a guest OS or container-based workloads. These instances are small and can be created in large quantities with minimal overhead.

Core Principles

• Isolation: Firecracker uses KVM for workload protection. A companion program called “jailer” adds another separation layer through Linux user-space controls.

• Minimalist Design: Firecracker omits unneeded devices and guest features, allowing each micro VM to occupy less than 5 MiB of memory. This approach lowers resource use and reduces exposure to attacks.

• Fast Startup: A minimal device model supports high-speed provisioning. Firecracker can launch about 150 microVMs per second per host and begin application code within 125 ms.

• Compute Oversubscription: Hosts can allocate more compute resources than physically available. This practice suits workloads with modest CPU usage and makes resource utilization more flexible.

Summary: VMs vs. Containers vs. MicroVMs

Below is a concise comparison of micro VMs, containers, and VMs, highlighting design, isolation, performance, and use cases.

	Design	Isolation	Performance	Use Cases
MicroVM	Lightweight and isolated. Each instance includes its own kernel, guest OS, and virtualized hardware.	Strong separation. Each runs independently, protecting both the host and other instances.	Lower overhead than standard VMs but more resource-intensive than containers. Quick startup times.	High-security or multi-tenant workloads.
Container	Lightweight runtime that shares a host OS kernel.	Operates at the application level, relying on the host OS kernel for isolation.	Efficient resource usage with fast startup. Uses host system resources directly.	Consistent application packaging and deployment across environments.
VM	Fully virtualized, with a dedicated OS, kernel, and virtual hardware for each instance.	Separates the host and guest environments completely.	Higher resource overhead. Startup may take longer than containers or micro VMs.	Workloads requiring thorough isolation from the host system.

A case study: Browser virtualization with Bromium

For instance, a Bromium-based browser microvirtualization setup applies policies that label processes as trusted. Untrusted tasks—such as opening files or accessing webpages—automatically launch in a micro VM. This approach confines malware or other harmful code within the micro VM, preventing damage to the rest of the system.

A computer can run dozens or hundreds of micro VMs at once. They are generated on demand for user-initiated tasks and must appear instantly. To keep latency low and avoid bottlenecks, micro VMs only use a minimal range of operating system resources and cannot communicate with other processes.

A case study : Firecracker microVMs

Before micro VMs, developers had to pick between containers, which offered resource efficiency and speed, or traditional VMs, which delivered better isolation and security. AWS also faced this trade-off and early Lambda and Fargate deployments relied on dedicated EC2 instances per customer. As demand grew, users requested faster scaling, lower latency, and more features—such as provisioned concurrency. AWS needed a virtualization approach that would maintain security without sacrificing performance.

To address this need, AWS created Firecracker., a virtual machine monitor (VMM) developed in Rust. Rust offers security, flexibility, efficient memory usage, and speed. Firecracker builds on crosvm and uses the Linux Kernel-based Virtual Machine (KVM) to create micro VMs.

How does microVM work ?

Some Terminology around virtualization

A virtualization solution often includes a hypervisor kernel, a virtual machine monitor (VMM), and device drivers.

The bare-metal hypervisor runs in privileged mode within the host OS and has direct access to hardware. The VMM also runs on the host OS, typically in non-privileged mode. Together, the hypervisor and VMM allocate resources to VMs, and each VM hosts its own guest OS.

Virtual machines isolate workloads in two key ways:

Operational isolation: One VM’s heavy CPU usage does not slow other VMs on the same host. This prevents the “noisy neighbor” effect.
Security isolation: One VM cannot read or infer data from another VM. It also stops privilege escalations so that data remains within each VM boundary. Security isolation helps avoid side-channel leaks, such as those stemming from shared caches in hyperthreading. Attackers could otherwise observe cache access patterns to gain insights into sensitive data.

Red Hat Reference Architecture for VM

Red Hat’s “All you need to know about KVM userspace” is one of the most referenced articles on the internet about open source virtualization. Linux KVM, QEMU, and libvirt form the backend of Red Hat’s virtualization stack.

KVM (Kernel-based VM)

KVM is the kernel-level virtualization layer. It also serves as an interface between hardware and QEMU:

Hardware virtualization: Uses CPU extensions (Intel VT-x or AMD-V) for hardware-assisted virtualization and offers an interface to QEMU for VM creation and management.
CPU and memory virtualization: Allocates and isolates these resources with minimal overhead.
Other kernel-level tasks: Manages VM exits, VM scheduling, and interrupt routing to virtual devices. It can also expose kernel features—such as memory management and process scheduling—to QEMU.

QEMU (Quick EMUlator)

QEMU is a user-space process that acts as both a device emulator and a virtual machine monitor:

Device emulation: Emulates hardware components (disks, network adapters, GPUs) that guest operating systems use.
VM setup and execution: Initializes and runs virtual machines along with their guest environments.
Performance optimizations: Passes hardware calls to KVM when possible, which boosts overall performance.

A kernel hypervisor (KVM) with a user-space VMM (QEMU) is a typical arrangement. Red Hat’s reference diagram does not detail how QEMU handles block devices and Red Hat articles mention virtio for that purpose.

libvirt

libvirt (not to be confused with virtio) is a manager for VMs across multiple virtualization platforms, including KVM, Xen, VMware ESXi, and Hyper-V. Red Hat uses libvirt more extensively as a “jailer” to confine QEMU processes. Since QEMU interacts with guest input, it may face malicious behavior. libvirt is hidden from the guest environment, which makes it well suited to limit QEMU’s access. SELinux and file permissions add extra restrictions to stop QEMU from accessing other VMs.

KVM, QEMU, and libvirt form the backbone of Red Hat’s reference model. Two additional Linux kernel projects often appear in similar solutions:

cgroups (control groups): Restricts CPU, memory, disk I/O, and network usage for specific process groups. This helps prevent resource contention across workloads.
nftables: Filters and sorts network packets, offering firewall-like functions that prevent unwanted traffic on the host. It operates more on external traffic than on communication between VMs.
Linux network namespaces and seccomp-bpf are commonly used for isolation among VMs sharing a host.

MicroVM architecture: AWS Firecracker example

AWS Firecracker is an open source virtual machine monitor (VMM) specialized for serverless workloads and often cited as one of Amazon’s most influential open source projects.

High CPU and memory overhead for VMs at the VMM and kernel layers
Slow VM startup, often measured in seconds
Large hypervisors and VMMs, usually written in memory-unsafe programming languages and exposed to security risks

AWS addresses these challenges by keeping Linux KVM but replacing QEMU with Firecracker—a minimal VMM written in Rust. Firecracker includes:

Device emulation for disk, networking, and a serial console for keyboard
REST-based configuration for managing microVMs (start, stop, configure). This replaces some of the functionality offered by libvirt
Rate limiting for network and disk I/O, allowing granular control over throughput and request volume. For this Firecracker implements its own solution for simplicity, rather than using cgroups

For network and block devices, Firecracker uses virtio. Virtio provides an open API for exposing emulated devices from hypervisors. Virtio is simple, scalable, and offers good performance through its use of paravirtualization.

On the networking side, Firecracker uses a TAP virtual network interface and encapsulates the guest OS (and the TAP device) inside their own network namespace.
For storage, AWS opts for block devices rather than filesystem passthrough as a security measure. Filesystems codebases can be large, which raises the host kernel’s risk. Offering guests only block IO avoids exposing those filesystem components, reducing potential attack points.

Firecracker includes a built-in rate limiter to manage network and storage usage for every microVM on the same host. Administrators can set limits to allow bursts or to enforce specific bandwidth and operations constraints.

A dedicated jailer adds an extra layer of security against unwanted VMM behavior. It applies Linux primitives such as chroot, pid, and network namespaces, then restricts calls to the host with seccomp-bpf. This differs from Red Hat’s use of libvirt for sandboxing.

MicroVMs run on any bare metal server. Firecracker started with Intel CPU support and later introduced AMD and ARM. It currently supports 64-bit systems from all three vendors.

Firecracker follows two guiding principles: reuse existing Linux components wherever helpful, and opt for minimal replacements when resource usage or potential attack surfaces need reduction. This approach delivers a secure and compact solution for serverless workloads.

Conclusion

MicroVMs are recognized for their security, speed, and scalability:

Security: They maintain a small attack surface. All Firecracker processes run under cgroups and seccomp BPF, and only a narrow set of system calls is permitted.

Speed: Benchmarks show launch times under 125 ms, supporting up to 150 microVMs per second.

Scale and Efficiency: Each microVM consumes fewer than 5 MiB of memory, allowing many instances on a single server. Firecracker’s built-in rate limiter allocates network and storage resources across all microVMs on the same host.

MicroVMs combine the density and speed typically associated with containers, plus the isolation and security often linked to traditional VMs. This lightweight virtualization supports serverless environments by letting providers run more workloads on bare metal with lower overhead.

jmanteau

Mon coin de toile - A piece of Web