Snapdragon and Qualcomm branded products are products of
Qualcomm Technologies, Inc. and/or its subsidiaries.
Co-written with Prakruthi Deepak Heragu
If you’ve started working with Android Virtualization Framework (AVF), then you probably know that it offers a set of APIs so you can configure and launch protected virtual machines (VMs). Many app developers have use cases they want to move out of main Android into isolation and protected VMs with the help of a hypervisor, and AVF makes that possible.
You’ve probably also worked with the protected kernel-based virtual machine (pKVM), which is the reference hypervisor for AVF. It’s well-suited to computing use cases, but in the Qualcomm Innovation Center, Inc. (QUIC), we have replaced pKVM with Gunyah Hypervisor Software – our own, more versatile hypervisor – on our Qualcomm Technologies’ chipsets. With Gunyah, we can use one hypervisor across use cases as varied as automotive, mobile broadband, IoT and wearables.
We’ve upstreamed Gunyah for general use by the Android community. In this post, we’ll describe how Gunyah works, how it fits in AVF and how you can use it for protected use cases like isolated compilation on VMs in Android 14.
(This is a summary of our presentation “Adding Third-Party Hypervisor to Android Virtualization Framework” at the Linux Plumbers Conference. Details at bottom.)
Protected memory in Android Virtualization Framework
What does AVF require in a hypervisor? Among other things, it must support:
- Dynamic VM allocation – First, the hypervisor must be capable of launching a virtual machine. Note that AVF does not require guests to be Linux-based.
- MMIO access – It must be able to handle memory-mapped input/output (MMIO) access from vCPUs, to support virtIO, a virtualization standard for network and disk device drivers.
- Shutdown – It must allow for restarting VMs.
- Essential, architecture-specific devices – Those devices must include interrupt controllers and timers.
- Protected VMs – It must enforce memory isolation to ensure that the VM is what you think it is; that includes authentication and anti-rollback (Android Verified Boot). Protected VMs attest that they were loaded by a trusted boot chain.
The VMs use a lot of memory that you don't want Android to be able to access. The diagram below shows the problem of what memory needs to be isolated from the host to be able to launch a protected VM.
Different, scattered regions get mapped into guest memory and are no longer accessible to the host.
Isolated compilation is the first use case supported by AVF on commercial devices shipping with Android 14. AVF VMs are guaranteed to have protection from Android OS. In the future, more secure computing tasks will be able to run on one or more VMs.
Using Gunyah in AVF
Instead of pKVM, Qualcomm Technologies, Inc. has been using Gunyah, a type-1 hypervisor developed by QUIC as a small microkernel.
From the outset, we designed Gunyah differently from other hypervisors. Its main advantages include:
- De-privileged VMs, including the primary VM
- Isolation of memory and I/O
- Interrupt assignment
- EL2-based scheduling and hypervisor-assisted device emulation
- Performance optimized for automotive (MISRA), mobile and IoT
- Full operation without needing Linux as the host VM
As shown in the block diagram below, Gunyah includes a resource manager (RM) for access to all the vCPUs and for memory management of VMs.
Android addresses either the RM or the hypervisor itself for all virtualization tasks. The RM is its own EL1 VM, running in isolation from all other VMs and implementing the strong, capability-based resource management policy for EL2 that Gunyah provides.
Unlike other hypervisors, the device isolation in Gunyah takes advantage of Snapdragon® hardware. The system memory management units (SMMUs) and input/output memory management unit (IOMMUs) in that hardware are standardized for the architecture, but Snapdragon hardware stands out among the few processors that use them.
To accommodate AVF we made changes to Gunyah, including device emulation provided by Android and stolen-time accounting for the protected VM.
Workflow when launching the VM
This flow diagram shows the order of operations when Gunyah launches a protected VM:
The Android bootloader (ABL) loads PVM firmware (pvmfw.bin) before launching the protected VM for AVF. Along with that, to establish a chain of trust, the bootloader verifies the firmware and attaches a boot certificate chain to it. When Gunyah sends a request to run the protected VM, it first loads the PVM firmware with the Boot Certificate Chain (BCC).
Execution then moves to the protected VM. In the Android 14 use case of isolated compilation, a Microdroid VM compiles and signs binaries during a system update. It saves the binaries to storage and after rebooting for the system update, Microdroid launches again and verifies the compiled binaries before the host installs them on the device. This process allows for system updates with less downtime because the compiling and signing can happen while Android is still usable.
New additions to support Gunyah in AVF
The block diagram below illustrates the architecture of AVF with Gunyah:
We submitted more than a dozen changes to crosvm, the virtual machine monitor (similar to QEMU-KVM) in Android userspace. The changes included adding a new hypervisor crate to use our Linux driver for Gunyah, which abstracts all communication to Gunyah.
In Linux, we've been actively upstreaming all of our changes to kernel.org.
Your turn: Try Gunyah
To summarize, Gunyah is a versatile hypervisor that can launch multiple protected virtual machines on AVF, supported in Android 14. We have open-sourced Gunyah, added support in crosvm and we are working on adding the Linux support to launch and manage VMs.
We want to support more use cases with Gunyah, so download it and send us your contributions and review comments. You’ll find steps in the Gunyah open-source tree on how to launch a secondary VM; you can run any computing task on that VM. You can find the latest (as of this blog post) kernel patches here.
For the future, our development efforts are focused in several directions, including managing memory for the guest through guestmemfd and dynamically growing the VM based on demand. We welcome community input and look forward to seeing what we can all build together. For more details, see our presentation “Adding Third-Party Hypervisor to Android Virtualization Framework” at Linux Plumbers Conference, with links to our slide deck and video.
And if you’re curious about the name, “gunyah” is a term for a small, temporary shelter, traditionally used by Australian Aboriginal people. The name reflects both the impermanence and versatility of the third-party hypervisor for AVF, and it’s a nod to our colleagues in Qualcomm Australia who developed it.
Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries. Gunyah is a product of Qualcomm Innovation Center, Inc.
