Virtualization of BSD - BSDCan
Transcript of Virtualization of BSD - BSDCan
© 2018 BlackBerry. All Rights Reserved. 1© 2019 BlackBerry. All Rights Reserved. 1Confidential – Internal Use Only
Virtualization of BSDUsing the QNX Hypervisor
Quentin Garnier
Senior Kernel [email protected]
May 2019
© 2018 BlackBerry. All Rights Reserved. 2© 2019 BlackBerry. All Rights Reserved. 2Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 22
• The virtualization environment: QNX Neutrino and QNX Hypervisor
• Goals for the exercise
• Stories from the trenches
Agenda
© 2018 BlackBerry. All Rights Reserved. 3© 2019 BlackBerry. All Rights Reserved. 3Confidential – Internal Use Only
QNX Host Environment
© 2018 BlackBerry. All Rights Reserved. 4© 2019 BlackBerry. All Rights Reserved. 4Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 44
• Some vocabulary:● Host system● Virtualization manager (qvm)● Virtual machine● Guest system
• What the host system provides:● Virtualization manager● Drivers for possible shared hardware resources● Anything else the system designer wants to have
About the QNX Hypervisor design (1/2)
© 2018 BlackBerry. All Rights Reserved. 5© 2019 BlackBerry. All Rights Reserved. 5Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 55
About the QNX Hypervisor design (2/2)
QNX Neutrino microkernel ( procnto )
Devices
Hardware (Aarch64, x86-64)
App 1 ... App n
Guest OS (QNX)
BSP
qvm process
Drivers
Virtual devices
Drivers
App 1 ... App n
Guest OS (Linux)
qvm process
Drivers
Virtual devices
HypervisorHypervisor module ( libmod_qvm.a )
Message handler threads
Shared
Private memory
Pass-through
EmulatedShared
Emulated
© 2018 BlackBerry. All Rights Reserved. 6© 2019 BlackBerry. All Rights Reserved. 6Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 66
• Targeting QNX guests and Linux guests● This is what the industry wants
• The Hypervisor runs as a process in the host system, with virtual CPUs being scheduled as normal threads
● A special privilege elevation interface allows running guest code
• Minimal environment, therefore no or minimal virtual firmware● For instance, no emulated BIOS whatsoever on x86_64
A few design choices for the Hypervisor (1/2)
© 2018 BlackBerry. All Rights Reserved. 7© 2019 BlackBerry. All Rights Reserved. 7Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 77
• Minimal environment, therefore no or minimal virtual firmware● QNX on x86 is booted through Multiboot● Linux/x86_64 has its own protected-mode loading protocol
• How does time flow in a guest?● It’s complicated...
A few design choices for the Hypervisor (2/2)
© 2018 BlackBerry. All Rights Reserved. 8© 2019 BlackBerry. All Rights Reserved. 8Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 88
• As little emulated hardware as possible● It’s not just about being lazy: emulation is slow
• Customers either pass-through hardware or use VirtIO devices
Virtual Machines
© 2018 BlackBerry. All Rights Reserved. 9© 2019 BlackBerry. All Rights Reserved. 9Confidential – Internal Use Only
Goals
© 2018 BlackBerry. All Rights Reserved. 10© 2019 BlackBerry. All Rights Reserved. 10Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1010
• Emulation gaps● Comes from focusing on a very limited number of guests
• Finding actual bugs● Same cause, but it means better coverage of the existing code
• QNX is cool, hypervisors are cool, BSDs are cool.
Why am I doing this?
© 2018 BlackBerry. All Rights Reserved. 11© 2019 BlackBerry. All Rights Reserved. 11Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1111
• Get a multi-user prompt● I’ll settle for some dmesg excitement
• Find bugs in our Hypervisor, possibly in the guests● It’s all a matter of point of view!
• Look at performance if time permits
Objectives with BSD guests
© 2018 BlackBerry. All Rights Reserved. 12© 2019 BlackBerry. All Rights Reserved. 12Confidential – Internal Use Only
Guest Experience
© 2018 BlackBerry. All Rights Reserved. 13© 2019 BlackBerry. All Rights Reserved. 13Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1313
• The easiest to start with:● Very familiar with the x86 Hypervisor code● Quite familiar with NetBSD internals● build.sh – other BSDs, take note!
Booting NetBSD/amd64
© 2018 BlackBerry. All Rights Reserved. 14© 2019 BlackBerry. All Rights Reserved. 14Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1414
• Not just about NetBSD/amd64
• While NetBSD/i386 can be booted through Multiboot, NetBSD/amd64 has it own protocol
• Frustrating because:● It starts in protected mode● It wants the same data it would get through Multiboot, just in a slightly
different format
The bootloader problem
© 2018 BlackBerry. All Rights Reserved. 15© 2019 BlackBerry. All Rights Reserved. 15Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1515
• A module to load NetBSD kernels
• Emulation of MSR 0x10 (IA32_TIME_STAMP_COUNTER)● In our Hypervisor, MSRs that are not passed-through or emulated
result in an exception● This is an emulation gap
Work needed to get NetBSD/amd64 booting (1/3)
© 2018 BlackBerry. All Rights Reserved. 16© 2019 BlackBerry. All Rights Reserved. 16Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1616
• NetBSD doesn’t handle not having MTRRs● MTRRs are not emulated because it’s a lot of work and not really
used these days● However, the Linux kernel ties the support of MTRRs to its ability
to use the PAT
• Handling REP OUTS correctly● Long debugging session
Work needed to get NetBSD/amd64 booting (2/3)
© 2018 BlackBerry. All Rights Reserved. 17© 2019 BlackBerry. All Rights Reserved. 17Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1717
• A small workaround for the virtio-block driver● size_max is a 32-bit unsigned
• The DSDT always exposes a PCI bridge, but it’s only created if there is a PCI device in the virtual machine
● NetBSD would crash trying to access the non-existent PCI host controller
Work needed to get NetBSD/amd64 booting (3/3)
© 2018 BlackBerry. All Rights Reserved. 18© 2019 BlackBerry. All Rights Reserved. 18Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1818
• It wants to start in long mode, and has its own set of data structures to pass memory information and command line
• Sorry, FreeBSD folks, I skip● Already done the data structure spelunking for NetBSD, and I’d
have to write even more code to start a guest in long mode● Neither are terribly difficult, and the long mode environment could
be a time saver booting Linux in the future
• Also, having to install FreeBSD somewhere just so I can recompile a kernel is frustrating
Booting FreeBSD/x86_64
© 2018 BlackBerry. All Rights Reserved. 19© 2019 BlackBerry. All Rights Reserved. 19Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 1919
• OpenBSD was interesting in addition to NetBSD because it has its own implementation of the ACPI OSPM
• The boot protocol to get the OpenBSD kernel running is very similar to NetBSD, but everything is just slightly different.
• The only problem was comprobe1() which compares the value read from IIR to 0x38 and thinks that it means the receive buffer is not empty.
● I haven’t seen any 8250 documentation that would indicate that● 0x20 actually means a 64-byte FIFO on the 16750● 0x38 was there in revision 1.1 in NetBSD, I didn’t go any further in
history
Booting OpenBSD/amd64
© 2018 BlackBerry. All Rights Reserved. 20© 2019 BlackBerry. All Rights Reserved. 20Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 2020
• It seems the FreeBSD/aarch64 kernel wants an EFI loader
• I couldn’t figure out an easy way to get an EFI environment● The Linaro EFI build for the Foundation Model expects to have code
running at EL3, which is not something we ever consider emulating● U-Boot might be an option to investigate more in the future
• Another skip, but I’m not biased against FreeBSD, okay?
Booting FreeBSD/aarch64
© 2018 BlackBerry. All Rights Reserved. 21© 2019 BlackBerry. All Rights Reserved. 21Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 2121
• NetBSD/aarch64’s locore.S is easier to read, it can work with only the pointer to the FDT in x0
• The QNX Hypervisor emulates (or pretends to) the Foundation Model, but there’s no code handling it among all the ARM_PLATFORM definitions
• NetBSD’s pl011 driver disables the transmit FIFO, which exposed a bug in our emulation
Booting NetBSD/aarch64
© 2018 BlackBerry. All Rights Reserved. 22© 2019 BlackBerry. All Rights Reserved. 22Confidential – Internal Use Only
What’s next?
© 2018 BlackBerry. All Rights Reserved. 23© 2019 BlackBerry. All Rights Reserved. 23Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 2323
• NetBSD/amd64 seems sluggish during boot● i8254_delay() is expensive, especially because of the way time flows
in our Hypervisor
• There’s a long pause in both OpenBSD and NetBSD on amd64● It seems to be because of the minimal emulation of the 8042
Performance considerations
© 2018 BlackBerry. All Rights Reserved. 24© 2019 BlackBerry. All Rights Reserved. 24Confidential – Internal Use Only © 2018 BlackBerry. All Rights Reserved. 2424
• Get a FreeBSD to boot
• Create guest images for our test team● The number of problems found proved it was a worthy experiment
Maybe for next time
© 2018 BlackBerry. All Rights Reserved. 25© 2019 BlackBerry. All Rights Reserved. 25Confidential – Internal Use Only
©2019 BlackBerry Limited. Trademarks, including but not limited to BLACKBERRY, BBM, BES, EMBLEM Design, ATHOC and SECUSMART are the trademarks or registered trademarks of BlackBerry Limited, its subsidiaries and/or affiliates, used under license, and the exclusive rights to such trademarks are expressly reserved. All other trademarks are the property of their respective owners. iPad and iPhone are trademarks of Apple, Inc., registered in the U.S. and other countries. Android is a trademark of Google Inc. The Android robot is reproduced or modified from work created and shared by Google and used according to terms described in the Creative Commons 3.0 Attribution License. Windows is a registered trademark of Microsoft Corporation in the United States and other countries. BlackBerry is a trademark of Research In Motion Limited.
Questions