QEMU Assisted Jails for Ports and

PackagesSean Bruno, sbruno@freebsd.org

Overview and Presentation Goals

● Kernel Components (11-Current)○ Image Activators

● Userland Components (11-Current)○ binmiscctl(8)○ qemu-static-user○ poudriere-devel

● Nullfs and Chroot○ Treat the environments as normal jails.

Contributors and Generally Awesome People

Stacey Son - binmiscctl(8) and imgact_binmisc(4) authorJuergen Lock - QEMU maintainer for BSD, many, many patchesEd Maste - QEMU patches and cat herderPeter Wemm - Sigtramp patch, because wow.Alexander Kabaev - QEMU patchesAdrian Chadd - Ignored my pleading for help with kern_imgact.cBaptiste Daroussin - Inflicting me with a ports commit bit

Contributors and Generally Awesome People

Dimitry Andric - CLANG Help and UpdatesAndrew Turner - GCC and Ports PatchesMikael Urankar - Mysql PatchesBryan Drewery - Taking me on a journey through poudriereWarner Losh - Created the native-xtools target (doubled speed)Ian Lapore - ARMv6 Assembly HelpBrooks Davis - Inspiration and initial guidance

Kernel Components (11-Current)

imgact_binmisc.ko● Kernel Hook to redirect execution● Examines ELF Header● If match on preconfigured setting, modifes

ARGV[0] with emulator helper

Kernel Components - Continued

binmiscctl(8) will load and configure settings, e.g.binmiscctl add armv6 --interpreter "/usr/local/bin/qemu-arm" --magic "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00"--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff"--size 20 --set-enabledbinmiscctl add mips --interpreter "/usr/local/bin/qemu-mips" --magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08" --mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" --size 20 --set-enabled

Kernel Components - Continued

binmisctl listname: mipsinterpreter: /usr/local/bin/qemu-mipsflags: ENABLED USE_MASKmagic size: 20magic offset: 0magic: 0x7f 0x45 0x4c 0x46 0x01 0x02 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x02 0x00 0x08mask: 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0x00 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xfe 0xff 0xff

Userland Components (11-Current)

QEMU● Use emulators/qemu-static-user● Installs statically linked binaries● Emulation is a journey not a destination● IOCTL emulation is still in progress

Userland Components - Continued

Poudriere-devel● Easiest way to get started building jails● Knows how to to understand binmiscctl(8)● Knows to copy QEMU into jails● Creates clean backup, in case of accident● Use ZFS, save yourself some pain

Build the jail -- poudriere-devel

● poudriere jail -c -j 11armv632 -m svn -v head -a arm.armv6 -x

● poudriere jail -c -j 11mips32 -m svn -v head -a mips.mips -x

● poudirere jail -c -j 11mips64 -m svn -v head -a mips.mips64 -x

● poudriere ports -c -m svn (so you have something to build)

Configure the jail and mount

poudriere will create:zroot/poudriere/jails/11armv632 112176079 1096098 111079980 1%/usr/local/poudriere/jails/11armv632

zroot/poudriere/ports/default 118616467 7536487 111079980 6% /usr/local/poudriere/ports/default

Manually mount devfs/ports:mount -t devfs devfs <path to jail/devmount -t nullfs /usr/local/poudriere/ports/default <path to jail>/usr/ports

Chroot and enjoyroot@dirty.ysv:/home/sbruno/fbsd_head # uname -aFreeBSD dirty.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r271414: Thu Sep 11 02:48:22 UTC 2014 sbruno@dirty.ysv.freebsd.org:/usr/obj/usr/src/sys/DIRTY amd64

root@dirty.ysv:~ # chroot /usr/local/poudriere/jails/11armv632

root@dirty:/ # uname -aFreeBSD dirty.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r271414: Thu Sep 11 02:48:22 UTC 2014 sbruno@dirty.ysv.freebsd.org:/usr/obj/usr/src/sys/DIRTY arm

Poudriere Demo

● poudriere bulk -a -j 11armv632● nginx rendering of status● sharp edges remain, signal handling

Building and testing - DEMO

● chroot and build, its magic● needs better documentation● man page updates● binmiscctl(8) syntax is precise and you will

always copy/paste it from somewhere

Future Development

● CLANG for MIPS will make more ports build● AARCH64 (ARM64) support will start soon,

or send me patches for QEMU now.● IOCTL debug handling, human readable

format

Questions?

There’s plenty of things that I’ve probably skipped, what do you want to do with this feature?