The Android graphics path, in depth

44
The Android graphics path in depth The Android graphics path 1 Copyright ©2011-2014, 2net Limited

description

Slides from Android Builder's Summit 2014 in San Jose, CA In this talk I describe the internal workings of the Android graphics stack from the Application layer down through the stack to pixels on the screen. It is a fairly complex journey, taking in two different 2D rendering engines, applications calling OpenGL ES directory, passing buffers on to the system compositor, SurfaceFlinger, and then down to the display controller or frame buffer.

Transcript of The Android graphics path, in depth

Page 1: The Android graphics path, in depth

The Android graphics pathin depth

The Android graphics path 1 Copyright ©2011-2014, 2net Limited

Page 2: The Android graphics path, in depth

License

These slides are available under a Creative Commons Attribution-ShareAlike 3.0license. You can read the full text of the license herehttp://creativecommons.org/licenses/by-sa/3.0/legalcode

You are free to

• copy, distribute, display, and perform the work

• make derivative works

• make commercial use of the work

Under the following conditions

• Attribution: you must give the original author credit

• Share Alike: if you alter, transform, or build upon this work, you may distributethe resulting work only under a license identical to this one (i.e. include thispage exactly as it is)

• For any reuse or distribution, you must make clear to others the license terms ofthis work

The orginals are at http://2net.co.uk/slides/android-graphics-abs-2014.pdfThe Android graphics path 2 Copyright ©2011-2014, 2net Limited

Page 3: The Android graphics path, in depth

About Chris Simmonds

• Consultant and trainer• Working with embedded Linux since 1999• Android since 2009• Speaker at many conferences and

workshops

"Looking after the Inner Penguin" blog at http://2net.co.uk/

https://uk.linkedin.com/in/chrisdsimmonds/

https://google.com/+chrissimmonds

The Android graphics path 3 Copyright ©2011-2014, 2net Limited

Page 4: The Android graphics path, in depth

Overview

• The Android graphics stack changed a lot in JellyBean as a result of project Butter

• This presentation describes the current (JB) graphicsstack from top to bottom

• Main topics covered

• The application layer

• SurfaceFlinger, interfaces and buffer queues

• The hardware modules HWComposer and Gralloc

• OpenGL ES and EGL

The Android graphics path 4 Copyright ©2011-2014, 2net Limited

Page 5: The Android graphics path, in depth

The big picture

SurfaceFlinger

Activity

OpenGL vendorlibraries

OpenGL ES

GPU driver

HWComposer

ION

Gralloc alloc

FB driver

Gralloc FB

Androidframework

Vendor HALlibrary

Kerneldriver

Activity

ActivityManager

WindowManager

BufferQueue

The Android graphics path 5 Copyright ©2011-2014, 2net Limited

Page 6: The Android graphics path, in depth

Inception of a pixel

• Everything begins when an activity draws to a surface

• 2D applications can use

• drawing functions in Canvas to write to a Bitmap:android.graphics.Canvas.drawRect(), drawText(), etc

• descendants of the View class to draw objects suchas buttons and lists

• a custom View class to implement your ownappearance and behaviour

• In all cases the drawing is rendered to a Surfacewhich contains a GraphicBuffer

The Android graphics path 6 Copyright ©2011-2014, 2net Limited

Page 7: The Android graphics path, in depth

2D rendering path

Activity

Canvas

Surface

SkiaOpenGL ES

Vendor Open GL

GPU driver

Activity

HWUI

The Android graphics path 7 Copyright ©2011-2014, 2net Limited

Page 8: The Android graphics path, in depth

Skia and hwui

• For 2D drawing there are two rendering paths

• hwui: (libwhui.so) hardware accelerated usingOpenGL ES 2.0

• skia: (libskia.so) software render engine

• hwui is the default

• Hardware rendering can be disabled per view,window, activity, application or for the whole device

• Maybe for comparability reasons: hwui producesresults different to skia in some (rare) cases

The Android graphics path 8 Copyright ©2011-2014, 2net Limited

Page 9: The Android graphics path, in depth

3D rendering path

• An activity can instead create a GLSurfaceView anduse OpenGL ES bindings for Java (theandroid.opengl.* classes)

• Using either the vendor GPU driver (which mustsupport OpenGL ES 2.0 and optinally 3.0)

• Or as a fall-back, using PixelFlinger, a software GPUthat implements OpenGL ES 1.0 only

• Once again, the drawing is rendered to a Surface

The Android graphics path 9 Copyright ©2011-2014, 2net Limited

Page 10: The Android graphics path, in depth

3D rendering path

Activity

Surface

OpenGL ES

Vendor Open GL

GPU driver

Activity

Android GL

PixelFlinger

The Android graphics path 10 Copyright ©2011-2014, 2net Limited

Page 11: The Android graphics path, in depth

Composition

SurfaceFlinger

Wallpaper

Launcher

Navigationbar

Statusbar

The Android graphics path 11 Copyright ©2011-2014, 2net Limited

Page 12: The Android graphics path, in depth

SurfaceFlingerframeworks/native/services/surfaceflinger

• A high-priority native (C++) daemon, started by initwith UID=system

• Services connections from activities via Binderinterface ISurfaceComposer

• Receives activity status from Activity Manager

• Receives window status (visibility, Z-order) fromWindow Manager

• Composits multiple Surfaces into a single image

• Passes image to one or more displays

• Manages buffer allocation, synchronisation

The Android graphics path 12 Copyright ©2011-2014, 2net Limited

Page 13: The Android graphics path, in depth

SurfaceFlinger binder interfaces

SurfaceFlinger

Client

ISurfaceComposer ISurfaceComposerClient IDisplayEventConnection

e.g. activity

createConnection()createDisplayEventConnection()

createSurface()destroySurface()

getDataChannel()

The Android graphics path 13 Copyright ©2011-2014, 2net Limited

Page 14: The Android graphics path, in depth

ISurfaceComposer

• ISurfaceComposer

• Clients use this interface to set up a connection withSurfaceFlinger

• Client begins by calling createConnection() whichspawns an ISurfaceComposerClient

• Client calls createGraphicBufferAlloc() to create aninstance of IGraphicBufferAlloc (discussed later)

• Client calls createDisplayEventConnection() to createan instance of IDisplayEventConnection

• Other methods include captureScreen() andsetTransactionState()

The Android graphics path 14 Copyright ©2011-2014, 2net Limited

Page 15: The Android graphics path, in depth

ISurfaceComposerClient

• ISurfaceComposerClient

• This interface has two methods:

• createSurface() asks SufraceFlinger to create a newSurface

• destroySurface() destroys a Surface

The Android graphics path 15 Copyright ©2011-2014, 2net Limited

Page 16: The Android graphics path, in depth

IDisplayEventConnection

• IDisplayEventConnection

• This interface passes vsync event information fromSurfaceFlinger to the client

• setVsyncRate() sets the vsync event delivery rate:value of 1 returns all events, 0 returns none

• requestNextVsync() schedules the next vsync event:has no effect if the vsync rate is non zero

• getDataChannel() returns a BitTube which can beused to receive events

The Android graphics path 16 Copyright ©2011-2014, 2net Limited

Page 17: The Android graphics path, in depth

BufferQueueframeworks/native/include/gui/BufferQueue.h

• Mechanism for passing GraphicBuffers toSurfaceFlinger

• Contains an array of between 2 and 32GraphicBuffers

• Uses interface IGraphicBufferAlloc to allocate buffers(see later)

• Provides two Binder interfaces

• IGraphicBufferProducer for the client (Activity)

• IGraphicBufferConsumer for the consumer(SurfaceFlinger)

• Buffers cycle between producer and consumer

The Android graphics path 17 Copyright ©2011-2014, 2net Limited

Page 18: The Android graphics path, in depth

BufferQueue state diagram

FREE

QUEUED

DEQUEUEDACQUIRED

IGraphicBufferProducer::dequeueBuffer()

IGraphicBufferConsumer::releaseBuffer()

IGraphicBufferProducer::queueBuffer()

IGraphicBufferConsumer::acquireBuffer()

IGraphicBufferProducer::cancelBuffer()

The Android graphics path 18 Copyright ©2011-2014, 2net Limited

Page 19: The Android graphics path, in depth

BufferQueue

• Default number of buffer slots since JB is 3(previously 2)

• In JB you can compile Layer.cpp withTARGET_DISABLE_TRIPLE_BUFFERING to return to 2 slots

• Call setBufferCount() to change the number of slots

• BufferQueue operates in two modes:

• Synchronous: client blocks until there is a free slot

• Asynchronous: queueBuffer() discards any existingbuffers in QUEUED state so the queue only holds themost recent frame

The Android graphics path 19 Copyright ©2011-2014, 2net Limited

Page 20: The Android graphics path, in depth

GraphicBuffer

frameworks/native/include/ui/GraphicBuffer.h

• Represents a buffer, wraps ANativeWindowBuffer

• Attributes including width, height, format, usageinherited from ANativeWindowBuffer

The Android graphics path 20 Copyright ©2011-2014, 2net Limited

Page 21: The Android graphics path, in depth

Composition

• On a vsync event, SurfaceFlinger callshandleMessageRefresh() which goes through acomposition cycle:

• preComposition(): sort layers by Z order and callonPreComposition() for each

• doComposition(): loop through displays: if there is adirty region, mark it to be drawn then callpostFameBuffer() to do the drawing

• postComposition(): loop through layers in Z order andcall onPostComposition()

The Android graphics path 21 Copyright ©2011-2014, 2net Limited

Page 22: The Android graphics path, in depth

Layerframeworks/native/services/surfaceflinger/Layer.h

• Each Layer has

• Z order

• Alpha value from 0 to 255

• visibleRegion

• crop region

• transformation: rotate 0, 90, 180, 270: flip H, V: scale

• SurfaceFlinger composits the layers using

• HWComposer, if it supports the operation

• Fall back to the GPU, via OpenGL ES (version 1.0only, for historical reasons)

The Android graphics path 22 Copyright ©2011-2014, 2net Limited

Page 23: The Android graphics path, in depth

HWComposerhardware/libhardware/include/hardware/hwcomposer.h

• HWComposer is a vendor-supplied library, at run-timein /system/lib/hw/hwcomposer.[product name].so

• Optional: in all cases there are fall-backs if HWC isabsent

• HWC does several different things

• sync framework (vsync callback)

• modesetting, display hotplug (e.g. hdmi)

• compositing layers together using features of thedisplay controller

• displaying frames on the screen

The Android graphics path 23 Copyright ©2011-2014, 2net Limited

Page 24: The Android graphics path, in depth

prepare() and set()

• SurfaceFlinger calls HWComposer in two stages

• prepare()

• Passes a list of layers

• For each layer, HWComposer returns

• HWC_FRAMEBUFFER: SurfaceFlinger should write thislayer (using OpenGL)

• HWC_OVERLAY: will be composed by HWComposer

• set()

• Passes the list of layers for HWComposer to handle

• set() is used in place of eglSwapBuffers()

The Android graphics path 24 Copyright ©2011-2014, 2net Limited

Page 25: The Android graphics path, in depth

vsync

• Since JB 4.1 SurfaceFlinger is synchronised to a60Hz (16.7ms period) vsync event

• If HWComposer present, it is responsible for vsync

• Usually using an interrupt from the display: if no h/wtrigger, fake in software

• vsync() is a callback registered with HWComposer

• Each callback includes a display identifier and atimestamp (in ns)

• If no HWComposer, SurfaceFlinger uses 16mstimeout in s/w

The Android graphics path 25 Copyright ©2011-2014, 2net Limited

Page 26: The Android graphics path, in depth

Displays

• HWComposer defines three display typesHWC_DISPLAY_PRIMARY e.g. built-in LCD screen

HWC_DISPLAY_EXTERNAL e.g. HDMI, WiDi

HWC_DISPLAY_VIRTUAL not a real display

• For each display there is an instance ofDisplayDevice in SurfaceFlinger

The Android graphics path 26 Copyright ©2011-2014, 2net Limited

Page 27: The Android graphics path, in depth

IGraphicBufferAlloc and friendsframeworks/native/include/gui/IGraphicBufferAlloc.h

• Binder interface used by SurfaceFlinger to allocatebuffers

• Has one function createGraphicBuffer

• Implemented by class GraphicBufferAllocator, whichwraps the ANativeWindowBuffer class

• Uses Gralloc.alloc to the the actual allocation

• Underlying buffer is referenced by a buffer_handle_twhich is a file descriptor (returned by gralloc alloc)

• Binder can pass open file descriptors from process toprocess

• Access buffer data using mmapThe Android graphics path 27 Copyright ©2011-2014, 2net Limited

Page 28: The Android graphics path, in depth

Buffer usage and pixel format

frameworks/native/include/ui/GraphicBuffer.h

USAGE_HW_TEXTURE OpenGL ES texture

USAGE_HW_RENDER OpenGL ES render target

USAGE_HW_2D 2D hardware blitter

USAGE_HW_COMPOSER used by the HWComposer HAL

USAGE_HW_VIDEO_ENCODER HW video encoder

frameworks/native/include/ui/PixelFormat.h

PIXEL_FORMAT_RGBA_8888 4x8-bit RGBA

PIXEL_FORMAT_RGBX_8888 4x8-bit RGB0

PIXEL_FORMAT_RGB_888 3x8-bit RGB

PIXEL_FORMAT_RGB_565 16-bit RGB

PIXEL_FORMAT_BGRA_8888 4x8-bit BGRA

The Android graphics path 28 Copyright ©2011-2014, 2net Limited

Page 29: The Android graphics path, in depth

Gralloc

hardware/libhardware/include/hardware/gralloc.h

• Gralloc is a vendor-supplied library, at run-time in/system/lib/hw/gralloc.[product name].so

• Does two things

• gralloc alloc: allocates graphic buffers

• gralloc framebuffer: interface to Linux framebufferdevice, e.g. /dev/graphics/fb0

• gralloc alloc allocates all graphic buffers using akernel memory manager, typically ION

• Selects appropriate ION heap based on the bufferusage flags

The Android graphics path 29 Copyright ©2011-2014, 2net Limited

Page 30: The Android graphics path, in depth

OpenGL ES

• The Khronos OpenGL ES and EGL APIs areimplemented in these libraries

• /system/lib/libEGL.so

• /system/lib/libGLESv1_CM.so

• /system/lib/libGLESv2.so

• /system/lib/libGLESv3.so (optional from JB 4.3onwards: actually a symlink to libGLESv2.so)

• In most cases they simply call down to thevendor-supplied libraries in /system/lib/egl

The Android graphics path 30 Copyright ©2011-2014, 2net Limited

Page 31: The Android graphics path, in depth

EGL• EGL is the Khronos Native Platform Graphics

Interface

• Rendering operations are executed in an EGLContext

• In most cases the EGLContext is based on thedefault display

• The mapping from the EGL generic display type isdone inframeworks/native/opengl/include/EGL/eglplatform.h

typedef struct ANativeWindow* EGLNativeWindowType;

• EGLNativeWindowType is defined insystem/core/include/system/window.h

The Android graphics path 31 Copyright ©2011-2014, 2net Limited

Page 32: The Android graphics path, in depth

OpenGL vendor implementation

• The vendor OpenGL libraries form the interface to theGPU

• Responsible for

• creating display lists

• scheduling work for the GPU

• managing buffer synchronisation (typically usingfences, see background at the end)

• Usually there is a kernel driver which handles lowlevel memory management, DMA and interrupts

• The kernel interface is usually a group of ioctlfunctions

The Android graphics path 32 Copyright ©2011-2014, 2net Limited

Page 33: The Android graphics path, in depth

• Questions?

The Android graphics path 33 Copyright ©2011-2014, 2net Limited

Page 34: The Android graphics path, in depth

Background: fences

The Android graphics path 34 Copyright ©2011-2014, 2net Limited

Page 35: The Android graphics path, in depth

Buffer synchronisation

• There are many producers and consumers ofgraphics buffers

• Pre JB sync was implicit: buffer not released untiloperation complete

• Did not encourage parallel processing

• JB introduced explicit sync: each buffer has a syncobject called a fence

• Means a buffer can be passed to the next user beforeoperations complete

• The next user waits on the fence before accessingthe buffer contents

The Android graphics path 35 Copyright ©2011-2014, 2net Limited

Page 36: The Android graphics path, in depth

Synchronisation using fences

• Represented by file handles: can be passed betweenapplications in binder messages

• Can also be passed from applications to drivers

• Each device driver (display, camera, video codec...)has its own timeline

• A fence may have synchronisation points on multipletimelines

• Allows buffers to be passed between multiple devices

The Android graphics path 36 Copyright ©2011-2014, 2net Limited

Page 37: The Android graphics path, in depth

Timeline and sync point

• Timeline

• Per-device (display, GPU, camera, ...)

• Monotonically increasing 32-bit value

• Incremented after each event (essentially it is a countof the jobs processed by the device)

• Sync point

• A point on a timeline

• Becomes signalled when the timeline passes it

The Android graphics path 37 Copyright ©2011-2014, 2net Limited

Page 38: The Android graphics path, in depth

Fence

• Fence

• A collection of one or more sync points, possibly fromdifferent timelines

• Represented by a file descriptor so an application canwait using poll()

• Two fences can be merged to create a new fence thatdepends on all the sync points of the original pair

The Android graphics path 38 Copyright ©2011-2014, 2net Limited

Page 39: The Android graphics path, in depth

Fence: example

FBtimeline

GPUtimeline

Sync pt

Sync pt

SyncFence

The Android graphics path 39 Copyright ©2011-2014, 2net Limited

Page 40: The Android graphics path, in depth

Background: ION

The Android graphics path 40 Copyright ©2011-2014, 2net Limited

Page 41: The Android graphics path, in depth

Memory constraints

• Often necessary for a buffer to be accessed byhardware

• Example: graphics buffer and display controller orGPU

• Hardware may constrain memory access

• Example: hardware without IOMMU usually needsphysically contiguous memory

• To avoid copying, the memory must be allocated forthe most constrained device

The Android graphics path 41 Copyright ©2011-2014, 2net Limited

Page 42: The Android graphics path, in depth

ION

• Previous memory allocators include pmem(Qualcomm), cmem (TI), and nvmap (NVIDA)

• ION provides a unified interface for these needs

• Different allocation constraints

• Different caching requirements

• But the programmer still has to make the right choices

The Android graphics path 42 Copyright ©2011-2014, 2net Limited

Page 43: The Android graphics path, in depth

Types of heap

• ION_HEAP_TYPE_SYSTEM

• memory allocated via vmalloc

• ION_HEAP_TYPE_SYSTEM_CONTIG

• memory allocated via kmalloc

• ION_HEAP_TYPE_CARVEOUT

• memory allocated from a pre reserved carveout heap

• allocations are physically contiguous

The Android graphics path 43 Copyright ©2011-2014, 2net Limited

Page 44: The Android graphics path, in depth

Heap flags

• ION_FLAG_CACHED

• mappings of this buffer should be cached, ION will docache maintenance when the buffer is mapped forDMA

• ION_FLAG_CACHED_NEEDS_SYNC

• Cache must be managed manually, e.g. usingION_IOC_SYNC

The Android graphics path 44 Copyright ©2011-2014, 2net Limited