Skia & Freetype - Android 2D Graphics Essentials
-
Upload
kyungmin-lee -
Category
Technology
-
view
3.607 -
download
4
description
Transcript of Skia & Freetype - Android 2D Graphics Essentials
Skia & FreeType Android 2D Graphics Essentials
Kyungmin Lee
Software Platform Lab., LG Electronics [email protected]
The 10th Kandroid Conference
Goal
The 9th Kandroid Conference
S/W 플랫폼에 있어 GUI(Graphical User Interface) 시스템은 해당 플랫폼의 성능을 결정짓는 매우 중요한 요소이다. Android 역시 새로운 버전을 발표할 때마다 GUI 시스템의 성능을 꾸준히 개선하여 최근에 릴리즈된 Jelly Bean에 와서는 iOS에 필적할만한 성능을 보여주고 있다. 이번 세션에서는 이벤트 처리 기술과 더불어 GUI 시스템의 한 축을 이루고 있는 2D Graphics 기술을 Android에서 어떻게 활용하고 발전시켜 왔는지에 대해 살펴본다. 1. Skia: 2D Graphics Library
Skia in Android Skia API Overview Skia Rendering Pipeline Skia Backends: Raster, OpenGL, PDF, ...
2. FreeType: Font Engine FreeType in Android FreeType API Overview Font Rasterization: Scan-conversion, Hinting, …
3. Android 2D Graphics Essentials Evolution of Drawing Models:
GPUI (UI on the GPU), Display List, Display List Properties, View Layers, … Android 2D Graphics Architecture
(From Conference Program Overview)
The 10th Kandroid Conference
http://code.google.com/p/skia/ Skia is Greek for “shadow”
The Skia Graphics Engine is a compact open source graphics library written in C++. It was originally developed by Skia Inc., which was subsequently acquired by Google in 2005, who then released the software as open source licensed under the New BSD License free software license. Now known as skia, it is currently used in Mozilla Firefox, Google Chrome, Chrome OS, Chromium OS, and Android; and the Skia library is present on the BlackBerry PlayBook though the extent of its usage is unclear. Skia's drawing has back-ends for a standard CPU-based software rasterizer, PDF, and OpenGL
(From Wikipedia)
Skia is a complete 2D graphic library for drawing Text, Geometries, and Images. • 3x3 matrices w/ perspective • antialiasing, transparency, filters • shaders, xfermodes, maskfilters, patheffects • subpixel text
Device backends for Skia currently include: • Raster • OpenGL • PDF • XPS • Picture (for recording and then playing back into another Canvas)
(From Project’s Home)
Mike Reed
Skia in Android
The 10th Kandroid Conference
Removed
Added
2006-11 r3
Skia (svn) Skia
in Android
2008-11 r4 2008-10
r2
r100
r1000
2009-03
2011-03
2006-09 r1 2006-09
r2000 2011-07
r3000 2012-01
r4000 2012-05
r5000 2012-08
r6022 2012-10 ~700
total commit #
…
…
???
Skia API Overview
The 10th Kandroid Conference
A Canvas encapsulates all of the state about drawing into a device (bitmap). This includes a reference to the device itself, and a stack of matrix/clip values. For any given draw call (e.g. drawRect), the geometry of the object being drawn is transformed by the concatenation of all the matrices in the stack. The transformed geometry is clipped by the intersection of all of the clips in the stack. While the Canvas holds the state of the drawing device, the state (style) of the object being drawn is held by the Paint, which is provided as a parameter to each of the draw() methods. The Paint holds attributes such as color, typeface, textSize, strokeWidth, shader (e.g. gradients, patterns), etc. Drawing basic primitives include rectangles, rounded rectangles, ovals, circles, arcs, paths, lines, text, bitmaps and sprites. Paths allow for the creation of more advanced shapes. Each path can be made up of multiple contours (or continuous sections), each consisting of linear, quadratic, and cubic segments
Canvas
Drawing
Primitives
Bitmap
Paint
Skia API Overview
The 10th Kandroid Conference
SkRect rect; rect.set(0, 0, 150, 120); canvas->drawRect(rect, shapePaint); canvas->drawRoundRect(rect, 15, 15, shapePaint); canvas->drawOval(rect, shapePaint); canvas->drawCircle(60, 60, 60, shapePaint); // Arc without a wedge canvas->drawArc(rect, 0, 255, false, shapePaint); // Arc with a wedge from the center canvas->drawArc(rect, 0, 255, true, shapePaint); canvas->drawLine(0, 0, 150, 120, shapePaint); const char str[] = "Hello World!"; canvas->drawText(str, strlen(str), 75, 145, textPaint); // Load a bitmap from an image and draw it only if the load succeeds SkBitmap bitmap; if (SkImageDecoder::DecodeFile("app/native/icon.png", bitmap)) { canvas->drawBitmap(*bitmap, 0, 0, NULL); }
SkPath path; path.moveTo(50, 50); // Specify endpoint using absolute coordinates path.lineTo(100, 100); // Specify endpoint using a relative offset from the last point path.rLineTo(50, 50); // Specify endpoint using absolute coordinates path.quadTo(120, 125, 150, 150); // Specify endpoint using a relative offset from the last point path.rQuadTo(20, 25, 50, 50); // Specify endpoint using absolute coordinates path.cubicTo(175, 175, 140, 120, 200, 200); // Specify endpoint using a relative offset from the last point path.rQuadTo(25, 25, -10, -30, 50, 50); canvas->drawPath(path, shapePaint);
Skia Rendering Pipeline
The 10th Kandroid Conference
SkCanvas
draw*(…, SkPaint)
Path Generation
Rasterization Shading
http://www.xenomachina.com/2011/05/androids-2d-canvas-rendering-pipeline.html
Initial Path
Apply
Path Effects
Apply
Stroke Styles
Path
Scan
Conversion
Rasterizer Initial Mask
Apply
Mask Filters
Mask
Apply
Shaders
Apply
Color Filters
Initial
SRC Image
SRC Image
DST Image
Intermediate
Image
Modified
DST Image
Apply
XferMode Blending
Transfer
Skia Backends
The 10th Kandroid Conference
To render with software Skia 1) Create a native window and then 2) Wrap a pointer to its buffer as an SkBitmap 3) Initialize an SkCanvas with the bitmap
To render with hardware-accelerated Skia 1) Create a GLES2 window or framebuffer and 2) Create the appropriate GrContext,
SkGpuDevice, and SkGpuCanvas
SkCanvas SkDevice
SkGpuCanvas SkGpuDevice GrContext
SkBitmap
FreeType?
The 10th Kandroid Conference
http://code.google.com/p/skia/ A Free, High-Quality, and Portable Font Engine
FreeType is a software library written in C that implements a font rasterization engine. It is used to render text on to bitmaps and provides support for other font-related operations.
FreeType 2 is a software font engine that is designed to be small, efficient, highly customizable, and portable while capable of producing high-quality output (glyph images). It can be used in graphics libraries, display servers, font conversion tools, text image generation tools, and many other products as well. Note that FreeType 2 is a font service and doesn't provide APIs to perform higher-level features like text layout or graphics processing (e.g., colored text rendering, ‘hollowing’, etc.). However, it greatly simplifies these tasks by providing a simple, easy to use, and uniform interface to access the content of font files. By default, FreeType 2 supports the following font formats: TrueType (and collections), Type 1, CID-keyed Type 1, CFF, OpenType (both , TrueType and CFF variants), SFNT-based bitmap, X11 PCF, Windows FNT, BDF (including anti-aliased ones), PFR, Type 42 (limited support)
David Turner
Robert Wilhelm
Werner Lemberg
(From Project’s Home)
(From Wikipedia)
FreeType in Android
The 10th Kandroid Conference
Removed
Added
2007-07 2.3.5
FreeType2 FreeType2
in Android
2008-06 2.3.6
2008-10
???
…
2009-06
2.3.9
2010-03
2.3.12
2010-09
2.4.2
2011-01
2.4.4
2011-08 2.4.6
2011-11 2.4.7
2011-12
2.4.8
2012-03
2.4.9
2009-03
2010-02
2010-08
2010-11
2010-07
2010-10
2010-11
2012-03
2.4.10 2012-06
~1100 ~50
2000-11 2.0.0 …
commit #
since 2007-07
FreeType API
The 10th Kandroid Conference
Glyph Outline (Lines + Curves)
Quadratic Bé zier Curve Cubic Bé zier Curve
FreeType API
The 10th Kandroid Conference
FT_Library library; FT_Face face; FT_GlyphSlot slot; FT_Error error; char* filename, text; int pen_x, pen_y, n, num_chars; ... /* initilization code */ error = FT_Init_FreeType( &library ); if ( error ) ... /* error handling code */ error = FT_New_Face( library, filename, 0, &face ); if ( error ) ... /* error handling code */ error = FT_Set_Char_Size( face, 50 * 64, 0, 100, 0 ); if ( error ) ... /* error handling code */ slot = face->glyph; for ( n = 0; n < num_chars; n++ ) { error = FT_Load_Char( face, text[n], FT_LOAD_DEFAULT ); if ( error ) continue; /* ignore errors */ error = FT_Render_Glyph( face, text[n], FT_RENDER_MODE_NORMAL ); if ( error ) continue; /* ignore errors */ draw_bitmap( &slot->bitmap, pen_x + slot->bitmap_left, pen_y - slot->bitmap_top ); pen.x += slot->advance.x >> 6; } show_image(); FT_Done_Face ( face ); FT_Done_FreeType( library );
Initialize the library
Load a font face
Set the character size (e.g., 50pt at 100dpi)
Load a glyph image into the slot
Convert the glyph’s outline to a bitmap
Draw the bitmap to a target surface
Release resources
1
2
3
4
5
6
7
Font Rasterization: Hinting
The 10th Kandroid Conference
Font hinting (also known as instructing) is the use of mathematical instructions to adjust the display of an outline font so that it lines up with a rasterized grid. At low screen resolutions, hinting is critical for producing a clear, legible text. It can be accompanied by antialiasing and (on liquid crystal displays) subpixel rendering for further clarity. Hints are usually created in a font editor during the typeface design process and embedded in the font. A font can be hinted either automatically (through processed algorithms based on the character outlines) or set manually. Most font editors are able to do automatic hinting, and this approach is suitable for many fonts. However, commercial fonts of the highest quality are often manually hinted to provide the sharpest appearance on computer displays. Verdana is one example of a font that contains a large amount of hinting data, much of which was accomplished manually by type engineer Tom Rickner.
(From Wikipedia)
How Views are Drawn in Android pre-3.0?
The 10th Kandroid Conference
http://developer.android.com/guide/topics/ui/how-android-draws.html
ViewRootImpl
performDraw()
View
onDraw(Canvas)
SkCanvasGlue
draw*(…, SkPaint)
SkCanvas
draw*(…, SkPaint)
SkDraw
draw*(..., SkPaint, ...)
SkDevice
draw*(..., SkPaint, ...)
SkRasterizer
rasterize(…)
View
draw(Canvas)
Canvas
draw*(…, Paint)
Android Framework (android.jar)
libandroid_runtime.so
libskia.so
SkBlitter
blit*(…)
CPU Rasterization + GPU Composition
Why New Drawing Models in Android post-3.0?
The 10th Kandroid Conference
GPUI
Display List
View Layers
Display List
Properties
Increased
Number of
Pixels
Source: Accelerated Android Rendering, Google I/O 2011
CPU Rasterization + GPU Composition GPU Rasterization + GPU Composition
…
GPUI – UI on the GPU (since 3.0)
The 10th Kandroid Conference
ViewRootImpl
performDraw()
View
onDraw(Canvas)
android_view_GLES20Canvas
_draw*(…, SkPaint)
OpenGLRenderer
draw*(…, SkPaint)
GLES2/gl2.h
glXXX(…)
View
draw(Canvas)
HardwareCanvas
draw*(…, Paint)
Android Framework (android.jar)
libandroid_runtime.so
libhwui.so
libGLESv2.so
android:handwareAccelerated="true" GPU Rasterization + GPU Composition
*GPUI turned on by default from Android 4.0
Display List in General
The 10th Kandroid Conference
A display list (or display file) is a series of graphics commands that define an output image. The image is created (rendered) by executing the commands. … A display list can represent both two- and three-dimensional scenes. Systems that make use of a display list to store the scene are called retained mode systems as opposed to immediate mode systems.
http://en.wikipedia.org/wiki/Display_list
http://developer.android.com/guide/topics/graphics/hardware-accel.html
Source: MSDN, http://msdn.microsoft.com/en-us/library/windows/desktop/ff684178(v=vs.85).aspx
(From Wikipedia)
Display List in Android (since 3.0)
The 10th Kandroid Conference
A display list records a series of graphics related operation and can replay them later. Display lists are usually built by recording operations on a android.graphics.Canvas. Replaying the operations from a display list avoids executing views drawing code on every frame, and is thus much more efficient.
DisplayList::Op[Non-Drawing]
<<enumeration>>
+Save
+Restore
+RestoreToCount
+SaveLayer
+SaveLayerAlpha
+Translate
+Rotate
+Scale
+Skew
+SetMatrix
+ConcatMatrix
+ClipRect
DisplayList::Op[Drawing]
<<enumeration>>
+DrawDisplayList
+DrawLayer
+DrawBitmap
+DrawBitmapMatrix
+DrawBitmapRect
+DrawBitmapData
+DrawBitmapMesh
+DrawPatch
+DrawColor
+DrawRect
+DrawRoundRect
+DrawCircle
+DrawOval
+DrawArc
+DrawPath
+DrawLines
+DrawPoints
+DrawText
+DrawTextOnPath
+DrawPosText
+ResetShader
+SetupShader
+ResetColorFilter
+SetupColorFilter
+ResetShadow
+SetupShadow
+ResetPaintFilter
+SetupPaintFilter
+DrawGLFunction
OpenGLRenderer
drawDisplayList(…)
DisplayList
replay(…)
DisplayListRenderer
draw*(…, SkPaint)
void* buffer
SkWriter32
write*(…)
SkReader32
read*()
GLES20RecordingCanvas
draw*(…, Paint)
android_view_
GLES20Canvas_draw*(…)
GLES20Canvas
drawDisplayList(…)
draw*(…, SkPaint)
GLES2/gl2.h
glXXX(…)
Display List Properties (since 4.1)
The 10th Kandroid Conference
Source: For Butter or Worse, Google I/O 2012
+ bool mClipChildren; + float mAlpha; + int mMultipliedAlpha;
+ bool mHasOverlappingRendering; + float mTranslationX, mTranslationY; + float mRotation, mRotationX, mRotationY; + float mScaleX, mScaleY;
+ float mPivotX, mPivotY; + float mCameraDistance; + int mLeft, mTop, mRight, mBottom; + int mWidth, mHeight;
+ int mPrevWidth, mPrevHeight; + bool mPivotExplicitlySet; + bool mMatrixDirty; + bool mMatrixIsIdentity;
+ uint32_t mMatrixFlags; + SkMatrix* mTransformMatrix; + Sk3DView* mTransformCamera; + SkMatrix* mTransformMatrix3D; + SkMatrix* mStaticMatrix;
+ SkMatrix* mAnimationMatrix; + bool mCaching;
Without DisplayList Properties
With DisplayList Properties
View Layers (since 3.0)
The 10th Kandroid Conference
View.setLayerType(int type, Paint p) [API Level 11]
type View is H/W-Accelerated
View is NOT H/W Accelerated
LAYER_TYPE_NONE • Rendered in H/W • Directly into surface
• Rendered in S/W • Directly into surface
LAYER_TYPE_HARDWARE • Rendered in H/W • Into hardware texture
• Rendered in S/W • Into bitmap
LAYER_TYPE_SOFTWARE • Rendered in S/W • Into bitmap
• Rendered in S/W • Into bitmap
Source: Accelerated Android Rendering, Google I/O 2011
*Measured when drawing a ListView with Android 3.0 on
a Motorola XOOM DisplayList::Op + DrawLayer + DrawBitmap
Layers = Off-screen Buffers or Caches
Android 2D Graphics Architecture
The 10th Kandroid Conference
Application (View)
Render Script
Canvas
OpenGL ES HWUI
Skia EGL
AGL HGL
Pixel Flinger GPU
Surface
SurfaceFlinger
HW Composer
OpenGL ES
EGL
HGL AGL
GPU Pixel Flinger
Overlay Frame Buffer
• HGL = Hardware OpenGL ES
• AGL = Android’s Software
OpenGL ES
Android 2D Graphics Libraries
What-How-Why-Why not?
The 10th Kandroid Conference
http://sangminpark.wordpress.com/2011/09/02/what-how-why-why-not/
What?
How?
Why? Why not?
When? Who?
Past Present Future
References
• How about some Android graphics true facts?, Dianne Hackborn, Nov. 2011,
https://plus.google.com/105051985738280261832/posts/2FXDCz8x93s
• Android 3.0 Hardware Acceleration, Romain Guy, Mar. 2011, http://android-
developers.blogspot.kr/2011/03/android-30-hardware-acceleration.html
• Android 4.0 Graphics and Animations, Romain Guy and Chet Haase, Nov. 2011,
http://android-developers.blogspot.kr/2011/11/android-40-graphics-and-animations.html
• Android Graphics, Animations and Tips & Tricks, Romain Guy and Chet Haase, Dec. 2010,
http://www.curious-creature.org/2010/12/02/android-graphics-animations-and-tips-tricks/
• Learning about Android Graphics Subsystem, Bhanu Chetlapalli, Apr. 2012,
http://developer.mips.com/2012/04/11/learning-about-android-graphics-subsystem/
• iOS vs. Android ICS: Hardware Accelerated Graphics Pipelines, Nathan de Vries, Nov.
2011, http://atnan.com/blog/2011/11/10/ios-vs-android-ics-hardware-accelerated-
graphics-pipelines/
• Android's 2D Canvas Rendering Pipeline, Laurence Gonsalves, May 2011,
http://www.xenomachina.com/2011/05/androids-2d-canvas-rendering-pipeline.html
• How FreeType's rasterizer work, David Turner, Feb. 2007, FreeType2 source/docs
• Android Graphics, Jim Huang, Sep. 2011, http://0xlab.org/~jserv/android-graphics.pdf
The 10th Kandroid Conference
The 10th Kandroid Conference