Trends and Standards for 3D Trends and Standards for 3D Graphics for HandsetsGraphics for Handsets

Christophe Quarre’Advanced System Technology, Shanghai Lab

STMicroelectronics

Christophe.quarre@st.com

State Key Lab of CAD&CG, Zhejiang University1-3 March 2005

2

Outline

Handhelds VS Desktops.... new industry

Challenges for ISVs

Standards for graphics mobile.... and 3D acceleration

The key players... ISVs & IHVs

Conclusion

3

Mobile 3D industry

Mobile phones are about to be transformed in personal compute devices

Graphics capabilities will be a key ingredient – just as on the PC

Before we had 2D, but now we want 3D !

The Mobile 3D industry is embryonic - but moving fast!

We are where PC graphics were in 1996 But evolving about three times faster!

4

Game market in Korea

0

50

100

150

200

250

300

350

400

450

500

Online game Mobile Video(PS2,XBOX)

Arcade game PC game

200120022003

<From : 2003 Korea Game White Paper>

[USD million]

Surging Video and mobile game market

5

Desktop vs. Handheld Systems

Desktop Handheld

CPU INTEL1GHz up to >3GHz

ARM as de-facto standard100MHz to 500+MHz

FPU YES Today ARM9 (Nomadik 1 @ ST) NOTomorrow ARM11(Nomadik 2 @ ST) YES

CACH, MEMORY, BANDWIDTH

Very big,power hungry

~10s of KB of eSRAM~1 MB of eDRAM is now feasible~100s of MB DDR-SDRAM Ultra low power design library

RESOLUTION SVGA>1024 * 768 pixels

QVGA (320*240)VGA (640*480)

GPU (e.g.) Nvidia: GeForce 6800,ATI: Radeon X800

IMG, Falanx, ATI, nVidia, Bitboys and others

3D API OpenGL1.5 / 2.0DirectXJava3D

OpenGL ES 1.0/1.1and future 2.0D3DM

JSR184 M3G

6

Challenges on Handheld Platforms

Small battery, small CPU

Management of Host CPU unload max work on Host CPU

and hardware resources can save 90% of battery life !

Limited system memory size and bandwidth

Memory architecture will make the difference !!

Cache memory; buffers compression

A wide range of Mobiles and gfx architectures, from Low-End High-End

Means scalable and flexible solutions !!!

7

Challenges for Software Vendors 1/2

If we consider 3 likely classes of handheld device

Class A : High-performance CPU, GPU with vertex processing + FPU Class B : High-performance CPU, GPU with vertex processing Class C : CPU, rasterizer

Classes B and C, low cost will likely ship in higher volume,

therefore may offer more revenue opportunities for software vendors but yet platforms do not have floating-point support (FPU)

Class A device may win out for better capacity and performance

Software vendors must cover all the bases to guarantee success

Observation

8

Challenges for Software Vendors 2/2

How to develop or port rich and attractive applications with such a spread of computational capabilities?

System Software must be scalable with Unified driver architecture to rapidly be adapted to different system architectures

Middleware is a key component

E.g. RenderWare (Criterion) optimized for Nokia N-gage

Action

9

Standardization

Different class of mobiles and Handsets, different capabilities

Many Graphics accelerator Suppliers

Graphics low power for mobile is a new industry

Need for standardizations

10

An Important and leading Group

Member-funded industry consortium

Lead specification of open standard, multi-platform, royalty-free low level APIs for HW acceleration

Leverage graphics technology by working closely with ISVs IHVs

2002Embedded 3D

Graphics Acceleration

2004Embedded 2D Vector

Graphics Acceleration

2000Streaming Media

Control and Video Acceleration

2004Embedded Media

Primitive Acceleration

11

Khronos Members

Promoting Members

Contributing Members

12

OpenML Standard for Dynamic media authoring

royalty-free, cross-platform programming environment

Abstraction layer for capturing, transporting, processing, displaying, and synchronizing digital media :–

audio/video streams 2D/3D graphics streams

OpenGL extensions for accelerated video processing

Lowest level of unified functionality, thinnest layer on top of HW

Application

Windows system

OpenGL OpenGL Extension

MLdc ML

Synchronisation

13

OpenVG 1/2 Standard for Vector Graphics acceleration

Why OpenVG ?

2D vector graphics is necessary for high quality GUIs or Font display on small display

Need hardware acceleration for useful performance Existing vector graphics APIs are proprietary and developers must rewrite

code for each platform Existing APIs usually designed for software rendering

Vector Graphics Exemple

Original image SVG zoom Bitmap zoom

14

OpenVG 2/2 Standard for Vector Graphics acceleration

OpenVG: Royalty-free, cross-platform, low-level APIs for hardware acceleration for high quality 2D vector graphics.

Interface for vector graphics libraries such as Flash and SVG

Same framework as OpenGL|ES combine with 3D APIs (Blending, lighting, texturing…)

OpenVG1.0 specs is currently under review by selected ISVs. Official release expected in March 2005 !

15

OpenMAX Standard for Media acceleration primitive

OpenMAX: royalty-free, cross-platform, common APIs for multimedia application

standardizes access to media processing primitives used extensively in graphics, audio and image libraries and video codecs (eg. MPEG-4)

Benefit: Accelerate porting of multimedia SW

Currently in development, expected release in End 2005

VideoMPEG-4

H.264Etc.

VoiceG.7xx

Etc

AudioMP3AACetc.

ImageJPEG

JPEG2k etc

Mul

timed

iaA

PI’s

RISC DSP

SMP H/Waccel M

ultim

edia

Acc

eler

ator

s

The problem

PORTING PROBLEM! slow down multimedia sw development

POORLY optimized

16

OpenMAX Standard for Media acceleration primitive

Accelerating a broad range of media types

3DSmall footprint 3D for embedded systems

VideoDynamic Media

Authoring

Vector 2DLow-level hardware

acceleration API

Media Engines – CPUs, DSP, Hardware Accelerators etc.

Accelerated media primitives for cross-platform acceleration

of media libraries

Image librariesVideo Codecs

Sound Libraries

Applications

Three media libraries called by applications. Can be ported

directly to silicon.

17

OpenGL ES Standard for Embedded 3D graphics acceleration

Khronos has created a small-footprint subset of OpenGL Created with the blessing and cooperation of the OpenGL ARB

Full functionality for 3D games On a wide variety of platforms – including fixed point processors

Eliminate Redundancy

Eliminate Workstation Functionality

ARBFeedback and Ratification

EmbeddedFocus

WorkstationFocus

Small footprint e.g. 50KB software

engines

18

OpenGL ES Roadmap Update OpenGL ES every year by default

To expose rapidly developing handheld platforms capabilities BUT ONLY introduce features with proven demand from ISVs or IHVs

Guarding against unnecessary pieces of HW Track and adapt developments in desktop OpenGL

OpenGL ES 2.0OpenGL ES 2.0

OpenGL 2.0OpenGL 2.0

OpenGL ES 1.1OpenGL ES 1.1

OpenGL ES 1.0OpenGL ES 1.0

OpenGL 1.3OpenGL 1.3

OpenGL 1.5OpenGL 1.5

Enabling software AND hardware implementations – including small-footprint, low-end fixed point platforms

Shader programmability for embedded devices

Widely available cross-platform 3D

graphics API

Full high-level shading language capability to harness the power of programmable hardware

Mid-03 Mid-04 Mid-05

Increasing emphasis on enabling emerging hardware with video

support and enhanced 3D pipeline

19

Java The multiplatform language

Game developers prefer Java, better for creating and managing tools and high level gfx libraries

But still need low level access to the 3D HW

Binding to OpenGL and OpenGL|ES JSR 231 – OpenGL (Expert Group created in Oct 2003) JSR 239 – OpenGL ES (Expert Group created in March 2004)

Wrappers around native OpenGL ES implementation

Higher level, Scene Graph API JSR 184 – Mobile 3D Graphics API for J2METM (Released in Nov 03)

can be built on a native OpenGL ES implementation can be layered on JSR 239Reduce application size for OTA downloading

20

SW Standards layers for 3D Mobile graphics

C/C++ Applications

Scenegraph APIsM3G (JSR 184)

MiddlewareLibraries

GamesEngines

JavaApplications

Hardware OpenGL ES

Engines

Software OpenGL ES

Engines

Operating Systems

Low-level3D Graphics

API

High-levelGraphicsLibraries

Applications

J2ME

Hardware Hardware

21

Graphics SW (Engine, Middleware…) Providers

Games developers

Game Engine, Middleware, Dev. tools

OpenGL|ES implementation and driver, Authoring tools...

JSR184/239

Mascot CapsuleMascot CapsuleX-FORGEX-FORGE

22

Graphics HW acceleration Providers

AcceleonAcceleon G30/G40™G30/G40™

PowerVR MBX / MBX LitePowerVR MBX / MBX Lite

GoForce 4000/3000GoForce 4000/3000

IMAGEON 2300IMAGEON 2300

And And of course !!of course !!

Mali50/100Mali50/100

23

Mobile Multimedia chips Providers

ARM9 (200MHz), VFP, MBX

ARM11 (330MHz), TMS310 (220MHz), VFP, MBX

ARM11 (500MHz), MMDSP(ST), VFP, 3D Acc (ST)

Hitachi semiconductor Mitsubishi semiconductor

SH-CPU (600MHz), SH-DSP, VFP, MBX

24

Conclusion

3D graphics for mobile is a new industry,

for a new market...

To be competitive, SW systems need to be scalable, with

unified low level standards,

unified drivers...

Middleware is the key for success,

to provide efficient data conditioning...