WebGL Meetup GDC 2012 - Khronos Group

© Copyright Khronos Group, 2012 -

WebGL Meetup GDC 2012


Khronos API Ecosystem Trends

Neil Trevett Vice President Mobile Content, NVIDIA

President, The Khronos Group


Mobile Gaming Innovation Vectors

Console-Class Performance,

Quality, Controllers and TV

connectivity

HTML5 and WebGL Web Apps that can be discovered on the Net and run on any platform

Vision Cameras as sensors, Gesture Processing, Augmented Reality

Sensor Fusion Devices become ‘magically’ context aware – location, usage, position

New platform capabilities being driven by SILICON and APIs

OpenVL


Khronos - Connecting Software to Silicon • Creating open, royalty-free API standards

- Focus on graphics, dynamic media, compute and sensor hardware

• Low-level - just above raw silicon

- “Foundation” functionality needed on every platform

• Safe forum for industry cooperation

- ‘By the industry for the industry’

- Open to any company to join

- IP framework to protect

members and industry

APIs enable software developers to turn silicon

functionality into rich end user experiences


Board of Promoters

Over 100 members – any company worldwide is welcome to join

Apple

http://www.amd.com/

http://www.antixlabs.com/

http://www.altsoftware.com/

http://www.google.com/

http://www.hu1.com/

http://www.fujitsu.com/

http://global.mitsubishielectric.com/

http://www.nds.com/

http://www.s3graphics.com/

http://www.st.com/

http://www.sasken.com/

http://www.toshiba.com/

http://www.vivantecorp.com/

http://www.marvell.com/index.jsp

http://www.umu.se/umu/index_eng.html

http://www.gshark.com/


HTML5 – Cross OS App Platform • Increasing diversity of devices

creates a demand for a true cross OS

programming platform

• BUT need more than “more HTML”

Rich Experiential Processing Multi-core CPUs

Rich 2D and 3D GPU GPU Computing

Multiple HD cameras Image and vision processing

Video encode/decode Audio encode/decode

Inertial and positional sensors

How can the Browser rapidly assimilate such diverse functionality?

Traditional Web-content


Leveraging Native API Investment into HTML5 • HTML5 evolving into cross-platform programming platform

- Gradually exposing complete system capabilities

• Opportunity to synergize Web and native APIs development

- Leverage native API investments, reduce developer learning cycles

• Khronos and W3C creating close liaison

Native APIs shipping or working group underway

JavaScript API shipping or working group underway

WebVL? Vision

Processing

WebAudio Advanced JavaScript

Audio

WebMAX? Camera

control and video

processing

Possible future JavaScript APIs

Device and Sensor APIs

Device Orientation

Working Groups

Native

JavaScript

OpenVL


WebGL Deployment • WebGL 1.0 Released at GDC March 2011

- Mozilla, Apple, Google and Opera working closely with GPU vendors

• Typed array 1.0 spec ratified by Khronos in May

- Supporting bulk data transfer between threads (workers)

- Many use cases - background mesh loading, generation, deformation, physics ...

• 1.0.1 release of WebGL spec and conformance suite imminent

- 100% robust stance on security

- Fixing bugs in 1.0.0 conformance suite

- Implementations will report getContext("webgl") (not experimental)

http://caniuse.com/#search=webgl WebGL is not enabled by default in Safari

http://caniuse.com/


WebCL – Parallel Computing for the Web • Khronos has launched new WebCL initiative

- First announced in March 2011

- API definition underway

• JavaScript binding to OpenCL

- Security is top priority

• Many use cases

- Physics engines to complement WebGL

- Image and video editing in browser

• Stay close to the OpenCL standard

- Maximum flexibility

- Foundation for higher-level middleware


OpenVL • Vision Hardware Acceleration Layer

- Enable hardware vendors to implement

accelerated imaging and vision algorithms

• OpenVL can be used by high-level

libraries or applications directly

- Primary focus on enabling mobile and

embedded systems

• OpenCV is widely used open source

library for vision projects

- Future version will leverage OpenVL

OpenCV open source library

Application

OpenVL

Open source sample implementation?

Hardware vendor implementations

Other higher-level CV libraries


StreamInput Connects Sensors to Apps

Advanced Sensors Everywhere Standard cameras, depth cameras

motion and position, touch, microphones wireless controllers

Apps Need Sophisticated Access to Sensor Data

Without coding to specific systems or sensor hardware

Apps request semantic sensor information StreamInput defines list of possible semantic requests

“Am I in an elevator?” “Give me gestures and face position”

Sensor graph created to provide sensor stream High-quality sensor data stream using optimized, sensor smart middleware

Apps gain ‘magical’ situational awareness – and portability


Current StreamInput Participants • Aiming for specification release in 2012


Possible Implementation of Vision Stack

Semantics and fusion of camera and positional sensors

Accelerated computer vision algorithms

Parallel computation

OpenVL

High-level computer vision library

Use OpenCL to implement CV HAL with

parallel execution

Accelerate OpenCV

library with CV HAL

Functions

Implement StreamInput vision sensor modules with

OpenCV

Data and event interop with CL /

GL / ES for display and compute processing

Camera input from OpenMAX AL or other camera subsystems


Khronos Ecosystem of Standards

Embedded and Mobile 3D

Cross platform desktop 3D

Plugin-free 3D Web Content

Heterogeneous Parallel

Computing Camera, Imaging and Streaming Media

Web Compute

Hardware accelerated vision functions

OpenVL

Unified Sensor and Input Processing

Graphics Sensors

Compute Imaging and vision

!

!

Na

tive

A

PIs

Web APIs


Current Khronos Liaisons • W3C

- HTML5/JavaScript bindings: WebGL, WebCL, sensors, video and camera

- Browser Acceleration: CSS shaders, composition, audio, SVG, Canvas

• Web3D

- Liaison agreement regarding X3D and COLLADA

- Declarative 3D community group at W3C - 3D scene graph in the DOM - WebGL

• ISO/IEC JTC1

- SC4/TC184 3D Visualization Specifications - Copyright License COLLADA executed

- SC24/WG9 – Augmented Reality working group – using Khronos APIs

• OMA and OGC

- Augmented reality cooperation - complementary activities


WebGL and Declarative 3D • Declarative 3D

- W3C Community Group – open to all

• Very synergistic

- WebGL can accelerate Declarative 3D

• Declarative 3D seems to be a major opportunity for X3D community

- By re-using existing HTML5 machinery – rather than duplicate – have a chance to

be very widely adopted

Canvas

2D 3D

Scenegraph

Immediate


Need for 3D Transmission Standard • Approaching chaos in data encodings used in WebGL browsers and apps

- We need to enable native decompression in browsers rather than JavaScript

• Possible key requirements

- Full-scene - Geometry, textures, materials, animations, physics etc.

- Compression of textures and geometry

- Streaming support with LOD flexibility

• Some formats in use – but no widespread consensus

- COLLADA, KML, MPEG-4, VRML, JSON, X3D binary, PowerVR POD, GZIP etc. etc.

• Is the industry ready to work on this? Is the need real?

Audio Video Images 3D

All above defined by MPEG – should 3D be an MPEG standard too? Must be royalty free! Leverage MPEG-4 Part 16 ‘AFX’, Part 25 – a lot of investment – but why no momentum?

MP3 H.264 PNG/JPEG ?


COLLADA WebGL Synergy • COLLADA synergy with WebGL authoring

- COLLADA import into common WebGL frameworks such as three.js?

- JSON as transmission format - robust COLLADA to JSON conversion?

• Lots of discussion needed around deployment formats

- Compression, streaming – use the Transmission Standard

WebGL Meetup GDC 2012 - Khronos Group

Documents

Transcript of WebGL Meetup GDC 2012 - Khronos Group