MPEG Technologies and Roadmap for AR

5th ARStandards Meeting, March 19-20, Austin, US

Marius Preda

Institut TELECOM

What is MPEG?

A family of standards published by ISO/IEC dealing with:

• Coding/compression of elementary media: audio/video (MPEG-1, 2 and 4), 2D/3D graphics (MPEG-4)• Hybrid (natural & synthetic) scene description, user interaction (MPEG-4)• Metadata (MPEG-7)• Media management and protection (MPEG-21)• Sensors and actuators (MPEG-V)• Virtual Worlds (MPEG-V)• Advanced User interaction (MPEG-U)• Media-oriented middleware (MPEG-M)• Coding and Delivery in Heterogeneous Environments• Dynamic Adaptive Streaming over HTTP• …

What is MPEG?

An activity continuing for 24 yearsSeveral hundreds of companies from all over the worldMore than 2300 active users

A proven manner to organize the work in order to deliver useful and used standards- 4 meetings per year, around the world, 500+ participants- subgroups with clear objectives - well defined procedure

Hundreds of millions of devices contain MPEG technologies (cameras, mobile phones, DVD/Blue-Ray players, STB and TVs, …)

MPEG tools related to AR

MPEG-1/2(AV content)

1992/4

VRML

1997

• Part 11 - BIFS: -Binarisation of VRML -Extensions for streaming -Extensions for server command -Extensions for 2D graphics• Part 2 - Visual: - 3D Mesh compression - Face animation

1998

• Part 2 – Visual - Body animation

1999

MPEG-4

MPEG-4

First form of augmentation of broadcast signal

MPEG tools related to AR

MPEG-4

2003

•AFX 2nd Edition: - Animation by morphing - Multi-texturing

2005

• AFX 3rd Edition - WSS for terrain and cities - Frame based animation

2007

MPEG-4

MPEG-4

• Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearence, animation

• AFX 4th Edition - Scalable complexity mesh coding

2011

MPEG-4A rich set of 3D Graphics representation and compression tools

MPEG tools related to AR

MPEG-4

2003

•AFX 2nd Edition: - Animation by morphing - Multi-texturing

2005

• AFX 3rd Edition - WSS for terrain and cities - Frame based animation

2007

MPEG-4

MPEG-4

• Part 16 - AFX: - A rich set of 3D graphics tools - Compression of geometry, appearence, animation

• AFX 4th Edition - Scalable complexity mesh coding

2011

MPEG-4

2009

• Part 25 - Compression of third-party XML (X3D, COLLADA)

MPEG-4

2004

• Part 16 - X3D Interactive Profile

MPEG-4

MPEG tools related to AR

MPEG-V

2011

• 2nd Edition: - GPS - Biosensors - 3D Camera

201x

• Compression of video + depth

201x

MPEG-V

3D Video

• 1st Edition - Sensors and actuators - Interoperability between Virtual Worlds

• Feature-points based descriptors for image recognition

201x

CDVS

MPEG-U

2012

• Advanced User Interface

A rich set of sensors, actuators

MPEG tools related to AR

MPEG-V

2011

• 2nd Edition: - GPS - Biosensors - 3D Camera

201x

• Compression of video + depth

201x

MPEG-V

3D Video

• 1st Edition - Sensors and actuators - Interoperability between Virtual Worlds

• Feature-points based descriptors for image recognition

201x

CDVS

MPEG-U

2012

• Advanced User Interface

2011

Augmented Reality• A study about the MPEG offer in the field of AR

MPEG vision on AR

Using MPEG technologies it is possible to envision having a generic AR browser. Instead of downloading a new and heavy application for every context, users could simply point to a URL. The browser would retrieve a scenario from the internet, start the video acquisition, track objects, recognize few of them from visual signatures, recover the camera pose, download 3D graphics, compose a new scene, get inputs from various sensors, and constantly adapt the interaction possibilities and objects from a remote server to offer an optimal AR experience.

Instead of developing a new application for each use case and smart-phone platform, the industry could rely on MPEG-compliant authoring tools, and MPEG-compliant browsers to reach the maximum number of customers.

MPEG vision on AR

MPEG-4/MPEG-7/MPEG-21/MPEG-U/MPEG-V MPEG Player

CompressionAuthoring Tool

Produce

Download

• Document use-cases• Extract requirements• Profile existing tools to address the requirements• Issue Call for Proposals for new technologies• Develop an Application Format for AR as part of

MPEG-A

MPEG Plans for AR

• Broadcast AR

AR use-cases

• Realistic augmentation

AR use-cases

• AR/MR Games

AR use-cases

Requirements for Augmented Reality Application Format (N12423)– on scene description– on media– on metadata– on remote access – on compression – on management and protection– on integration and communication 

AR Requirements

AR Context

• Representation of synthetic and natural objects• Real-time BIFS/3D mesh based streaming• Objects from remote/local storage, stored/real-time • Inherent object scalability• User interaction/server generated scene changes• Physical context captured/affected by a broad range

of sensors/actuators

Main features of MPEG AR technologies

Call for Proposals for Additional Augmented Reality Technologies (N12588, February 2012, San Jose)

- ARAF metadata

- ARAF management and protection

ARAF CfP

• WD2.0• Some technical details and a first implementation of

ARAF on mobile phone will be presented tomorrow in Session 8

ARAF