ITU-T Workshop on ILE

Macao, China

Masayuki Tanimoto

Nagoya University

Nagoya Industrial Science Research Institute

Nagoya, Japan

October 24, 2017

FTV (Free viewpoint Television)

and Its Standardization in MPEG

FTV Video Produced by IEEE Signal Processing Society

“Signal Processing in Free Viewpoint Television”

https://www.youtube.com/watch?v=T0VPjHwrc-o

1. Introduction of FTV

2. Principle of FTV

3. FTV Technologies and Demo

4. FTV Standardization in MPEG

5. Conclusions

Outline

1. Introduction of FTV

Proce

ssin

g

Capture

Display

Free

viewpoint

camera

FTV (Free viewpoint TV)

FTV on a

Mobile Player

FTV (Free-viewpoint TV)Visual media that enable users to view a

3D scene by freely changing the

viewpoint as if they were actually there.

2DTV/SDTV

３DTV

HDTV

FTV

UHDTV

HD-FTV UHD-FTV

HD-3DTV UHD-3DTV

360-degreevideo

360-3D video

360-FTV

Number of pixels / FoV (Field of View)

Num

ber

of

view

s /

Vie

win

g zo

neProgress of Visual Media

Many and Dense Views Needed for Viewing

with Motion Parallax and without Eye Fatigue

different views

different viewpoints

viewing zone

object

motion

parallax

Significances of FTV

Ultimate 3DTV that transmits an infinitenumber of views and ranks as the top ofvisual media

Natural interface between human andenvironment

Immersive media that provide veryrealistic VR experiences

“ Multiview Imaging and 3DTV,” IEEE Signal Processing

Magazine, Vol.24, No.6, pp.10-21 (November 2007)

2. Principle of FTV

position

(x, y, z)

direction

(q, f)

one ray

[Real Space] [Ray-Space]

one point

Ray-space is a parameter space where one ray

is expressed by one point.

Ray-Space Representation

group of rays through one point plane

virtual camera

object

captured ray

1. View generation 2. Ray interpolation

created rays

diffusion surface

Concept of “Group of Rays through One Point”

1

2

34

5

X

Y

Z

( , )x y

( , )q

Visual Field(Horizontal)

Visual Field(Vertical)

Cameras

Position

Direction

Viewing Zone

23

4 5

1

u ( tan ) q

xy

f x y u( , , )

Visual Field(Horizontal)

Visual Field(Vertical)

Viewing Zone

Ray-Space

45

32

1

Ray Capture in Linear Camera Arrangement

Real Space

Ray-Space

Horizontal

section

x

u=tanθ

xqtan

y

x

u=tanθ

line detection

+ interpolation

Ray Interpolation

Orthogonal Ray-Space and Horizontal Cross-Section

x

u=tan θ

y

The horizontal cross-sections of orthogonal ray-space have line structures.

f (x , y , u=tan θ)

Section Image 1

Section Image 2

Section 1

Section 2y

x

u=tan θ

View Generation in Linear Camera Arrangement

Camera Arrangements for Capture

(b) Circular/spherical arrangement

for convergent view

(a) Linear/planar arrangement

for parallel view

The horizontal cross-sections of spherical ray-space

have sinusoidal structures.

0°

180°

360°

0° 180° 360°

θ

θ

η

ξ

Horizontal Cross-Sections of Spherical Ray-Space

ξ

η

ξ

θ

f (ξ , η , θ)

Section Image 1

Section Image 2

Section 1

Section 2

View Generation in Circular Camera Arrangement

3. FTV Technologies and Demo

100-Camera System

Linear Planar

Circular

FTV Demo: Aquarium

Original Camera Views (15 Cameras)

FTV Demo: Aquarium

Generated Free Views

Original Views

Generated Views - Interpolated -

Generated Views - Forward and Backward -

FTV on a laptop PC FTV on a mobile player FTV on an all-aroundwith mouse control with touch panel control 3D display (Seelinder)

FTV on a 2D display with head tracking

FTV on a multi-view 3D display with head tracking

FTV on a multi-view 3D display without head tracking

Interface of FTV

Super-Multiview 3D Display: The SeeLinder

4. FTV Standardization in MPEG

FTV first phase (2004/03-2009/05)

MVC (Multi-view Video Coding) standard to

compress multiple video

FTV second phase (2007/04-2015)

3DV (3D Video) standards for multiview display

application

FTV third phase (2013/08-)

FTV standards for Super-Multiview, Free

Navigation and 360 3D Video applications

MPEG has been developing FTV standards since 2001.

First Phase of FTV

(MVC)

Encoder

Sender side Receiver side

Trans-

mission

Decoder

multi-view

Coding of multiview video

MVC (Multi-view Video Coding)

Second Phase of FTV

(3DV)

Depth

EstimationEncoder

Stereo

cameraSender side Receiver side

Multi-view

display

Trans-

mission

DecoderView

Synthesis

multi-view

multi-depth

MVD (multi-view + depth)

Joint coding of multi-view and depth

3D-AVC, 3D-HEVC

Super-Multiview

Free Navigation

Scene

Represent

ation

EncoderTrans-

mission

DecoderView

Synthesis

Sender side Receiver side

360 3D

Third Phase of FTV

(FTV)

Capture

Display

FTV targets immersive 3D viewing by super-

multiview, free navigation and 360 3D.

“Revolutionizing Football”

produced by 2022 FIFA World Cup Japan Bid Committee in 2010

2022 FIFA World Cup Japan Bid Committee Planed to Deliver

the Excitement on Soccer Stadium to the World by FTV

1D Super-Multiview Display (Only Horizontal Parallax)

80 directions （Holografika）

All-Around Super-Multiview Display

(360 views) Holo-Table (3Dragons LLC)

2D-SMV Display (Full Parallax)

Integral 3DTV (400x250 views)

(NHK)

Free Navigation for Soccer (KDDI)

Exploration Experiments and Call for

Evidence were conducted.

Test material, reference software and

evaluation methods were developed.

After the success of Call for Evidence,

FTV activity moved to MPEG-I in

January 2017 and has been in charge of

its video part.

FTV is immersive media that transmit all ray

information of a 3D scene and enable very

realistic 3D viewing.

FTV was developed based on the ray-space

representation.

MPEG has been creating various FTV

standards.

FTV activity moved to MPEG-I for further

standardization.

5. Conclusions