FTV (Free viewpoint Television) and Its Standardization in ??1. Introduction of FTV 2. Principle of...

download FTV (Free viewpoint Television) and Its Standardization in ??1. Introduction of FTV 2. Principle of FTV 3. FTV Technologies and Demo 4. FTV Standardization in MPEG 5. Conclusions Outline

of 39

  • date post

    13-Feb-2018
  • Category

    Documents

  • view

    227
  • download

    4

Embed Size (px)

Transcript of FTV (Free viewpoint Television) and Its Standardization in ??1. Introduction of FTV 2. Principle of...

  • ITU-T Workshop on ILE

    Macao, China

    Masayuki Tanimoto

    Nagoya University

    Nagoya Industrial Science Research Institute

    Nagoya, Japan

    October 24, 2017

    FTV (Free viewpoint Television)

    and Its Standardization in MPEG

  • FTV Video Produced by IEEE Signal Processing Society

    Signal Processing in Free Viewpoint Television

    https://www.youtube.com/watch?v=T0VPjHwrc-o

  • 1. Introduction of FTV

    2. Principle of FTV

    3. FTV Technologies and Demo

    4. FTV Standardization in MPEG

    5. Conclusions

    Outline

  • 1. Introduction of FTV

    Proc

    essin

    g

    Capture

    Display

    Free

    viewpoint

    camera

    FTV (Free viewpoint TV)

  • FTV on a

    Mobile Player

    FTV (Free-viewpoint TV)Visual media that enable users to view a

    3D scene by freely changing the

    viewpoint as if they were actually there.

  • 2DTV/SDTV

    DTV

    HDTV

    FTV

    UHDTV

    HD-FTV UHD-FTV

    HD-3DTV UHD-3DTV

    360-degreevideo

    360-3D video

    360-FTV

    Number of pixels / FoV (Field of View)

    Num

    ber

    of

    view

    s /

    Vie

    win

    g zo

    neProgress of Visual Media

  • Many and Dense Views Needed for Viewing

    with Motion Parallax and without Eye Fatigue

    different views

    different viewpoints

    viewing zone

    object

    motion

    parallax

  • Significances of FTV

    Ultimate 3DTV that transmits an infinitenumber of views and ranks as the top ofvisual media

    Natural interface between human andenvironment

    Immersive media that provide veryrealistic VR experiences

  • Multiview Imaging and 3DTV, IEEE Signal Processing

    Magazine, Vol.24, No.6, pp.10-21 (November 2007)

    2. Principle of FTV

  • position

    (x, y, z)

    direction

    (q, f)

    one ray

    [Real Space] [Ray-Space]

    one point

    Ray-space is a parameter space where one ray

    is expressed by one point.

    Ray-Space Representation

    group of rays through one point plane

  • virtual camera

    object

    captured ray

    1. View generation 2. Ray interpolation

    created rays

    diffusion surface

    Concept of Group of Rays through One Point

  • 1

    2

    34

    5

    X

    Y

    Z

    ( , )x y

    ( , )q

    Visual Field(Horizontal)

    Visual Field(Vertical)

    Cameras

    Position

    Direction

    Viewing Zone

    23

    4 5

    1

    u ( tan ) q

    xy

    f x y u( , , )

    Visual Field(Horizontal)

    Visual Field(Vertical)

    Viewing Zone

    Ray-Space

    45

    32

    1

    Ray Capture in Linear Camera Arrangement

    Real Space

    Ray-Space

  • Horizontal

    section

    x

    u=tan

    xqtan

    y

    x

    u=tan

    line detection

    + interpolation

    Ray Interpolation

  • Orthogonal Ray-Space and Horizontal Cross-Section

    x

    u=tan

    y

    The horizontal cross-sections of orthogonal ray-space have line structures.

  • f (x , y , u=tan )

    Section Image 1

    Section Image 2

    Section 1

    Section 2y

    x

    u=tan

    View Generation in Linear Camera Arrangement

  • Camera Arrangements for Capture

    (b) Circular/spherical arrangement

    for convergent view

    (a) Linear/planar arrangement

    for parallel view

  • The horizontal cross-sections of spherical ray-space

    have sinusoidal structures.

    0

    180

    360

    0 180 360

    Horizontal Cross-Sections of Spherical Ray-Space

  • f ( , , )

    Section Image 1

    Section Image 2

    Section 1

    Section 2

    View Generation in Circular Camera Arrangement

  • 3. FTV Technologies and Demo

  • 100-Camera System

    Linear Planar

    Circular

  • FTV Demo: Aquarium

    Original Camera Views (15 Cameras)

  • FTV Demo: Aquarium

    Generated Free Views

  • Original Views

  • Generated Views - Interpolated -

  • Generated Views - Forward and Backward -

  • FTV on a laptop PC FTV on a mobile player FTV on an all-aroundwith mouse control with touch panel control 3D display (Seelinder)

    FTV on a 2D display with head tracking

    FTV on a multi-view 3D display with head tracking

    FTV on a multi-view 3D display without head tracking

    Interface of FTV

  • Super-Multiview 3D Display: The SeeLinder

  • 4. FTV Standardization in MPEG

  • FTV first phase (2004/03-2009/05)

    MVC (Multi-view Video Coding) standard to

    compress multiple video

    FTV second phase (2007/04-2015)

    3DV (3D Video) standards for multiview display

    application

    FTV third phase (2013/08-)

    FTV standards for Super-Multiview, Free

    Navigation and 360 3D Video applications

    MPEG has been developing FTV standards since 2001.

  • First Phase of FTV

    (MVC)

    Encoder

    Sender side Receiver side

    Trans-

    mission

    Decoder

    multi-view

    Coding of multiview video

    MVC (Multi-view Video Coding)

  • Second Phase of FTV

    (3DV)

    Depth

    EstimationEncoder

    Stereo

    cameraSender side Receiver side

    Multi-view

    display

    Trans-

    mission

    DecoderView

    Synthesis

    multi-view

    multi-depth

    MVD (multi-view + depth)

    Joint coding of multi-view and depth

    3D-AVC, 3D-HEVC

  • Super-Multiview

    Free Navigation

    Scene

    Represent

    ation

    EncoderTrans-

    mission

    DecoderView

    Synthesis

    Sender side Receiver side

    360 3D

    Third Phase of FTV

    (FTV)

    Capture

    Display

    FTV targets immersive 3D viewing by super-

    multiview, free navigation and 360 3D.

  • Revolutionizing Football

    produced by 2022 FIFA World Cup Japan Bid Committee in 2010

    2022 FIFA World Cup Japan Bid Committee Planed to Deliver

    the Excitement on Soccer Stadium to the World by FTV

  • 1D Super-Multiview Display (Only Horizontal Parallax)

    80 directions Holografika

  • All-Around Super-Multiview Display

    (360 views) Holo-Table (3Dragons LLC)

  • 2D-SMV Display (Full Parallax)

    Integral 3DTV (400x250 views)

    (NHK)

  • Free Navigation for Soccer (KDDI)

  • Exploration Experiments and Call for

    Evidence were conducted.

    Test material, reference software and

    evaluation methods were developed.

    After the success of Call for Evidence,

    FTV activity moved to MPEG-I in

    January 2017 and has been in charge of

    its video part.

  • FTV is immersive media that transmit all ray

    information of a 3D scene and enable very

    realistic 3D viewing.

    FTV was developed based on the ray-space

    representation.

    MPEG has been creating various FTV

    standards.

    FTV activity moved to MPEG-I for further

    standardization.

    5. Conclusions