UHD als Format-agnostisches Produktionsverfahren und als eine
Transcript of UHD als Format-agnostisches Produktionsverfahren und als eine
Fraunhofer
Heinrich Hertz Institute
©
Fraunhofer
Heinrich Hertz Institute
Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587 Berlin, www.hhi.fraunhofer.de
UHD als Format-agnostisches
Produktionsverfahren und als eine Plattform für
interaktives Video
Ralf Schäfer
1
Fraunhofer
Heinrich Hertz Institute
© ©
Immersive Telepresence Systems
2 Ralf Schäfer
Immersive sound
Immersive displays & surrounding (panorama) video
Users feel like „being there“ – standing in the middle of a scene or
sitting at the best seat
Users not having access to an „immersive screen“ can navigate with
their smaller screen in the Ultra High Definition panorama
Fraunhofer
Heinrich Hertz Institute
© ©
Outline
3
Introduction - Beyond HD
Ultra High Definition (UHD) @ HHI – TiME Lab
FASCINATE project: Format agnostic video production and
distribution
Shooting of UHD panoramas – OMNICAM
Production examples
Combining panorama and broadcast camera views
Streaming for interactive navigation in UHD panoramas
Further developments
Conclusions
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© ©
Video Beyond HD
4
Video formats evolve beyond HD
New standards like 4k (3840x2160) and 8k (7680x4320) are
available
4k equipment is already available today, 8k is targeted for 2020
In the future flexible displays will be available which can be used
like wall paper
The European project FASCINATE is working on a completely new
production approach: Format Agnostic Video
Format Agnostic Video means: Combination of any number of
video sensors to create Ultra High Resolution Video of arbitrary
spatial and temporal resolution
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© ©
Real-time acquisition and projection of high-
resolution video panoramas
Ultra high resolution video @ HHI
5
Omnidirectional
6k camera
system
(OMNICAM)
CineBox for
stitching and
segmentation
Transmission
CineBox for
playout,
decoding,
warping and
blending
7k multiprojection system
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© © 6
Immersive Viewing at HHI‘s TiME Lab
Surround view by 180°panorama projection in 7k in 2D and 3D and
surround sound by IOSONO WFS system
mirrors
for HD projectors
(7x2)
Row of 120
loudspeakers Curved screen (3m x 12m)
acoustically transparent
TiME: Tomorrow‘s immersive Media Experience
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© © 7
TiME Lab –
a Laboratory for the Creative Community
TiME Lab is a place, where creative people can experiment with a 180°video
panorama and a WFS sound system.
Partners are
• HFF Konrad Wolf
• BTK Hochschule
• Berliner Philharmoniker
• Berlinale Talent Campus
• Lichtmond
• Neue Roadmovies/Wim Wenders
• Red Bull Media
• KUK - Filmproduktion
• Moro Images
• Arri
• ........
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© ©
Format Agnostic Video Production
9
A layered scene representation is used to represent the camera clusters
used to record a scene
A powerfull set of metadata is used to describe the single sensor data,
their positions, viewing directions and their geometrical relations
Video streams and metadata can be transmitted and used to create
appropriate formats for further processing and/or for available displays
Such a format agnostic video representation is especially suitable for
interactive viewing of large panoramas
Ralf Schäfer
CC: camera cluster
Cam: camera
Video
Scene CC 1
CC 2
CC N
Cam 1.1
Cam 1.n
Fraunhofer
Heinrich Hertz Institute
© ©
Test shoot at Chelsea
FascinatE test shoot at Premier League Match, Oct. 2010
Real live event
Usage of professional equipment by outside broadcaster
Visual sensors
OMNICAM by HHI
ALEXA by ARRI
4 HD broadcast cameras by BBC
Audio sensors
12 Stadium shot gun mics around
the pitch
Stereo mics
2 Soundfield® mics
1 Eigenmike®
Ralf Schäfer 10
Fraunhofer
Heinrich Hertz Institute
© ©
Test shoot arrangement
OMNICAM Broadcast cam Alexa
Soundfield Stereo pair
Eigenmike
Ralf Schäfer 11
Fraunhofer
Heinrich Hertz Institute
© © 19.2.2010 13
Postprocessing: Stitching and
geometric correction
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© © 14
Postprocessing: Photometric &
colorimetric correction
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© ©
Real-Time Stitching Engine
16
Realtime
stitching of 6
HD cameras on
a single PC
platform -
demonstrated
@ IBC 2011
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© ©
Collaboration with HFF
18
Production of feature film „Orlac Reloaded“ presented at Talent
Campus of Berlinale 2010
OMNICAM
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© ©
Shooting a concert of BPO*
19
OMNICAM
*Berlin Philharmonic Orchestra
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© ©
Improving resolution of OMNICAM
First shootings have been performed with an OMNICAM using 1st
generation micro HD cameras, which have strong limitations in spatial
resolution, temporal resolution (25 fps) and dynamic range.
Especially the limitation to 25 frames caused severe motion blur and
motion judder.
Meanwhile a new OMNICAM using 6 ALEXA-m cameras is available
The ALEXA-m has a high dynamic range (15 stops), a 3k sensor and can
shoot at 50 fps or higher
Two first shootings with the new OMNICAM have already been made:
• Berliner Philharmoniker & Sasha Waltz @ ARENA in Berlin
• Night of the Proms @ Royal Albert Hall in London
Ralf Schäfer 20
Fraunhofer
Heinrich Hertz Institute
© ©
OMNICAM with ALEXA-M
New rig with 6 high end electronic film cameras using 6 segments for 180° panoramas:
• 6 x ALEXA-M
• vertical FOV: 60° (portrait format)
• horizontal FOV 30° per segment => 180°
• resolution: 6k x 2k
• upgradable up to 360°
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© © 23 Ralf Schäfer
Live
Preview
Director‘s room
Panorama Shooting @ Arena / Berlin
Fraunhofer
Heinrich Hertz Institute
©
Fraunhofer
Heinrich Hertz Institute
24
Panorama Shooting – BBC Proms / London
Ralf Schäfer
Royal Albert Hall
Fraunhofer
Heinrich Hertz Institute
© ©
Comparison of resolutions
Full broadcast
camera image
6k x 2k panorama
Detail from
panoramic image
Merging of Broadcast Camera(s) and
Panorama
Ralf Schäfer 26
Fraunhofer
Heinrich Hertz Institute
© ©
Registration between cameras
Region of OMNICAM image
Ralf Schäfer 27
Fraunhofer
Heinrich Hertz Institute
© ©
Registration: Colour Matching Problems
Simple overlay shows problems with photometric matching and parallax
Ralf Schäfer 28
Fraunhofer
Heinrich Hertz Institute
© ©
Histogram matching can largely solve photometric issues
Registration with Colour Correction
Ralf Schäfer 29
Fraunhofer
Heinrich Hertz Institute
© ©
Parallax problems are more fundamental: depth variation makes global
registration impossible
Clearly visible when trying to take low frequencies from OMNICAM and
high frequences (detail) from broadcast camera
Registration:Parallax Problems
Ralf Schäfer 30
Fraunhofer
Heinrich Hertz Institute
© ©
Registration with Parallax Correction
Correction can be applied for each depth plane:
Estimate depth of the scene part in the panorama, at which the broadcast
camera points
Use Depth Image Based Rendering (DIBR) to compensate the parallax
Demo: Blending from broadcast camera into OMNICAM panorama
Ralf Schäfer 31
Fraunhofer
Heinrich Hertz Institute
© © 34
Shooting of UHD Panoramas for
Interactive Usage
production
phone/tablet
Interactive TV
distribution
network and/or
Internet
HD
UHD
Ralf Schäfer
OMNICAM
Fraunhofer
Heinrich Hertz Institute
© ©
Interactive Tiled Streaming of ROIs
Control API to
Combiner
Monitor combiner output:
Stitched video of the ROI
Panorama Sequence
total of 385 spatially segmented and
compressed video files of resolution
480x270 from 10 resolution layers
Ralf Schäfer 35
Fraunhofer
Heinrich Hertz Institute
© ©
Server-side Panning and Zooming
High resolution panoramic content
Real-Time Reframing and Coding
Frame width / bitrate / quality
2D navigation / zoom commands
Remote A/V Proxy Thin Clients
- Set-up per client
- Interactive View streaming per client
- Decoded once for all clients
Ralf Schäfer 36
Fraunhofer
Heinrich Hertz Institute
©
Fraunhofer
Heinrich Hertz Institute Example for Intertactive Streaming
37 Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
©
Fraunhofer
Heinrich Hertz Institute
38
3D CGI Production – Lichtmond 2
"Precious Life" – joint project with
Alan Parsons
Cooperation with Lichtmond for the
production of a 180°animation in
3D
WFS production of sound
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© © 39 Ralf Schäfer
• First prototype of OMNICAM 3D
with 3 camera pairs has been
developed
• Presentation of first 3D
panoramas took place during
Berlinale on February 17th, 2011
(PRIME Symposium:
http://ip.hhi.de/PRIME_Workshop
/index_de.html
• 180°3D OMNICAM will be
available by mid 2013
OMNICAM 3D
Fraunhofer
Heinrich Hertz Institute
© © 40
Mirror Surface
concentric
arrangement
of stereo
sub-systems
The 3D OmniCam
3D OMNICAM
Ralf Schäfer
left view right view
stereo content from crossed mirror segments
Stitching of 3D panoramas
Fraunhofer
Heinrich Hertz Institute
©
Fraunhofer
Heinrich Hertz Institute Compact OMNICAM
41
• Compact 360°version
• 10 HD cameras
• Vertical FOV: 60° (portrait format)
• Horizontal FOV: 36° per segment
• Resolution: 10k x 2k
• Diameter: 50 cm
• Height: 40 cm
<<< size comparison
Fraunhofer
Heinrich Hertz Institute
©
Fraunhofer
Heinrich Hertz Institute
42
Mobile TiME Lab
180°panorama
14 HD projectors
3D projection using Infitec
3D audio (32 loudspeakers) with
IOSONO-WFS
Premiere at IFA 2012
Part of the program of
Medienwoche
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© © 43
Conclusions
Video technology is evolving beyond HD
New video formats such as 4k and 8k have been standardized
In a long term wall displays will be available
For HHI video systems beyond HD are a strategic R&&D topic
The European project FASCINATE is working on a new concept, which is called
“Format Agnostic Video” and which is especially suitable for sport events such
as soccer
The complete chain is being developed in FASCINATE
Several panorama productions using the OMNICAM have been made
Technologies for combining video from panorama and broadcast cameras have
been developed
A test bed for adaptive streaming using interactive navigation has been
developed
Further developments (Mobile TiME Lab, 3D OMNICAM, compact OMNICAM)
are on its way
Ralf Schäfer
Fraunhofer
Heinrich Hertz Institute
© © 44
Contact:
Dr. Ralf Schäfer
Fraunhofer HHI
Einsteinufer 37
10587 Berlin, Germany
Phone: +49 30 31002-560
www.hhi.fraunhofer.de
Thank you!
Ralf Schäfer
Acknowledgement: Parts of this presentation are based on work within the FP7
project FASCINATE funded by the European Commission