1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T....
-
Upload
grant-horn -
Category
Documents
-
view
223 -
download
4
Transcript of 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T....
1
Anahita: A System for 3D Video Streaming with Depth Customization
K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda
QCR & MIT & SFU5 November 2014
2
3D content is popular especially in theaters
Motivations
Controlled, homogenous environments
3
… is a lot more challenging because of …
complexity of perceiving and rendering depth
on different displays
&
in dynamic network conditions
Internet 3D Video Streaming …
Sample 3D Displays
4
How do we see 3D?
5
6
Approximate Model for Depth [Holliman 04]
d
e
Left eye
Right eye
p
Left eye
Positive disparity
zd
e
Right eye
p
z
Negative disparity
Comfort zone depends on:
• Presented content• Viewing condition• Screen distance• …
“The zone of comfort: Predicting visual discomfort with stereo displays” by Shibata et al. 2011
Depth & Comfort Zone [Shibata 11]
7
8
Many different display technologies - From anaglyph to auto stereoscopic
Rendering 3D Content
3D Display: Active/Passive Glasses
9
Autostereoscopic
• Parallax Barriers:
• Blocks light in certain direction
3D Display: Glasses-Free
• Lenticular Lens:
• Reflects light to certain direction
10
3D Display: Multi-views
11
12
Dynamic conditions- Bandwidth- Loss rate- Delay
Network Dynamics
13
Depth perception heavily depends on - Display size and technology - User preferences
Current 3D streaming systems are simple add-on to 2D streaming - Poor depth/visual perception and/or discomfort
In Summary …
Customize depth for different displays- Big TV … to phone
Serve different formats- Stereo, V+D, MV + D, …
Adapt to network dynamics - Wired, wireless, …, varying bandwidth
Manage many versions - Compared to 1—3 versions for 2D videos
3D Streaming Systems Need to ..
14
Side By Side
Top Bottom
Frame Sequential
Anaglyph
Row-Interleaved
Column-Interleaved
Video Plus Depth
Display Size
TV
Desktop
Laptop
Tablet
Cell Phone
Display Technology
5 x 7 x 3 = 105 versions for each video
Example
High Quality
Medium Quality
Low Quality
Network Bitrate
15
Enable all devices to render best possible 3D videos
From mobiles to big TVs
Our Challenge
16
Our Solution …
17
18
Show the need for depth optimization - Subjective study on different displays
Anahita: complete system for 3D streaming- Supports most current displays/technologies - Provides efficient management of 3D versions- Enables personalized depth - Is scalable and dynamic (using DASH)
Method for depth expansion and compression- Optimizes depth: from mobile phones to big TVs - Preserves scene structure (important for sports)- Does not introduce visual artifacts- Is computationally inexpensive
Our Contributions
19
Subjective study- 10 subjects- 2 displays (phone & 55” TV)- Series of short 3D clips (from soccer)- Show six different versions
• Original
• Depth compression
• Depth expansion (4 cases)
- Versions are shown in random to subjects- Subjects are asked which version is preferred
The Need for Depth Customization
20
Best version is NOT the original one Depth customization depends on display size
Results
21
SPSS: Structure Preserving Scene Shifting- Expands or compresses depth in stereo video- Simple image processing operations- Preserves lines and planes
- Suitable for field sports: soccer, football, tennis, …
Depth Customization
22
Control disparity of pixels - increase/decrease depth
Two operations- Slant (affects vertical component of disparity) - Stretch (affects horizontal component of disparity)
SPSS: Basic Idea
Depth Gradient
g = (gx, gy)23
Depth Gradient
g = (gx, gy) 24
SPSS: Slant
Left eye image Right eye image
25
gy is changed
SPSS: Slant
26
gx is changed
SPSS: Stretch
27
28
Calculate gradient g = (gx, gy) of scene disparity
Compute slant and stretch factor
Remap left/right views using:
SPSS: Main Steps
Before
After
SPSS: Sample Results
29
Before
After
SPSS: Sample Results
30
31
SPSS applies to scenes with planar depth long shots in field sports Our analysis of multiple full games
- 60—70% of the game
Close-ups and short shots do not benefit from SPSS
Anahita: has automatic shot classifier
SPSS: Coverage
Anahita: System Architecture
32
Anahita: 3D Version Manager
33
34
Based on DASH Versions are organized in segments Most suitable 3D version is chosen for each client
Client can personalize: ask for more/less depth- Served using DASH
Adaptive Streamer
35
Server - Deployed on Amazon- Implements many 3D video processing operations, and
our SPSS depth customization method
Clients- Mobile, tablet (mobile apps)- Stereo TV (web)- Desktop display (web)- Auto stereoscopic 55” display (custom app)
Implementation: Whole System
36
5 displays- mobile, tablet, 15.6” laptop, 27” desktop, 55” TV
3 soccer 3D video clips from YouTube- Man United vs. Wigan (60 sec)- Chelsea vs. Wigan (24 sec)- Chelsea vs. Plymouth (20 sec)
15 subjects- viewed all clips (original & optimized) on all displays in
random- Ranked depth from 1-poor to 5-Excellent
Evaluation: Subjective Study
37
Significant improvements in all cases- up to 35%
Sample Results
38
3D streaming systems are more complex than 2D
Showed the need for depth adaptation
Proposed method to customize depth
Designed 3D streaming system
Evaluation study shows large gains
Conclusions
Anahita: Meaning
• Ancient Persia: Anahita is source of all water, where warm and clear streams flow through golden channels to all lands and oceans on Earth
• Internet age: Anahita is source of all 3D videos, where high-quality streams of 3D videos flow through network channels to all types of displays