Immersive actor feedback system
-
Upload
katelynn-victory -
Category
Documents
-
view
48 -
download
3
description
Transcript of Immersive actor feedback system
![Page 1: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/1.jpg)
1 Research & Innovation
Immersive actor feedback system
• What sees the actor on set ?
( Answer! )
![Page 2: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/2.jpg)
2 Research & Innovation
Immersive actor feedback system
• Viewpoint dependent projection– Virtual reality: CAVE system
• Integration into Chroma-keying environment– Retro-reflective cloth (BBC patent)
![Page 3: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/3.jpg)
3 Research & Innovation
Immersive actor feedback system
• Retro-reflective cloth– “truematte” developed by BBC
Camera with LED ring
LED off
LED on
![Page 4: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/4.jpg)
4 Research & Innovation
Immersive actor feedback system
Projector
3D model of Actor
3D model of virtual scene
ViewpointRenderer
Head position
Mask generation
![Page 5: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/5.jpg)
5 Research & Innovation
Immersive actor feedback system• View dependent projection
The Ambassadors (1533), by Hans Holbein the Younger. National Gallery London
![Page 6: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/6.jpg)
6 Research & Innovation
Immersive actor feedback
• Head tracking
Fast template matching filter on voxel data
![Page 7: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/7.jpg)
7 Research & Innovation
Production Visualisation Tools
• real-time, low-quality
Monitor
Actor Feedback
Director’s View
3D model of real scene
3D model of virtual scene
On-set
Planning,On-set
![Page 8: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/8.jpg)
8 Research & Innovation
Immersive actor feedback system
![Page 9: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/9.jpg)
9 Research & Innovation
Bamzooki
• BBC Children Production
![Page 10: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/10.jpg)
10 Research & Innovation
Free-viewpoint video for sports
• ‘Virtual Replay’ of football scenes– Allows viewer interactive
review
– Manually modelled offline
– Not realistic
![Page 11: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/11.jpg)
11 Research & Innovation
Free-viewpoint video for Sport visualisation
• Piero
• Overcome limitations: use multi-camera system, iview project
![Page 12: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/12.jpg)
12 Research & Innovation
Applying multi-camera techniques to sport scenarios
• Test shoot England-Wales in Old Trafford 9 Oct 2004
![Page 13: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/13.jpg)
13 Research & Innovation
Where to put the cameras?
• Broadcast coverage cameras
• Pro:– Works without extra setup
• Cons:– Unreliable framing
![Page 14: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/14.jpg)
14 Research & Innovation
Applying multi-camera techniques to sport scenarios
• Calibration with wand• Problems:
– Cameras knocked overnight– Cameras not rigidly mounted
![Page 15: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/15.jpg)
15 Research & Innovation
Line-based calibration
• Colour-based edge detection
• Line-fitting• Calibration of camera
parameters (T,R,f) against dimensions of the pitch
• Runs at video frame rate
![Page 16: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/16.jpg)
16 Research & Innovation
Processing - Segmentation
• Chroma-key against green
![Page 17: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/17.jpg)
17 Research & Innovation
Processing – 3D Reconstruction
• 3D representations– Billboard (flat polygon, sprite)– Visual hull (Shape from silhouette)
![Page 18: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/18.jpg)
18 Research & Innovation
Results
From 16 cameras, visual hull + graph-cut shape optimisation
(results from iview project, jointly with University of Surrey)
![Page 19: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/19.jpg)
19 Research & Innovation
Results
![Page 20: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/20.jpg)
20 Research & Innovation
Summary
• Computer vision techniques are important building blocks for the automated generation of visual effects
• For the integration of video and graphics several optical phenomena have to be harmonised. In descending order of importance:– Camera perspective, occlusions, shadows, ..
• Visual feedback systems integrated into the production pipeline are important to allow interaction with virtual objects
• Free-viewpoint video allows new insights– Goal: Stream 3D to the viewer at home
![Page 21: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/21.jpg)
21 Research & Innovation
![Page 22: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/22.jpg)
22 Research & Innovation
3D Modelling
• Graph-cut
• Details: A Bayesian Framework for Simultaneous Matting and 3D Reconstruction, J.-Y. Guillemaut, A. Hilton, J. Starck, J. Kilner, and O. Grau, 3DIM’07
![Page 23: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/23.jpg)
23 Research & Innovation
Results
• Video from ORIGAMI demo production
![Page 24: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/24.jpg)
24 Research & Innovation
Applications: Requirements
I
coarse rich detailed
3-D
II
III IV
2 1/2-D2-D
Applications:
a) virtual studiob) On-set Visualisationc) Pre-Visualisationd) Interactive e) Digital effects
Optical phenomenon:
1) camera perspective2) occlusions3) shadows4) reflections
1 2 3 4
I
II
III
IV
a
b
e
Shape classes:
d c
![Page 25: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/25.jpg)
25 Research & Innovation
Introduction
• Conventional work flow in (feature) TV or film productions
PlanningPhase
On-setPhase
PostProduction
Animations /Virtual Set
Storyboard
FinalProgramme
expensive
![Page 26: Immersive actor feedback system](https://reader030.fdocuments.net/reader030/viewer/2022032612/56813431550346895d9b2062/html5/thumbnails/26.jpg)
26 Research & Innovation
Introduction
• ORIGAMI approach– Mixed-reality techniques for visualisation through the production
PlanningPhase
On-setPhase
PostProduction
Animations / Virtual Set
Storyboard
FinalProgramme