2D to 3D conversion of formula 1 footage

2D to 3D conversion of Formula 1 footage

Serge Hendrickx

1

Presentation Overview

• Thesis objective

• Conversion process

• Conclusion and result 2

• 3D rotation

• Car detection and tracking

• Inpainting

• Overlay detection and conversion of background

Objective: 2D to 3D conversion of Formula 1 footage

2D recorded and broadcasted images 3D experience

3

Objective: 2D to 3D conversion of Formula 1 footage

4

What is special about Formula 1?

Undeformable objects with near perfect 3D models -> Investigate how these could be used in this conversion process -> Not just generating best-looking 3D

Introduction to Formula1 footage Example footage

5





• 3D rotation


• Inpainting


3D effect

• Translation + rotation

7

Retinal Disparity

3D effect


8

Retinal Disparity

1) Nearest-neighbor

2) 3D Projection

3D effect


9

Retinal Disparity

1) Nearest-neighbor

2) 3D Projection

Nearest neighbor rotation

10

For each pixel: 1) Find corresponding RGB 2) Detect nearest match in

rotated view 3) Copy pixelvalue

XYZ represented by RGB during rendering

11

Nearest neighbor rotation

3D effect


12

Retinal Disparity

1) Nearest-neighbor

2) 3D Projection

3D projection

13

Traverse rendering pipeline

(X,Y) –coordinate in new view

(X,Y,Z) coordinate

(X,Y) –coordinate in known view

Using ModelView and Projection matrices of new view

Using ModelView and Projection matrices of known view

3D projection

14

Traverse rendering pipeline

Source:

Generated:

3D rotation

• 3D projection : + Fast + Produces clear image - Holes

15

• Neighborhood: - Very slow - Artifacts + Fills in holes

3D rotation

• 3D projection : + Fast + Produces clear image - Holes

16

• Neighborhood: - Very slow - Artifacts + Fills in holes

-> Combine both methods

17

Source:

3D rotation: final result

Rotated 10 degrees:

18


Retinal Disparity

3D effect

19


Retinal Disparity

3D effect

-> Exact position and pose of car needed





• 3D rotation


• Inpainting


Car detection and tracking

• Car detection

• Tracking throughout fragment

21


• Car detection


22

Car detection

23

Sliding window -> binary classifier: contains car yes or no?

Car detection

24


Car detection

25


Car detection

26


Render car from all different angles

27

Car detection

Car detection

28


-> Does it match one of the renders?

Car detection

29

How to compare? Pixel per pixel -> Not resistant to illumination changes and other differences Solution: Histogram of Oriented Gradients (HoG)

HoG based object detection

30

Histogram of Oriented Gradients (HoG)

Gradient computation

Gradient binning


31

Parts-based detection

Star-based representation:

Real example:


32

Analysis of detection method on F1 footage

33

Analysis of distibution of certainty-scores of 1 particular render on 1 frame

Example: best certainty-score is 10 -> only 4% of matches score > 9 -> ony 14% of matches score > 5

Analysis of detection method on F1 footage

34

Score deduction in neighboring renders


• Car detection


35

Car tracking

36

36,000 renders (360*100 angles) * 500 frames = 18,000,000 detections -> coarse selection of renders and frames for first pass.

Tier 1

Car tracking

37

36,000 renders (360*100 angles) * 500 frames = 18,000,000 detections -> coarse selection of renders and frames for first pass.

Tier 1

Car tracking

38

Tier 2 Starting from best detection: 1) Rerun detection on best frame 2) Detect surrounding frames

Surrounding frames: - Car is at nearly same position - Car viewed under nearly same angle

Car tracking

39

Tier 3: smoothing Smooth angles Smooth boudingbox location and size

Horizontal angle

Vertical angle

Car tracking

40

Tier 3: smoothing Smoothed angles can still be used Exact size and position needed

Horizontal angle

Vertical angle

Car tracking

41

Tier 3: smoothing Keypoint tracking

Track points using feature-tracking algorithm Calculate new positions using rendering pipeline

Car tracking

42

Tier 3: smoothing Keypoint tracking

Car tracking

43

Tier 3: smoothing Car rotation and template matching

1) Best match

2) Rotate to desired angle 3) Find rotated car in cropped image

Car tracking

44


Car tracking

45


1) Detect car 2) Rotate car to desired angle 3) Average multiple rotated cars

Car tracking

46

Tier 3: smoothing





• 3D rotation


• Inpainting


48


Retinal Disparity

3D effect

Inpainting

49

Single frame methods:

Geometrical inpainting: Patch-based inpainting:

Inpainting

50


Geometrical inpainting: Patch-based inpainting:

Inpainting

51


Objective comparison Only outermost mixels important

Inpainting

52

Video inpainting

Inpainting

53

Video inpainting

Inpainting

54

Video inpainting

Inpainting

55

Video inpainting

Inpainting

56

Video inpainting





• 3D rotation


• Inpainting

• Non-object based 3D conversion

Non-object based 3D conversion

• Overlay detection

• Conversion of background to 3D

58




59

Overlay detection

60

Graphical overlay: - Not always present - If present, always at fixed location

Template: averaged over 100 frames




61

Conversion of background to 3D

62

Make3D algorithm - Texture gradients - Texture variations - Reduced contrast





• 3D rotation


• Inpainting


Overview of 3D conversion process

64

1) Inpaint car and overlay 2) Generate depthmap 3) Shift background 4) Inpaint background 5) Add overlay 6) Add car 7) Merge to 3D





• 3D rotation


• Inpainting


Result and conclusion

66

Is it a feasible method of 2D-3D conversion?

Yes, but 1) very time-consuming -> also the reason why most of this thesis featured 1 single fragment 2) differences between chosen render-viewpoint and car can become noticeable -> can be solved/prevented by using realistically rendered car instead of the actual car from the frame Car angle, size and location known at each frame -> could be used for other purposes, for example advanced graphical overlays

Result: video on 3D tv

Questions?

67

2D to 3D conversion of formula 1 footage

Technology

Transcript of 2D to 3D conversion of formula 1 footage