The Free Safety Problem The Free Safety Problem Using Gaze Estimation as a Meaningful Input to a Homing TaskUsing Gaze Estimation as a Meaningful Input to a Homing Task

Albert GoldfainAlbert Goldfain

CSE 668: Animate Vision PrinciplesCSE 668: Animate Vision Principles

Final Project PresentationFinal Project Presentation

Defining the problemDefining the problem

American Football TerminologyAmerican Football TerminologyA “A “quarterbackquarterback” is an offensive player who is responsible for ” is an offensive player who is responsible for throwing the ball to “receivers”.throwing the ball to “receivers”.A “A “receiverreceiver” is an offensive (moving target) player who runs in a ” is an offensive (moving target) player who runs in a predefined pattern (or “route”) known to the quarterback and the rest predefined pattern (or “route”) known to the quarterback and the rest of his team. The receiver attempts to catch the ball if the of his team. The receiver attempts to catch the ball if the quarterback decides to throw to him.quarterback decides to throw to him.The “The “free safetyfree safety” is a defensive player who is responsible for ” is a defensive player who is responsible for impeding the targeted receiver from catching the ball. The free impeding the targeted receiver from catching the ball. The free safety is called “free” because he is not assigned to any particular safety is called “free” because he is not assigned to any particular receiver, and must make the decision of who to guard during play. receiver, and must make the decision of who to guard during play. He is labeled “safety” because he is usually the last line of defense He is labeled “safety” because he is usually the last line of defense and stands 6 or 7 yards behind all of the other defenders.and stands 6 or 7 yards behind all of the other defenders.

The goal of this project is to explore the visual aspects of the free safety’s decision making process

as he selects a receiver to guard.

Existing Literature/ResearchExisting Literature/Research

Gaze / Pose EstimationGaze / Pose Estimation– Perception of head orientation.Perception of head orientation. [Wilson et al] [Wilson et al]– Motion Segmentation and Pose Recognition with Motion History Motion Segmentation and Pose Recognition with Motion History

GradientsGradients. [Bradski, Davis]. [Bradski, Davis]– Head pose estimation without manual initialization.Head pose estimation without manual initialization. [Fitzpatrick] [Fitzpatrick]– 3D Face pose estimation and tracking from a monocular camera.3D Face pose estimation and tracking from a monocular camera. [Ji, [Ji,

Hu]Hu]– Pose determination of human faces by using vanishing pointsPose determination of human faces by using vanishing points. [Wang, . [Wang,

Sung]Sung]– Many, many more.Many, many more.

Computational Vision in SportsComputational Vision in Sports– RoboCupRoboCup [www.robocup.org] [www.robocup.org]– Computers Watching FootballComputers Watching Football. [MIT Media Lab/Vision and Modelling . [MIT Media Lab/Vision and Modelling

Group]Group]

Initial ObservationsInitial Observations

The problem is too large without domain knowledge/constraints.The problem is too large without domain knowledge/constraints.

A robotic (embodied computational) free safety would have to act very A robotic (embodied computational) free safety would have to act very quickly…no time for recovery paradigm approach…a good application for quickly…no time for recovery paradigm approach…a good application for active/animate vision.active/animate vision.

From the free safety’s point of view, the task of guarding a receiver can be From the free safety’s point of view, the task of guarding a receiver can be done using only two dimensions (width and depth). This is why a coach done using only two dimensions (width and depth). This is why a coach can draw x’s and o’s on a chalkboard without confusing his players. can draw x’s and o’s on a chalkboard without confusing his players.

Domain ConstraintsDomain Constraints

Players wear uniformsPlayers wear uniforms

Markings on field can be used to determine distances with Markings on field can be used to determine distances with precisionprecision

Good lighting can be assumedGood lighting can be assumed

SidelinesSidelines

2D internal representation for free safety2D internal representation for free safety

QB head pose restrictionsQB head pose restrictions

Free Safety SubtasksFree Safety Subtasks

Quarterback Centered Coordinate Quarterback Centered Coordinate SystemSystem

1

(0)(0)

n

i

is

xx

n

• At each instant t, wide receiver i’s location is given by the parametric coordinates (xi(t),yi(t)) in this space.

• The initial pre-snap values for the safety position are given by

(0)s losy D

Safety Centered Polar Coordinate Safety Centered Polar Coordinate SystemSystem

• Coordinates for wide receiver i can be expressed with a distance from the safety ri and rotation angle s

• We can convert from safety coordinates to quarterback coordinates using

( ) cos

( ) sin

i i si

i i si

x t r x

y t r y

Finding the QuarterbackFinding the QuarterbackSafety could use one of two techniques:

• Use constraints of football formations to determine who is behind the center (a lineman on the line of scrimmage) at time t0

• Search image for a player that has the ball immediately after time t0

Once the quarterback is found, the head pose image must be segmented out.

Given a new image…

…find best correlation to a stored image

Image Database: Views of Quarterback Pose at 10 deg. Angle Increments.

Using Image Difference Matching Using Image Difference Matching as a Similarity Metricas a Similarity Metric

• Difference will be minimized when test image and stored image are the most similar. Thus the darkest of these image differences should be closest match.• The catch: often times, symmetries in image differences at certain angles will yield false positives. Perhaps image difference is a weak similarity metric!• The solution: Look at the angular “nearest neighbors” of a proposed solution.

Some Test Results for Image Some Test Results for Image Difference ClassificationDifference Classification

Test Head Pose 1

0

20

40

60

80

100

1 3 5 7 9 11 13 15 17 19

Stored Image Index

Ab

so

lute

Im

ag

e

Dif

fere

nce

Test Head Pose 2

0

20

40

60

80

100

1 3 5 7 9 11 13 15 17 19

Stored Image Index

Ab

solu

te Im

age

Dif

fere

nce

Acute Angle Obtuse Angle

Test Image Stored Image Test Image Stored Image

Labeling the Receivers in Labeling the Receivers in Quarterback’s FOVQuarterback’s FOV

Safety begins turning motion based on Safety begins turning motion based on θθqq

If no receiver is found, safety turns back to the If no receiver is found, safety turns back to the quarterback to estimate quarterback to estimate θθqq once again. once again.

If multiple receivers are found, a suggested If multiple receivers are found, a suggested egomotion “action” is output and egomotion “action” is output and θθqq is estimated is estimated again.again.If only one receiver is found, the safety approaches If only one receiver is found, the safety approaches that receiver.that receiver.

Further Work / Potential Further Work / Potential ApplicationsApplications

Working on decision making egomotion “action” function Working on decision making egomotion “action” function for the safety. Such a function needs to balance the for the safety. Such a function needs to balance the early goal of maintaining a large field of view with the early goal of maintaining a large field of view with the later goal of approaching the wide receivers.later goal of approaching the wide receivers.

This function needs to be time based. If the defensive This function needs to be time based. If the defensive team is doing its job, the quarterback should be running team is doing its job, the quarterback should be running out of time to throw. out of time to throw.

I am willing to take any suggestions I am willing to take any suggestions I believe further research of this problem could yield I believe further research of this problem could yield important results in robotics, video game AI and important results in robotics, video game AI and computer vision in general.computer vision in general.