Landing a UAV on a Landing a UAV on a Runway Using Image Runway Using Image RegistrationRegistration

Andrew Miller, Don Harper, Mubarak Andrew Miller, Don Harper, Mubarak ShahShahUniversity of Central FloridaUniversity of Central FloridaICRA 2008ICRA 2008

OverviewOverview

►System for landing a UAV on a runwaySystem for landing a UAV on a runway Small RC airplane Small RC airplane Only sensor is a fixed, forward-looking Only sensor is a fixed, forward-looking

cameracamera Finds the runway using SIFT registrationFinds the runway using SIFT registration Linear control systemLinear control system

►ExperimentsExperiments Microsoft Flight Simulator (no flight model)Microsoft Flight Simulator (no flight model) Partial implementation on a real UAVPartial implementation on a real UAV

Human operator for high level control

Laptop Computer for Vision Processing and Control Algorithms

Ground Station

Block DiagramBlock Diagram

RC Plane with Camera

30 frames per second720x480 pixels RGBDownsample to 360x240

Remote Control (72 Mhz)

Analog Video (NTSC 900 Mhz)

Main StepsMain Steps

1.1. Locate the runway in each video Locate the runway in each video frameframe

2.2. Estimate the attitude of the UAVEstimate the attitude of the UAV

3.3. Steer the UAV towards the runway Steer the UAV towards the runway maintaining the correct glideslopemaintaining the correct glideslope

1. Locate the Runway1. Locate the Runway

►Base point and vanishing pointBase point and vanishing point(location and orientation)(location and orientation)

Base point

Vanishing point

Planar HomographyPlanar Homography► The 3x3 planar homography matrix The 3x3 planar homography matrix

projects every point in the reference projects every point in the reference frame to the corresponding point in frame to the corresponding point in the incoming video framethe incoming video frame

Reference FrameWarped Reference Frame

Test Frame

Find the Homography usingFind the Homography usingSIFT and RANSACSIFT and RANSAC

► SIFT Feature MatchingSIFT Feature Matching 200-500 feature points, 100-200 matches200-500 feature points, 100-200 matches Chosen greedily, least ambiguous firstChosen greedily, least ambiguous first

► Planar homography between Planar homography between correspondencescorrespondences

► RANSAC to discard outliersRANSAC to discard outliers

Stack of Reference FramesStack of Reference Frames

► Prepare a reference frames from a videoPrepare a reference frames from a video Annotate the runway and vanishing pointAnnotate the runway and vanishing point

Sample the frames (more samples at lower Sample the frames (more samples at lower altitudes)altitudes)

Terrain features are important (not just the Terrain features are important (not just the runway)runway)

Stack of Reference FramesStack of Reference Frames

Using the StackUsing the Stack

►Keep track of the current indexKeep track of the current index Highest number of SIFT matches = most Highest number of SIFT matches = most

similar viewpointsimilar viewpoint

►Only need to compare adjacent framesOnly need to compare adjacent frames

Previous Closest Match SIFT Matching

2. Estimate the UAV Attitude2. Estimate the UAV Attitude► 6 Degrees of Freedom6 Degrees of Freedom

Pitch, Bank, Heading, Elevation, Distance, CoursePitch, Bank, Heading, Elevation, Distance, Course► StrategyStrategy

Ignore Ignore DistanceDistance Find Find PitchPitch and and BankBank from the horizon line (x-axis) from the horizon line (x-axis) Find Find ElevationElevation, , HeadingHeading, , Course Course from the runwayfrom the runway

Intuitive GeometryIntuitive Geometry

►Relationship between runway Relationship between runway appearance and UAV attitudeappearance and UAV attitude This is how human pilots land visuallyThis is how human pilots land visually

Too High On Target Too Far Right

Formal GeometryFormal Geometry

► 3D Projection3D Projection C = Internal CalibrationC = Internal Calibration

R = External CalibrationR = External Calibration

► Small Angle ApproximationSmall Angle Approximation Assume the UAV is flying smooth and levelAssume the UAV is flying smooth and level

2. Estimate the UAV Attitude2. Estimate the UAV Attitude

►Recover the orientation parametersRecover the orientation parameters

►Vanishing point of the runwayVanishing point of the runway

►Beginning of the runwayBeginning of the runway

Find the HorizonFind the Horizon► Horizon estimation algorithm by Ettinger, et al.Horizon estimation algorithm by Ettinger, et al.► Based on Differing Color DistributionsBased on Differing Color Distributions► Used to recover two (pitch / bank)Used to recover two (pitch / bank)

Correct Horizon Wrong Horizon

3. Control the UAV3. Control the UAV► Cascaded Linear Feedback Cascaded Linear Feedback

ControllerController Two separate chainsTwo separate chains Two gainsTwo gains

► ProportionalProportional► IntegralIntegral

► IntuitiveIntuitive If UAV is too far right, steer If UAV is too far right, steer

leftleft If UAV is too high, pitch If UAV is too high, pitch

downdown Bank angle is derivative of Bank angle is derivative of

heading, heading is heading, heading is derivative of coursederivative of course

Pitch is derivative of Pitch is derivative of elevationelevation

PI 1

PI 1

PI 2

PI 3

PI 2

Course

Heading

Bank

Elevation

Pitch

Autopilot GUIAutopilot GUI

Algorithm PerformanceAlgorithm Performance

►Multiple stagesMultiple stages Control loops run at 50 HzControl loops run at 50 Hz

►Integrates smoothly even while input stays Integrates smoothly even while input stays samesame

Horizon detection runs at 10 HzHorizon detection runs at 10 Hz►Pitch and bank are the most sensitivePitch and bank are the most sensitive

Runway detection runs at 2 HzRunway detection runs at 2 Hz►Elevation and course are the least sensitiveElevation and course are the least sensitive

Simulator ResultsSimulator Results

►Microsoft Flight SimulatorMicrosoft Flight Simulator►Simulator-in-the-loop (separate Simulator-in-the-loop (separate

computer)computer)

ICRA08_1140_VI_fi.mp4ICRA08_1140_VI_fi.mp4 Horizon 2007 09 Sep 11 Tue 08.13pm.aviHorizon 2007 09 Sep 11 Tue 08.13pm.avi

Simulator ResultsSimulator Results

► Error from earth curvatureError from earth curvature

Actual UAV ExperimentsActual UAV Experiments

►Only using partial implementationOnly using partial implementation Horizon stabilizationHorizon stabilization Road following (no runway available)Road following (no runway available) Only brief periods of autonomous controlOnly brief periods of autonomous control

Horizon Stabilization - Horizon Stabilization - ResultsResults

ConclusionsConclusions

►Successful but imprecise landingsSuccessful but imprecise landings►Performance is applicable to CessnaPerformance is applicable to Cessna

Slower and more stable than actual UAVsSlower and more stable than actual UAVs

►Assumption of linear system is not Assumption of linear system is not applicable near the runwayapplicable near the runway This is why the aircraft oscillates before This is why the aircraft oscillates before

landinglanding

►Future workFuture work Incorporate flight model into controller designIncorporate flight model into controller design