Allen HansonChris Jaynes

Mauricio MarengoniEdward RisemanHoward Schultz

Frank Stolle

University of Massachusetts3D Building Reconstruction

With: Keith Hoepfner and Heddi Plumb

Ascona 2001

Talk Outline

Introduction: RADIUS and Ascender IOverview of Ascender IIGeometric Reconstruction of RooftopsExperimentsConclusions

Lab URL: http://vis-www.cs.umass.eduAPGD URL: http://vis-www.cs.umass.edu/projects/ascender/ascender.html

Research Goal(s)

EO, Digital Elevation Maps

IFSAR, Multi/Hyperspectral

3D Geometric Site Model Reconstruction

With multiple objects,

Using multiple strategies,

From multiple images

RADIUS: Ascender I

System for Automatic Site Modelling

Multiple Images Geometric Models

2D polygon detection line extraction corner detection perceptual grouping

epipolar matchingmulti-image triangulation

geometric constraints precise photogrammetry

extrusion to ground planeprojective texture mappingCVIU 72(2) 1998

Ascender Works

Extensive evaluation Good detection but high false alarms Good geometric accuracy

Ground Truth Ascender Results

.

10

20

30

40

50

60

70

80

90

73 Ground Truth Polygons

100

110

713

711

927525

Combined, 95%

Combined, 89%

.1 .2 .3 .4 .5 .6 .7 .8 .9

Detection Rates

Ascender Doesn’t Work

Delivered to NEL (NIMA) Applied to classified imagery Performance not as expected

High failure rate

(buildings not detected) High false positive rate

WHY? Imagery didn’tconform to design constraints!

No Buildings Detected!

Some Observations

IU Algorithms work within correct context constrained contexts constrained object classes

Domain knowledge provides constraints from domain, partial results, strategies

Many strategies, only correct ones used selective application correct parameters fuse results from individual strategies into

complete reconstruction

Key to Success

A system which:

applies the right strategy

at the right time

in the right place

with the right constraints

Ascender II Overview

Dealing with Context: Ascender II

Goals 3D Geometric Site Model Reconstruction Focus: More complex building structures Context sensitive control strategies for applying algorithms

Multiple strategies Multiple images Multiple sensors

Knowledge Representation, Control and Inferencing layered Bayesian networks

EO, Digital Elevation Maps

IFSAR, Multi/Hyperspectral

Observations

Buildings = Location + Geometry

Reconstruction = Localization + 3D Recovery

3D Recovery is a two-stage process

Where are the buildings? What is their structure?

Ascender II Overview

Bayes NetworkControl SystemCP

CPCP

CP

Ascender II Knowledge BaseAvailable Data

ImageryElevation dataSite Model

ASCENDER ISystem

andStrategies

Ascender IIRCDESocket I/O

CP: Control Policy

IU Algorithms work within correct context. Domain knowledge provides constraints. Choose good strategy from many alternatives.

Basic Principles

Ascender II Components

Visual Subsystem Library of IU Algorithms Geometric Database (data, models) Display/user interface through RCDE

Knowledge Base Control system Belief networks in Hugin Reasoning mechanisms over regions of discourse

Regions of Discourse Subset of available data Image regions, a model, partial interpretation

Knowledge Representation

PossibleValuesH1={j,..,k}

H1

Variable of interest

Schema/Belief Node

Evidence Policy

H3

H4

H2

First level of classification hierarchy

Second level (object subclass)

Combination of belief networks and visual schemas, implemented in Hugin.

Encodes: Domain Knowledge Acquired Site Knowledge Control Mechanism

Allows both diagnostic and causal inference.

Hierarchical topology allows simpler networks which reduces propagation time and controls complexity.

Evidence policy defines how IU strategies gather relevant evidence according to context.

Algorithms to Evidence

Define the context in which it can run currently straightforward context only disallows an algorithm too simple in the long run

Method for deriving certainty value needed to update knowledge network

Given an IU algorithm....

Algorithms

Algorithms operate on regions of discourse Ascender I Surface Fits Line Extraction and Count Junction Count Average Junction Contrast Region Height Region Height to Width ratio Edge Terminals ……………...

3D Geometric Rooftop Reconstruction

Fusing DEM and Optical Data

Registered Optical and Elevation Reconstruct Rooftops (and buildings)

Optical Image (Fort Hood) Elevation Data (Fort Hood)

Technical Overview

Registered Optical and DEM

Parameterized Model Library

Site Model Visualization

Model Indexing

Elevation Surface and Histogram

Mod

el F

it: O

ptim

izat

ion

Surface Primitive Library

51 Models in 8 Parameterized Classes

Gable

Ascender + Terrest Results

Experiments

Fort HoodAvenches

Fort Benning

Experiments

Approach Site model from Ascender I with no filtering and loose polygon acceptance criteria => high detection rate but lots of errors (false positives) Classify Ascender I regions into

{building, parking lot, open field, unknown} Classify buildings into

{multi-level building, single-level building} Then into {flat, peak, round, flat-peak, other} Site model from Ascender II after classification/reconstruction

Degraded Ascender I ModelFort Hood

22 True Positive Buildings1 Incorrect Model (A) (multi-level reconstructed as single level)

No Knowledge and Loose Polygon Acceptance Criteria

A

20 False Positive Buildings

Bayes Nets for Experiments

Planar Fit

Evidence Policy: none

Multi-LevelSingle

Building

TJunctions

GoodBad

YesNo

Region

BuildingOpen FieldParking LotComplexUnknown

Evidence Policy: none

Height

Evidence Policy:multi-EO: triangulateHeight()DEM: medianHeight()

LowMediumHigh

Planar Fit

GoodBad

Evidence Policy:

DEM: robustFit(plane)

WidthLine

Count

LJunctions

NumberT Junc.

ContrastT Junc.

Evidence Policy:EO: junctions(T)

<5>5None

<50>50Zero

Evidence Policy: Evidence Policy:EO: junction_count(T) EO: junction_contr(T)

First level of classification hierarchy

Second level (object subclass)

Level 1

Level 2

Next Slide

Level 3 of Network

Rooftop

FlatPeakRoundFlat-peakOther

Evidence Policy: none

Planar Fit

GoodBad

Evidence Policy:

DEM: robustFit(plane)

Level 3

Middle>

CornerYesNo

SideLine

YesNo

CenterLine

YesNo

ShadowFormat

PrismRoundFlat

Multi-level

YesNo

Evidence Policy: none

Planar Fit A

GoodBad

Evidence Policy:

DEM: robustFit(plane)

Level 3

Height A>

Height BYesNo

CenterLineRatio

YesNo

Planar Fit B

GoodBad

Evidence Policy:

DEM: robustFit(plane)

Multiple calls, one for each section

Visual Subsystem:Algorithms and Evidence Policies

Stereo-OpticalDEM

DEM MedianHeight

Multi-OpticalEpipolar Align.

Region-EdgeTriangulate Hgt.

USGS DTM DTM at regioncenter

HeightP{}

xF{x}

P{}

Height

Evidence policies associate algorithms with belief.

Function P maps request for evidence from knowledge base to algorithm.

F(x) maps algorithm result to belief value.

Example Algorithms: Parallel Lines T junctions DEM-Median-Height Region-Edge-Triangulate-Height Planar Fit Error

Experiments

Detection rates per Object Class (Semantic Accuracy) Three-Dimensional Accuracy

For each DataSet For each Object Class (buildings)

How do network priors influence performance? Uniform (no knowledge) versus Estimated Priors Adjustment of Priors for each dataset versus uniform

Three Datasets Fort Benning MOUT Fort Hood Facility Avenches Boat Yard

Priors and Operators Used

Network Priors by Dataset

Operators Applied by Dataset

Site Name Building ParkingLot

OpenField

SingleVehicle

Other

Fort Hood 0.35 0.2 0.2 0.15 0.1Avenches 0.32 0.25 0.38 0.03 0.02Fort Benning 0.35 0.2 0.2 0.15 0.1

SiteName

Total(avg.)

Building ParkingLot

OpenField

SingleVehicle

FortHood

3.9 4.25 4.00 3.29 0.0

Avenches 4.7 5.22 5.00 4.0 0.0FortBenning

2.9 3.00 0.0 0.0 2.0

SkipF.Hood

Ascender II ClassificationContext Sensitive Strategies

43 Regions, 4 Misclassified, 1 Unknown

88% Accuracy

Ascender II Reconstruction

3 False Positives23 True Positive Buildings0 Incorrect Models (single, multi level)3 New Functional Areas

Site Reconstruction:Fort Hood

Reconstruction Accuracy:Fort Hood

Object Class TruePositive

FalsePositive

FalseNegative

Completeness Correctness

Building (General) 23 3 10 70.0% 88.5%Peak Roof 0 0 1 0% 0%Flat Roof 22 2 9 71.0% 91.6%Cylinder Roof 0 1 0 0% 0%

Detection Rates by Object Class

Error MetricIV Planimetric 0.69IV Altimetric 0.52IV Absolute 0.864Centerline 0.491PercentageOverlap

98.4%

Median Accuracy Measures , Building Classes

SkipAvenches

ISPRS Avenches Dataset

Two Overlapping Optical Images (60% sideward)

Corresponding DEM Calibration provided by ETH 200x220 meter “Boatyard” Ground Sample Dist: ~ 0.13 meters Reference model provided by ETH

Regions and Classifications:Avenches

Parking Lot

Peak Bldg.

Peak Bldg.

Peak Bldg.

Peak Bldg.

Open Field

Flat Bldg.

Flat Bldg.

Flat Bldg.

Flat Bldg.

Flat Bldg.

Peak Bldg.

Site Reconstruction:Avenches

Skip Benning

Fort Benning MOUT Site

Two overlapping downlooking optical images DEM computed from image pair using Terrest Camera resection (SRI) Reference Model (SRI) GSD, optical view, 0.31 meters

Optical DEM

Original Image Ascender I Polygons

Polygons from SAR

+

Final Polygons

MOUT Input Polygons

MOUT Site: Classification

Object Class True False False Positive Positive Negative

Building (General) 19 1 0Peak Roof 14 1 0Flat Roof 5 0 0Single Car 1 1 2

Flat roof building detectedas peak.

Low walled entrance (open field)Detected as single vehicle.

Missed detection of two vehicles.

Overall classification accuracy: 91%

Recovery of 4 attached buildings.

Model Indexing:MOUT Region 8

Constructed Surface Mesh Corresponding Gaussian Sphere

PlaneFlat PeakPeak (15)Peak (35)

.939

.819

.742

.601

Correlation Scores

MOUT Site: Accuracy

Detection Rates, Full Evaluation

Detection Rates, FOA Regions Only

Object Class TruePositive

FalsePositive

FalseNegative

Completeness Correctness

Building(General)

18 1 4 81.8% 94.7%

Peak Roof 14 1 0 100% 93.3%Flat Roof 5 0 0 100% 100%Single Car 1 1 2 33.3% 50%Overall Rates 20 2 2 90.1% 90.1%

Object Class TruePositive

FalsePositive

FalseNegative

Completeness Correctness

Building(General)

18 1 0 100.0% 94.7%

Peak Roof 14 1 0 100% 93.3%Flat Roof 5 0 0 100% 100%Single Car 1 1 0 100.0% 50%Overall Rates 20 2 0 90.1% 90.1%

MOUT Site: Reconstruction

MOUT Reconstruction

SkipKodak

Examples on Three Image Chips

Hospital

FineArts

Industrial

N

Building Localization

Optical Lines 2D PolygonsBuilding FP Hypotheses

+Final Building

Location Hypotheses

+ Classifier:TreesBuildingsGround

Building FP Hypotheses

Elevation BlobsBuildings (st. lines)Trees (no st. lines)

v Building FP Hypotheses

Reg

istr

atio

n

FP = Footprint

Building Localization: Polygons

Group

Focus (optional)

Graph Polys

GeometricRecovery

Classification

Canny Edges Grouped Lines Hypotheses

Building FootprintOn Range Data

Essentially Ascender I 2D Polygon Detection Algorithm with additional external inputs providing additional constraints.

Building Localization: Blobs

Bald Earth Model

Thresholded using bald earth

modelFiltered using

Long Lines and Region Size

First Threshold

Lines from Optical

Range Data

Localization Results

Hospital

Industrial

Fine Arts

Final Building Models

Conclusions

Vision works when properly constrained and focused Still a lot of work to do on:

Basic Algorithms Knowledge Representations Representing and Using Context and Constraints Inferencing and Causal Reasoning System Architectures

Conclusions, Part 2

No shortage of interesting research problems!

Lab URL: http://vis-www.cs.umass.edu

3D Geometric Rooftop Reconstruction

Model-Based Indexing

Use DEM and focus-of-attention to recognize rooftop shape.

Building rooftops can be modeled as the union of a small set of

parameterized surfaces.

Variety arises from different combinations and parameterizations.

Index into a library of rooftop models.

Provide initial parameters for final reconstruction (robust fitting)

Regions classified as single-building are then passed to the Model-

Indexing algorithm for recognition.

Library of models is filtered based on the distribution

at the single-building node.

DEM + Optical 2: Complex Roofs

Registered Optical and Elevation Reconstruct Rooftops (and buildings)

Optical Image (Fort Hood) Elevation Data (Fort Hood)

Technical Overview

Registered Optical and DEM

Parameterized Model Library

Site Model Visualization

Model Indexing

Elevation Surface and Histogram

Mod

el F

it: O

ptim

izat

ion

Model-Based Indexing

Use DEM and focus-of-attention to recognize rooftop shape.

Building rooftops can be modeled as the union of a small set of

parameterized surfaces.

Variety arises from different combinations and parameterizations.

Index into a library of rooftop models.

Provide initial parameters for final reconstruction (robust fitting)

Regions classified as single-building are then passed to the Model-

Indexing algorithm for recognition.

Library of models is filtered based on the distribution

at the single-building node.

Visual Subsystem:Algorithms and Evidence

Policies

Evidence policies associate algorithms with belief.

Function P maps request for evidence from knowledge base to algorithm.

F(x) maps algorithm result to belief value.

Example Algorithms: Parallel Lines T junctions DEM-Median-Height Region-Edge-Triangulate-Height Planar Fit Error

Experimental Method

Use “reference model” of each site as basis for evaluation

Centerline Distance to measure the difference between two arbitrary

polygons/wireframes

tessellate target and reference, take average of minimum distance for all

points between target and reference, and reference to target.

measures: size, shape, location error

Detection: C-dist(T,R) < 0.2*cube-root(volume(T))

Planimetric, Altimetric, and Intervertex error

Voxel Overlap

Completeness = 100 x (True Positives)/(True Positives + False

Negatives)

Correctness = 100 x (True Positives)/(True Positives + False Positives)

Algorithm Library

Library contains both lightweight and heavyweight processes

Algorithms expected to perform only under constrained conditions

Algorithms contain descriptions of context under which they can be applied