The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für...

99
Institut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D Building Reconstruction Franz Rottensteiner a , Gunho Sohn b , J. Jung b , Markus Gerke c , Caroline Baillard d , Sebastien Benitez d , Uwe Breitkopf a a Institute of Photogrammetry & GeoInformation, University Hannover, Germany b GeoICT Lab, York University, Toronto, Camada c University of Twente, Faculty ITC; EOS Dept., Enschede, The Netherlands d SIRADEL, Rennes, France Franz Rottensteiner XXII ISPRS Congress Melbourne 2012 26 August 2012

Transcript of The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für...

Page 1: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

The ISPRS Benchmark on Urban Object Classification and

3D Building Reconstruction

Franz Rottensteiner a, Gunho Sohn b, J. Jung b, Markus Gerke c, Caroline Baillard d, Sebastien Benitez d, Uwe Breitkopf a

a Institute of Photogrammetry & GeoInformation, University Hannover, Germany b GeoICT Lab, York University, Toronto, Camada

c University of Twente, Faculty ITC; EOS Dept., Enschede, The Netherlands d SIRADEL, Rennes, France

Franz Rottensteiner XXII ISPRS Congress Melbourne 2012 26 August 2012

Page 2: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Outline • Introduction

• Test Data

• Object Detection − Task and Evaluation Methodology − Evaluation of Building Detection − Evaluation of Tree Detection

• 3D Building Reconstruction − Task and Evaluation Methodology − Results − Discussion

• Conclusion

Page 3: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Outline • Introduction

• Test Data

• Object Detection − Task and Evaluation Methodology − Evaluation of Building Detection − Evaluation of Tree Detection

• 3D Building Reconstruction − Task and Evaluation Methodology − Results − Discussion

• Conclusion

Page 4: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

• Benchmarks are widely used in Computer Vision to make the results of different solutions comparable

• Examples: − Middlebury Stereo Vision test

(Scharstein & Szeliski, 2002)

− PASCAL VOC (Everingham et al., 2010)

• Urban object extraction in photogrammetry & remote sensing?

Benchmarks

Page 5: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Benchmark on (Urban) Object Extraction? • Benchmarks for object extraction by OEEPE/EuroSDR:

− Building extraction (Kaartinen et al., 2005): scanned aerial images + airborne laserscanner (ALS) data

− Road extraction (Mayer et al., 2006): orthophotos from scanned aerial images and from satellite images

− Updating of building data bases (Champion et al., 2009): digital aerial / satellite images, ALS + (scanned) orthophoto

• Outdated as far as airborne sensors are concerned

Need for new benchmark data sets for urban object extraction

Page 6: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Benchmarks on (Urban) Object Extraction! • Data set of ISPRS WG III/4 “Complex scene analysis and 3D

reconstruction”: http://www.itc.nl/ISPRS_WGIII4/tests_datasets.html

• The data have been downloaded by ~140 participants

• Tasks 1) Urban object detection: buildings, trees, roads, cars 2) 3D building reconstruction: roof plane boundaries

• Results submitted by the participants are evaluated by WG III/4 based on reference data (cf. http://tinyurl.com/ISPRStest)

• This presentation: report on the results

Page 7: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Outline • Introduction

• Test Data

• Object Detection − Task and Evaluation Methodology − Evaluation of Building Detection − Evaluation of Tree Detection

• 3D Building Reconstruction − Task and Evaluation Methodology − Results − Discussion

• Conclusion

Page 8: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Test Data: Overview • Two urban test data sets consisting of

− High-resolution digital aerial images + orientation data

− ALS point cloud

− Digital surface model (DSM) from last ALS pulses

• Reference data were generated by manual stereo plotting

• Data sets:

1) Vaihingen / Enz (Germany): typical “old world” scenario

2) Downtown Toronto (Canada): typical “new world” scenario

Page 9: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Data Set 1: Vaihingen / Enz (Germany) • Aerial images: − Z/I DMC (c = 120 mm) − 20 images in 4 strips − GSD: 8 cm − 11 bit CIR (pan-sharpened) − Accuracy of orientation: ± 1 pixel

Area 1

Area 2

Area 3

Roads

Page 10: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Vaihingen / Enz: Image Configuration

Strip 3 10030060 10030061 10030062 10030063

Strip 25

10250132

10250133

10250134

10250135

10250130

10250131

Strip 5 10050103 10050104 10050105 10050106 10050107

Strip 4 10040081 10040082 10040083 10040084 10040085

• 4 strips (1 cross strip)

• 70 % forward overlap

• 60% side lap fourfold overlap guaranteed!

Page 11: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Data Set 1: Vaihingen / Enz (Germany) • Aerial images: − Z/I DMC (c = 120 mm) − 20 images in 4 strips − GSD: 8 cm − 11 bit CIR (pan-sharpened) − Accuracy of orientation: ± 1 pixel

• ALS data: − Leica ALS 50 − 5 strips (30% overlap) − 4 - 7 points/m2, multiple pulses

with intensities − Accuracy: ± 3 cm

Area 1

Area 2

Area 3

Roads

Page 12: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Vaihingen / Enz: ALS Configuration • 5 strips

• DSM interpolated at 25 cm

Strip 10

Strip 9

Strip 5 Area 2 Area 3

Strip 7

Strip 3

Area 1

Roads

Area 2

Area 3

Area 1

Roads

Page 13: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Vaihingen: Test Area 1 • Dense urban development, historic inner city area

Area 1: 125 m x 200 m Reference: 37 building outlines accuracy: ± 10 cm

Reference: 105 trees

accuracy: ± 1 m

Page 14: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Vaihingen: Test Area 1 • Dense urban development, historic inner city area

Reference (roofs) accuracy: ± 10 cm

Area 1: 125 m x 200 m

Page 15: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Vaihingen: Test Area 2 • High-rise residential buildings

Reference: 14 building outlines accuracy: ± 10 cm

Reference: 162 trees

accuracy: ± 1 m

Area 2: 170 m x 190 m

Page 16: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Vaihingen: Test Area 2 • High-rise residential buildings

Reference (roofs) accuracy: ± 10 cm

Area 2: 170 m x 190 m

Page 17: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Vaihingen: Test Area 3 • Residential buildings with detached houses

Reference 56 building outlines accuracy: ± 10 cm

Reference 155 trees

accuracy: ± 1 m

Area 3: 150 m x 220 m

Page 18: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Vaihingen: Test Area 3 • Residential buildings with detached houses

Reference (roofs) accuracy: ± 10 cm

Area 3: 150 m x 220 m

Page 19: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Data Set 2: Downtown Toronto (Canada) • Aerial images: − Microsoft Vexcel UltraCam-D (c = 101.4 mm) − 13 images in 3 strips − GSD: 15 cm − 8 bit RGB (pan-sharpened) − Accuracy of orientation: ± 1 pixel

Page 20: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Downtown Toronto: Image Configuration

• 3 strips

• 60 % forward overlap

• 30% side lap occlusions due to high buildings

Strip 1

Strip 3

Strip 2

Page 21: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Data Set 2: Downtown Toronto (Canada) • Aerial images: − Microsoft Vexcel UltraCam-D (c = 101.4 mm) − 13 images in 3 strips − GSD: 15 cm − 8 bit RGB (pan-sharpened) − Accuracy of orientation: ± 1 pixel

• ALS data: − Optech’s ALTM-ORION M − 6 strips (30% overlap) − 6 points/m2, multiple pulses

with intensities − Accuracy: ± 5 cm

Page 22: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Downtown Toronto: ALS Configuration • 6 strips

• DSM interpolated at 25 cm

Page 23: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Downtown Toronto: Test Area 4 • Mixture of low and high-storey buildings, Park

Reference 58 building outlines accuracy: ± 20 cm

Area 4: 530 m x 600 m

Page 24: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Downtown Toronto: Test Area 4 • Mixture of low and high-storey buildings, Park

Reference (roofs) accuracy: ± 20 cm (X / Y)

± 15 cm (Z)

Area 4: 530 m x 600 m

Page 25: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Downtown Toronto: Test Area 5 • Downtown area, high-rise buildings

Reference 38 building outlines accuracy: ± 20 cm

Area 5: 530 m x 600 m

Page 26: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Downtown Toronto: Test Area 5 • Downtown area, high-rise buildings

Reference (roofs) accuracy: ± 20 cm (X / Y)

± 15 cm (Z)

Area 5: 530 m x 600 m

Page 27: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Outline • Introduction

• Test Data

• Object Detection − Task and Evaluation Methodology − Evaluation of Building Detection − Evaluation of Tree Detection

• 3D Building Reconstruction − Task and Evaluation Methodology − Results − Discussion

• Conclusion

Page 28: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Task of the Participants • Detection of urban objects:

− Buildings

− Trees

− (...)

• Deliverables: one of the following per object class

− Object outlines as (2D or 3D) polygons in object space

− Label image of objects + georeferencing information

− (binary mask of object pixels) + georeferencing information

Page 29: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation Methodology • Evaluation of thematic accuracy:

Completeness = TP / (TP + FN) Correctness = TP / (TP + FP) Quality = TP / (TP + FN + FP)

TP: Number of True Positives FP: Number of False Positives FN: Number of False Negatives

• Evaluation of geometrical accuracy (TP objects only): − RMS error of distances

… from extracted outlines to reference outline (buildings), no considering points with distance > 3 m

… from extracted centres of gravity to reference (trees)

Page 30: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation Methodology • Completeness / Correctness / Quality are determined for three

entities:

− per-area: based on counting TP/FP/FN pixels in a raster image (buildings only)

− per-object: an object is counted as a TP if at least 50% of its area coincide with an object in the other data set

− per-object (50 m2): These measures show the performance for the most relevant objects (covering an area > 50 m2)

• Mayer et al, 2006: Requirements for practical relevance:

− Completeness ≥ 70%

− Correctness ≥ 85%

Page 31: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Results Submitted by Participants Submitted Abbr. Affiliation Areas

by 1 2 3 4 5 A. Moussa CAL University of Calgary B+T B+T B+T D. Bulatov FIE Fraunhofer Inst. Ettlingen B J. Niemeyer HAN University of Hannover B+T B+T B+T D. Grigillo LJU University of Ljubljana B+T B+T B+T C. Liu TON Tongji University B B B W. Yao TUM TU Munich B+T B+T B+T B B P. Dorninger VSK TU Vienna B B B Q. Zhan WHU Wuhan University B+T B+T B+T

• Only one participant delivered results for Toronto (areas 4 + 5)

• One participant only delivered results for area 3

• Some participants only delivered results for building detection

Page 32: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Data Used by the Participants Submitted Abbr. Affiliation Areas

by 1 2 3 4 5 A. Moussa CAL University of Calgary B+T B+T B+T D. Bulatov FIE Fraunhofer Inst. Ettlingen B J. Niemeyer HAN University of Hannover B+T B+T B+T D. Grigillo LJU University of Ljubljana B+T B+T B+T C. Liu TON Tongji University B B B W. Yao TUM TU Munich B+T B+T B+T B B P. Dorninger VSK TU Vienna B B B Q. Zhan WHU Wuhan University B+T B+T B+T

• Original images only (1) • DSM (ALS) + orthophoto (4) • DSM (ALS) (1) • ALS points (2)

Page 33: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Processing Strategies Submitted Abbr. Affiliation Areas

by 1 2 3 4 5 A. Moussa CAL University of Calgary B+T B+T B+T D. Bulatov FIE Fraunhofer Inst. Ettlingen B J. Niemeyer HAN University of Hannover B+T B+T B+T D. Grigillo LJU University of Ljubljana B+T B+T B+T C. Liu TON Tongji University B B B W. Yao TUM TU Munich B+T B+T B+T B B P. Dorninger VSK TU Vienna B B B Q. Zhan WHU Wuhan University B+T B+T B+T

• Supervised, without segmentation (3) • Unsupervised, with segmentation (5)

Page 34: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection - Vaihingen • Area-based evaluation (average Areas 1-3)

405060708090

100

CAL FIE HAN LJU TON TUM VSK WHU

ComparCorrarQar

Completeness [%] Correctness [%] Quality [%]

%

Page 35: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection - Vaihingen • Object-based evaluation (average Areas 1-3)

405060708090

100

CAL FIE HAN LJU TON TUM VSK WHU

CompobjCorrobjQobj

Completeness [%] Correctness [%] Quality [%]

%

Page 36: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection - Vaihingen • Object-based evaluation for objects ≥ 50 m2 (average Areas 1-3)

• Large structures can be detected reliably by most methods

• All methods except WHU fulfill the requirements by (Mayer et al., 2006) for large buildings

405060708090

100

CAL FIE HAN LJU TON TUM VSK WHU

Compobj50Corrobj50Qobj50

Completeness [%] Correctness [%] Quality [%]

%

Page 37: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection - Vaihingen • RMSXY: error of planimetric distances (average Areas 1-3)

• Order of magnitude: 2-4 x resolution of ALS data

• Full potential of images is not exploited

0.50

0.60

0.70

0.80

0.90

1.00

1.10

CAL FIE HAN LJU TON TUM VSK WHU

RMSRMSXY [m]

m

Page 38: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection - Toronto • Evaluation for TUM (average of Areas 4 and 5) (RMSXY = 1.5 m)

• More difficult scenario than Vaihingen −No Infrared band −Occlusions, extremely high-rising buildings

• Nevertheless, requirements by (Mayer et al., 2006) fulfilled

405060708090

100

area-based object-based objects > 50 m2

compcorrQ

Completeness [%] Correctness [%] Quality [%]

%

Page 39: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

CAL

TP FP FN

HAN LJU

Evaluation of Building Detection – Area 1

TUM VSK WHU TON

Page 40: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

TON TUM VSK WHU

CAL

Evaluation of Building Detection – Area 2

HAN LJU

TP FP FN

Page 41: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection – Area 3

CAL HAN LJU

TON

FIE

TUM WHU VSK

Page 42: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection – Areas 4 & 5 • Results by TUM

Area 4

Area 5

Page 43: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Building Detection • Building outlines for pixel-based approaches

Page 44: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

CAL

TP FP FN

HAN LJU

Evaluation of Building Detection – Area 1

TUM VSK WHU TON

• Building outlines for pixel-based approaches

Page 45: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Building Detection • Building outlines for pixel-based approaches

• Complex structures in combination with steep terrain

Page 46: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

CAL

TP FP FN

HAN LJU

Evaluation of Building Detection – Area 1

TUM VSK WHU TON

Page 47: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Building Detection • Building outlines for pixel-based approaches

• Complex structures in combination with steep terrain

• Dense tree canopies Problems for ALS-based methods

Page 48: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

CAL

TP FP FN

HAN LJU

Evaluation of Building Detection – Area 1

TUM VSK WHU TON

Page 49: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Building Detection • Building outlines for pixel-based approaches

• Complex structures in combination with steep terrain

• Dense tree canopies Problems for ALS-based methods

• Roofs covered by grass

Page 50: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

TON TUM VSK WHU

CAL

Evaluation of Building Detection – Area 2

HAN LJU

TP FP FN

Page 51: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Building Detection • Building outlines for pixel-based approaches

• Complex structures in combination with steep terrain

• Dense tree canopies Problems for ALS-based methods

• Roofs covered by grass

• Large trees next to buildings for segmentation-based methods

Page 52: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection – Area 3

CAL HAN LJU

TON

FIE

TUM WHU VSK

Page 53: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems • Building outlines for pixel-based approaches

• Complex structures in combination with steep terrain

• Dense tree canopies Problems for ALS-based methods

• Roofs covered by grass

• Large trees next to buildings for segmentation-based methods

• Small building structures

Page 54: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection – Area 3

CAL HAN LJU

TON

FIE

TUM WHU VSK

Page 55: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Building Detection • Building outlines for pixel-based approaches

• Complex structures in combination with steep terrain

• Dense tree canopies Problems for ALS-based methods

• Roofs covered by grass

• Large trees next to buildings for segmentation-based methods

• Small building structures

• Occlusion / perspective distortion with high-rise buildings

Page 56: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection – Areas 4 & 5 • Results by TUM

Area 4

Area 5

Page 57: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Building Detection • Building outlines for pixel-based approaches

• Complex structures in combination with steep terrain

• Dense tree canopies Problems for ALS-based methods

• Roofs covered by grass

• Large trees next to buildings for segmentation-based methods

• Small building structures

• Occlusion / perspective distortion with high-rise buildings

• Shadow

• Uncommonly low buildings in a CBD environment

Page 58: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection – Areas 4 & 5 • Results by TUM

Area 4

Area 5

Page 59: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Detection - Vaihingen • Influence of used input data (area-based, average Areas 1-3)

• Observed differences are not conclusive due to small number of samples

40

50

60

70

80

90

100

images (1) DSM + ortho(4)

DSM (1) ALS points(2)

ComparCorrarQar

%

Completeness [%] Correctness [%] Quality [%]

Page 60: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

405060708090

100

segmentation /unsupervised (5)

pixels/points /supervised (3)

ComparCorrarQar

Evaluation of Building Detection - Vaihingen • Influence of processing strategy (area-based,

average Areas 1-3)

• No significant difference

%

Completeness [%] Correctness [%] Quality [%]

Page 61: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Tree Detection - Vaihingen • Object-based evaluation (average Areas 1-3)

• Very low performance for all methods compared to building detection

30405060708090

100

CAL HAN LJU TUM WHU

CompobjCorrobjQobj

Completeness [%] Correctness [%] Quality [%]

%

Page 62: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Tree Detection - Vaihingen • Object-based evaluation (average Areas 1-3)

• 50 m2 corresponds to a crown diameter of about 8 m

• Only CAL and TUM fulfill the requirements by (Mayer et al, 2006)

30405060708090

100

CAL HAN LJU TUM WHU

Compobj50Corrobj50Qobj50

Completeness [%] Correctness [%] Quality [%]

%

Page 63: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

TP FP FN

CAL HAN

Evaluation of Tree Detection– Area 1

LJU

TUM WHU

Page 64: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

TP FP FN

CAL HAN

Evaluation of Tree Detection– Area 2

LJU

TUM WHU

Page 65: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

TP FP FN

Evaluation of Tree Detection– Area 3

CAL HAN LJU

TUM WHU

Page 66: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Tree Detection • Similar problems as for buildings

• Complex scene content in combination with shadow

Page 67: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

TP FP FN

CAL HAN

Evaluation of Tree Detection– Area 1

LJU

TUM WHU

Page 68: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems in Tree Detection • Similar problems as for buildings

• Complex scene content in combination with shadow

• Small trees

Page 69: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

TP FP FN

Evaluation of Tree Detection– Area 3

CAL HAN LJU

TUM WHU

Page 70: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Object Detection: Discussion • Main buildings per plot can be detected rather reliably by most

methods, most methods can be practically relevant

• Small buildings remain a problem

• Occlusions in CBD scene Multiple overlap required?

• Comparison of data sets and processing strategies remains inconclusive

• Full geometrical accuracy potential of images not yet exploited

• Trees are more problematic than buildings, practical relevance questionable for most methods

• Area 2 (high-rise residential) offers most favourable conditions for automation

Page 71: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Outline • Introduction

• Test Data

• Object Detection − Task and Evaluation Methodology − Evaluation of Building Detection − Evaluation of Tree Detection

• 3D Building Reconstruction − Task and Evaluation Methodology − Results − Discussion

• Conclusion

Page 72: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Task of the Participants • 3D Reconstruction of Buildings (LoD2 of CityGML standard):

− Detailed roof structures

− No roof overhangs, balconies

• Deliverables:

− Roof plane outlines as 3D polygons in object space

Page 73: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation Methodology • Quality of roof plane segmentation:

− Completeness / Correctness / Quality of all roof planes (50% overlap required for TPs) of roof planes larger than 10 m2

− (Topological parameters: not reported here)

• Geometrical Accuracy:

− RMSXY: RMS errors of the planimetric distances of roof plane boundary points from reference

− RMSZ: RMS errors of height differences between synthetic DSMs

− RMSZ is also affected by segmentation errors

Page 74: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

• Two participants delivered results for Toronto (areas 4 + 5)

• Two participants only delivered results for area 3

Submitted Abbr. Affiliation Areas by 1 2 3 4 5

W. Zhang BNU Beijing Normal University X J.-Y. Rau CKU Cheng Kung Univ. Taiwan X X X X X D. Bulatov FIE Fraunhofer Inst., Ettlingen X S. Oude Elberink ITCE1 ITC, Enschede X X X S. Oude Elberink ITCE2 ITC, Enschede X X X B. Xiong ITCX ITC, Enschede X X X P. Dorninger VSK TU Vienna X X X G. Sohn YOR York University X X X X X

Results Submitted by Participants

Page 75: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Submitted Abbr. Affiliation Areas by 1 2 3 4 5

W. Zhang BNU Beijing Normal University X J.-Y. Rau CKU Cheng Kung Univ. Taiwan X X X X X D. Bulatov FIE Fraunhofer Inst., Ettlingen X S. Oude Elberink ITCE1 ITC, Enschede X X X S. Oude Elberink ITCE2 ITC, Enschede X X X B. Xiong ITCX ITC, Enschede X X X P. Dorninger VSK TU Vienna X X X G. Sohn YOR York University X X X X X

Data used by the participants

• Images only (2) • Images + ALS points (1) • ALS points (5)

Page 76: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Submitted Abbr. Affiliation Areas by 1 2 3 4 5

W. Zhang BNU Beijing Normal University X J.-Y. Rau CKU Cheng Kung Univ. Taiwan X X X X X D. Bulatov FIE Fraunhofer Inst., Ettlingen X S. Oude Elberink ITCE1 ITC, Enschede X X X S. Oude Elberink ITCE2 ITC, Enschede X X X B. Xiong ITCX ITC, Enschede X X X P. Dorninger VSK TU Vienna X X X G. Sohn YOR York University X X X X X

Degree of Automation

• Semi-automatic (2) • Fully automatic (6)

Page 77: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Submitted Abbr. Affiliation Areas by 1 2 3 4 5

W. Zhang BNU Beijing Normal University X J.-Y. Rau CKU Cheng Kung Univ. Taiwan X X X X X D. Bulatov FIE Fraunhofer Inst., Ettlingen X S. Oude Elberink ITCE1 ITC, Enschede X X X S. Oude Elberink ITCE2 ITC, Enschede X X X B. Xiong ITCX ITC, Enschede X X X P. Dorninger VSK TU Vienna X X X G. Sohn YOR York University X X X X X

Building Models

• Generic (polyhedral) (4) • Primitives (1) • Adaptive (3)

Page 78: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Reconstr. - Vaihingen • Evaluation for all roof planes (average Areas 1-3)

• Correctness > 80% for all methods

• Many undetected roof planes

30405060708090

100

BNU CKU FIE ITCE1 ITCE2 ITCX VSK YOR

CompobjCorrobjQobj

Completeness [%] Correctness [%] Quality [%]

%

Page 79: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Reconstr. - Vaihingen • Evaluation for roof planes >10 m2 (average Areas 1-3)

• Correctness > 89% for all methods

• Number of undetected planes is still relatively large

30405060708090

100

BNU CKU FIE ITCE1 ITCE2 ITCX VSK YOR

Compobj10Corrobj10Qobj10

Completeness [%] Correctness [%] Quality [%]

%

Page 80: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Reconstr. - Vaihingen • Geometrical errors(average Areas 1-3)

• Order of magnitude (planimetry): 2 – 4 x point spacing

• Height errors are relatively high, but also have influence of segmentation errors

• Full accuracy potential not exploited

0.0

0.5

1.0

1.5

BNU CKU FIE ITCE1 ITCE2 ITCX VSK YOR

RMSXYRMSZRMSXY [m] RMSZ [m]

Page 81: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Reconstr. - Toronto • Evaluation for roof planes (average Areas 4 & 5)

• Correctness > 80% for all methods

• Slightly lower completeness / correctness values than in Vaihingen

• RMS errors larger than Vaihingen

30405060708090

100

BNU YOR

CompobjCorrobjQobj

30405060708090

100

BNU YOR

Compobj10

Corrobj10

Qobj10

Compl. [%] Correct. [%] Quality [%]

% %

0.00.20.40.60.81.01.21.4

RMSXYRMSZRMSXY [m] RMSZ [m]

all planes planes > 10 m2

Page 82: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems • Small roof planes (YOR)

Page 83: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems • Small roof planes (YOR)

• Small appendices to larger roof planes (YOR)

Page 84: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems • Small roof planes (YOR)

• Small appendices to larger roof planes (YOR)

• Undersegmentation (ITCE1)

Page 85: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems • Small roof planes (YOR)

• Small appendices to larger roof planes (YOR)

• Undersegmentation (ITCE1)

• Wrong segmentation (VSK, FIE)

Page 86: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems • Small roof planes (YOR)

• Small appendices to larger roof planes (YOR)

• Undersegmentation (ITCE1)

• Wrong segmentation (FIE)

• Missing regularisation (YOR)

Page 87: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Common Problems • Small roof planes (YOR)

• Small appendices to larger roof planes (YOR)

• Undersegmentation (ITCE1)

• Wrong segmentation (FIE)

• Missing regularisation (YOR)

• Incorrect combination of planes (VSK)

Page 88: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Reconstr. - Vaihingen • Impact of used data (average Areas 1-3, roof planes > 10 m2)

• Comparison not conclusive: −Image-based methods only available for area 3 (FIE) or

semi-automatic −Only one method combining ALS + images

30405060708090

100

images(2)

images+ ALS

(1)

ALSpoints

(5)

Compobj10Corrobj10Qobj10

%

Compl. [%] Corr. [%] Quality [%]

0.0

0.5

1.0

1.5

images(2)

images+ ALS

(1)

ALSpoints

(5)

RMSXYRMSZRMSXY [m] RMSZ [m]

Page 89: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Reconstr. - Vaihingen • Impact of degree of automation (average Areas 1-3, roof

planes > 10 m2): Semi-automatic (S) vs. Fully-automatic (F)

• Semi-automatic processing can achieve a better accuracy both in planimetry and height

30405060708090

100

S (2) F (6)

Compobj10Corrobj10Qobj10

0.0

0.5

1.0

1.5

S (2) F (6)

RMSXYRMSZRMSXY [m] RMSZ [m]

Compl. [%] Corr. [%] Quality [%]

%

Page 90: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Reconstr. - Vaihingen • Impact of degree of automation (average Areas 1-3, roof

planes > 10 m2): Semi-automatic (S) vs. Fully-automatic (F)

• Semi-automatic processing can achieve a better accuracy both in planimetry and height

30405060708090

100

S (2) F (6)

Compobj10Corrobj10Qobj10

0.0

0.5

1.0

1.5

S (2) F (6)

RMSXYRMSZRMSXY [m] RMSZ [m]

Compl. [%] Corr. [%] Quality [%]

%

Page 91: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Evaluation of Building Reconstr. - Vaihingen • Impact of building models (average Areas 1-3, roof planes

> 10 m2): Generic (G) vs. Primitives (P) vs. Adaptive (P)

• Comparison is not really conclusive − Three adaptive methods are variant of the same algorithm − Only one method based on primitives

30405060708090

100

G (4) P (1) A (3)

Compobj10Corrobj10Qobj10

0.0

0.5

1.0

1.5

G (4) P (1) A (3)

RMSXYRMSZ

Compl. [%] Corr. [%] Quality [%]

% RMSXY [m] RMSZ [m]

Page 92: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Discussion • Building reconstruction works well for simply-shaped buildings

− Generally simple shape

− Dormers are small compared to dominant roof planes

• Complex shapes and small objects do not simply lead to more generalized models

• Accuracy potential of the sensors not yet fully exploited

• Results are generally sufficient for a ‘nice’ visualisation

• Fully automatic generation of topologically and geometrically correct models in complex environments

• Most favourable conditions: area 2 (high-rise residential bld.)

Page 93: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Outline • Introduction

• Test Data

• Object Detection − Task and Evaluation Methodology − Evaluation of Building Detection − Evaluation of Tree Detection

• 3D Building Reconstruction − Task and Evaluation Methodology − Results − Discussion

• Conclusion

Page 94: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Conclusion • Comparison of state-of-the-art methods

• Interested in individual methods? See sessions on the ISPRS benchmark

• Benchmark for urban object extraction is still on-going

• The data may still be obtained via the WWW

• Results are made available via the WWW

• Changed conditions of use: DGPF has allowed the use of the Vaihingen data for any scientific purpose

Page 95: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Introduction Test Data Object Detection Conclusion Building Reconstruction

Future Work • The data set should remain a standard data set for making

different methods comparable

• We are contemplating an online evaluation system

• Expansion of the data set: − New data (incl. reference!) ? − Provide DSM from matching and true orthophoto − Provide reference for labelling (rather than object detection)

• Special issue on the outcomes of the ISPRS benchmark

− Journal still to be negotiated

− We will announce this soon

Page 96: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

Acknowledgements • The Vaihingen data set was provided by the German Society

for Photogrammetry, Remote Sensing and Geoinformation (DGPF) (Cramer, 2010): http://www.ifp.uni-stuttgart.de/dgpf/DKEP-Allg.html.

• The reference for Vaihingen was generated by RAG Steinkohle AG and SIRADEL (www.siradel.com).

• The Toronto data set was provided by Optech Inc., First Base Solutions Inc. and York University.

• The reference for Toronto was created by York University.

Page 97: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

References • Bulatov D., Solbrig P., Gross H., Wernerus P., Repasi E., Heipke C., 2011. Context-based urban

terrain reconstruction from UAV-videos for geoinformation applications. IAPRSIS XXXVIII-1-C22 (on CD-ROM)

• Cramer, M., 2010. The DGPF test on digital aerial camera evaluation – overview and test design. Photogrammetrie – Fernerkundung – Geoinformation 2(2010):73-82.

• Champion, N., Rottensteiner, F., Matikainen, L., Liang, J., Hyyppä, J., Olsen, B., 2009. A test of automatic building change detection approaches. IAPRSIS XXXVIII-3/W4:145-150.

• Dorninger, P., Pfeifer, N., 2008. A comprehensive automated 3D approach for building extraction, reconstruction, and regularization from airborne laser scanning point clouds. Sensors 8(11):7323-7343.

• Everingham, M., Van Gool, L., Williams, C., Winn, J., Zisserman, A., 2010. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision 88(2): 303-338.

• Griffin, G., Holub, A., Perona, P., 2007. Caltech-256 Object Category Dataset. Technical report 7694, California Institute of Technology, http://authors.library.caltech.edu/7694.

• Grigillo, D., Kosmatin Fras, M., Petrovič, D., 2011. Automatic extraction and building change detection from digital surface model and multispectral orthophoto. Geodetski vestnik 55: 28-45.

• Gröger, G., Kolbe, T. H., Czerwinski, A., Nagel, C. 2008. OpenGIS city geography markup language (CityGML) encoding standard, Version 1.0.0, OGC Doc. No. 08-007r1 http://www.opengeospatial.org/standards/citygml (10/01/2012).

Page 98: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

References • Jwa, Y., Sohn, G., Cho, W. and Tao, V., 2008. An implicit geometric regularization of 3D building

shape using airborne LiDAR Data. IAPRSIS XXXVII-B3A:69-76. • Kaartinen, H., Hyyppä, J., Gülch, E., Hyyppä, H., Matikainen, L., Hofmann, A. D. Mäder, U.,

Persson, A., Söderman, U., Elmqvist, M., Ruiz, A., Dragoja, M., Flamanc, D., Maillet, G., Kersten, T., Carl, J., Hau, R., Wild, E., Frederiksen, L., Homgaard, J., Vester, K., 2005. Accuracy of 3D city models: EuroSDR comparison. IAPRSIS XXXVI-3/W19:227–232.

• Liu, C., Shi, B., Xuan, X., Nan, L., 2012. LEGION segmentation for building extraction from LiDAR data. Accepted for publication in IAPRSIS XLIX-B3.

• Liu, X., Wang, D. L., 1999. Range image segmentation using a relaxation oscillator network. IEEE Transactions on Neural Networks 10 (3):564-573.

• Mayer, H., 2008. Object extraction in photogrammetric computer vision. ISPRS J. Photogrammetry & Remote Sens. 63(2):213-222.

• Mayer, H., Hinz, S., Bacher, U., Baltsavias, E., 2006. A test of automatic road extraction approaches. IAPRSIS XXXVI-3: 209–214.

• Moussa, A., El-Sheimy, 2012. A new object based method for automated extraction of urban objects from airborne sensors data. Accepted for publication in IAPRSIS XLIX-B3.

• Niemeyer, J., Wegner, J.D., Mallet, C., Rottensteiner, F., Soergel, U., 2011. Conditional random fields for urban scene classification with full waveform LiDAR data. U. Stilla et al. (Eds.): PIA 2011, LNCS 6952, pp. 233-244.

Page 99: The ISPRS Benchmark on Urban Object Classification and 3D ... · PDF fileInstitut für Photogrammetrie und GeoInformation The ISPRS Benchmark on Urban Object Classification and 3D

Institut für Photogrammetrie und GeoInformation

References • Oude Elberink, S., Vosselman, G., 2009. Building reconstruction by target based graph matching

on incomplete laser data: analysis and limitations. Sensors, 9(8):6101-6118. • Oude Elberink, S., Vosselman, G., 2011. Quality analysis on 3D building models reconstructed

from airborne laser scanning data. ISPRS J. Photogrammetry & Remote Sens. 66(2) 157-165. • Rau, J.-Y., Lin, B.-C., 2011. Automatic roof model reconstruction from ALS data and 2D ground

plans based on side projection and the TMR algorithm. ISPRS J. Photogrammetry & Remote Sens. 66 (6) (supplement):s13-s27.

• Rutzinger, M., Rottensteiner, F., Pfeifer, N., 2009. A comparison of evaluation techniques for building extraction from airborne laser scanning. IEEE J. Selected Topics in Applied Earth Observations & Remote Sens. 2(1):11-20.

• Scharstein, D., Szeliski, R., 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International J. Computer Vision, 47(1/2/3):7-42,

• Sohn, G., Huang X. and Tao, V., 2008. Using binary space partitioning tree for reconstructing 3D building models from airborne LiDAR data. PE & RS 74(11):1425-1440.

• Zhang, W., Grussenmeyer, P., Yan, G., Mohamed M., 2011. Primitive-based building reconstruction by integration of Lidar data and optical imagery. IAPRSIS XXXVIII-5/W12.