Point Cloud Processing 2018 March 13th, 2018 POINT … Vision community), ... Photogrammetry...

3/13/2018

1

Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING

POINT CLOUD

ACQUISITION & STRUCTURING

Fabio REMONDINO

3D Optical Metrology (3DOM)

Bruno Kessler Foundation (FBK)

Trento, Italy

Email: [email protected]

http://3dom.fbk.eu

1

with contributions from FBK-3DOM members:

Isabella Toschi, Fabio Menna, Emre Oezdemir,

Daniele Morabito, Elisa Farella, Erica Nocerino

Point Cloud Processing 2018

March 13th, 2018 – TU Delft, the Netherlands

Fabio Remondino - POINT CLOUD ACQUISITION & STRUCTURING2

hardware / sensors

dense image matching algorithms

CONTENTS

POINT CLOUD

ACQUISITION & STRUCTURING

classification / segmentation

mesh / polygonal model generation

3/13/2018

2


generation / updating of 3D city models

forest mapping / vegetation analytics

monitoring corridor infrastructures

volume computations (mining, landslides, etc.)

heritage documentation and valorization

Building Information Modeling (BIM)

flood modeling

change detection

tunnel inspection

monitoring coastal erosion

dike monitoring

etc.

WHERE DO WE USE POINT CLOUDS?



3D mapping / quantification of snow / ice lost on the Marmolada glacier

Point cloud 2014

Point cloud 2014 vs 2009

3/13/2018

3


http://3dom.fbk.eu/repository/3Dpointclouds/celva/index.html


3D mapping of WWI fortifications: underground tunnels and under-forest trenches,

classification of trench structures and gallery components (enrances, riflemen

emplacements, barracks, etc,


Brain anatomy with photogrammetric point clouds at 0.05 mm resolution to (i)

study the white matter for an exhaustive understanding of the brain diseases and (ii)

identify axons in the white matter responsible to “transport” information across the brain

http://3dom.fbk.eu/repository/brain/index.html


[Nocerino, E., Menna, F., Remondino, F., Sarubbo, S., De Benedictis, A., Chioffi, F., Petralia, V., Barbareschi, M., Olivetti, E.,

Avesani, P., 2017: Application of photogrammetry to brain anatomy. ISPRS Int. Archives of PRS&SIS, XLII-2-W4, pp. 213-219]

http://3dom.fbk.eu/repository/brain/index.html#s_2-s_3

http://3dom.fbk.eu/repository/brain/index.html#s_2-s_3

3/13/2018

4


TECHNIQUES FOR POINT CLOUD GENERATION

0.1 m 1 m 10 m 100 m 1 km 10 km 100 km 1000 km

10 Mil

1 Mil

100 000

10 000

1 000

100

10

1Ob

ject

/ S

cen

e C

om

ple

xity

[p

oin

ts/o

bje

ct]

Object / Scene Size

Close-range

photogrammetry

and

terrestrial laser scanners

Aerial

photogrammetry

and LiDAR

Satellite

3D Remote Sensing

Tactile / CMM

Hand

measurements

Topography GNSS

UAV

7


From 2000 there was a growing in popularity of LiDAR/TLS sensors for the production of

dense point clouds and photogrammetry could not efficiently deliver similar results

Many researchers shifted their research interests to LiDAR/TLS, resulting in a decline of new

advancements / developments of new photogrammetric methods

LiDAR/TLS became the dominant technology for 3D recording and modelling, replacing

photogrammetry in many application areas

The bottleneck of photogrammetry was represented by massive manual data processing,

high technical skills required, long processing time, etc.

Over the past 10 years, improvements in hardware and software (primarily pushed from the

Computer Vision community), have improved image-based tools and algorithms to the point

that nowadays photogrammetry and LiDAR/TLS can deliver comparable geometrical 3D

results

TECHNIQUES: IMAGING VS RANGING

3/13/2018

5


PHOTOGRAMMETRY LiDAR / Laser Scanning

Origin born ca 1850’s born ca 1960’s

Maturity

60’s-70’s (BBA); 90’s digital

sensors; 2000+ automated

methods / SfM / DIM

2000’s with first commercial TLS and

LiDAR/ALS solutions

Measurement

principleTriangulation

TOF (long-range) and triangulation

(short-range)

Spectrum /

RadiometryMultispectral (VIS-NIR)

Generally @ laser wavelength, rarely

multispectral

3D information To be derived direct

Scale absent (to be provided) Implicit (1:1)

Redundancy Multi-ray / Multi-view Single measure

Dependency Light, geometry, texture Distance, material, object reflectance

Statistics/ Quality

parametersFor each 3D

Generally one value for the entire

cloud

Point density 10-100 pts/sqm 1-50 pts/sqm

Precision/Accuracy XY most accurate than Z (depth) Z (depth) most accurate than XY

Target detection Top-most surface (DSM) Multiple targets per pulse (DTM/DSM)



Photogrammetry ↔ passive method (passive sensors) ↔ image-based method

Scanning ↔ active method (active sensors) ↔ range-based method

Image data acquisition

Image pre-processing

Calibration and orientation

Dense 3D point cloud

generation

Surface generation, feature

extraction and texture mapping

Visualization, GIS products,

replicas, inspection, virtual

restoration, etc.

Range data acquisition

(dense 3D point cloud)

Editing and alignment

Surface generation, feature

extraction and texture mapping

Visualization, GIS products,

replicas, inspection, virtual

restoration, etc.1:8 1:10


10

ACQUISITION

PROCESSING

STRUCTURING

VISUALIZATION

3/13/2018

6


POINT CLOUD ACQUISITION - SENSORS

PASSIVE HYBRID ACTIVE

AE

RIA

L

• Single frame cameras

• Multi-view cameras

(oblique)

• Single frame +

LiDAR

• Multi-view + LiDAR

• Traditional linear Airborne Laser

Scanning

• SPL/Geiger-mode Airborne

Laser Scanning

TE

RR

ES

TR

IAL

• DSLR cameras

• Panoramic cameras

• Mobile Mapping

systems

• Hand-held /

backpack system

• RGB-D sensors

• TOF laser scanner (long-range)

• Triangulation laser scanners

(short-range)

• Structured light systems (short-

range)



AE

RIA

L



(oblique)

• Single frame +

LiDAR



Scanning


Laser Scanning

TE

RR

ES

TR

IAL

• DSLR cameras


• Mobile Mapping

systems

• Hand-held /

backpack system

• RGB-D sensors



(short-range)


range)

Vexcel Ultracam Osprey

Midas Octoblique


3/13/2018

7



AE

RIA

L



(oblique)

• Single frame +

LiDAR



Scanning


Laser Scanning

TE

RR

ES

TR

IAL

• DSLR cameras


• Mobile Mapping

systems

• Hand-held /

backpack system

• RGB-D sensors



(short-range)


range)

Nadir

camera

Oblique

cameras

LiDAR

Leica CityMapper Leica CountryMapper




AE

RIA

L



(oblique)

• Single frame +

LiDAR



Scanning


Laser Scanning

TE

RR

ES

TR

IAL

• DSLR cameras


• Mobile Mapping

systems

• Hand-held /

backpack system

• RGB-D sensors



(short-range)


range)

Leica BackPack Pegasus Trimble Timms Siteco Road-Scanner Riegl VMX-2HA


3/13/2018

8



AE

RIA

L



(oblique)

• Single frame +

LiDAR



Scanning


Laser Scanning

TE

RR

ES

TR

IAL

• DSLR cameras


• Mobile Mapping

systems

• Hand-held /

backpack system

• RGB-D sensors



(short-range)


range)

• Laser output split in n x m array of laser

beamlets (SigmaSpace / Leica: 10 x 10;

Harris: 32 x 128)

• from 10 to 100 pts/sqm, up to 6 mil. pts/sec

• 5 times more effective than traditional

LiDAR

• Range of operation: 2000-4500 m AGL @

200 knot speed



DENSE IMAGE MATCHING (DIM)

http://www.mtzgeo.com/history.cfm

1950’s:

Analogue image matching

and stereoplotter

1960’s:

First digital cross-correlation

1980’s:

Least squares matching

&

Multi-photo matching

1990’s:

Digital stereo processing

systems

2000’s:

Close range photogrammetry,

convergent images, MVS, SGM

Today:

From feature matching to dense stereo

POINT CLOUD ACQUISITION – IMAGE ALGORITHMS

3/13/2018

9


Image rectification / epipolar

Select a matching criteria (what and how do I match?)

Use some assumptions (e.g. constant depth/disparity,

continuity constraint, etc.)

Apply local / global algorithms (aggregation) - iterative

updating

Apply optimization algorithms - energy (cost) formulation,

Markov Random Fields, graph algorithms, least squares,

etc.

Consider multi-view stereo (MVS)

Efficiency thru dynamic programming, GPU and FPGA

implementations

POINT CLOUD ACQUISITION – IMAGE ALGORITHMS

[Remondino, F., Spera, M.G., Nocerino, E., Menna, F., Nex, F., 2014: State of the art in high

density image matching. The Photogrammetric Record, Vol. 29(146), pp. 144-166]


not only geometry data (3D coordinates), but also attributes, e.g.

GEOMETRY and ATTRIBUTES

PHOTOGRAMMETRY LiDAR

RGB intensity

reconstruction uncentainty returns

redundancy time

intersection angles strip

classes (post-processing) classes (post-processing)

normals (post-processing) normals

3/13/2018

10


Photogrammetry – attributes from bundle adjustment and dense image matching

GEOMETRY and ATTRIBUTES

intersection angles (20-90 deg) precision of 3D points (mean 0.06 mm)

redundancy of 3D points (3-65)

[Menna, F., Nocerino, E., Remondino, F., Dellepiane, M., Callieri, M. and Scopigno, R., 2016: 3D Digitization of an Heritage

Masterpiece - a Critical Analysis on Quality Assessment. ISPRS Int. Archives of PRS&SIS, Vol. XLI-B5, pp. 675-683]


POINT CLOUD STRUCTURING

Normally point clouds are unstructured 3D data (with few exceptions)

Structuring can be seen as

generation of an organized dataset (e.g. 2.5D grid DSM)

generation of structured 3D data (e.g. mesh model, 3D building models, etc.)

Bergamo (Italy) - AVT flight with Vexcel Osprey

and dense point cloud @ 10 cm resolution

Bergamo (Italy) - AVT flight with Vexcel Osprey

and 3D semantic modeling of building based on

primitive fitting

[Toschi, I., Ramos, M.M., Nocerino, E., Menna, F., Remondino, F., Moe, K., Poli, D., Legat, K., Fassi, F., 2017: Oblique

photogrammetry supporting 3D urban reconstruction of complex scenarios. ISPRS Int. Archives of PRS&SISI, Vol. XLII-1-

W1, pp. 519-526]

3/13/2018

11







Trento (Italy) - AVT flight with Vexcel Osprey @ 10 cm GSD

Huge and unique mesh/polygonal model

3D modeling of each single building (LOD2)

with parametric shapes fitted on the DSM

vs







generation of classified / segmented point clouds

Dortmund (ISPRS benchmark) – 3D dense point cloud and classification results using

deep learning (supervised) methods

3/13/2018

12


Technology is super rapidly democratizing and point clouds are

nowadays everywhere for many applications and at disposal of

many end-users

Point clouds are the real surveying product and have more added

value than derived products as they keep details and they are not

interpolated

From a business point of view, probably, the money maker is not

anymore the data acquisition part but the added value we can give

to point clouds for domain-specific applications

How do we enrich point clouds with attributes useful to end-users?

Which are the attribute that should be (always) linked to geometry?

How do we reliably and efficiently extract semantic information to

support domain-experts?

How do we store, visualize and transmit enriched point cloud data

to non-specialist users?

How do we provide analytics for multi-temporal point clouds?

CONCLUSIONS & OPEN ISSUES

Point Cloud Processing 2018 March 13th, 2018 POINT … Vision community), ... Photogrammetry...

Documents

Transcript of Point Cloud Processing 2018 March 13th, 2018 POINT … Vision community), ... Photogrammetry...