Perception-enabledKnowledge - TUM · Dejan Pangercic, Nico Blodow, Zoltan-Csaba Marton, Ferenc...

Perception-enabled KnowledgeIAS Internal Workshop

Dejan Pangercic, Nico Blodow, Zoltan-Csaba Marton, Ferenc Balint-Benczedi

Intelligent Autonomous Systems Group

Technische Universität München

June 14, 2012

Semantic Mapping Perception of Objects of Daily Use Dynamic Scenes and Spatio-Temporal Memory (Nico) Misc

Outline

1. Semantic MappingSemantic Mapping

2. Perception of Objects of Daily UseInteractive SegmentationObject Categorization - Graph-PartsObject Categorization -Hough VotingODUFinderObject Recognition using Barcodes

3. Dynamic Scenes and Spatio-Temporal Memory (Nico)GPU-based PerceptionCompression of Point CloudsUnstructured Information Management applications

4. Misc

Perception



Semantic MappingSemantic map representation

Abstract knowledgeabout object classes

Object instances and componenthierarchy

Poses in the environmentand their changes over time

Related: TBOX/SBOX, Galindo et al (RAS

2008)

[Pangercic IROS2012]

Perception



Semantic MappingSystem Version 1

��AB��C��D�

E��F��D�

��A�DC��D�D��D��CD��

�

��A�BC�C

DCEF�C��EC�F�

�F��D��A�

�B��B��F��C��

��F��C��AC��BF�F�E��EC��FA�F��

D�E��DC�A�AC�F��F�

AD��D�EE��CAC��D�

�F�F�C��B��C�A�B�C�F�� !��"F�F�

�C��AC��

!��"F�F�

�F��C�F��

C�A�#�$��F�

*

[Blodow IROS2011]

Perception



Semantic MappingAcquisition of Sensor Data

360deg Scans plus ICP optimizations:• overlaping regions• zeroed roll and pitch

Perception



Semantic MappingNext Best View - Visibility Kernel

Costmap-based approach1. max fringe points costmap, 2. occupied voxels costmap → minimumintersection

vϑ

φ

φ2

dmaxdmin

*Perception



Semantic MappingPoint Cloud Data Interpretation - Floor, Ceiling

• classifying points close to min/max along Z axis as floor and ceiling

Perception



Semantic MappingPoint Cloud Data Interpretation - Walls, ROIs

• classifying planes perpendicular to X and Y axis orientation of the wholepoint cloud and close to min/max along those axes as walls

• classifying tabletops as those between 0.75m-1m heightPerceptionDejan Pangercic, Nico Blodow, Zoltan-Csaba Marton, Ferenc Balint-Benczedi


Semantic MappingPoint Cloud Data Interpretation - Fixtures

*• remaining vertical surfaces and tabletops classified as areas of interest

• findind fixtures - handles and knobs

• various handle and knob detectors

Perception



Semantic MappingDoor and Drawer Hypotheses

• using region growing• median intensity and median average distance

Perception



Semantic MappingSystem Version 2

��ABCD��DEAB

F��

��D��A��DEAB

F��

��ECD��DEABF��

��D��CD��E�� !�BBEB�F��"�

��DE��!�DEAB#A$�!%��BEB�F��

&�D��DEAB�A��!�'�BD� !�B�C

F��

&�D��DEAB�A��(�B$!�CF��

�)#*�#��+�!�+�BD�$�EB,BA�A-F��DEAB��

&AA��B$�&��(.�AD/�C�CF��"�

)-��D�&�D��DEAB�B$

��A�BEDEAB

)-��D�A��&�E!.�0C��&�D��DEAB��B$��A�BEDEABF �B��E�1��)��2��

&AA��B$�&��(.�AD/�C�C��!E$�DEABF��3�

�C4��!!�BD��

��A��CCEB�

��

A�BCDCB�E��

��

*

[Pangercic CogSys2012, Pangercic IROS2012]

Perception



Semantic MappingRegistration, Reconstruction, Texture Re-projection

Testbed Kitchens Poisson surface re-construction

Blending-based tex-ture reprojection

Perception



Semantic MappingLearning of Articulation Models - Arm Control

cabinet objectinteraction

observegripperpose yi

generatenext controlpoint xCEP

i

estimatearticulationmodel M̂, θ̂

[Sturm JAIR2011]

• Using impedance controller fromWillow Garage (pr2 cockpit stack)

• Initialization: gently pull thehandle backwards by moving theCartesian equilibrium pointtowards the robot

• Record the trajectory of robot’sgripper y1:n with yi ∈ SE(3)

VIDEO: *

Perception



Semantic MappingLearning of Articulation Models - Model Fitting




i


Iteratively:• (Re-)estimate the kinematic modelM ∈ {rigid, prismatic, rotational}

• Estimate model-specific parametervector θ ∈ Rd (encoding radius,rotation axis) of the articulatedobject:M̂, θ̂ = argmaxM,θ p(M, θ | y1:n)

• Fit the parameter vector of allmodel candidates using anMLESAC estimator

Perception



Semantic MappingLearning of Articulation Models - Model Selection




i


• Select the best model accordingthe Bayesian information criterion(BIC)

• Use the model to predict thecontinuation of the trajectory andto generate the next Cartesianequilibrium point xCEPn+1 .

• Finally, determine opening angle /opening distance

• Output: Kinematic model

Perception



Semantic MappingDoor and Drawer Hypotheses Validation through Interaction

*PerceptionDejan Pangercic, Nico Blodow, Zoltan-Csaba Marton, Ferenc Balint-Benczedi


Semantic MappingFinal Segmented Map

Perception



Semantic MappingFinal KnowRob Map

Acquisition Code:bosch registration, bosch surface reconstruction,bosch texture reconstruction, mappingRepresentation Code:http://www.ros.org/wiki/knowrob

Perception


http://www.ros.org/wiki/knowrob


Semantic MappingFuture Work

In collaboration with KTH:

• classification of rooms using shape, size and set of objects

• language for probabilistic representation and reasoning

• room and object novelty detection

• integration of planning

Perception



Semantic MappingTutorial 1

In progress . . . - by end of June 2012.

• RGB + D + Plane-based + Edge-based Registration• Automatic segmentation of fixtures and dimensions• Automatic learning of articulation models

Perception



Outline




4. Misc

Perception



Interactive SegmentationSystem

��A��BCD�E�

A��F��F��B��

A��F��

��F��F��

�B��D�� !��F��

��F��"��F#��E

$�D%��E��

��F��&��E

��ABCABD��E�� F��ABC

C��CD�E�

C��

D��D�AB��EA��A�B

��A��A�B

�'('��

*

[Bersch IROS2012, Bersch RSS2012]

Perception



Interactive SegmentationContact Point and Push Direction

• Segmentation of cluttered scenes

• Contour estimation

• Finding concave corners

• Bisector gives a push direction

Perception



Interactive SegmentationClustering Algorithm

t = 0

A0

A1

t = 1

t = 2

��AB�CD

• Features u and v, depicted as redcircles, are randomly selected

• From their trajectories Su and Sv,a rigid transformation At iscalculated

• If u and v are on the same object,all other features will moveaccording the sequence of At

Perception



Interactive SegmentationResults

Perception



Interactive SegmentationFuture Work

• dealing with textureless and transparent objects

• real-world scenes

• integration of an arm planner

• constrained spaces

Perception



Interactive SegmentationTutorial 2

Available on:http://ros.org/wiki/pr2_interactive_segmentation/Tutorials.

• Form 3 groups with 1 Kinect, 1 tripod and 2 objects each.

Thanks to Karol Hausman!

Perception


http://ros.org/wiki/pr2_interactive_segmentation/Tutorials


Object Categorization - Graph-PartsIntroduction

Figure: Overview of the process ofscanning segmenting and categorizingobjects in clutter

• Object categorization in clutteredscenes where accuratesegmentation can be difficult toachieve.

• Over-segmentation and multiplehypotheses better than relying ona single, possibly erroneoussegmentation

• Approach based on scene- orpart-graphs, using additive RGBDfeature descriptors and hashing

[Marton SC2012]

Perception



Object Categorization - Graph-PartsObjectives

��AB�C�DECF

�

�

�

FFF

A

�

��

��

��

FFF

�

FFF

FFF

�CC�A��A�D��

��D��

�

�

�

FF

A

D��D��D��D�D��D

��D��D��D�D��

��D��D��D�D��

�FF

��D��D��D�D��


��D��D��D�D��

��D��D��D�D��

�FF

��D��D��D�D��


��D��D��D�D��

��D��D��D�D��

�FF

��D��D��D�D��

FFF

FFF

FFF

FFF

FFF

FFF

FFF

• an efficient part-based objectclassification method for clutteredscenes, taking into accountrelations between parts;

• a graph-theoretic hashing methodthat allows model refinement whilehaving competitive classificationperformance;

• evaluation of geometric, color, andmulti-modal features andclassification approaches;

• exemplifying the advantages ofmultiple views, multiple segmentgroupings, and domain adaptation;

Perception



Object Categorization - Graph-PartsRadius-based Suface Descriptor (RSD)

Theory and Approximation

• Estimate minimum and maximum curvature radius from angle/distancepairs:

��

��

�

�

�

� �� A�

�

�B�

�B�

�B�

�B�

�B�

�B�

�B�

�B

�BA

�

CDEF��C��

��E��C�F�� CF�!�"��#��F�$��F��E�

%�"��&�F'��F(��E��C��#�F'��FC

��CF��)��E��F�*��F��F(��+��,��)�CF��

Near-linear relation between distance and angle: the radius [m]

d(α) =√2r√

1− cos(α) ⇒ d(α) = rα+rα3

24+O(α5) ⇒ d = r · α

[Marton IROS2010:]

Perception



Object Categorization - Graph-PartsRadius-based Surface Descriptor (RSD)

Principle Radii

• Local variation of normal angles by distance (similar to PFH and “spin images with

normals”):

Synthetic Data

plane sphere sphere side corner cylinder cylinder top cylinder side edge handle

Real Data

small cylinder medium cylinder big cylinder handle1 handle2 handle3

The tilt angles of the lines starting from bottom left corner correspond to the physical radii:

smallest tilt that still covers occupied cells to the min. radius, while the biggest to the max.

Perception



Object Categorization - Graph-PartsStatic Scenes

(a) Scene 1

(b) Scene 2

(c) Scene 3

Perception



Object Categorization - Graph-PartsScenes From Multiple Views

��

��

��

��

��

��

�

ABCDEF

A�F

��

��

Figure: As the camera is moved (left), multiple frames can be captured thatcover different parts of the objects in the scene (right).

Perception



Object Categorization -Hough VotingPart-Based Object Detection using Hough Voting and Model Fitting

Table Chair Sideboard

[RAM2011]

• Same task as before, with larger objects:furniture pieces have to be identified forwhich (similar, but not exactly matching)CAD models are available.

• Here pose estimation is added by modelmatching and geometric verification.

• Parts are considered separately, and only3DOF are possible, so more descriptivefeatures can be used (stattistics about thedistribution of the 3D points).

• Integration is achieved by parts beingcategorized (unsuperwised) and voting forlikely positions.

Perception



Object Categorization -Hough VotingPart Codebook of Example CAD Models

• Realistic scan simulation, unsupervised segmentation, building acodebook of parts, learning of spatial relations

• Probabilistic Hough voting to obtain likely object locations and aweighted list of their parts

Perception



Object Categorization -Hough VotingModel Fitting for Verification and 3DOF Pose

Assigning parts/points to objects and rejecting false positives

Perception



Object Categorization -Hough VotingReal-world Results: Office

Perception



Object Categorization -Hough VotingReal-world Results: Seminar Room

• remaining false matches due to high occlusionsPerception



Object Categorization -Hough VotingImproving Results by Taking Multiple Scans

• Decreasing occlusion, increasing number of parts• Similarly as for small objects: merging the votes from multiple scans

• The model fitting and verification steps also do not assume oneviewpoint or one segmentation per scan

Perception



Object Categorization -Hough VotingFuture Work

• test on objects of daily use

• include color features

• explore the “power-of-data”

• integrate online building of models and sharing (e.g. via RoboEarth)

Perception



Object Categorization -Hough VotingTutorial 3

Available on: http://www.ros.org/wiki/furniture_classification.Thanks to Vlad Usenko!

Perception


http://www.ros.org/wiki/furniture_classification


ODUFinderObject Recognition with ODUFinder - Idea

Perception



ODUFinderSystem

��AB�CDE�AEF

�DBA��A��ACECE�

�E��AEF��B��DF��CE�

�ABDD��C��

�ED ��F��A��

�CE�B�!DCE��B�D�F�AEF�C"A��

�ABDF��#DBA�C$A�CDE

AEF%�BDFCE�

��"AE�CB�E&D"A�CDE

'!!�AAEB��DF��

[Pangercic IROS2011, Pangercic IROS2012]

Perception



ODUFinderVocabulary Tree and SIFT

+ + TF-IDF+Metric Score (e.g. L1)

Perception



ODUFinderOver-Segmentation-Based Object Candidate Detection

r2(x) = (r2max − r2min)(K(1− logsig(x−A))) + r2min (1)

where logsig is defined as:

logsig(x) =1

1 + e−x, (2)

Perception



ODUFinderOver-Segmentation-Based Object Candidate Detection - cont.

Perception



ODUFinderSemantic Map Prior

*Perception



ODUFinderTopic-based Integration with KnowRob

• Automatically created ontology of>7500 objects from the online shopgermandeli.com

• Class hierarchy from categories +perishability, weight, price, origin,...

• Code:http://www.ros.org/wiki/

objects_of_daily_use_finder,KnowRob: comp germandeli

[Tenorth, RAM2011a]

Perception


http://www.ros.org/wiki/objects_of_daily_use_finderhttp://www.ros.org/wiki/objects_of_daily_use_finder


Object Recognition using BarcodesIdea

Perception



Object Recognition using BarcodesSystem

��AB�CDE�AEF

�DBA��A��ACECE�

�E��AEF��B��DF��CE�

�ABDD��C��

�ED ��F��A��

�CE�B�!DCE��B�D�F�AEF�C"A��

�ABDF��#DBA�C$A�CDE

AEF%�BDFCE�

��"AE�CB�E&D"A�CDE

'!!�AAEB��DF��

[Pangercic IROS2012]

Perception



Object Recognition using BarcodesBarcode Decode

Perception



Object Recognition using BarcodesServer Communication

• Request: http://www.barcoo.com/api/get_product_complete?pi=4101530002475&pins=ean&format=xml&source=ias-tum

• Response:

• Information parsed: image, category, product name, brandPerception


http://www.barcoo.com/api/get_product_complete?pi=4101530002475&pins=ean&format=xml&source=ias-tumhttp://www.barcoo.com/api/get_product_complete?pi=4101530002475&pins=ean&format=xml&source=ias-tum


Object Recognition using BarcodesIntegration with KnowRob

• OWL file-based

• Topic-based using json server(TBD)

Perception



Object Recognition using BarcodesTutorial 4

Available on: http://www.ros.org/wiki/objects_of_daily_use_finder/Tutorials/Barcode-based%20object%20recognition.For this tutorial you will need to connect to the “lab” wireless network(pw: artificialintelligence) and disable your wired connection.Thanks to Nacer Khalil.

Perception


http://www.ros.org/wiki/objects_of_daily_use_finder/Tutorials/Barcode-based%20object%20recognitionhttp://www.ros.org/wiki/objects_of_daily_use_finder/Tutorials/Barcode-based%20object%20recognition


Outline




4. Misc

Perception



Misc

List of things we did not talk about . . . :

• object tracking

• http://ros.org/wiki/cop

• plethora of algorithms in PCL (pointclouds.org)

• people/hands tracking

• DAFT feature

• . . .

Perception


http://ros.org/wiki/coppointclouds.org


Thank You

Contact:http://ias.cs.tum.edu/people/[pangercic|marton|blodow|balintbe]

Perception


http://ias.cs.tum.edu/people/

Semantic MappingSemantic Mapping

Perception of Objects of Daily UseInteractive SegmentationObject Categorization - Graph-PartsObject Categorization -Hough VotingODUFinderObject Recognition using Barcodes

Dynamic Scenes and Spatio-Temporal Memory (Nico)GPU-based PerceptionCompression of Point CloudsUnstructured Information Management applications

Misc

Perception-enabledKnowledge - TUM · Dejan Pangercic, Nico Blodow, Zoltan-Csaba Marton, Ferenc...

Documents

Transcript of Perception-enabledKnowledge - TUM · Dejan Pangercic, Nico Blodow, Zoltan-Csaba Marton, Ferenc...