Semantizing Complex 3D Scenes using Constrained Attribute...
Transcript of Semantizing Complex 3D Scenes using Constrained Attribute...
Semantizing Complex 3D Scenesusing Constrained Attribute Grammars
A. Boulch, S. Houllier,R. Marlet and O. Tournaire
slight variant of the materialpresented at SGP 2013
Motivation
Semantizing complex objects in 3D scenes
Bare geometry
Semantized geometry
2
Motivation
Semantizing complex objects in 3D scenes
Bare geometry
Semantized geometry
(rendered)
2
Motivation
Semantizing complex objects in 3D scenes
⌅ Building industry⇤
for the renovation market
⌅ point cloud ! building model
⇤for architects
⌅ building sketch ! rendering⌅ Game industry
⇤for graphic designers
⌅ basic level design ! rendering⌅ Object mining in shape databases
⇤for semantic queries
⌅ 3D object ! semantic labeling
Semantized geometry
2
Motivation
Semantizing complex objects in 3D scenes
⌅ Building industry⇤
for the renovation market
⌅ point cloud ! building model⇤
for architects
⌅ building sketch ! rendering
⌅ Game industry⇤
for graphic designers
⌅ basic level design ! rendering⌅ Object mining in shape databases
⇤for semantic queries
⌅ 3D object ! semantic labeling
Semantized geometry
(rendered)
2
Motivation
Semantizing complex objects in 3D scenes
⌅ Building industry⇤
for the renovation market
⌅ point cloud ! building model⇤
for architects
⌅ building sketch ! rendering⌅ Game industry
⇤for graphic designers
⌅ basic level design ! rendering
⌅ Object mining in shape databases⇤
for semantic queries
⌅ 3D object ! semantic labeling
Semantized geometry
(rendered)
2
Motivation
Semantizing complex objects in 3D scenes
⌅ Building industry⇤
for the renovation market
⌅ point cloud ! building model⇤
for architects
⌅ building sketch ! rendering⌅ Game industry
⇤for graphic designers
⌅ basic level design ! rendering⌅ Object mining in shape databases
⇤for semantic queries
⌅ 3D object ! semantic labelingSemantized geometry
2
Outline
Constrained attribute grammars
Scene interpretation
Bottom-up parsing
Experiments
3
Outline
Constrained attribute grammars
Scene interpretation
Bottom-up parsing
Experiments
4
Grammars to express hierarchical decomposition
Sentence
Subject Verb Complement
NounAdjective
The super computer rules Discovery One
Building
Facade Roof Floor
Window WallArticle
4
Complex, non-hierarchic relations between components
Sentence
Subject Verb Complement
NounAdjective
The super computer rules Discovery One
Building
Facade Roof Floor
Window Wall
agreement (singular)
adjArticle
5
Constrained attribute grammars
G = (N,T ,P , S)
⌅ N: nonterminals ($ complex forms) e.g., window, wall
⌅ T : terminals ($ geometric primitives) e.g., polygon, cylinder
⌅ P : production rules ($ hierarchical decomposition and constraints)⌅ S : start symbols ($ root shapes) e.g., building
6
Basic rules
Decomposition of a complex object y of type Y into its constituentsxi of type Xi :
Y y �! X1
x1
, . . . ,Xn xn
Example:
step s �! riser r , tread t
Rule application:⌅ top-down view:
y decomposes into x1
, . . . , xn⌅ bottom-up view:
given some x1
, . . . , xn, create a new object y
riser
tread
7
Basic rules
Decomposition of a complex object y of type Y into its constituentsxi of type Xi :
Y y �! X1
x1
, . . . ,Xn xn
Example:
step s �! riser r , tread t
Rule application:⌅ top-down view:
y decomposes into x1
, . . . , xn⌅ bottom-up view:
given some x1
, . . . , xn, create a new object y
riser
tread
tread ?
riser ?
7
Constraints
Conditional rule application (conjonction of predicates):
Y y �! X1
x1
,X2
x2
, . . . h cstr1
(x1
), cstr2
(x1
, x2
), . . . i
Example:
riser r �! polygon p h vertical(p) itread t �! polygon p h horizontal(p) istep s �! riser r , tread t
h edgeAdj(r , t), above(t, r) i
riser
tread
8
Attributes
Features attached to each grammar element:⌅ at creation time (primitives)⌅ at rule application (synthetized attributes)
Examples:⌅ length, width⌅ bounding box⌅ ... r.widthr.length
9
Predicates
Predicates on grammar elements:⌅ adj, edgeAdj⌅ orthogonal, parallel⌅ vertical, horizontal⌅ ...Predicates on attributes:⌅ �, >, , ...⌅ =, 6=⌅ ...
t.length
r.length
Example: r .length == t.length
10
Collections of similar elements
Grouping elements via recursion (Y as set of X s):Y y �! X x
Y y �! X x ,Y y2
Grouping elements via specific collection operators:Y y �! coll(X ) xs
Useful operators: maximal collections⌅ maxconn: maximal set of connected components⌅ maxseq: maximal sequence⌅ ...Example:
stairway sw �! maxseq(step, adjEdge) ss
11
Collections of similar elements
Grouping elements via recursion (Y as set of X s):Y y �! X x
Y y �! X x ,Y y2
Grouping elements via specific collection operators:Y y �! coll(X ) xs
Useful operators: maximal collections⌅ maxconn: maximal set of connected components⌅ maxseq: maximal sequence⌅ ... [see complete stairway grammar]
Example:stairway sw �! maxseq(step, adjEdge) ss
11
Outline
Constrained attribute grammars
Scene interpretation
Bottom-up parsing
Experiments
12
Scene interpretation: parse tree
Tree representation of a grammatical analysis of the scene:⌅ leaves: terminals representing primitives⌅ non-leaf nodes: instanciations of grammar rules
12
Scene interpretations: parse forest
Set of parse treeswith systematic sharing (DAG)
Compact representationof all possible interpretations
13
Ambiguity and the exclusivity constraint
Example (assuming no height ordering constraint):
step s �! riser r , tread t h edgeAdj(r , t) i step 2
step 1
Exclusivity constraint:
at most 1 occurrenceof a grammar elementper interpretation
Stairway
Step 2Step 1
Riser 2 Tread 1 Riser 1
14
Outline
Constrained attribute grammars
Scene interpretation
Bottom-up parsing
Experiments
15
Parse forest computation
Bottom-up parsing:construction of the parse forestfrom leaves (terminals) to roots (start symbols)
⌅ create one terminal for each primitives⌅ iteratively create new grammar elements from existing ones
⇤e.g., given grammar rule step s �! riser r , tread t
given existing instances riser r23, tread t18create new instance step s5
⌅ merge identical trees on the fly⌅ stop iterating when no rule applies
15
Rule application order
⌅ Simple rules: use any order⌅ Maximal operators: wait for all subelements to ensure maximality
⇤reverse topological sort of nonterminal dependency graph
maxseq
16
Rule application order
⌅ Simple rules: use any order⌅ Maximal operators: wait for all subelements to ensure maximality
⇤reverse topological sort of nonterminal dependency graph
maxseq
16
Rule application order
⌅ Simple rules: use any order⌅ Maximal operators: wait for all subelements to ensure maximality
⇤reverse topological sort of nonterminal dependency graph
maxseq
16
Mastering the combinatorial explosion
Usual drawback of bottom-up analysis: combinatorial explosion⌅ all trees⌅ all sets, all sequences, ...⌅ all combinations
Our solution:⌅ tree sharing: construction of a parse forest (exp. ! lin.)⌅ maximal operators, with efficient implementation (> exp. ! polyn.)⌅ constraint propagation: predicate ordering ) early pruning
17
Maximal operators
maxseq
vs
allseq
18
Maximal operators
maxseq
vs
max(allseq)
18
Constraint propagation as predicate orderingA simple 2D example
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
19
Constraint propagation as predicate orderingExample of poor ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. orthogonality
2. adjacency3. verticality
Complexity:O(#seg2)
constraints to test
20
Constraint propagation as predicate orderingExample of poor ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. orthogonality
2. adjacency3. verticality
Complexity:O(#seg2)
satisfied constraints
20
Constraint propagation as predicate orderingExample of poor ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. orthogonality2. adjacency
3. verticality
Complexity:O(#seg2 +
#seg ⇥maxDeg)
= O(#seg2)
constraints to test
20
Constraint propagation as predicate orderingExample of poor ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. orthogonality2. adjacency
3. verticality
Complexity:O(#seg2 +
#seg ⇥maxDeg)
= O(#seg2)
satisfied constraints
20
Constraint propagation as predicate orderingExample of poor ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. orthogonality2. adjacency3. verticality
Complexity:O(#seg2 +
#seg ⇥maxDeg +#seg)
= O(#seg2)constraints to test
20
Constraint propagation as predicate orderingExample of poor ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. orthogonality2. adjacency3. verticality
Complexity:O(#seg2 +
#seg ⇥maxDeg +#seg)
= O(#seg2)satisfied constraints
20
Constraint propagation as predicate orderingExample of good ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. verticality
2. adjacency3. orthogonality
Complexity :O(#seg)
constraints to test
21
Constraint propagation as predicate orderingExample of good ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. verticality
2. adjacency3. orthogonality
Complexity :O(#seg)
satisfied constraints
21
Constraint propagation as predicate orderingExample of good ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. verticality2. adjacency
3. orthogonality
Complexity :O(#seg +
#seg ⇥maxDeg)
= O(#seg ⇥maxDeg)
constraints to test
21
Constraint propagation as predicate orderingExample of good ordering
pair p �! seg s1
, seg s2
, h orthogonal(s1
, s2
), adj(s1
, s2
), vertical(s1
) i
1. verticality2. adjacency3. orthogonality
Complexity :O(#seg +
#seg ⇥maxDeg +#seg ⇥maxDeg)
= O(#seg ⇥maxDeg) ⌧ O(#seg2)satisfied constraints
21
Constraint propagation as predicate ordering
1. Unary predicates
2. Invertible predicates that are partially instantiated3. General predicates
= constraints implying only 1 element: complexity O(#element)
Example:riser r �! polygon p h vertical(p) i
22
Constraint propagation as predicate ordering
1. Unary predicates2. Invertible predicates that are partially instantiated
3. General predicates
= constraints with small cardinality when some arguments are fixed
Example:
step s �! riser r , tread t h edgeAdj(r , t) i
22
Constraint propagation as predicate ordering
1. Unary predicates2. Invertible predicates that are partially instantiated3. General predicates
= remaining relations
Example:
step s �! riser r , tread t h orthogonal(r , t) i
22
Outline
Constrained attribute grammars
Scene interpretation
Bottom-up parsing
Experiments
23
Semantization pipeline
Semantizedmodel
Polygonextraction Parsing
Grammar
CAD Model(triangle soup)
Point cloud Primitivedetecion
23
CAD modelsPreprocessing
⌅ Region growing over trianglesfor polygon creation
⌅ Computation of exact andapproximate adjacency graphs
24
CAD modelsDetection of stairs (more examples in supplem. material)
25
CAD modelsDetection of walls, roofs and openings (more examples in supplem. material)
26
Real data: photogrammetry⌅ Preprocessing (point cloud)
⇤clustering using RANSAC
(or region growing)
⇤polygons bounded
by alpha shapes
⌅ Problems:⇤
missing primitives
⇤false primitives
⇤wrong adjacencies
⌅ Solution:⇤
use of a relaxed grammar
⌅ looser bounds⌅ 1-2 missing items OK
⇤22 openings out of 31
27
Quasi-real data: simulated LIDAR
⌅ Planes by region growing in depth image⌅ Polygons as oriented bounding rectangles⌅ Adjacency based on pixels in depth image
28
Size and parsing time (CAD models)
# of # of Parsing time (s)Name triangles polygons stairs openingsLcG 48332 9705 5 15LcA 111979 26585 14 42LcC 385541 111732 33 306LcD 313012 75257 25 111LcF 286996 84347 39 322
29
Precision and recall (%, CAD models)
# of # of Stairs OpeningsName stairs steps Prec. Rec. # Prec. Rec.LcG 3 45 100 93 83 100 90LcA 6 84 100 100 62 98 83LcC 30 210 100 100 196 100 98LcD 5 61 93 100 74 100 93LcF 7 98 100 50 99 100 96
30
Future work
⌅ Principled way to deal with partial or missing primitives⌅ Exploitation of occlusion/visibility information⌅ Scoring of interpretations: pick best tree(s)
31
Conclusion
Constrained attribute grammars:⌅ appropriate to semantize complex objects⌅ high-level specification language
⇤being expert is enough, computer scientist not required
⌅ efficient even on large models
This work:⌅ well-delimited first step: perfect data⌅ extensions required for incomplete/noisy data
On the web⌅ http://imagine.enpc.fr/⌅ sites.google.com/site/boulchalexandre/
32