Bayesian Networks for Sketch Understanding

Christine AlvaradoMIT Student Oxygen Workshop12 September 2003

Sketching in Design

Mechanical Engineering

Software

A Challenge In Sketch Understanding

Noisy Input

There is no one threshold for shapes or constraintsInterpretation depends on context

Naïve Approach

Why not just try all possibilities?

Naïve Approach

Why not just try all possibilities?

Arrow?

Naïve Approach

Why not just try all possibilities?

Arrow?

Naïve Approach

Why not just try all possibilities?

Arrow?

Naïve Approach

Why not just try all possibilities?

Arrow?

Naïve Approach

Why not just try all possibilities?

Arrow?

Naïve Approach

Why not just try all possibilities?

Si ik

nMust consider interpretations

n = number of strokes/segments

S = set of shapes

ki = subcomponents in

shape Si

Naïve Approach

Why not just try all possibilities?

Si ik

nMust consider interpretations

n = number of strokes/segments

S = set of shapes

ki = subcomponents in

shape Si

And this only considers shapes independently

Previous Approaches Use Rigid Segmentation

Single stroke shapes Palm Pilot Graffiti Long et. al. [1999]

Explicit Segmentation Quickset: Cohen et. al. [2001]

Pause between strokes

Recognition Using Partial Interpretations

Recognition: Build partial interpretations (PIs) as the

user draws based on easily recognizable low-level shapes

Prune unlikely PIs and use likely PIs to find misrecognized low-level shapes

Evaluating PIs Graphical Models: Missing data = unobserved nodes Interpretation influenced by top-down

and bottom-up information

BN fragments [similar to PRMs, Getoor et. al. 1999]

(Define Arrow (Components (Line shaft) (Line head1) (Line head2)) (Constraints (connects shaft.p1 head1.p1) (connects shaft.p1 head2.p1) (= head1.length head2.length) (< head1.length shaft.length) (< (angle head1 shaft) 90) (< (angle shaft head2) 90) (> (angle head1 shaft) 0) (> (angle shaft head2) 0)))

L1:L2:L3:

C1:C2:C3:C4:C5:C6:C7:C8:

Arrow

L1 L2 L3 C1 C2 C3 C8…

Instantiated and linked together as recognition proceeds[Hammond and Davis, 2003]

Primitive shapes/Constraints

Observation node added when primitive linked to stroke

P(Obs|Prim) determined through data collection

L1

Obs

Example

Sq. error(Stroke a)

Sq. error(Stroke b)

Line(l1)Connects

l1 l2

Arrow

Line(l2) Line(l3)

Quad

LineLine

RemainingArrow

Constraints

Force(F)

Force-pushes-body

Body(B)Touches F B

0.99

0.95

0.95

0.970.99

0.5

0.59

Observation

Example

Sq. error(Stroke a)

Sq. error(Stroke b)

Sq. error(Stroke c)

Line(l1)Connects

l1 l2

Arrow

Line(l2) Line(l3)

Quad

LineLine

RemainingArrow

Constraints

Force(F)

Force-pushes-body

Body(B)Touches F B

0.99

0.95

0.95

0.970.99

0.5

0.59

Observation

Example

Sq. error(Stroke a)

Sq. error(Stroke b)

Sq. error(Stroke c)

Line(l1)Connects

l1 l2

Arrow

Line(l2)Line(l3)

Quad

LineLine

RemainingArrow

Constraints

Force(F)

Force-pushes-body

Body(B)Touches F B

1

1

1

0.971

0.47

0.61

0.95

Observation

Example

Sq. error(Stroke a)

Sq. error(Stroke b)

Sq. error(Stroke c)

Line(l1)Connects

l1 l2

Arrow

Line(l2)Line(l3)

Quad

LineLine

RemainingArrow

Constraints

Force(F)

Force-pushes-body

Body(B)Touches F B

Observation

1

1

1

11

0.47

0.95

1

Sq. error(Stroke d)

Ellipse

0.99

0.970.99

Observation

Current/Future Work

Expand domain/include other domains

Gather sketches from users

Conclusion

Graphical models evaluation Partial Interpretations Context-guided search

More drawing freedom + More robust recognition =

More natural interfaces (i.e. The goal of OXYGEN)