Learn how to make your drawings come alive…

NEW COURSE:SKETCH RECOGNITIONAnalysis, implementation, and comparison of sketch recognition

algorithms, including feature-based, vision-based, geometry-based, and timing-based recognition algorithms; examination of methods to combine results from various algorithms to improve recognition using

AI techniques, such as graphical models.

Sezgin

• Finds corners in a polygon or in a complex shape.

• First: Polygon

Direction

Direction of each stroke segment = arctan2(dy,dx)

Add check to make sure graph continuous (e.g., add 2pi)

Curvature

Change in direction for each segment

Speed

Speed of each segment (already computed in Rubine)

Threshold Curvature

• Threshold = mean curvature

Threshold Speed

• Threshold = 90% of mean

Select Vertices for Each• Max of all sections above threshold • Fd = curvature points

Select Vertices for Each• Max of all sections above threshold• Fs = speed points

Initial Fit

• Intersection of Fd and Fs (and of course the endpoints)

Local neighborhood: i-k, i+k

• Si is a stroke point

• k is number of stroke point on either side to search

• Le = Euclidian distance from Si-k to Si+k

• K not defined– Lets try k = 3 … AND

– Variable k where min(Le > 6 pixels)

Curvature Certainty Metric for Fd

• di = curvature at point i

• l = stroke segment curve length from Si-k to Sj+k

• Lets call it CCM(vi)

Speed Certainty Metric for Fs

• Lets call it SCM(vi)

Vertex Possibilities:

• Fd: Possibilities based on curvature

• Fs: Possibilities based on speed

• CCM(Fd) : Curvature Certainty Metric

• SCM(Fd) : Speed Certainty Metric

• H0 = Initial Hybrid Fit = intersection of Fd and Fs (and endpoints)

Selecting new points to add

• Hi’ = H0 + max(SCM(Fs – H0) )

• Hi’’ = H0 + max(CCM(Fd – H0))

• Pick highest scoring from each list

• Try to add it to the H0

Make Line Segments for Hi’ and Hi’’

• For each vertex vi in Hi – create a list of line segments between vi and vi+1

• If there are n points in the vertex selection, there will be n-1 line segments

Calculate Errorfor Both Hi’ and Hi’’

• S = total stroke length (not Euclidean distance)

• ODSQ = orthogonal distance squared

• ODSQ(s,Hi) = distance of stroke point s to appropriate line segment (previous slide)

Distance from point to line segment

• if isPointOnLine(point, line)• theDistance = 0;• else• array1 = getLineAxByC(line);• array2 = getAxByCPerpendicularLine(line, point);• • A = [array1(1), array1(2); array2(1), array2(2)];• b = [array1(3); array2(3)];• intersectsPoint = linsolve(A, b)';• if isPointOnLine(intersectsPoint, line)• theDistance = getLineLength([intersectsPoint; point]);• else • dist(1) = getLineLength([line(1,:); point]);• dist(2) = getLineLength([line(2,:); point]);• theDistance = min(dist);• end• end

Select either Hi’ or Hi’’

• Pick the one with the lower error

• (in the paper this is also the one with the `higher score’ – score in this sense is just an internal ranking metric)

• H1 = Hi’ or Hi’’ (the one with the lower error)

• We have added one vertex point.

Repeat

• Create new Hi’ and Hi’’

• Hi’ = H1 + max(SCM(Fs – H1) )

• Hi’’ = H1 + max(CCM(Fd – H1))

• (Note one of them will be the same as the previous time.)

• Recompute error and continue

When to Stop

• Stop when error is below the threshold.• … what is threshold?• Not in paper• Lets

– Compute the error for H0 = e0

– Compute the error for Hall (all the chosen v) = eall

– We want something in the middle: close to eall

– .1*(e0-eall) + eall

– You will try other thresholds in your implementation

Implications

• Anything that you can describe geometrically you can build sketch system for

Curves

• Stroke between corners can be curve or line

• Is line if l2/l1 is close to 1. – Lets use .9 < l2/l1 *1.1 < 1

• Else is curve

Sezgin Bezier Curve Fitting• Want to replace with a Bezier curve• http://www.math.ubc.ca/people/faculty/cass/gfx/bezier.html

– Bezier Demo• 2 endpoints and 2 control points

Bezier curve equation• http://www.cl.cam.ac.uk/Teaching/2000/AGraphHCI/SMEG/node3.ht

ml• http://www.moshplant.com/direct-or/bezier/math.html

• P0 (x0,y0) = p1, P1 = c1, P2 = c2, P3 = p2

• x(t) = axt3 + bxt2 + cxt + x0• y(t) = ayt3 + byt2 + cyt + y0• cx = 3 (x1 - x0)

bx = 3 (x2 - x1) - cxax = x3 - x0 - cx - bx

• cy = 3 (y1 - y0)by = 3 (y2 - y1) - cyay = y3 - y0 - cy - by

Sezgin Bezier Curve Fitting

• Control points• Endpoints: u = p1, v = p2• T1= tangent vector – initial direction of stroke at

point p1• T2 = tangent vector of p2 – initial direction of

stroke at point p2• K = stroke length / 3

– 3 is common approximation

• c1=k*t1 + v• c2 = k*t2 + u

Want to Test our Approximation

• Perhaps this is really a very complex curve which can’t be fit with a simple Bezier curve– E.g., the treble clef of a musical staff

• Discretize the curve. (It doesn’t say into how many parts – I leave that up to you.)– Linear approximation for each part

If error too high

• Break our curve down the middle into two curves, and try again.

Matlab Curve Fitting• function [estimates, model] = fitcurvedemo(xdata, ydata)• % Call fminsearch with a random starting point.• start_point = rand(1, 2);• model = @expfun;• estimates = fminsearch(model, start_point);• % expfun accepts curve parameters as inputs, and outputs sse,• % the sum of squares error for A * exp(-lambda * xdata) - ydata, • % and the FittedCurve. FMINSEARCH only needs sse, but we want to • % plot the FittedCurve at the end.• function [sse, FittedCurve] = expfun(params)• A = params(1);• lambda = params(2);• FittedCurve = A .* exp(-lambda * xdata);• ErrorVector = FittedCurve - ydata;• sse = sum(ErrorVector .^ 2);• end• end

Circles and Ellipses

• Least squares fit

• Make a bounding box of stroke and form

• Oval is formed from that bounding box.

• Circle is easy: – Find out how far the circle is from the center =

d, return (d-r)^2– Ellipse more complicated

Project Suggestions

• Build a finite state machine recognizer for the computability class to easily draw and hand in their diagrams.

• Build a physics drawing program that attaches to a design simulator (we have interactive physics 2005)

• Build a fashion drawing program. You draw clothes on a person, and it puts them one the person.

Project Ideas

• Build a robot drawing and simulation program. You draw the robot and have a number of gestures to have it do different things

• Gesture Tetris

Project Suggestions

• Use both rubine and geometrical methods in recognition

• Develop new ways for editing.

• Build new low level recognizers

Projects!

• Proposal due: Sept 22

• Visualization Contest– Ivc.tamu.edu/competition

• Smart Boards coming

• IAP (Industrial Affiliates Program) Demo

Projects!

• 2 types:– Cool application

• Sketch front end to your own research system• Fun application to go on smart board/vis contest

– Gesture Tetris– TAMU gesture-based map/directory info– Computability/Physics/EE/MechEng simulator

– New recognition algorithm• Significant change to old techniques to make a

new application

Final Project Handin• Implementation… Build it… • In class Demonstration (5-10 minutes)• Previous work

– Find at least 3 relevant papers (not read inc class)– Assign one for class to read, you lead short discussion

• Test– Run your recognition system on data.– Find out what data you need (e.g., UML class diagrams)– Each student in class will supply others data– + Find 6 more people outside (to give 15 different people)

• Paper– Introduction (why important)– Previous Work– Implementation– Results– Conclusion

Syllabus

• http://www.cs.tamu.edu/faculty/hammond/courses/SR/2006