Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd...
-
Upload
erika-dalton -
Category
Documents
-
view
217 -
download
1
Transcript of Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd...
![Page 1: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/1.jpg)
Sketch Recognition for Digital Circuit Diagrams in the Classroom
Christine AlvaradoHarvey Mudd College
March 26, 2007
Joint work with the HMC Sketchers group:Aaron Wolin, Ned Burns, Jason Fennell, Max Pfleuger, Devin
Smith, Paul Wais, Howard Chen, Matt Weiner, Prof. Sarah Harris
![Page 2: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/2.jpg)
Sketching In Education
Digital Circuit Design
![Page 3: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/3.jpg)
Educational Technologies
"Most of the time the lab was more about battling Xilinx than actually learning anything useful" –HMC, E85 student
![Page 4: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/4.jpg)
Problem: Design a 1-bit full adder
![Page 5: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/5.jpg)
Cin B A Cout Sum
0 0 0 0 0
0 0 1 0 1
0 1 0 0 1
0 1 1 1 0
1 0 0 0 1
1 0 1 1 0
1 1 0 1 0
1 1 1 1 0
Correct!
AND-2
AND-2
XOR-2
XOR-2
OR-2
![Page 6: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/6.jpg)
Goals
Build a sketch-based circuit simulation tool that:– is robust enough for real use – allows students to sketch as freely as possible– is easier to use than current software
We need:– An integrated circuit simulation system – Improved free-sketch recognition algorithms– An understanding of user interface issues
![Page 7: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/7.jpg)
Integrated System Overview
Front End
Circuit Recognitionand Translation
Simulation (Xilinx)
Verilog file
hand-drawn sketch
![Page 8: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/8.jpg)
Integrated System Overview
Front End
Simulation (Xilinx)
Verilog file
hand-drawn sketch
Recognize Symbols
Construct Circuit
Translate to Verilog
Free-sketch Recognition
Diagram Parsing
User Interface Design
![Page 9: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/9.jpg)
Sketch Recognition Subtasks
Stroke Fragmentation
Stroke Grouping
Symbol RecognitionNOR
![Page 10: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/10.jpg)
A Typical Approach to Recognition
Stroke Fragmentation
Stroke Grouping
Symbol Recognition
![Page 11: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/11.jpg)
A Typical Approach to Recognition: Problem
incorrect grouping
![Page 12: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/12.jpg)
Why is Grouping Hard?
• No clear boundaries in space…
![Page 13: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/13.jpg)
Why is Grouping Hard?
• No clear boundaries in space or time
0
500
1000
1500
2000
2500
3000
3500
4000
4500
0 1 2 3 4 5 6 7 8 9 10 11 12 13
User ID
Pa
use
Tim
e (
mse
c)
Same Shape New Shape
![Page 14: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/14.jpg)
A Typical Approach to Recognition: Problem
???
Stroke Grouping cannot be done without recognition(But recognition cannot be done without stroke grouping)
![Page 15: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/15.jpg)
Our Approach
Stroke Grouping
Symbol Recognition
Stroke Fragmentation
Single-StrokeRecognition
![Page 16: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/16.jpg)
Our Approach
Stroke Grouping
Symbol Recognition
Stroke Fragmentation
Single-StrokeRecognition
![Page 17: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/17.jpg)
Single-Stroke Recognition• Goal: Label each stroke as WIRE, GATE or SIGNAL
• Method: Conditional Random Field Approach based on Yuan Qi, Martin Szummer, Thomas P. Minka.
Diagram Structure Recognition by Bayesian Conditional Random Fields June 2005 Proc Comp. Vision Pattern Recogn. (CVPR) C. Schmid and S. Soatto and C. Tomasi 191--196
gategategate wire
gate
wire
wire
signal
![Page 18: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/18.jpg)
Conditional Random Field (CRF)
• Determines P(y|x)– y: vector of labels (wire or gate), one for each
fragment (stroke)– x: set of all observations (stroke features)
![Page 19: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/19.jpg)
Single-Stroke Classification Demo
![Page 20: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/20.jpg)
Training the CRF: Data Collection and Labeling
![Page 21: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/21.jpg)
Data Collection
• Goal: Free sketching in engineering education• Method:
– Distributed Tablet Computers to ~35 students in HMC E85 (digital circuit design) and E84 (analog circuit design)
– Collected sketches from notes, homeworks, and labs
But what about labeling?
![Page 22: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/22.jpg)
Labeling Tasks
• Stroke Fragmentation
• Stroke Grouping and Labeling
![Page 23: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/23.jpg)
Labeler Demo
![Page 24: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/24.jpg)
Designing the UI
• User Study to examine:– Recognition Triggers (button, gesture, pause)– Feedback mechanisms (color, symbol, text)– Error Rates and Types
• Preliminary Results– Users prefer active recognition trigger– Trigger must be reliable– Users rarely trigger recognition
![Page 25: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/25.jpg)
(Some Immediate) Future Work
Stroke Grouping
Symbol Recognition
Stroke Fragmentation
Single-StrokeRecognition
Multi-class recognitionwire vs. gate vs. signal
![Page 26: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/26.jpg)
Conclusion
• Single-stroke recognition Improved grouping + recognition
• Direct manipulation labeling more complete datasets
• Robust free-sketch recognition lower barriers to learning
![Page 27: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/27.jpg)
Extra Slides
![Page 28: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/28.jpg)
Nodes for every stroke (fragment)
![Page 29: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/29.jpg)
Edges between related fragments
![Page 30: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/30.jpg)
Example of a label set
![Page 31: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/31.jpg)
CRF Probability Distribution
Eji
xfw
Vi
xgv jii eeZ
xyP},{
)()( ,
)(
1)|(
y Eji
xfw
Vi
xgv jii eeZ
},{
)()( ,)(
The probability of a set of labels given dataWant to maximize P(y|x)
Normalize by sum over all possible label sets. Nasty term
Need approximation to make this computationally feasible
Local compatibility with labels Compatibility based on contextNormalizing term
![Page 32: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/32.jpg)
Feature functions
• CRF cannot use raw stroke data
• Feature functions extract useful numerical data
• Vector of data extracted for each node and pair of adjacent nodes
P(y|x) What are these observations?
![Page 33: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/33.jpg)
Parameters
• Relative usefulness of features for classification needs to be accounted for
• Parameters act as weights for individual features• Weighted features combined with a sum• Represented with a dot product
i
ii baba
![Page 34: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/34.jpg)
Site Potentials
• Measure compatibility between labels and a node
• The exponential makes the math nicer• All potentials combined with a product
Vi
xgv ie )(
Site Potential
feature functionweight vector
![Page 35: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/35.jpg)
Interaction Potentials
• Where the CRF gets its power• Uses context by measuring compatibility
between pairs of adjacent nodes and pairs of labels
• Mathematically, same story as site potentials
Eji
xfw jie},{
)(,
Interaction Potential
feature functionweight vector
![Page 36: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/36.jpg)
What does a CRF need?
• Gather data on the sketch and individual strokes (feature functions)
• Determine weights (training)
• Maximize P(y|x) in a computationally feasible way (inference)– Not going to talk about this
![Page 37: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/37.jpg)
Feature Functions
• Can’t pass stroke data directly into the CRF
• Feature functions translate raw stroke data into simple linear values that the CRF can act on
• We required returned values to be in the range of [-1, 1]– In theory other ranges work, but we had
problems with them
![Page 38: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/38.jpg)
Eji
xfw
Vi
xgv jii eeZ
xyP},{
)()( ,
)(
1)|(
• The CRF must respond linearly to the values returned by feature functions
• This can be problematic if the returned value has physical meaning, like the length of a stroke– To deal with features like length we created a couple
of different features for whether the length was within a certain range
Mathematical Limitations
![Page 39: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/39.jpg)
Site Feature: Turning
• Calculates the total quantity of rotation in a stroke
• After calculating the value of Turning, we returned four different values for different regions
• To see why we need to do this, consider the red, blue, and green strokes below
![Page 40: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/40.jpg)
Interaction Feature: T-Junction
• Detects whether two strokes are configured in a T-Junction with each other– Might occur where a wire meets a gate
• Note that this function is non-symmetric– We have to differentiate the cross from the
stem of a T-Junction– We use two identical versions of this function
with the arguments reversed
![Page 41: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/41.jpg)
Parameters
• We still need a list of weights or parameters relating every site feature to every label, and every interaction feature to every pair of labels
• Must learn parameters from labeled data
Eji
xfw
Vi
xgv jii eeZ
xyP},{
)()( ,
)(
1)|(
![Page 42: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/42.jpg)
Likelihood function
• The likelihood function is a representation of how well a given set of parameters classifies a given data set
• We actually use (-log(likelihood)) to make the math simpler
• Training allows us to find
yxvwP ,
yxvwP ,log
yxvwPvw
,logminarg,
![Page 43: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/43.jpg)
Training: Idea
• How can we minimize ?– Take the derivative and set it to 0?
• Equation is too complicated
– Gradient descent: Locally follow the gradient down to the lowest point (hopefully!)
yxvwP ,log
[w,v]optimal parameters
[evaluated ontraining data]
yxvwP ,log
![Page 44: Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers.](https://reader038.fdocuments.net/reader038/viewer/2022110209/56649e1b5503460f94b09813/html5/thumbnails/44.jpg)
• More / better feature functions
• Computational issues– Numerical under- and overflow
• Multi-Pass CRFs– Find Gates and Wires– Train the CRF again on the gates,
distinguishing the type of gate
• Circuit understanding, interface with Xilinx
Future Work