OpenCount Improved support for Machine-Assisted Ballot-Level Audits
description
Transcript of OpenCount Improved support for Machine-Assisted Ballot-Level Audits
![Page 1: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/1.jpg)
OPENCOUNTIMPROVED SUPPORT FOR MACHINE-ASSISTED BALLOT-LEVEL AUDITS
EVT/WOTE 2013. Washington DC. 8/13/2013.
Eric Kim, Nicholas Carlini, Andrew Chang, George Yiu,Kai Wang†, David WagnerUniversity of California, Berkeley†University of California, San Diego
![Page 2: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/2.jpg)
Talk Overview Motivation
How can OpenCount help the audit process?
Challenges Important: Accuracy and scalability
Pipeline Overview Election Experiences Questions
![Page 3: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/3.jpg)
What is OpenCount? Software that tabulates elections Generates ballot-level cast vote
recordsCVR 00001President of the United States Mitt RomneyMember, County Central Com. Shawn Nelson David John Shawver Greg Sebourn Steve Hwangbo
![Page 4: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/4.jpg)
Motivation Want to perform a post-election audit Statistical ballot-level audit Risk-limiting audit
Typically only have to examine tens to hundreds of ballots (depends on margin)
More efficient than alternative CA: Each county hand-counts all ballots from1%
of precincts
![Page 5: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/5.jpg)
Motivation (cont.) Ballot-level audits require: access to the
voting system’s interpretation of each ballot
Cast Vote Record (CVR) for each ballot Electronic record of the cast votes
![Page 6: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/6.jpg)
Motivation (cont.) Cast Vote Record (CVR)
CVR 00001President of the United States Mitt RomneyMember, County Central Com. Shawn Nelson David John Shawver Greg Sebourn Steve Hwangbo
=?
![Page 7: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/7.jpg)
Motivation (cont. ) Problem: current deployed voting
systems do not output CVRs for each ballot
Only output election totals
![Page 8: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/8.jpg)
Motivation (cont. ) Can’t “upgrade” existing systems
Most vendors are focusing on next-gen systems
EAC certification process (U.S. Election Assistance Commission) would make upgrade expensive
![Page 9: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/9.jpg)
Motivation (cont. ) What is one to do?
If you can’t improve it, rebuild it!
![Page 10: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/10.jpg)
OpenCount Tabulates elections
Input: Scanned ballot images Output: Cast Vote Records, election totals.
Built specifically with ballot-level audits in mind
Open-source software (free!) http://code.google.com/p/opencount/
![Page 11: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/11.jpg)
First Attempt: Blank Ballots Collect one blank ballot from each
ballot style Blank Ballot: Unmarked ballot
Style A
Style B
![Page 12: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/12.jpg)
With Blank Ballots… (1/6)
Style A
Style B
![Page 13: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/13.jpg)
With Blank Ballots… (2/6)
Style A
Style B
![Page 14: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/14.jpg)
With Blank Ballots… (3/6)
Style A
Style B
![Page 15: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/15.jpg)
With Blank Ballots… (4/6)
Style A
Style B
![Page 16: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/16.jpg)
With Blank Ballots… (5/6)
Style A
Style B
![Page 17: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/17.jpg)
With Blank Ballots… (6/6)
Style A
Style B
![Page 18: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/18.jpg)
Previous Work EVT/WOTE 2012 (Bellevue, Washington)
First introduction of the OpenCount (2012) system
“Operator-Assisted Tabulation of Optical Scan Ballots”. Kai Wang, Eric Kim, Nicholas Carlini, Ivan Motyashov, Daniel Nguyen, David Wagner.
Required collecting all blank ballots
![Page 19: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/19.jpg)
Previous Work (cont.) Problem: Did not scale to large
elections Collecting blank ballots is a huge burden
for election officials Blocked some counties from participating
Overall, too much required effort
![Page 20: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/20.jpg)
A Second Attempt New approach: No blank ballots
![Page 21: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/21.jpg)
How can we do this?
![Page 22: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/22.jpg)
No Blank Ballots
Style A
Style B
![Page 23: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/23.jpg)
No Blank Ballots How to find:
Voting Targets? Contests?
![Page 24: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/24.jpg)
OpenCount Pipeline Overview of system Election experiences
California risk-limiting audit pilot program
![Page 25: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/25.jpg)
Scan Ballots (1/6) Use any commercial, off-the-shelf
scanner
![Page 26: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/26.jpg)
Ballot Grouping (2/6)
![Page 27: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/27.jpg)
Ballot Grouping (2/6)
![Page 28: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/28.jpg)
Ballot Grouping (2/6)
![Page 29: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/29.jpg)
Ballot Grouping (2/6)
~124,000
Ballots
~200 Styles
![Page 30: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/30.jpg)
Ballot Grouping (2/6)
![Page 31: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/31.jpg)
Ballot Grouping (2/6)
![Page 32: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/32.jpg)
Ballot Grouping (2/6) Implemented vendor-specific barcode
decoders Diebold ES&S Hart Sequoia
![Page 33: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/33.jpg)
Layout Annotation (3/6) Goal
Specify location of contests and voting targets
Perform data entry of contest text Only need to annotate one ballot from
each style
![Page 34: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/34.jpg)
Layout Annotation (3/6)
How to find voting targets automatically?
![Page 35: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/35.jpg)
Layout Annotation (3/6)
1.) User selects empty voting target
![Page 36: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/36.jpg)
Layout Annotation (3/6)
1.) User selects empty voting target
![Page 37: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/37.jpg)
Layout Annotation (3/6) Search for empty voting target on ballots Template Matching
Grid-search Search for this:
![Page 38: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/38.jpg)
Layout Annotation (3/6)
Verify Match
es
![Page 39: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/39.jpg)
Layout Annotation (3/6)
Problem: Voter marks interfere with template
matching
![Page 40: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/40.jpg)
Layout Annotation (3/6)
Problem: Voter marks interfere with template
matchingIdea: Voters
vote differently. Can find
missing targets on other ballots with the same
style
![Page 41: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/41.jpg)
Layout Annotation (3/6)
Ballot A
Idea: Voters vote differently.
Can find missing targets on other ballots with the same
style
![Page 42: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/42.jpg)
Layout Annotation (3/6)
Ballot B
Idea: Voters vote differently.
Can find missing targets on other ballots with the same
style
![Page 43: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/43.jpg)
Layout Annotation (3/6)
Union of detections from
A + B
Idea: Voters vote differently.
Can find missing targets on other ballots with the same
style
![Page 44: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/44.jpg)
Layout Annotation (3.5/6) Contest text data entry
Contest title, candidate names
Judge of the Superior Court (Office No. 1)- Deborah J. Chuang- Eugene Jizhak
![Page 45: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/45.jpg)
Layout Annotation (3.5/6) Can’t rely completely on OCR Manually labeling each contest takes
forever Number of distinct contests is small
A few hundred at most Contests are duplicated on many ballot
styles “President of the US”
![Page 46: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/46.jpg)
Layout Annotation (3.5/6)
Should only have to label this contest once!
![Page 47: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/47.jpg)
Layout Annotation (3.5/6) Want to detect contest duplicates Simple idea: compare contest images
Pixel-difference (L2 norm)
![Page 48: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/48.jpg)
Layout Annotation (3.5/6)
- Diff = 0.058
MATCH
![Page 49: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/49.jpg)
Layout Annotation (3.5/6)
- Diff = 0.175NOT
MATCH
![Page 50: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/50.jpg)
Layout Annotation (3.5/6) Problem: contest visual appearance
varies Word spacing, line wrapping, candidate re-
ordering
Different Line Wrap
![Page 51: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/51.jpg)
Layout Annotation (3.5/6)
- Diff = 0.146NOT
MATCH
![Page 52: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/52.jpg)
Layout Annotation (3.5/6) Our approach: utilize OCR + edit-
distance
![Page 53: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/53.jpg)
Layout Annotation (3.5/6) Our approach: utilize OCR + edit-
distance
![Page 54: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/54.jpg)
Layout Annotation (3.5/6) Our approach: utilize OCR + edit-
distance
![Page 55: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/55.jpg)
Layout Annotation (3.5/6) Our approach: utilize OCR + edit-
distance
Match!
![Page 56: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/56.jpg)
Ballot Interpretation (4/6) Goal
Determine if voting targets are “filled” or “empty”
Filled
Empty
![Page 57: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/57.jpg)
Ballot Interpretation (4/6)
Separating Line
Sorted by Average
Pixel Intensity
![Page 58: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/58.jpg)
Ballot Interpretation (4/6)
![Page 59: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/59.jpg)
Ballot Interpretation (4/6)
![Page 60: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/60.jpg)
Ballot Interpretation (4/6)
![Page 61: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/61.jpg)
Ballot Interpretation (4/6)
Filled
Empty
![Page 62: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/62.jpg)
Generate CVRs (5/6) Output CVRs
![Page 63: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/63.jpg)
Perform Audit (6/6) Finally, perform the audit! Done!
![Page 64: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/64.jpg)
Election Experiences OpenCount has been used to support
risk limiting pilot audits in several California counties Alameda, Madera, Merced, Napa, San Luis
Obispo, Stanislaus, Ventura OpenCount’s results matched all
examined paper ballots perfectly
![Page 65: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/65.jpg)
Election Experiences (cont.)County #
Ballots# Ballot Styles
Total Time (2013)
Stanislaus 3,151 1 7m 18sMerced 7,120 1 12m 31sVentura 17,301 1 23m 6sAlameda 1,374 8 22m 1sSan Luis Obispo
10,689 27 30m 35s
Madera 3,757 1 6m 38sNapa 6,809 11 1h 56m 9sYolo 35,532 623 3h 36m
![Page 66: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/66.jpg)
Election Experiences (cont.)County #
Ballots# Ballot Styles
Total Time (2013)
Speedup (2012 / 2013)
Stanislaus 3,151 1 7m 18s 2.40xMerced 7,120 1 12m 31s 2.04xVentura 17,301 1 23m 6s 2.52xAlameda 1,374 8 22m 1s 1.29xSan Luis Obispo
10,689 27 30m 35s 2.78x
Madera 3,757 1 6m 38s 1.28xNapa 6,809 11 1h 56m 9s 2.78xYolo 35,532 623 3h 36m 16.24x
![Page 67: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/67.jpg)
Election Experiences (cont.)County #
Ballots# Ballot Styles
Total Time (2013)
Speedup (2012 / 2013)
Stanislaus 3,151 1 7m 18s 2.40xMerced 7,120 1 12m 31s 2.04xVentura 17,301 1 23m 6s 2.52xAlameda 1,374 8 22m 1s 1.29xSan Luis Obispo
10,689 27 30m 35s 2.78x
Madera 3,757 1 6m 38s 1.28xNapa 6,809 11 1h 56m 9s 2.78xYolo 35,53
2623 3h 36m 16.24x
![Page 68: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/68.jpg)
Election Experiences (cont.)County #
Ballots# Ballot Styles
Total Time (2013)
Human Time
Marin 29,121 398 11h 53m 5h 45mSanta Cruz 34,004 136 18h 50m 5h 27mLeon 124,200 216 14h 2s 1h 53mOrange 294,402 1,839 3d 22h 39s 1d 8h 25m
Previous version (2012) could not process elections of this size and
complexity.Progress!
![Page 69: OpenCount Improved support for Machine-Assisted Ballot-Level Audits](https://reader033.fdocuments.net/reader033/viewer/2022051219/56816276550346895dd2e6bf/html5/thumbnails/69.jpg)
Conclusion Improvements to the OpenCount system
Don’t have to collect blank ballots Reduce operator effort significantly
OpenCount is ready for election officials to use
Used in ballot-level risk-limiting audits Audits made possible by OpenCount