Risk-Limiting Audit (RLA) conducted at

Porter County, Indiana on

January 24-25, 2019

A Report to the Indiana Secretary of State

January 31, 2019

by

Voting System Technical Oversight Program (VSTOP)

Ball State University

1

Executive Summary

On Election Day, November 6, 2018, in Porter County, Indiana several election related issues occurred. These issues included poll workers not showing up at polling places, polls opening up late, problems with absentee and early voting ballot delivery to polling places for tabulation, and the late release of election results. As a result of these issues, Secretary of State Connie Lawson was asked to conduct a review of its election preparation procedures and other related matters. The Secretary and her staff discussed with VSTOP the possibility of a conducting a Risk-Limiting Audit (RLA) of a selected set of 2018 General Election races in Porter County. In a subsequent telephone conversation with VSTOP on December 5, 2018, Mr. Brandon Clifton (Deputy Secretary of State and Chief of Staff) requested that VSTOP conduct an RLA in Porter County Indiana. He also asked that a proposal from VSTOP for the RLA Project be submitted to the Secretary by December 14, 2018. The proposal was submitted on December 14th and approved by the Secretary.

Risk-Limiting Audits (RLAs) provide statistical assurance that election outcomes are correct by manually examining paper ballots or voter-verifiable paper records. RLAs do not guarantee that the electoral outcome is right, but they have a large chance of correcting the outcome if it is wrong. If the original outcome is wrong, there is a chance the audit will not correct it. Thus, the risk limit is the largest chance that an incorrect outcome escapes correction. For instance, if the risk limit is 10% and the outcome is wrong, there is at most a 10% chance (and typically much less) that the audit will not correct the outcome—at least a 90% chance (and typically much more) that the audit will correct the outcome. Thus, if the risk limit is 1%, then, in the long run at least 99 out of 100 wrong outcomes would be corrected by the audit.

The number of ballots required to conduct an RLA will vary based on the smallest margin of the contest selected and the risk limit. The smaller the margin, the more ballots to audit. The smaller the risk limit, the more ballots to audit.

On January 15, 2019, members of the VSTOP Team traveled to Porter County, Indiana to meet with County Clerk Jessica Bailey and other county election staff to discuss the upcoming RLA, observe ballot storage for purposes of developing a ballot manifest (a document that describes how ballots are stored and organized) and to answer questions. During this visit, a public presentation was made by VSTOP regarding the upcoming RLA process and questions were answered.

The VSTOP Team spent two days in Porter County, Indiana conducting the RLA on January 24 and 25, 2019. On January 24, 2019, VSTOP and the RLA team prepared for the RLA by obtaining ballots collected from all precincts which would be used to draw samples of ballots for the “Ballot Polling RLA” as well as a “Bayesian Audit.” Also on January 24, 2019 a public presentation was made by Mr. Jerome Lovato regarding the RLA process, procedures and questions were answered (see Appendix B).

The RLA Team audited five races which included: Public Question #1; County Prosecutor; County Auditor; County Recorder; and County Coroner. The sample sizes of ballots sampled for each race were based on the margin for each race and the risk limit. For example, the County Recorder race had a margin of 3% and with a risk limit of 10%, the initial sample to audit was 4,955 ballots. Due to time constraints, we were unable to complete the audits for County Recorder and County Coroner. However, we did complete the audits for Public Questions #1, County Prosecutor, and County Auditor confirming these election outcomes with high degrees of statistical confidence, and were able to obtain time and cost data for additional RLA research.

2

Introduction and Background

On Election Day, November 6, 2018, in Porter County, Indiana several election related issues occurred. These issues included poll workers not showing up at polling places, polls opening up late, problems with absentee and early voting ballot delivery to polling places for tabulation, and the late release of election results. As a result of these issues, Secretary of State Connie Lawson was asked to conduct a review of its election preparation procedures and other related matters. The Secretary and her staff discussed with VSTOP the possibility of a conducting a Risk-Limiting Audit (RLA) of a selected set of 2018 General Election races in Porter County as part of her review. In a subsequent telephone conversation with VSTOP on December 5, 2018, Mr. Brandon Clifton (Deputy Secretary of State and Chief of Staff) requested that VSTOP conduct an RLA in Porter County Indiana. He also asked that a proposal from VSTOP for the RLA Project be submitted to the Secretary by December 14, 2018. The proposal was submitted on December 14th and approved by the Secretary.

Risk-Limiting Audits

Risk-Limiting Audits (RLAs) provide statistical assurance that election outcomes are correct by manually examining paper ballots or voter-verifiable paper records. RLAs do not guarantee that the electoral outcome is right, but they have a large chance of correcting the outcome if it is wrong. If the original outcome is wrong, there is a chance the audit will not correct it. Thus, the risk limit is the largest chance that an incorrect outcome escapes correction. For instance, if the risk limit is 10% and the outcome is wrong, there is at most a 10% chance (and typically much less) that the audit will not correct the outcome—at least a 90% chance (and typically much more) that the audit will correct the outcome. Thus, if the risk limit is 1%, then, in the long run at least 99 out of 100 wrong outcomes would be corrected by the audit.

The number of ballots required to conduct an RLA will vary based on the smallest margin of the contest selected and the risk limit. The smaller the margin, the more ballots one must audit. In RLA parlance, this is referred to as the “diluted margin” and is the margin of an audited contest divided by a total number of tabulated ballots. Also, the smaller the risk limit the more ballots one must audit. Thus, if a diluted margin is narrow in a race and the risk limit is set low for the audit the more ballots one must sample and examine.

Computer software cannot be guaranteed to be perfect or secure, so voting systems should be software-independent – An undetected change or error in voting system software should be incapable of causing an undetectable change or error in an election outcome. An RLA leverages software independence by checking the audit trail strategically. Efficient RLAs do not require complicated calculations or in-house statistical expertise.

An RLA software program is used to calculate the number of ballots to audit, randomly select the ballots, provide a ballot lookup table, and notify the user of when an audit can stop. Thus, Risk-Limiting Audits depend on sampling methodology as well as statistical methodology.

In 2009, Colorado’s HB 09-1335 introduced RLAs to commence with the 2014 General Election. In 2013, Colorado conducted the first pilot RLA at Arapahoe County. More counties were added in 2015-16. Colorado developed rules, procedures and software to conduct an RLA for the 2017 Coordinated Election. The November odd-year election is generally referred to as the coordinated election in Colorado. Elections in Colorado are conducted by mail ballot.

3

In 2014, Cuyahoga County, Ohio, conducted a risk-Limiting audit for its gubernatorial race. Incumbent John Kasich received 51 percent of the votes cast in the county, and challenger Edward FitzGerald received 45 percent. The county Board of Elections needed to recount slightly more than 8,000 ballots before it could confidently determine that Governor Kasich had correctly been declared the winner. The board also audited the race for state treasurer, in which incumbent Joshua Mandel received 39 percent of the vote versus 61 percent for challenger Connie Pillich. In this less competitive contest, fewer than 2,500 ballots were needed to certify Pillich’s victory among county voters. The California Secretary of State recently completed a three-year pilot program that audited contests of varying size in counties throughout the state. In September 2017, Rhode Island became the second state to require Risk-Limiting Audits, for implementation by 2020, with a pilot in January 2019. In May 2018, Indiana conducted a pilot RLA in Marion County. Currently, there is proposed legislation to require the use of RLAs in all counties. While there is a large set of references on RLAs, the following two provide comprehensive introductions and details.

• A Gentle Introduction to Risk-Limiting Audits, by Mark Lindeman and Philip B. Stark, IEEE SECURITY AND PRIVACY, SPECIAL ISSUE ON ELECTRONIC VOTING, 2012. https://www.stat.berkeley.edu/~stark/Preprints/gentle12.pdf

• A Bayesian Method for Auditing Elections https://www.usenix.org/system/files/conference/evtwote12/rivest_bayes_rev_073112.pdf

There are four types of sampling methodologies in Risk-Limiting Audits: ballot polling, ballot comparison, batch polling, and batch comparison. Additionally, there are two types of statistical methods: Stark’s RLA and the Bayesian Audit. These are described below. In ballot polling sampling, one randomly draws ballots, examines ballots manually and tallies results for each ballot. For ballot comparison, ballots are randomly drawn, examined manually and each ballot is compared to its cast vote record (CVR). For batch polling, one randomly draws batches, examines results manually, and tallies results for each batch. Finally, for batch comparison, one randomly draws batches, examines ballots manually, tallies results for each batch and compares each batch to its batch report produced by the voting system. Ballot-level audits are more efficient than batch-level since they require examining fewer ballots. A comparison audit is more efficient but requires CVRs. Polling can be used if CVRs are not available. The Stark RLA provides strong statistical evidence that the election outcome is right, and has a high probability of correcting a wrong outcome. The risk limit is the largest chance that a wrong outcome will not be corrected. If the risk limit is 5% and the outcome is wrong, there is at most a 5% chance that the audit will not correct the outcome, and at least a 95% chance that the audit will correct the outcome. The Stark audit tool can be found at the following link: https://www.stat.berkeley.edu/~stark/Preprints/gentle12.pdf A Bayesian audit is a statistical tabulation audit that provides assurance that the reported contest outcome is correct, or else determines the correct contest outcome. A Bayesian risk limit is a desired upper bound on the probability that the audit will make an error (by accepting an incorrectly reported contest outcome as correct). The Bayesian audit tool can be found at the following link: https://arxiv.org/pdf/1801.00528.pdf

4

The Stark RLA is more popular and statistically rigorous. The Bayesian is more flexible and can be used in non-standard situations. RLAs are valuable because they are efficient and can detect problems with election outcomes, with a high degree of statistical confidence, without having to engage in an expensive and time-consuming full recount unless it is absolutely necessary. Further, the outcomes from RLAs can enhance voter confidence that votes were correctly counted and tabulated. Moreover, RLAs serve as a check on the integrity of election outcomes based on statistical methods which confirm winners.

The Porter County, Indiana RLA Methodology

When VSTOP received its charge to conduct an RLA in Porter County, Indiana, the Team held several telephone conference calls with the Secretary’s Office and election personnel in Porter County. These conversations centered on explaining Risk-Limiting Audits, providing resources and planning for the audit. Subsequent to these conversations, the VSTOP Team continued to plan and strategize for the audit with the assistance of Mr. Jerome Lovato and others. This planning and strategizing was essential to carrying out a successful audit since the VSTOP Team tried to anticipate as many potential issues with the audit as possible and to plan for these.

Porter County uses the ES&S Unity 3.4.1.0 Ballot Card Voting System. According to the VSTOP inventory, there are 140 M100 units and 125 AutoMark (Ballot Marking Device) units. There are 123 precincts in Porter County and the county stores ballots by precincts.

On January 15, 2019, members of the VSTOP Team traveled to Porter County, Indiana to meet with County Clerk Jessica Bailey and other county election staff to discuss the upcoming RLA, observe ballot storage for purposes of developing a ballot manifest and to answer questions. During this visit, a public presentation was made regarding the RLA process. Additionally, the 20 digit “random seed” was selected at this meeting by rolling a 10-sided die 20 times in order to begin the RLA process. The random seed is entered into the RLA audit tool and this creates an algorithm which determines the number of ballots needed to audit particular races based on the diluted margin and the number of ballots cast.

Subsequent to this visit, Clerk Jessica Bailey provided the VSTOP Team with a sample ballot from the November 6, 2018 general election so the order of races to be audited could be determined. On January 23, 2019, Clerk Bailey and other county election officials organized ballot stacks by precinct in the room reserved for the audit.

The VSTOP Team, County Clerk Jessica Bailey, Deputy Clerk Kathy Hartwig, and six election judges spent two days in Porter County, Indiana conducting the RLA on January 24 and 25, 2019. On January 24, 2019, VSTOP and the RLA team prepared for the RLA by obtaining ballots collected from all precincts which would be used to draw samples of ballots for the Ballot Polling RLA as well as a Bayesian Audit. Sampling of ballots and the entering of election results took place on both January 24 and 25, 2019. Ballot sampling for each precinct was conducted using the three-cut method which involves cutting a deck of ballots three times and selecting the top ballot for auditing after each cut. The number of times a ballot is pulled after each three-cut is determined by the tool which prescribes the number of ballots needed from the precinct for the races examined.

Please see Appendices B and C for more details of the methodology.

5

The Porter County, Indiana RLA Results

The RLA Team audited Public Question #1, County Prosecutor, County Auditor, County Recorder and County Coroner. The total number of ballots cast in Porter County, Indiana for the 2018 general election was 65,449.

Public Question #1

Public Question #1 in Porter County, Indiana was first examined (see Appendix C for full RLA details). For this vote, the RLA tool instructed the RLA Team to audit a sample of 47 ballots. The margin for this vote was 35.8% and the diluted margin was 28.73%. The audit was inconclusive after the initial sample of ballots were selected due to the high number of undervoted ballots (n=14 or about 30% of the sampled ballots). The team randomly sampled an additional 165 ballots, which were a part of the initial sample of County Prosecutor and County Auditor ballots. This brought the total number of sampled ballots to 212, which included 97 “Yes” votes, 59 “No” votes and 56 “undervotes.” The Stark and Bayesian methods functioned as expected and confirmed the “Yes” outcome in the Porter County precincts with high levels of statistical assurance (97% for the Ballot Polling RLA; 99.8% for the Bayesian Audit).

County Prosecutor

The first race examined was for County Prosecutor (see Appendix C for full RLA details). The RLA tool instructed the RLA Team to audit a sample of 713 ballots. The margin for this race was 8.18% and the diluted margin was 7.99%. The Stark and Bayesian methods functioned as expected and confirmed the “Gary S. Germann” outcome in Porter County with high levels of statistical assurance (99% for the Ballot Polling RLA; 99.9% for the Bayesian Audit).

County Auditor

The second race to be examined was for the County Auditor (see Appendix C for full RLA details). The RLA tool instructed the RLA Team to audit a sample of 1,186 ballots. The margin for this race was 6.34% and the diluted margin was 6.14%. The Stark and Bayesian methods functioned as expected and confirmed the “Vicki Urbanik” outcome in the Porter County precincts with high levels of statistical assurance (100% for the Ballot Polling RLA; 100% for the Bayesian Audit).

County Recorder

The third race examined was for the County Recorder (see Appendix C for full RLA details). This race had the smallest diluted margin requiring a larger sample size. The margin for this race was 3.12% and the diluted margin was 3.0%. The RLA tool instructed the RLA Team to audit 4,761 ballots. Due to time limitations, we elected to restrict the sample to the same size as that for County Coroner. The risk limit was not reached due to the size of the sample of ballots. The sample also turned up some blank ballots initially. At our stopping point of 2,397 ballots, Lily Schaefer (the reported loser) had tallied 55 more votes than Chuck Harris (the reported winner). The Ballot Polling RLA was inconclusive and we would need to sample at least 2,364 more ballots to determine if the audit met the risk limit. It is possible that even more ballots would need to be sampled to reach the risk limit.

6

County Coroner

The fourth race examined was for the County Coroner (see Appendix C for full RLA details). The RLA tool instructed the RLA Team to audit 2,399 ballots. The margin for this race was 4.44% and the diluted margin was 4.29%. This initial sample of ballots showed that the Ballot Polling RLA tabulated results were too close to provide strong statistical confidence that the outcome was correct. Additional ballots would need to be audited to have a high level of statistical confidence that the outcome is correct. Our experiments with the Stark tool showed that an additional 347 Republican ballots would be needed to achieve this confidence level. This would mean approximately 800 more ballots to be audited. See Appendix C for more details of the results.

Conclusion

The Porter County RLA was one of the most comprehensive local RLAs conducted in the United States and VSTOP was able to acquire valuable information about pre-election preparation, poll worker training, ballot chain-of-custody, post-election processes, RLA processes and timing and budget. The RLA covered five races with varying diluted margins from 3% to ~29% and diverse outcomes – two races were won by Democratic candidates and two races were won by Republican candidates. Of the races that were audited completely (Public Question #1, County Prosecutor and County Auditor), we can report with very high levels of statistical confidence that the winner is the winner and the loser is the loser. On the flip side, we would need to sample many more ballots to have that same level of confidence for the County Recorder and County Coroner races.

Several lessons were learned from this audit. First, the Three-Cut ballot sampling method used was easy to explain and understand for the teams that were drawing samples of ballots. As a result, the ballot sampling and tabulation were more efficient than expected. Even with the ease of use and quick ballot polling, more time was still needed to sample additional ballots for the Recorder and Coroner races due to a substantial number of undervotes. It is important to note that all ballot sampling and tabulation is done manually and this process is time consuming. Second, the pre-audit on-site visit was helpful with preparing the county to be able to start the audit on time. Thus, this was time well spent by the VSTOP Team as it prepared for the audit. Third, proper chain-of-custody for ballots ensures that the ballots are secure and provides a good cross-check for auditors. Chain-of-custody logs help auditors trace ballots to their original container and can also be used to create the ballot manifest, which is a foundational document for RLAs. Fourth, a ballot adjudication guide is essential for assisting auditors with determining voter intent and ensuring that ballots are tabulated correctly. This is something VSTOP developed before the audit (see Appendix B). Fifth, as is known among RLA experts and practitioners, the number of undervotes impacts Risk-Limiting Audits. That is, when conducting a ballot polling RLA ballots are sampled from stacks or bundles of ballots. Some of the ballots have fully voted races while in other instances there may be undervotes. The undervote count is part of the ballot sample even though those ballots do not contribute to the audit. In short, an undervoted ballot is part of the sample but there is no data from the undervoted ballot which contributes to the audit and undervoted ballots are equivalent to “missing data” in research. Consequently, more ballots must be sampled if there are a large number of undervotes which impacts the sample negatively.

7

The Porter County RLA Project Team

Dr. Jay Bagga and Dr. Bryan Byers, VSTOP Co-Directors Jessica Martin, VSTOP Project Manager Mani Kilaru, VSTOP IT Specialist Isaac Walling, VSTOP Computer Science Graduate Assistant Jerome Lovato, RLA Expert Consultant

Acknowledgments

The VSTOP Team wishes to acknowledge and thank Secretary Lawson and her staff for their continuous support. We acknowledge and appreciate MIT Institute Professor Ronald L. Rivest and his graduate assistant Mayuri Sridhar for their assistance with the Bayesian audit analysis. Special thanks go to the Porter County Clerk Jessica Bailey, Deputy Clerk Kathy Hartwig, the Clerk’s Office staff and the Porter County Voter Registration Office staff for their support, assistance and cooperation with the organization and execution of the RLA. We are grateful to the six Porter County residents for their excellent work in handling and sampling the ballots and for recording the results of several hundreds of sampled ballots. This project could not have been completed without the help of these individuals.

Appendices

Appendix A: Risk-Limiting Audit Project for Porter County Indiana

Appendix B: Porter County Indiana Risk-Limiting Audit Process with a presentation on January 24-25, 2019

Appendix C: Detailed Results of Porter County RLA

8

Appendix A Risk-Limiting Audit Project for Porter County Indiana

9

December 13, 2018

1

Risk Limiting Audit Project for Porter County Indiana General Election 2018

Introduction During the November 2018 General Election, Porter County, Indiana experienced a number of issues and problems on Election Day. Secretary Lawson was asked to conduct a review of its election preparation procedures and other related matters. The Secretary and her staff discussed with VSTOP the possibility of a conducting a Risk Limiting Audit (RLA) of certain 2018 General Election races in Porter County. In a subsequent telephone conversation with VSTOP on December 5, 2018, Mr. Brandon Clifton (Deputy Secretary of State and Chief of Staff) requested that VSTOP conduct an RLA in Porter County Indiana. He also asked that a draft proposal from VSTOP for the RLA Project be submitted to the Secretary by December 14, 2018. General Election Information VSTOP has gathered the following general information about Porter County’s election data and its Voting System equipment.

Total number of precincts:

123 (http://www.porterco.org/index.aspx?NID=365#) Boone = 5, Center = 32, Jackson = 5, Liberty = 7, Morgan = 3, Pine = 2, Pleasant = 3, Portage = 33, Porter = 7, Union = 7, Washington = 3, Westchester = 16)

The 2018 election results are available at the following links:

Results by Precinct: http://www.porterco.org/DocumentCenter/View/5255 Results Summary: http://www.porterco.org/DocumentCenter/View/5256

The Indiana election results data are available at:

https://indianaenr.blob.core.usgovcloudapi.net/site/index.html

Summary Voting Data

Description Number Total Number of Precincts 123 Number of Registered Voters 123,529 Ballots Cast 65,449 Voter Turnout 52.98%

10

December 13, 2018

2

Porter County’s Voting System ES&S Unity 3.4.1.0

M100: 140 units AutoMark (Ballot Marking Devices): 125 units

Initial RLA Organization Preparation by VSTOP VSTOP has already contacted the following members of the Marion County RLA Pilot team:

• Mr. Jerome Lovato, EAC • Prof. Ron Rivest, MIT • Ms. Mayuri Sridhar, MIT

Dr. Jay Bagga attended the Election Audit Summit meeting held at MIT on December 8-9, 2018. During this event Dr. Bagga held discussions with the above individuals. Mr. Lovato has agreed to help with the preparations for the RLA, the selection of races to be audited, and discussions with the County on the organizations and material needed for the audit. Dr. Bagga also discussed RLA details with Prof. Philip Stark of UC Berkeley, the inventor of the RLA Audit Process. Additionally, Prof. Rivest and Ms. Sridhar also expressed support for helping with a Bayesian RLA audit. Proposed Porter County RLA Team:

• The VSTOP Team • Mr. Jerome Lovato, EAC • A representative of Prof. Ron Rivest, MIT

Proposed Races to be Audited

In discussions with Mr. Jerome Lovato we have identified the following countywide races:

County Race Remarks County Prosecutor To be audited County Clerk Not to be audited per SOS request County Auditor To be audited County Recorder To be audited County Sheriff Not to be audited as the race was unopposed County Coroner To be audited County Assessor Not to be audited as the race was unopposed

All of the precincts will be audited. This way, every ballot has a fair chance of being audited.

In the table above, the closest contest is County Recorder with a margin of 1,964 votes. We would have to audit approximately 5,000 ballots. These numbers are based on the margin and a 10% risk limit performing a ballot polling RLA. To audit the County Auditor race, we would examine approximately 1,200 ballots. We could make this the key contest and still audit all of the other countywide contests and see what the audit tool tells us as far as confidence is concerned.

The above discussion helps us to estimate the time required to conduct this risk limit audit. Generally, an audit team can review about 1 ballot every 2-4 minutes (this includes retrieving

11

December 13, 2018

3

the ballot) once a team has established a routine. A safe estimate is a pace of 20 ballots per hour. Examining 1,200 ballots with 6 teams of two would take approximately 10-20 hours. Additional time savings may be obtained if we employ the k-cut method of ballot sampling since it would remove the cumbersome task of trying to count to a specific number in a stack of ballots. In the Marion County RLA pilot, the ballots were exclusively handled by Marion County election personnel. We expect to follow the same process in Porter County. Hence, this would require significant involvement of personnel from Porter County for the 6 teams proposed. Additionally, it is estimated that County level personnel will need between 2 and 3 hours of on-site training prior to the RLA.

We have the ability to look at additional races on each ballot. Statewide and cross-jurisdictional races will be a little more challenging to get a true feel of the RLA since we would have to account for results from other jurisdictions. These additional races are listed in the appendix.

We seek guidance from Secretary Lawson on which, if any, additional races should be audited.

Type of RLA Ballot Polling audit using Stark’s RLA Tool and possibly a Bayesian audit with the BPTool. A Comparison Audit cannot be conducted since there are no cast vote records. Risk Limits In the discussion above, a 10% risk limit was proposed. The exact risk limits will be determined based on the races selected and related margins in consultation with Mr. Lovato, Prof. Rivest and Ms. Sridhar. Proposed Timeline Date Activity December 5-December 13, 2018

Data gathering and Initial discussions within the VSTOP Team and with Jerome Lovato of EAC, and others

December 14, 2018 This draft proposal to be submitted to Secretary Lawson December 17-19, 2018 Review and Approval of Proposal by Secretary Lawson December 19, 2018 Secretary Lawson to inform Porter County to work with the

RLA team to prepare and conduct the RLA audit December 20, 2018-January 11, 2019

RLA team to prepare the materials for the audit and conduct meetings with Porter County Officials

January 14-25, 2019 Final preparations, including possible preparatory visits, and implementation of the audit on a date or dates TBD

January 25-30, 2019 Porter County RLA Report Preparation January 31, 2019 Report to be submitted to Secretary Lawson

12

December 13, 2018

4

Budget The budget is to be determined. Cost may include travel expenses for Mr. Jerome Lovato and possibly a representative of Prof. Rivest. Other costs may include county level personnel time. Appendix

Federal Races: • US SENATOR • US REP DST 1

State Races: • SECT OF STATE • STATE AUDITOR • STATE TREASURER • STATE REP DST 9 • STATE REP DST 10 • STATE REP DST 11 • STATE REP DST 19 • PUBLIC QUESTION #1

County Races: • SUPER. CT JUDGE 2 • SUPER. CT JUDGE 6 • COMMISSIONER DST 2 CENTER • COUNTY COUNCIL DST 1 • COUNTY COUNCIL DST 2 • COUNTY COUNCIL DST 3 • COUNTY COUNCIL DST 4

Please note that there are other races that are not included in the list above as those were not discussed in consultation with the Secretary of State.

13

Appendix B Porter County Indiana Risk-Limiting Audit Process with a

presentation on January 24-25, 2019

14

1

Porter County Indiana Risk Limiting Audit (RLA) Process

January 24-25, 2019

Introduction: The Indiana Voting System Technical Oversight Program (VSTOP) and Porter County are conducting a Ballot Polling RLA onsite at the Porter County Government Administration Building at 155 Indiana Ave. Valparaiso, Indiana. This RLA process involves the following phases, which are both open to the public.

1. Generation of a seed number

A seed number is the first step in creating a pseudo random number generator (PRNG). A PRNG is, “…an algorithm that uses mathematical formulas to produce sequences of random numbers. PRNGs generate a sequence of numbers approximating the properties of random numbers. A PRNG starts from an arbitrary starting state using a seed state.” 1 The seed number must be a random number. In this case, we will use a 10-sided die to generate a seed number. A 10-sided die has digits 0-9. Such a die was rolled 20 times by Porter County Personnel in a public meeting on January 15, 2019 to generate the 20-digit random seed number (see below). This along with the Ballot Manifest is necessary for the Stark Tool (see below).

2. Conducting the RLA

Prior to 1/24/19, VSTOP will enter the seed number into Stark’s tool to determine the number of ballots that need to be sampled. The RLA will be conducted on January 24-25 (See details below).

Agenda for January 15, 2019:

VSTOP Team to arrive at approximately 9:30 am CT • Check connectivity of VSTOP Team Laptops/Equipment • Random Seed Number Generation: Participants and Public will be invited to roll a 10-

sided die. Once it is rolled 20 times a 20-digit random seed number is created • Ballot Organization and Ballot Manifest Creation

1 https://www.geeksforgeeks.org/pseudo-random-number-generator-prng/

15

2

Agenda for January 24-25, 2019: Location: Government Administration Building at 155 Indiana Ave. Valparaiso, Indiana, Room 102-A Meeting Time: 8:30 am – 5:00 PM CT January 24 – Day 1 8:30 am Jessica Bailey conducts Welcome and presides over oath 8:45 am Jerome Lovato explains audit procedures and workers are split into Teams 9:30 am RLA conducted Noon Break for Lunch 1:00 pm RLA conducted 4:30 pm Wrap up January 25 – Day 2 8:30 am Welcome, Additional directions if needed (Jessica Bailey & Jerome Lovato) 8:45 am RLA conducted Noon Break for Lunch 1:00 pm RLA conducted 4:30 pm Wrap up

RLA Procedure

County: Porter, Indiana

Election: 2018 General Election

Number of Registered Voters: 123,529

Number of Ballots Cast: 65,449

Total Number of Precincts: 123

Stark’s Risk-Limit: 10%

Bayesian Risk-Limit: 5%

Seed: 20 digits long (randomly selected at the county on 1/15/19 location using dice)

The 20-digit seed number that was publicly generated at Porter County on January 15, 2019 is:

67645322607997686112

Ballot Manifest: A document that describes how the ballots in an election are organized and stored

16

3

RLA tools used: The ballot polling RLA tools are available at

Stark’s Tool: https://www.stat.berkeley.edu/~stark/Vote/ballotPollTools.htm

Bayesian Tool (BP Tool): http://audits.csail.mit.edu/Main/

Races to be Audited:

Candidate information obtained from results summary available at http://www.porterco.org/DocumentCenter/View/5256

Race/Question Outcome Margin Number of Ballots to be

Audited* Public Question #1 (State-wide)

YES 35,655 NO 16,854

18,801 47

County Prosecutor

BRIAN T. GENSEL (REP) 29,339 GARY S. GERMANN (DEM) 34,568

5,229 713

County Auditor KAREN MARTIN (REP) 29,647 VICKI URBANIK (DEM) 33,667

4,020 1,187

County Recorder CHUCK HARRIS (REP) 32,483 LILY M. SCHAEFER (DEM) 30,519

1,964 4,761

County Coroner CYNDI DYKES (REP) 33,007 RANDY WILKENING (DEM) 30,197

2,810 2,399

*This number is generated from Stark’s Tool

Ballot Polling RLA:

A ballot polling RLA is essentially an exit poll, but in this case, ballots are randomly sampled and the votes for the chosen contests are tabulated. Ballots can be randomly selected using multiple methods, but we will use the following methods.

Method 1: The Stark tool displays the sequence numbers of the ballots to be selected from each stack of ballots organized by precinct. These ballots are selected without replacement and the results for the five races are inspected and recorded for each ballot. If it becomes necessary to adjudicate the results, the Ballot Adjudication Protocol (see below) will be used.

Method 2: k-cut method (in our case k = 3)

GIVEN: A stack of n ballots (n is in the range 100 to 500 ballots, for example) GOAL: A randomly chosen ballot from the stack METHOD: Define a "cut" to be the operation (as with a deck of cards) of taking a segment of ballots from the bottom based on a random portion of the deck. For example, start with a stack of 400 ballots and cut the random portion of the deck at 30-60% from the bottom and put that segment on the top.

17

4

So, the procedure for 3-cut is: • Do three consecutive cuts. • Take the top ballot in the stack as the chosen ballot.

After viewing the ballot organization and creating the manifest the RLA team will determine the method or methods for ballot selection.

Procedure:

Step 1: Determine number of sample ballots by Precinct.

Step 2: Ballot Polling Method

a. For each Precinct, select random ballots one at a time using 3-cut method or predetermined number from Stark’s tool

b. Record the outcome of each selected ballot and for each race/public-question being audited in an Audit Trail paper document and in a Spreadsheet.

c. Comment: For all undervotes, overvotes and ballots whose results cannot be adjudicated, record those as exceptions in the spreadsheet

Step 3: Repeat Step 2 until the requisite number of sample ballots are selected/recorded

Step 4: Input the end results into Stark and BP tools

Step 5: If risk-limit is not reached, go to Step 2 for the requisite number of ballots indicated by the tools. Comment: There is a small probability that this process may lead to a full recount

Step 6: Once the risk-limit is reached or (in rare instances) a full recount is done, the RLA ends

Ballot Adjudication Protocol

The tabulation team will serve as adjudicators for ballots with questionable marks. Examples of questionable marks are: use of highlighter or red ink, marks around the intended target instead of in the intended target, and stray marks. If a ballot with a questionable mark is selected, the tabulation team will:

1. Verify if the ballot has been remade. If it has been remade, the tabulation team will retrieve the original ballot.

2. Determine the voter’s intent. If the tabulation team cannot agree on intent, then the county clerk will be the deciding factor.

3. Tabulate the ballot as the voter intended.

The VSTOP Team will take pictures of the questionable marks and record the outcome of the decision for complete transparency and report writing purposes. Special care will be given when photographing ballot marks so that the marks and not the complete ballot are photographed.

18

5

Report:

A Report of this RLA will be prepared by VSTOP and submitted to Secretary Lawson by January 31, 2019.

Audit Team:

Dr. Jay Bagga and Dr. Bryan Byers, Co-Directors, VSTOP

Jerome Lovato, Technical Consultant, VSTOP

Jessica Martin, Project Manager, VSTOP

Mani Kilaru, IT Specialist, VSTOP

Isaac Walling, Graduate Assistant, VSTOP

The following provided special assistance with the Bayesian Audit:

Dr. Ron Rivest, Professor, MIT

Mayuri Sridhar, Graduate Student, MIT

19

Porter County, IndianaRisk-Limiting Audit

January 24-25, 2019

Jerome Lovato – Elections Consultant for VSTOP

120

What is a Risk-Limiting Audit?

A risk-limiting audit (RLA) provides strong statistical evidence that the election outcome is right, and has a high probability of correcting a wrong outcome. There are two main types of RLAs: ballot-polling and comparison.

The risk limit is the largest chance that a wrong outcome will not be corrected. If the risk limit is 5% and the outcome is wrong, there is at most a 5% chance that the audit will not correct the outcome, and at least a 95% chance that the audit will correct the outcome.

221

Two Main Types of RLAs

• Ballot-Polling Essentially an exit poll except ballots, instead of people, are randomly selected and tabulated (polled).

• ComparisonIndividual ballots are randomly selected and compared to the cast vote record (CVR) for each ballot. A CVR is an export of data from the voting system showing how the voting system interpreted markings for each ballot.

322

The RLA Triangle

BallotManifest

RiskLimit

RLAToolkit

(spreadsheets, forms, software, calculator, CVR)

ElectionResults

423

Contact Information

Jerome Lovatomr.jeromelovato@gmail.com

524

Appendix C Detailed Results of Porter County RLA

25

Detailed Results of Porter County RLA

January 24-25, 2019 Introduction: This document reports the results of the RLA conducted in Porter County, Indiana on January 24-25, 2019. There are 123 precincts in Porter County and the county stores ballots by precincts. VSTOP used this organizational structure for the audit. All the methods described below use an input seed (a random number with at least 20 digits). This input seed is used to begin the process of generating random numbers of ballots to be sampled. In our case, this was achieved by rolling a 10-sided die which resulted in the input seed being 67645322607997686112. This input seed was used by Stark’s RLA and Rivest’s Bayesian method tools to generate a pseudo-random sample (see Appendix B). The following risk limits (see Appendix A) were used for each audit:

• A Risk limit of 10% for Stark’s method • A Bayesian Limit of 5% for Rivest’s method

Ballot Manifest: The Ballot Manifest as shown in the following table describes how ballots are stored and organized. Precinct Ballots Cast Precinct Ballots Cast Precinct Ballots Cast Boone 1 434 Center 16 431 Center 33 379 Boone 2 356 Center 17 535 Center 34 668 Boone 3 458 Center 18 732 Center 35 348 Boone 4 557 Center 19 527 Jackson 1 548 Boone 5 360 Center 20 779 Jackson 2 493 Center 1 468 Center 21 525 Jackson 3 489 Center 2 474 Center 22 664 Jackson 4 515 Center 3 454 Center 23 846 Jackson 5 589 Center 5 478 Center 24 708 Liberty 1 700 Center 6 779 Center 25 762 Liberty 2 213 Center 7 566 Center 26 504 Liberty 3 1060 Center 8 619 Center 27 467 Liberty 4 535 Center 9 655 Center 28 369 Liberty 5 752 Center 11 694 Center 29 515 Liberty 6 363 Center 13 578 Center 30 605 Liberty 7 445 Center 14 494 Center 31 642 Morgan 1 289 Center 15 642 Center 32 422 Morgan 2 753

26

Precinct Ballots Cast Precinct Ballots Cast Precinct Ballots Cast Morgan 3 697 Portage 21 553 Union 3 681 Pine 1 565 Portage 22 513 Union 4 617 Pine 2 634 Portage 23 408 Union 5 1001 Pleasant 1 633 Portage 24 584 Union 6 437 Pleasant 2 660 Portage 25 384 Union 7 496 Pleasant 3 459 Portage 26 409 Washington 1 648 Portage 1 351 Portage 27 707 Washington 2 631 Portage 2 394 Portage 28 294 Washington 3 766 Portage 3 404 Portage 31 270 Westchester 1 352 Portage 4 760 Portage 32 415 Westchester 3 587 Portage 5 595 Portage 33 292 Westchester 4 671 Portage 6 313 Portage 35 647 Westchester 5 450 Portage 7 433 Portage 36 561 Westchester 6 137 Portage 9 666 Portage 37 509 Westchester 7 763 Portage 10 472 Portage 38 373 Westchester 8 601 Portage 12 347 Porter 1 468 Westchester 9 764 Portage 13 522 Porter 2 605 Westchester 10 550 Portage 14 479 Porter 3 602 Westchester 11 612 Portage 15 425 Porter 4 710 Westchester 12 771 Portage 16 437 Porter 5 500 Westchester 13 383 Portage 17 464 Porter 6 661 Westchester 14 743 Portage 18 572 Porter 7 262 Westchester 16 497 Portage 19 299 Union 1 415 Westchester 17 89 Portage 20 525 Union 2 326 Westchester 18 390

The ballot polling procedure involved the following steps:

• Ballots were randomly drawn by using the Three-Cut method (see Appendix B). • Ballots were examined manually • Results for each ballot were tallied

27

Detailed Results by Race Public Question #1 Ballot Polling (Stark RLA Method): The details were input into the tool https://www.stat.berkeley.edu/~stark/Vote/ballotPollTools.htm# The sample size of ballots was calculated by entering reported votes by the candidate and a total number of votes cast. Here the sample size was 47.

The seed number was input into the tool in order to generate a pseudo-random sample of ballots. The ‘current sample number’ field was initialized to 0. The fields ‘Number of ballots’ and ‘Draw this many ballots’ were auto-initialized. The random ballot numbers were selected and sorted with duplicates removed when the draw sample button was clicked.

The sorted sample ballots were examined manually (audited) with the results as shown below. The audit was inconclusive after the initial sample of ballots were selected due to the high number of undervoted ballots (n=14 or about 30% of the sampled ballots). The team randomly sampled an additional 165 ballots, which were a part of the initial sample of County Prosecutor and County Auditor ballots. This brought the total number of sampled ballots to 212.

28

Bayesian Method (Rivest's Method): The Bayesian tool was initialized with the following fields:

29

After auditing 212 ballots, Stark’s tool and the BP Tool reached the risk limits of 10% and 5%. The Stark and Bayesian methods functioned as expected and confirmed the “Yes” outcome in the Porter County precincts with high levels of statistical assurance (97% for the Ballot Polling RLA; 99.8% for the Bayesian Audit). County Prosecutor Ballot Polling RLA (Stark’s Method): The details were input into the tool https://www.stat.berkeley.edu/~stark/Vote/ballotPollTools.htm# The sample size of ballots was calculated by entering reported votes by the candidate and a total number of votes cast. Here the sample size was 716.

The seed number was input into the tool in order to generate a pseudo-random sample of ballots. The ‘current sample number’ field was initialized to 0. The fields ‘Number of ballots’ and ‘Draw this many ballots’ are auto-initialized. The random ballot numbers were selected and sorted with duplicates removed

30

when the draw sample button was clicked. In our case, 713 ballots were selected after removing duplicates.

The sorted sample ballots were examined manually (audited) with the results as shown below:

Once one has reached the sample number of ballots, if the risk limit is not met one continues the selection of ballots, using the three-cut method until the risk limit has been reached. Bayesian Method (Rivest's Method): The Bayesian tool was initialized with the following fields:

31

Once one has reached the sample number of ballots, if the Bayesian limit is not met one continues the selection of ballots, using the three-cut method until the Bayesian limit has been reached. The Stark and Bayesian methods functioned as expected and confirmed the “Gary S. Germann” outcome in Porter County with high levels of statistical assurance (99% for the Ballot Polling RLA; 99.9% for the Bayesian Audit). County Auditor Ballot Polling (Stark RLA Method): The details were input into the tool https://www.stat.berkeley.edu/~stark/Vote/ballotPollTools.htm# The sample size of ballots was calculated by entering reported votes by the candidate and a total number of votes cast. Here the sample size was 1,196.

32

The seed number was input into the tool in order to generate a pseudo-random sample of ballots. The ‘current sample number’ field was initialized to 0. The fields ‘Number of ballots’ and ‘Draw this many ballots’ are auto-initialized. The random ballot numbers were selected and sorted with duplicates removed when the draw sample button was clicked.

The sorted sample ballots were examined manually (audited) with the results as shown below:

Once one has reached the sample number of ballots, if the risk limit is not met one continues the selection of ballots, using the three-cut method until the risk limit has been reached. In our case, Vicki Urbanik reached the 10% risk limit.

33

Bayesian Method (Rivest's Method): The Bayesian tool was initialized with the following fields:

34

Once one has reached the sample number of ballots, if the Bayesian limit is not met one continues the selection of ballots, using the three-cut method until the Bayesian limit has been reached. The Stark and Bayesian methods functioned as expected and confirmed the “Vicki Urbanik” outcome in the Porter County precincts with high levels of statistical assurance (100% for the Ballot Polling RLA; 100% for the Bayesian Audit). County Recorder Ballot Polling: (Stark RLA Method) The details were input into the tool https://www.stat.berkeley.edu/~stark/Vote/ballotPollTools.htm# The sample size of ballots was calculated by entering reported votes by the candidate and the total number of votes cast. Here the sample size was 4,955.

The seed number was input into the tool in order to generate a pseudo-random sample of ballots. The ‘current sample number’ field was initialized to 0. The fields ‘Number of ballots’ and ‘Draw this many ballots’ are auto-initialized. The random ballot numbers were selected and sorted with duplicates removed when the draw sample button was clicked. In our case, 4,761 ballots were selected after removing duplicates.

35

The sorted sample ballots were examined manually (audited) with the results as shown below. The RLA tool instructed the RLA Team to audit 4,761 ballots. Due to time limitations, we elected to restrict the sample to the same size as that for County Coroner. The risk limit was not reached due to the size of the sample of ballots. The sample also turned up some blank ballots initially. At our stopping point of 2,397 ballots, Lily Schaefer (the reported loser) had tallied 55 more votes than Chuck Harris (the reported winner). The Ballot Polling RLA was inconclusive and we would need to sample at least 2,364 more ballots to determine if the audit met the risk limit. It is possible that even more ballots would need to be sampled to reach the risk limit.

Once one has reached the sample number of ballots, if the risk limit is not met one continues the selection of ballots, using the three-cut method until the risk limit has been reached. In our case, due to time constraints only a portion of the sample number of ballots was examined. Bayesian Method (Rivest's Method): The Bayesian tool was initialized with the following fields:

36

Once one has reached the sample number of ballots, if the Bayesian limit is not met one continues the selection of ballots, using the three-cut method until the Bayesian limit has been reached. In this case, due to time constraints a portion of sample number of ballots was examined. County Coroner Ballot Polling (Stark RLA Method): The details were input into the tool https://www.stat.berkeley.edu/~stark/Vote/ballotPollTools.htm# The sample size of ballots was calculated by entering reported votes by the candidate and a total number of votes cast. Here the sample size was 2,435.

37

The seed number was input into the tool in order to generate a pseudo-random sample of ballots. The ‘current sample number’ field was initialized to 0. The fields ‘Number of ballots’ and ‘Draw this many ballots’ are auto-initialized. The random ballot numbers were selected and sorted with duplicates removed when the draw sample button was clicked. In our case, 2,399 ballots were selected after removing duplicates.

The sorted sample ballots were examined manually (audited) with the results as shown below. The RLA tool instructed the RLA Team to audit 2,399 ballots. The margin in for this race was 4.44% and the diluted margin was 4.29%. This initial sample of ballots showed that the Ballot Polling RLA tabulated results were too close to provide strong statistical confidence that the outcome was correct. Additional ballots would need to be audited to have a high level of statistical confidence that the outcome is correct. Our experiments with the Stark tool showed that an additional 347 Republican ballots would be needed to achieve this confidence level. This would mean approximately 800 more ballots to be audited.

Bayesian Method (Rivest's Method): The Bayesian tool was initialized with the following fields:

38

39