## Risk-Limiting Post-Election Audit

### Summary

The Risk-Limiting Audit (RLA) utilized by the Cuyahoga County Board of Elections (CCBOE) is based upon the Kaplan-Markov method as explained by Philip B. Stark and Mark Lindeman.[1] [2] Risk-Limiting Audits have a large, pre-determined chance of leading to a full recount whenever a full recount would show a different electoral outcome. A spreadsheet is created with election data/results to: determine how many batches must be audited to reach the desired confidence level; determine selection probabilities; track batches picked; record hand count results; and determine the effects of any changes as a result of the hand counts.

Materials Needed:

- All voted ballots
- Official Results showing each contest’s results per precinct by category
- Risk-Limiting Post-Election Audit Blank Spreadsheet
- Hand Count Tally Sheets
- Supplies for random selection (see below)
- Calculator
- Green pens

##### Sample

Risk-Limiting Post-Election Audit SAMPLE

#### Preparing for the Audit

For information on how to prepare for the audit click here.

#### Determining Audit Size

The number of ballots that must be audited for a risk-limiting post-election audit is based on a confidence level that has been decided upon. While this percentage may vary according to election jurisdiction, it is generally prescribed by the secretary of state or other head election officer. The Cuyahoga County Board of Elections uses a confidence level of 90% (i.e., it has at least a 90% probability of leading to a full recount if the apparent outcome is incorrect).

There is no simple formula for determining the audit size. Our risk-limiting spreadsheet must be used and filled in with the pertinent election data. (Instructions are included in the spreadsheet). This is because the audit size is dependent upon the sum of all error bounds, which in turn is based on the number of votes in each batch, the margin of victory in that batch, and the margin of victory within the contest as a whole. Once the data is entered, the spreadsheet will run the necessary calculations and determine how many batches are required to be audited assuming there are no differences between the audit and the official canvass.

#### Selecting Batches for Audit

The required number of precincts to be audited must be randomly selected by a bipartisan selection team in a time and place that is publicized and in a location that is open to the public.

- It is recommended that the total ballots cast per precinct be split into multiple batch types.[3] Each batch consists of a different category: absentee ballots, Election Day ballots, provisional ballots, post-Election Day ballots, and post-absentee ballots respectively.
- Whole batches shall be selected for the audit.
- Enough batches shall be selected for the audit equal at least to the threshold of the confidence level that must be audited for the election at issue in the county.
- A “Probability Proportional to Error Bound with Replacement” (PPEBWR) selection method is utilized to determine which batches should be audited.
- This method mandates that each batch in the contest to be audited be assigned a range of numbers that pertain to that batches error bound (this is automatically calculated in the spreadsheet).
- In sum, the greater the possibility of a miscount within a batch, the more numbers are assigned, and the more likely the batch is to be selected.
- “With Replacement” means each number has a chance to be picked more than once.

- The specific method used to randomly select batches can vary depending on election jurisdiction. As the PPEBWR selection method demands unequal representation among batches, the range of numbers to be picked from is usually fairly large due to rounding problems. The range of numbers that must be picked from could be as high as 1,000,000. This necessitates some changes to the selection process. Instead of assigning each random pick to a specific precinct or batch, each pick dictates one digit of the whole number. If the range of numbers to pick from is 0 – 999,999 one single digit would be picked six separate times. For example, if a if 0 is picked first, 7 is picked second, 9 is picked third, 1 is picked fourth, 5 is picked fifth and 4 is picked sixth, the number would be 079154.
- Some common methods of PPEBWR batch selection include:
- Using ping pong balls and a hopper, assign the numbers 0-9 to 10 ping pong balls, place all ping pong balls into the hopper, mix the ping pong balls together and select one ball at a time with each pick creating a single digit until a whole number is made, continuing until the threshold is reached or exceeded. Each ball must be replaced after every pick.
- Using a box and small slips of equal sized paper, assign the numbers 0-9 to ten slips of paper, place all slips of paper into the box, mix the papers together and select one slip at a time with each pick creating a single digit until a whole number is made, continuing until the threshold is reached or exceeded. Each slip of paper must be replaced after every pick.
- Using the respective number of ten-sided die with numbers 0-9, roll each die, with each dice being a single digit until a whole number is made, continuing until the threshold is reached or exceeded.
- This is the method currently in use by the CCBOE.

#### Auditing the Selected Precincts

Teams of election officials equally divided between the state’s two major political parties for each precinct batch to be audited will perform the following steps.

- Record the name of the batch on the “Hand Count Tally Sheet.”
- Compare the total number of votes cast in the candidate races and question or issue elections being audited to the number of voters listed in the poll book, poll list or signature poll book records.
- If more votes in a precinct appear for a particular race or issue than the number of marked names in the poll book, poll list or signature poll book records (showing voters who voted, including absentee and provisional voters), check the appropriate box on the tally sheet.
- Verify each candidate’s race, or a question or issue has been properly identified.
**NOTE: Observers and members of the public may observe the inspection of the ballots but may not handle ballots.**- For batches with multiple ballot types, the auditing counting team will process one ballot category at a time.
**(DEM)**Count the total number ballots in stacks of ten, crisscrossing each stack.**(REP)**Verify each stack contains ten ballots.- Total the ballot stacks and record this number on the tally sheet. A supervisor will compare the hand count of total ballots to the official certified results. If the totals match, continue to the next step. If there is a difference, the same team can do a new hand count.
- Separate the ballots by each possible response for the audited contest including stacks for over votes and under votes.
- Contests in which more than one response may be voted will require each response to be stacked and counted individually.
**(DEM)**Count the total number ballots in each response pile in stacks of ten, crisscrossing each stack.**(REP)**Verify each stack contains ten ballots.- Total the ballot stacks for each response and record on the tally sheet.
- A total of all ballots for each ballot type within the batch will be hand counted and placed in the TOTALS column of the tally sheet.
- A supervisor will compare the hand count tally to the official certified results.
- If the results match, return all ballots to their original packaging and move on to the next selected precinct.
- If there is a difference, a new hand count will commence by a different team.

Repeat these steps until all selected election units have been hand counted.

#### Audit Escalation

Risk-limiting audits have escalation procedures built into them. In the event of discrepancies between the Official Canvass and the Audit, the spreadsheet will automatically calculate, based on the level of confidence, whether selecting additional batches to audit is necessary.

[1] Stark, P.B. (December 2009). Risk-limiting postelection audits: p-values from common probability inequalities. IEEE Transactions on Information Forensics and Security, 4(4). Retrieved from http://www.stat.berkeley.edu/~stark/Preprints/pvalues09.pdf.

[2] Lindeman, M. (2010, January 10). A Kaplan-Markov auditing example using 2008 California data (version 1.2x 3/1/2010). Retrieved from https://sites.google.com/site/electionaudits/small-batch/kaplan-example-12x.pdf?attredirects=0.

[3] Electionaudits.org. Principles and best practices for post-election audits. Retrieved from http://electionaudits.org/principles.html.