Fatal Flaws of the VSAP Report

Analysis of the Voting Systems Assessment Project (VSAP) Report

by Judy Alter, Ed.D., Director Protect California Ballots

This two-part critical analysis of the Voting Systems Assessment Project (VSAP) Report first challenges the basic premise of the project, and describes the serious omissions, distortions, and misinterpretations that the RR/CC presented in the Summary Report of the studies in Voting Technology Project and, second, identifies the many research method­o­­logical errors in the reports that follow in the four addenda. The VSAP report supplies NO basis for change in the Los Angeles County voting system and, therefore, cannot in this form provide “a solid foundation for continuation of the project.” (p. 2 SR)



Dean Logan, Los Angeles County Registrar Recorder/County Clerk (RR/CC) submitted to the Los Angeles County Board of Supervisors a report: Voting Systems Assessment Project (VSAP) on July 9, 2010. This VSAP report includes the Summary Report (SR) by the RR/CC, and four more documents on which he bases this summary report:

  1. 2010 Survey of Performance of LA County Elections, Spring 2010 Study: Initial Survey Report. Internet (YouGov/Polimetrix) and telephone interviews of voters (Interviewing Services of America Inc.)

  2. Getting Ready for Tomorrow’s Voters: VSAP Focus Groups (12) Report (SA/Qualitative Insights)

  3. 2010 Poll worker Survey of Performance of LA County Elections: on-line survey

  4. Internal RR/CC Staff Discussion Groups: 64 staff members

The RR/CC claims to have established “an unprecedented project” to gather information from citizens and officials concerning their opinions about how to upgrade and improve, even replace the current InkaVote Plus voting system. The project partners with Massachusetts Institute of Technology (MIT) and California Institute of Technology (Cal Tech) in their Voting Technology Project (VTP) (established by the voting technology industry leaders). The James Irvine Foundation funded the VSAP with $150,000.

The Voting Technology Project (VTP) and RR/CC employed four private companies (three listed above and The Connections Group) to conduct each part of the research activities. The reports give no evidence about how the VTP or VSAP members selected these private companies.

In Research Activities (p. 3 SR) the RR/CC lists the diverse groups from which he is collecting “sound data.” Omitted from this VSAP report are responses from the local election officials, although they are quoted, and Community Organi­za­­­­tions, “election advocates,” and other election experts.

Problems in the RR/CC Summary Report (SR)

Questionable Premise:

The RR/CC declares that LA County needs a new voting system (p.1). In only a few places scattered throughout the 69-page report, does he justify the need for a new system. In fact a majority (55%) of the groups surveyed do not believe the current system needs replacing.

In Project Background, (p. 3 SR) the reason for establishing the project is in anticipation of “future regulatory changes and pending legal requirements that our current systems are unable to meet.” The RR/CC offers no supporting information for this statement beyond stating that LA County uses ”outdated computer equipment” in Voter Perception of the Process, (p. 6 SR) as part of the summary of the perspectives and outlook of the RR-CC staff.

In the first statement under Current InkaVote Plus System, (p.6 SR) the RR/CC claims that the “changing demographics and complexities suggest strong need to change the voting system” although he does not say who says this, why, and why now. He offers no information about how to fix the current system to adapt it to the changing requirements. The summary does include the advan­tages of the current system as well as the disadvan­tages.

Further the RR/CC asserts that a new system could help consolidate local and county elections and streamline poll worker training but offers no explanation for this claim. Many local elections are already consolidated with the county-wide elections. The RR/CC gives no examples of cost savings for the city clerks; new equipment will cost money. A new system would require new training for both the poll workers and local election officials and create additional financial burdens.

General Bias towards computer systems:

The name “Voting Technology Project” in the partnership with the LA County RR-CC shows the emphasis of the project: technology. As Brad Friedman (well known “election advocate”: of Bradblog.com) said at a meeting with the RR/CC on Aug. 11, 2010, “It is computer centric.” This technolog­ical bias flies in the face of the basic tenet of democratic elections that require public vote counting. Citizens cannot see or know if election results are accurately counted when the counting is done on computers. And the California Secretary of State severely limited the use of Direct Recording Electronic voting machines (DREs) after the 2007 Top to Bottom Review of all California election machinery because experts showed how insecure they were and how easily they could be hacked. The MTS tabulating system of LA County was incorrectly not part of the review.

In the second part of the VSAP surveys, when interviewers asked respondents what they valued in a new system, the overwhelming response among the public groups was “accuracy.” The second most important value was “security.” The focus group members also required a paper record of their votes. The basic technological focus of the study is, therefore, distorted and incomplete because with “techno­logy,” voters cannot know if their votes are counted accurately or if they are secure.

The RR/CC, SR, emphasizes how little voters trust hand counted paper ballots. This generalization distorts the questions in all the surveys because voters assess hand marked paper ballots in three of the four options: hand counted paper ballots, paper ballots counted by scanners, and InkaVote Plus (not defined). He glosses over whether the voters distrust hand counting or the marking of paper ballots. The preference for technology runs throughout the VSAP.

Distortions of the Studies’ Results

In the Summary of Findings (p. 6 SR), the RR/CC distorts the information in the actual study. Between 20 to 25% of the voters surveyed repeatedly showed in their “I don’t know” answers how little they know about voting systems, their processes, and accuracy. In the Voter Perceptions of the Process:

    • “Nine of ten voters are confident that their votes are counted as cast.” The actual Study does not give the results of questions 32a,b that ask about their confidence in the system.

    • “Poll workers exhibit a similar level of confidence. Specifically poll workers believe the current InkaVote Plus voting system is working fine.” The actual poll worker survey asked them about InkaVote, the name of the method by which voters mark their ballots, not InkaVote Plus, the name of the device that could count and even tabulate ballots and votes on ballots.

    • The RR/CC repeats that the voters had three choices when, in each survey, respondents had four choices.

    • In the discussion of the focus groups’ responses to “ballot marking devices” being the ideal for “ease of use” (p. 9 SR) the RR/CC leaves out contradictory evidence from a quoted voter who calls them “an expensive pencil.”

    • In the section, Marking the ballot, (p. 9) The voters in the phone and internet surveys were asked to describe their confidence level among three choices: “1) Hand counted paper ballots; 2) Hand marked optical Scan ballots; 3) Electronically marked Direct Recording Electronic (DRE) ballots.” (The RR/CC, SR, leaves out InkaVote Plus, the fourth choice.) The “focus groups presented voters with a slight variation: 1) Hand counted paper ballots; 2) Optical Scan-Ballot marking device which, blends a touchscreen interface with an opscan ballot; 3) Direct Recording Electronic (DRE) ballots.” The ballot-marking device is not a hand-marked ballot and, thus, is not a slight variation at all.

    • He claims the voters agreed that the county needs a “public-private partnership” (p.10 SR) when only the local election officials and the RR/CC staff discussed this issue.

Inaccurate information in the SR

In Recommendations/Next steps (p. 10 SR) the RR/CC assumes that the VSAP process described in the report is complete. It is not. Three of the groups he lists as being part of the VSAP are neither complete, nor described, nor have happened. Researchers need to provide the report of the meeting with local election officials. Staff members need to write up the meeting on August 11, 2010 with the “election advocates.” (Does he mean election integrity advocates?) And the RR/CC needs to consult with stakeholders and election experts and then report on these results. Therefore, the RR/CC needs to complete the study before consulting with the LA Board of Supervisors and the Chief Executive officer.

In the Conclusion (p. 11 SR) the RR/CC distorts information saying that the “voters trust that their votes are counted as intended.” In the actual report 25% of the phone interviewees and 41% of the Internet voters surveyed responded, “I don’t know” to this question. (Question 42d, Table 1, p. 16, Addendum 1)


The Summary Report by the RR/CC does not provide the Supervisors complete and accurate analysis of the information contained in the reports. Without reading and analyzing the entire series of reports in the VSAP booklet, readers will not know that the SR is insufficient, incomplete, and inaccurate; it does not show that the VSAP provides “a sound foundation for the complex decisions that lie ahead.” (p. 11 SR)

Review of the actual research study reports. In the VSAP

Problems with the research methods

The following methodological problems demonstrate that the research submitted does not provide a basis on which to proceed to make decisions about a new voting system for LA County.

1. High rate of uninformed respondents

At the top of page 4, Alvarez and Stewart (A & S, Addendum 1) state that when they asked voters if the current InkaVote Plus system should be replaced, 49% of the Internet and 25% of the phone respondents said that they did not know. A & S do not indicate if they defined “InkaVote Plus”: the name of the entire voting system used by the voters. It is also the commercial name of the precinct based reader, PBR, the scanner, which offers voters “second chance voting,” letting voters vote again if they over-vote or leave a ballot blank.

The survey methods, phone and internet, yielded such different levels of “I don’t know” answers that the researchers needed to perform a correlation analysis (a statistical tool to reconcile two distinct groups in research) to show that they could compare them even though “significantly more” internet respondents answered “I don’t know.”

The serious level of voters’ “I don’t know” answers actually provided the RR/CC and researchers valuable information about how uninformed voters are. This lack of voters’ knowledge demonstrates to researchers that they cannot claim that these studies in the VSAP are valid or reliable. The results demonstrate that voters are unable to participate in this kind of study because they lack basic knowledge about their own voting system, in addition to not knowing what its name is. In contrast, the staff members at the RR/CC headquarters do know about the voting system.

2. Inconsistencies in the questions asked.

The voters were asked to assess InkaVote Plus while the poll workers survey gave their opinions about InkaVote. See Addendum 2.

Respondents of the internet survey were shown pictures of the four voting options: a) hand counted paper ballots, b) hand marked paper ballots counted by scanners in the precincts or centrally, c) DREs, or d) InkaVote Plus. Telephone respondents did not see pictures.

Unlike the telephone and Internet respondents, the focus groups were asked about ballot marking devices instead of just optically scanned paper ballots.

3. Inaccurate information given respondents

When interviewers asked respondents about DREs, the surveys and interviewers defined them by using an inaccurate analogy that the DRE works like an ATM machine. That is just not true. Bank clients receive a receipt with their own account number on it. They receive a bank statement showing their withdrawals and deposits. Ballots of voters who vote on an electronic voting machine have no names; DREs offer no way for the voters to know if their ballots were accurately counted. In contrast, bank statements show the accuracy of clients’ deposits and withdrawals.

4. No evidence of information given or terms defined

On p. 5, A & S (Addendum 1) realized that the voters answered “I don’t know” at the highest level for InkaVote Plus. Perhaps that is because the voters do not know that name of the system on which they vote is called InkaVote Plus. Yet the researchers continue to generalize about the results in the study.

The authors describe the voters’ demand for a paper “trail.” Did the experts conducting the study clarify the major difference between a paper ballot and a paper trail, one with legal standing and one with none. Researchers have also shown that a paper “trail” may not reflect the votes cast by voters or record those votes accurately on the memory card or internal tally tape inside the DRE.

5. Report omits actual findings

On page 3 par. 4, A & S state that 84% of LA County voters felt very or somewhat confident that their votes were accurately counted yet they do not include in their report all the results of their survey and do not show the details of the answers in the tables to this question, 32 a, b, which asks about the levels of confidence in this report. See Addendum 1.

6. Artificial choices offered

In the “comparison among four systems” the authors fail to note that three of the four choices use hand marked paper ballots. I combined the total responses among these three choices. In every table in Addendum 1, except for one, (Table 5, By race analysis Asians,) the totals for the three paper-based ballot choices combine to show significantly higher percentages of preference among the respondents than for DRES. The authors’ analysis, therefore, distorts the information gathered by artificially dividing the paper-based choices into 3 groups.

7. Qualitative data inadequately reported

When the authors of Addendum 2 provide quotations made by the members of the focus groups, they do not report how often the statements were made during the 12 sessions. Thus, readers do not know if the quotations were frequent or rare in the discussions. Proper reporting of this kind of data mandates this minimal quantitative indication.

8. Participation selection process unclear

A & S explain that the respondents for the telephone and Internet surveys were randomly selected in Addendum 1. The authors of the focus group report, Addendum 2, do not provide sufficient information about how the private company, The Connections Group, recruited and assembled the 12 groups of registered voters.

Authors R. Michael Alvarez, Thad E. Hall, and Kathleen Moore (Addendum 3) do not explain how the poll workers were selected to participate in their online survey other than saying that the individuals were invited to complete it after they completed their online training. What percentage of these poll workers are newly trained with little experience with the voting system?

The SR by the RR/CC states that 24 local election officials participated in discussions. That report is not included in the VSAP nor is an explanation of how they were selected.

Sixty-four staff members of the RR/CC participated in discussions. See Addendum 4. The RR/CC does not state how these participants were selected.

9. Acknowledged limitations of VSAP

The researchers indicate the limitations of the study and state this in the following paragraph (p. 11 A & S Addendum 1):

These survey results are preliminary, in the sense that this is the first time that we are aware that surveys of this nature have been conducted in Los Angeles County. These results should thus be seen as a preliminary baseline, and as the VSAP effort evolves voter evaluation surveys like these should be conducted to assess how and in what ways voter opinions are changing overtime.”


Summary : Flawed Research

This analysis shows the serious problems with the research methods used in the VSAP: high rate of uninformed respondents, omission of findings, inconsistent questions, inaccurate information, lack of definitions, artificial choices, unclear participant selection process, inadequately reported quantitative data, and the acknowledged limitations by the researchers themselves. As a professor of research methods, I would have terminated the project if a student attempted to use the high number of uninformed respondents—minimally one in four--as is in the VSAP studies. These problems make it clear that the findings in the VSAP Reports cannot be used for even a preliminary basis for deciding on a new voting system. The main finding is how uninformed the voting public is about their voting processes. If the people directing the project intend to repeat it, they must provide lots of voter education. The researchers should then redo the questions, make them consistent across all groups, and provide accurate definitions and complete explana­tions. They should also make their respon­dent selection process completely transparent for each section in the VSAP compound report.

Judy Alter, M. A. Mills College; M.A.T.& Ed.D. Harvard School of Education; UCLA Emerita Associate Professor at UCLA, taught at Tufts University, University of Wisconsin Madison, taught from 1975 to 2000; among other subjects, taught research methods, directed over 100 M.A and Ph.D. theses and dissertations; published 23 refereed articles with qualitative and quan­titative research; refereed for two professional journals. Taught graduate research on Fulbright teaching fellowship in Portugal.

Since 2004 Dr. Alter has researched and made public several voting irregularities projects including Recount New Mexico, led citizens’ exit polls, studied the 1% manual tally in Los Angeles County, given over 120 talks, and leads Protect California Ballots, a registered unincorporated non-profit association registered with the CA Secretary of State. This report was written with Darlene Little and Michael Milroy, assisted by Ken Aaron and Rochelle Low.