© Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005...
-
Upload
julie-henderson -
Category
Documents
-
view
217 -
download
0
Transcript of © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005...
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Considerations In Designing a
Conformity Assessment System for Voting Equipment
Presented atThe Voting Systems Testing Summit
November 28-29, 2005Sacramento, CA
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Purpose of Presentation
This presentation discusses:
• Elements common to conformity assessment systems
• The EAC system for certification and decertification of voting equipment
• Considerations for state and local programs
• Potential for unintended consequences
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Certification of a product is a means of providing assurance that it complies with specified standards and other normative documents.
Several types of certification systems exist and are under review for the EAC system:
•Some comprise type testing only
•Other systems include initial testing and field surveillance
•Still others include initial testing of a product and assessment of its suppliers’ quality systems, followed by routine audits that take into account the factory quality system and the testing of samples from the factory and the open market.
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Components of a Certification System
•Initial Type Testing
•Assessment of the Supplier’s Quality System
•Field Information & Feedback
•User Involvement
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Key Participants
•EAC
•State certification authorities
•Testing Laboratories
•Vendors
•State & local election officials
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Key Issues for Certification System
•What is a minimal acceptable system?
•Are the testing lab/testers/lab assessors qualified?
•Will the vendor deliver units within manufacturing tolerances to those tested?
•How will the election officials know if non-compliant units are delivered and what corrective actions can it take?
•Will election officials and poll workers use the systems as intended?
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Primary ConsiderationNational Program The primary concern of the EAC
program is that the system design, as represented in the system submitted for testing, meets minimum requirements.
State Program State certification seeks to assure that:
1. Systems meet specific state requirements.
2. May be used as that state administers elections.
3. The vendor can give reasonable assurance that the units delivered meet and continue to meet the requirements over their useful life.
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Primary ConsiderationPurchase Evaluation Which system offers the best value
among systems offered.
Receiving Inspection Are the delivered systems functional and within manufacturing tolerance of those certified by the EAC and state officials?
Pre & Post Election Are the systems functioning correctly and unaltered from their certified condition?
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Coordinated Processes
Processes are necessary to implement the components of a conformity assessment system for voting systems and to provide reasonable assurance that the key concerns of such a system are carefully addressed.
The best confidence will be gained if the various authorities involved in certifying and selecting voting equipment coordinate their efforts so as to maximize the value contributed from each function.
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
EAC / NISTTGDC
Technical Guidelines
EAC / NISTTGDC
Technical GuidelinesCandidate System
Accredited
Test Laboratory
Accredited
Test Laboratory EACEAC Technical Review
Team
Technical Review
Team
Certified SystemSystem
Description
System
DescriptionNIST
Software Ref. Lib.
NIST
Software Ref. Lib.
Vendor
Quality & ConfigurationControl Processes
Vendor
Quality & ConfigurationControl Processes
State
Certification
State
Certification
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Certified SystemSystem
Description
System
DescriptionNIST
Software Ref. Lib.
NIST
Software Ref. Lib.
Vendor
Quality & ConfigurationControl Processes
Vendor
Quality & ConfigurationControl Processes
State
Certification
State
Certification
Field Deploymentand Use
Field Deploymentand Use
Purchase SelectionPurchase Selection
Purchase SelectionPurchase Selection
Pre & Post Election
Testing
Pre & Post Election
Testing
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
EAC / NISTTGDC
Technical Guidelines
EAC / NISTTGDC
Technical Guidelines
Accredited
Test Laboratory
Accredited
Test Laboratory
EACEACTechnical Review
Team
Technical Review
Team
NIST
NVLAP&
Software Ref. Lib.
NIST
NVLAP&
Software Ref. Lib.
VendorsVendors
State & LocalElection Officials
State & LocalElection Officials
CitizensCitizensLocal
Receiving Inspectors
Local
Receiving Inspectors
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
System Description
The ability to determine that a system is identical to that considered at other steps in the process is critical to the validity of the conformity assessment system.
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Typical FCC Submission
The FCC routinely requires external AND internal photos accompany all submissions.
For Software Controlled Radios source code is required for an equipment grant.
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Typical FCC Submission
External Photos
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Typical FCC Submission
Internal Photos
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Challenge for Vendors
Is there a way by which high confidence many be delivered that a system is exactly the same without
also transmitting proprietary information?
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Considerations
How does each step of the process assist the next?
Could the ITA’s include a section in each test report highlight information and observations that would be helpful to state certification personnel?
Could state certifiers transmit observations to those making purchasing decisions?
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Considerations
What is the repeatability of these evaluations?
How often will the same system receive the same evaluation if submitted at different time or to different ITA’s?
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Considerations
How does field performance and field experience information come back into the system to improve it?
How will field experience come into the system to improve EAC and state certification processes?
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Unintended Consequences
Cost of certification will limit the number of systems offered and may eliminate vendors.
With more rigorous testing and evaluation vendors will pay for fewer systems to be certified.
Vendors will bring new systems to market less often.
Some vendors may abandon this business.
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Unintended Consequences
Will the time and expense of certifying changes and upgrades delay good and needed changes?
What is the right balance between careful evaluation of all changes and not making these evaluations so costly that needed improvements are delayed or not introduced at all?
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Unintended Consequences
How will the necessary coordination of administrative processes and equipment requirements be maintained and evaluated?
Election management and equipment requirements must work together. How will this be assured? Where several alternative solutions are possible how will the linkage be maintained between equipment requirements and election management practices?
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
ProposalsDocuments are needed that define the interaction and expectations of each participant in the process.
Direct communications from each step in the process to succeeding steps may be very helpful.
Clarification is needed as to what is not done in preceding steps in the process.
TTEEMM CCoonnssuullttiinngg,, LLPP
© Copyright 2005 TEM Consulting, LP - All Rights ReservedPresentation To EAC Aug. 23, 2005 Hearing, Denver, CO Rev 1 – 08/16/05 - HSB
Conclusion
A well constructed certification system provides satisfactory answers to central issues:
•What is a minimal acceptable system?
•Are the testing lab/testers/lab assessors qualified?
•Will the vendor deliver units within manufacturing tolerances to those tested?
•Will the election officials know if non-compliant units are delivered and what actions can they take?
•Will election officials and poll workers use the systems as intended?