Thermal Impedance Measurements for Quality Assessment of ...
USPTO Quality Measurements
Transcript of USPTO Quality Measurements
National Association of Patent
Practitioners 20th Annual
Meeting and Conference
Martin Rater Office of Patent Quality Assurance, USPTO
July 28, 2016
1
Defining Quality
Biggest challenge has been generating metrics that meet a wide range of quality definitions
– Examiner’s view vs. Applicant’s view
– Big Q vs. little q
– Timeliness and value
– Reality vs. expectations
– Everything in between • I know it when I see it
• Akin to defining the color medium blue
Patent Examination Quality
Primary focus has been on examination quality – Examiners’ adherence to laws, rules, and procedures
– Tracked against some established standards for desired outcomes • Correctness – statutory compliance
• Clarity
• Consistency
• Reopening
• Rework
• Impacts on advancing prosecution
– Basis for historic “compliance” metrics reported by USPTO
Additional Challenges
• Objectivity vs. Subjectivity
• Leading vs. Lagging indicators – What we are doing rather than what we did
• Controlling for a wide range of factors – e.g. Technology; Examiner experience; applicant behavior; pilot programs
and initiatives
• Establishing causal effects
• Balloon-effect of pushing quality problems elsewhere
• Verification and validation of quality metrics
• There is no silver bullet
• Uniqueness of what we do
Office of Patent Quality Assurance (OPQA)
• 56 Review Quality Assurance Specialists (RQAS) – Average of 20 years of patent examination experience
– Demonstrated skills in production, quality, and training
– Assignments based on technology
• Major activities
– Review of examiner work product
– Coaching and mentoring
– Practice and procedure training
– Program evaluations, case studies, ad-hoc analyses
• Operate under an established ISO 9001-certified Quality
Management System
Data Sources • Patent Application and Monitoring System (PALM)
– Objective transactional data
• Review-based questionnaires/forms – OPQA
– Patent Operations
– Recent transition to Master Review Form
• Perception surveys – External stakeholders
– Patent examiners and managers
• Administrative data
• Big Data – Office Actions
Study Types
• Descriptive – Describe and explain conditions
• Time series / Longitudinal – Emphasis on growth or change
• Correlational – Explore relationships
• Causal – Cause and effect
Current case studies initiative encompasses multiple study types
Study Designs
• Quasi-experimental
– Lack random assignment
• e.g. program eligibility requirements
– Must control or account for characteristics such as
• Examination experience
• Technology
• Training/background
Sampling
• Random sampling – Larger sample sizes needed for estimating proportions
– Primary factors in sample size determination • How data will be used
• Resources necessary for data collection
– Maintain representativeness
– OPQA-generated metrics based on random sampling: • Compliance rate reviews
• Employee perceptions
• Training effectiveness
Sampling
• Stratified sampling – Superior to simple random sampling
– Requires weighting of data
– OPQA-generated metrics based on random sampling: • External Quality Survey
– Reduce sampling error and limit respondent burden
• Non-Probability Methods – Used sparingly to get general estimate of the results
• Convenience sampling
• Judgement sampling
Analysis of Quality Data
• Trend Analysis – Longitudinal data sets such as compliance rates, Quality Index Reporting
(QIR) database, quality surveys
– Statistically-significant differences • Significance is not always significant!
– Rolling 12-month statistics
• Pre-Post Comparisons – Evaluations of pilot programs
– Impacts of training initiatives and rules/policy changes
– Key points: • Establish a baseline
• Find a control group
Analysis of Quality Data
• Outlier analysis – Anomaly detection
– Primarily has been used for monitoring examination behaviors vis QIR database • Objective, comprehensive data set of transactions
– Big Data greatly enhances ability to detect anomalies • Not biased by what we think are the items of interest
• Custer and Factor Analysis – Cluster: classify
– Factor: reduce
• Correlation and Regression Analysis – Relationships
– Cause and effect
Reporting
• Frequency – Quarterly reporting most common for longitudinal data series
• Allow time for implementation of corrective and/or preventive actions
• Performance award plan inputs
• Level/Details – Maintain statistical validity
– Report only what the sample will support • Corps, TC, Art Unit, Examiner
– Combine reporting periods to enhance level of detail if sample size not sufficient
• Providing data sets for external analyses of data – Data hub
– Master Review Form
Next steps in USPTO Quality Metrics
• Discontinue Quality Composite
• More granularity
– Statute
– Action Type
– Technology
• Consistency, Reopening, Rework
• External perceptions for validation
• Transparency and sharing of data for external analyses