Computer Applications in Testing and Assessment James P. Sampson, Jr. Florida State University...
-
Upload
isabella-banks -
Category
Documents
-
view
219 -
download
0
Transcript of Computer Applications in Testing and Assessment James P. Sampson, Jr. Florida State University...
Computer Applications in Testing and Assessment
James P. Sampson, Jr.
Florida State University
Copyright 2002 by James P. Sampson, Jr., All Rights Reserved
2
Overview of Topics
• Computer-Assisted vs. Computerized
• Rationales for Computer-Assisted Testing
• Potential Benefits
• Potential Problems
• Future Use
3
Computer-Assisted vs. Computerized
• Both describe use of computer technology in assessment process
• Used with nearly the same frequency
• Using the term computerized may cause problems and foster misunderstanding
4
Computerized
• Implies that computers replace humans previously needed to complete a task
• May lead to practitioner under use or misuse
• An inaccurate depiction of computer technology in testing
5
Computerized
• Under Use– Uninformed rejection of computer
technology
– View computers as performing inappropriate functions
– Believe computers lead to dehumanization
6
Computerized
• Misuse
– View computers as effective in performing functions not intended by developer
– Use of computers to compensate fora lack of professional skill
7
Computer-Assisted
• Computer fulfills a support role
• Improves, not replaces the human element
• More likely to avoid under use or misuse
8
Rationales for CAT
1. Focus on assigning humans and computers to tasks best suited to the capabilities of each
2. Develop previously unavailable techniques to improve the quality of tests and assessments
9
Potential Benefits of CAT
• Overview of topics:
– Enhanced validity and reliability
– Enhanced client integration of assessment data in counseling
– Enhanced staff efficiency andcost-effectiveness
10
Enhanced Validity and Reliability
• Overview of topics:
– Test Administration and Scoring
– Administration of Interviews and Checklists
– Computer-Based Test Interpretation
11
Test Administration and Scoring
• Greater degree of standardization
• More accurate results for people with disabilities
• Availability of adaptive testing
• Collection of ancillary data (e.g., response manner and specific response recording)
12
Enhanced Validity and Reliability
• Overview of topics:
– Test Administration and Scoring
– Administration of Interviews and Checklists
– Computer-Based Test Interpretation
13
Administration of Interviews and Checklists
• More comprehensive topic coverage
• More consistent topic coverage
• Less evaluative tension from repeated questioning by the practitioner
• Results less affected by time of day
• More honest client response
14
Enhanced Validity and Reliability
• Overview of topics:
– Test Administration and Scoring
– Administration of Interviews and Checklists
– Computer-Based Test Interpretation
15
Computer-Based Test Interpretation (CBTI)
• Assists in the interpretation of test data
• Provides expanded & consistent knowledge base
• Includes research data andthe clinical experience of other clinicians
• Standardizes computer functioning
• Provides consistency
16
Types of CBTI
• Descriptive interpretations
• Clinician-modeled interpretations (renowned clinician type)
• Clinician-modeled interpretations (statistical model type)
• Clinical actuarial interpretations
17
Descriptive CBTI
• Descriptive sentences related to specific scales
• Scales interpreted independent of each other
• Typically cryptic in nature
18
Clinician-Modeled CBTI (Renowned Clinician Type)
• Replicates the interpretive judgment of a single clinician
• Based on the clinician with the most experience with the test
• Provides consistent results
• Mistakes also consistent
19
Clinician-Modeled CBTI(Statistical Model Type)
• Replicates interpretive judgmentsof a group of practitioners
• Includes a group comprised of experienced clinicians
• Statistically valid model
• Incorporates the best consensus possible among experts
20
Clinical Actuarial CBTI
• Provides narrative descriptions and clinical hypotheses
• Based on clinical research findings for particular score patterns
21
Potential Benefits of CBTI
• More comprehensive and objective
• Not subject to interpreter bias orextraneous circumstances
• Provides rapid data processing
• Ready for use in early counseling sessions
• Helps practitioners organize and systematicallyaccess extensive data bases
• Easily updated when new research is available
22
Potential Benefits of CBTI
• CBTI serves in a consulting role
• Practitioner integrates data from variety of sources to create assessment reports
• Computerized interpretive reports only used as adjuncts, not as the final word
23
Potential Benefits of CAT
• Overview of topics:
– Enhanced validity and reliability
– Enhanced client integration of assessment data in counseling
– Enhanced staff efficiency andcost-effectiveness
24
Enhanced Client Integration of Assessment Data in Counseling
• Computer applications used as one aspect of the therapeutic process
• CBTI reports in clinical settings are typically not given to most clients
• Interpretive reports are typically provided directly to clients in career counseling– provides basic understanding of results – outlines the decision-making process– suggests follow-up resources
25
Enhanced Client Integration of Assessment Data in Counseling
• Generalized test interpretations– video-based technology
26
Potential Benefits of CAT
• Overview of topics:
– Enhanced validity and reliability
– Enhanced client integration of assessment data in counseling
– Enhanced staff efficiency andcost-effectiveness
27
Enhanced Staff Efficiency and Cost-Effectiveness
• Computer-assisted testing is more cost-effective
• Saves testing time
• Improves time utilization of support staff
• Gives practitioners more time to prepare test interpretations, write test reports, and work directly with clients on more complex issues
28
Potential Problems Associated with Computer Applications
• Overview of Topics– Threats to Validity and Reliability
• administration and scoring• test interpretation• ethical issues• ethical standards
– Ineffective Implementation
29
Threats toValidity and Reliability
• Administration and Scoring
– Norms used from paper-and-pencil tests
– Blanket assumptions of equivalency
– Scoring errors
– Perceived inherent credibility - infallibility
– Client misunderstandings of instructions
30
Threats toValidity and Reliability
• Test Interpretation– Different reports from different software
developers
– Differences in CBTI reports and conclusion reached by clinical interviewer
– Computer may be perceived addinga level of scientific precision that does not exist
31
Threats toValidity and Reliability
• Misuse of CBTI
– Qualified practitioners may misuse CBTI as a result of being overworked
– Unqualified practitioners may use to offsettheir lack of training and experience
– Practitioner dependence on CBTI
32
Threats toValidity and Reliability
• Appropriate Use of CBTI– Computer serves as a consulting function– Practitioner has final responsibility for
communicating test results– Qualifications necessary for appropriate use– Prescreening– Not all clients can provide valid item
responses– Follow ethical standards
33
Potential Problems Associated with Computer Applications
• Overview of Topics– Threats to Validity and Reliability
• administration and scoring• test interpretation• ethical issues• ethical standards
– Ineffective Implementation
34
Ineffective Implementation
• Staff Resistance – Lack of staff participation in decision-making – Lack of experience with computers– Poor quality program documentation– time and effort required
• Poor staff support results in reduced acceptance
35
Effective Implementation
• Improves the likelihood of success– Careful planning– Group input in decision making– Reasonable rate of change
• Effective implementation involves interpersonal dynamics more than technology
36
The Future
• Hardware Development– Human voice– Video-based stimuli– Virtual reality
• Possible Problems– High cost– Extraneous auditory and visual data may
distract test taker– Stereotypes
37
The Future
• Software Development
– Expert and decision support systems
– Expert systems will be added to existingCBTI systems
– Expert systems can be used as a powerful training resource
38
Conclusion
• Do computer applications actually improve testing and assessment in counseling?
– Depends more on practitioner attitudes and skills than future advances in technology
– Without effective use, new technology is meaningless