Key Decision-Making in Assessment Design: Expert Lenses
-
Upload
touchstone-institute -
Category
Health & Medicine
-
view
514 -
download
0
Transcript of Key Decision-Making in Assessment Design: Expert Lenses
![Page 1: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/1.jpg)
PERSPECTIVES IN COMPETENCY ASSESSMENTDebra Sibbald, Executive Director, Assessment ProgramsBScPhm RPh ACPR MA (Adult Education) PhD (Curriculum Teaching & Learning)
February 2015
![Page 2: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/2.jpg)
KEY DECISION-MAKING IN ASSESSMENT DESIGN:
EXPERT LENSES
![Page 3: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/3.jpg)
Thus far you have heard opinions fromexpert authoritative sources on competency assessments:
how best practices should inform key decision making...
![Page 4: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/4.jpg)
Best Practices: Identify Important Design Principles
• Candidates• Outcomes• Competencies• Levels
• ‘Fit for purpose’ aligned blueprint• Methods• formats• Scales• Subject matter experts
• Writing• Validating• Standard setting• Iterative reflection and refinements
![Page 5: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/5.jpg)
HARMONIZATION OF COMPETENCY ASSESSMENTS:
![Page 6: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/6.jpg)
Collective CommunityIndividual
Agents
EXPERTS
![Page 7: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/7.jpg)
INDIVIDUAL AGENTSlikely perceive the problem relative to their context
![Page 8: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/8.jpg)
COLLECTIVE COMMUNITYA Collaboration
![Page 9: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/9.jpg)
![Page 10: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/10.jpg)
Individual Agents:
• Relevant decision-makers /organizations• Major Stakeholders (directly affected and held accountable)• Experts in their contexts • Others (groups/individuals who provide commentaries)
• HARMONIZATION PROCESS: • EXTERNAL EXPERTS (TOUCHSTONE)• Consider each agent’s vested interest and what that group wants• ‘Fit-for-purpose’ alignment • Acceptable to all• Understood and acted upon by all
![Page 11: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/11.jpg)
EXAMPLE:OPTOMETRY EVALUATING EXAM FOR INTERNATIONAL OPTOMETRY GRADUATES
![Page 12: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/12.jpg)
Major Stakeholders
1. Candidates – International Optometry Graduates
2. College of Optometrists of Ontario (Registration Committee)
3. International Optometric Bridging Program (IOBP)
4. CEO / ECO (Canadian Examiners Optometry) / (Examinateurs Canadiens en Optométrie)
5. Practicing Optometrists (from Canadian schools or from schools in North America accredited by the Accreditation Council on Optometric Education (ACOE)
6. Practicing Optometrists who are graduates of IOBP
7. University of Waterloo
![Page 13: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/13.jpg)
Major Stakeholders8. Pan Canadian Provincial Jurisdictions (FORAC – CORA)
9. Office of the Fairness Commissioner
10. TOUCHSTONE• Executive Director of Assessment• Director Health Professional Affairs• Psychometricians
11. TOUCHSTONE: Board of Directors• Chief Executive Officer• Directors Finance/Resources• Managers – Operations and Standardized Patients
12. Blueprint Subject Matter Experts:• Participants for EE from Optometric Practice
![Page 14: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/14.jpg)
Other Stakeholders
13. Evaluating Exam Chief Examiners
14. Subject Matter Experts Writers
15. Evaluating Exam Examiners
![Page 15: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/15.jpg)
Major Stakeholder Major Stakeholder Minor Stakeholder
Candidates from IEO Pan Canadian Provincial Jurisdictions (FORAC – CORA)
Evaluating Exam Chief Examiners
College of Optometrists of Ontario(Registration Committee)
Office of the Fairness Commissioner
Subject Matter Experts Writers
International OptometricBridging Program (IOBP)
TOUCHSTONE-Director of Assessment-Director Health Professional Affairs-Psychometricians
Evaluating Exam Examiners
CEO / ECO(Canadian Examiners Optometry) / (Examinateurs Canadiens en Optométrie)
TOUCHSTONE Board of Directors- Chief Executive Officer- Directors - Finance/Resources- Managers – Operations/SPs
Practicing Optometrists from ACOC schools
Blueprint Subject Matter Experts:Participants for EE from Optometric Practice
Practicing Optometrists who are graduates of IOBP
![Page 16: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/16.jpg)
![Page 17: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/17.jpg)
Harmony, balance, sustainability
![Page 18: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/18.jpg)
WHO ARE THE AGENTS IN OUR AUDIENCE?Turn on your clickers!
![Page 19: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/19.jpg)
![Page 20: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/20.jpg)
What is your main focus of interest?A. RegulatoryB. EducationC. ClinicalD. AssessmentE. Administrative
![Page 21: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/21.jpg)
From what viewpoint is the domain of your concern?
A. Economic forcesB. Social/sociological structures and practicesC. Political structures and policiesD. Educational affordances and constraintsE. Cultural beliefs or supports
![Page 22: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/22.jpg)
TODAY YOU HAVE BEEN PRESENTED WITH A GREAT DEAL OF INFORMATION FROM VARIOUS DIMENSIONS.
![Page 23: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/23.jpg)
i-Clicker Polling - CommunityMicrophone -
Agents
It’s time to pause and engage you in a guided reflection
![Page 24: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/24.jpg)
Recall the issues and perspectives presented(s)
How did you react during the presentation?
Why did you feel this way?
Future changes based on these insights?
Reflection & Feedback on This Morning
![Page 25: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/25.jpg)
Identify
Change
Analyze
Reflect
![Page 26: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/26.jpg)
Identify:
The most surprising piece of information that you learned today
Or
The perspective or view which is not aligned with your current practices or beliefs
![Page 27: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/27.jpg)
What was your primary Reaction?
![Page 28: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/28.jpg)
Why do you have this reaction or concern? The domain your view arises from is :A. Economic
forcesB. Social/sociological
structures and practicesC. Political
structures and policiesD. Educational
affordances and constraintsE. Cultural
beliefs or supports
![Page 29: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/29.jpg)
What are your main issues?
![Page 30: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/30.jpg)
Considering the reasons for these surprises or differences, will you change your beliefs or practices? A. YES
B. No
![Page 31: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/31.jpg)
If yes, what will be your main goal?
![Page 32: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/32.jpg)
What will be likely implications of this decision ?
![Page 33: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/33.jpg)
You identify your primary challenge as
A. Problems created by your own decisions and behaviours
B. Problems created by external forces
![Page 34: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/34.jpg)
What is the primary characteristic of your challenges?
A. Problems can be solved in whole or in part
B. Problems beyond our controls
![Page 35: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/35.jpg)
On what step towards change has today’s discussion had impact, if any?
A. You recognize you now face an important decision
B. You now more clearly or accurately can identify alternatives
C. You now are better equipped to evaluate alternatives
D. You are now ready to act on the best alternative
E. No impact on steps toward change
![Page 36: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/36.jpg)
![Page 37: Key Decision-Making in Assessment Design: Expert Lenses](https://reader030.fdocuments.net/reader030/viewer/2022032619/55c3a441bb61eb37338b4730/html5/thumbnails/37.jpg)
Audience has the floor!
You Say….. I Say….