Peer Reviewer Training(f2f) Quality Matters : Inter-Institutional Quality Assurance in Online...
-
Upload
rosalind-stewart -
Category
Documents
-
view
222 -
download
0
Transcript of Peer Reviewer Training(f2f) Quality Matters : Inter-Institutional Quality Assurance in Online...
Peer Reviewer Training(f2f)
Quality MattersQuality Matters: : Inter-Institutional Quality Assurance Inter-Institutional Quality Assurance
in Online Learningin Online Learning
Updated January 2007
Proposed for use by
all QM trainers.
After this Peer Reviewer Training…
You should be able to:
• Describe the critical materials, processes, and administrative elements of the Quality Matters online course quality assurance program.
• Apply the Peer Course Review rubric to review online courses.
• Write useful recommendations for course improvement.
• Explain the QM scoring system.
• Describe the Peer Course Review process and your role in it.
QM Materials Overview
• Today’s Agenda
• Binder Overview
• QM Website
• QM Contact Information
• QM Resources
Introductions
• Pair up
• Share name, institution, job, and best distance learning practice.
• Introduce your partner to the rest of us. Briefly (in one sentence) describe your partner’s best practice.
About About Quality Matters Quality Matters
Peer Course Peer Course ReviewReview
FeedbackFeedback
CourseCourse
Course Meets Course Meets Quality ExpectationsQuality Expectations
Course Course RevisionRevision
Instructional Designers
InstitutionsFaculty Course Developers National Standards &
Research Literature
Rubric
Faculty ReviewersTraining
Quality Matters:Quality Matters: Peer Course Peer Course
Review ProcessReview Process
For Our Purposes, Quality Is…
• More than average; more than “good enough”
• An attempt to capture what’s expected in an effective online course at about an 85% level
• Based on research and widely accepted standards
85 %
What this process is NOT
• Not about an individual instructor
(it’s about the course)
• Not about faculty evaluation
(it’s about course quality)
• Not about “win/lose” or “pass/fail”
(it’s about continuous improvement in a supportive environment)
QM Collegial Review vs. Faculty Evaluation
A QM Review is• Ongoing
• Focus: course design
• Outcome: course improvement
• Non-threatening• Team approach that
includes faculty• Full disclosure to faculty
A Faculty Evaluation is• Single point in time
• Focus: delivery
• Outcome: decision on performance for promotion/tenure
• Win/lose situation• Confidential/secretive
Design vs. Delivery
The faculty member is integral to both design and delivery.
Course Design … is the forethought and planning that a faculty member puts into the course.
Course Delivery …is the actual teaching of the course, the implementation of the design.
QM is about DESIGN - not delivery or faculty performance
Distinguish Between Design vs. Delivery
Example: Discussion Board
Design: A discussion board has been planned into the course; students have been told how they should participate and how they can expect the faculty to participate.
Delivery:How often the faculty member actually participates in the discussion; what the faculty member actually says to students.
The Peer Review Team
• 3 faculty peer reviewers:– must be experienced online instructors– must attend QM training– one MUST be external to the course’s originating
institution– there must be a subject matter expert (SME)
on the team. NOTE: The SME could also be the external reviewer.
AND • Faculty course developer:
– access to rubric prior to review– involved in pre-review discussions– consulted during review
Your Point of View as a QM Peer Course Reviewer
• Take the students’ point of view• Advocate for the student• If you can’t find evidence that the
standard is met, don’t assume it is or isn’t there….. ask the faculty member.
Factors Affecting Course Quality
• Course design QM REVIEWS THIS
• Course delivery (i.e. teaching, faculty performance)
• Course content• Course management system• Institutional infrastructure• Faculty training and readiness• Student engagement and readiness
Underlying Principles of QM• The QM toolset and process are:
– based in national standards of best practice, the research literature and instructional design principles
– designed to promote student learning
– integral to a continuous quality improvement process
– part of a faculty-driven, peer review process
• Course does not have to be “perfect” but better than just “good enough.” (Standards met at about 85% level or better.)
Underlying Principles of QM• Process designed to ensure all reviewed
courses will eventually meet expectations
• Collegial review process, not an evaluation process
• Review team must include an external peer reviewer
• Course faculty or instructor considered part of the review team
What’s In It For Institutions …
• External validation
• Strengthen accreditation package
• Raise QA as a priority activity
• Access to a sustainable, replicable, scalable QA process
• Inform online course training & practices
• Provide professional development activities
QM Process Provides
• Institutional toolset and process to meet quality expectations:– Online course design– Student learning– Improved instruction– Assessment and feedback loops– Professional development
What’s In It For Faculty …• Improve online courses
• External quality assurance
• Expand professional community
• Review other courses and gain new ideas for own course
• Participation useful for professional development plan and portfolio
• Receive $150 for each completed peer course review
QM in Transition
• 2003 – August 2006– QM project funded by FIPSE grant money– materials and some services freely available
• August 2006 and beyond– QM project funded by MarylandOnline– Some limited materials will be freely available– Other materials available to individuals and
institutions at nominal fees– Institutional membership affords full access to
materials and services
The InstructorThe Instructor Worksheet Worksheet
Instructor Worksheet
• Important part of review
• Includes info such as:– Institutionally mandated objectives, materials,
practices, policies– Materials outside course pages– Types of interaction used & instructor’s
statement on the appropriateness of interaction in the course
– Additional items that may require review
Instructor Worksheet
• Read it first
• Refer to it during the review
• Use in team discussions
The Rubric The Rubric
The Rubric
• Eight standards: – Course Overview and Introduction– Learning Objectives – Assessment and Measurement– Resources and Materials– Learner Interaction– Course Technology– Learner Support– Accessibility
Key components must align.
What is Alignment?
Critical course elements
work together to ensure
that students achieve
the desired learning outcomes.
Course Learning Objectives (II)
Resources, Materials (IV) & Technology (VI)
Assessment and Measurement (III)
Learner Interactions & Activities (V)
Key Sections that Must Align
Key Standards that Must Align• Objectives
– Standard II.1: Measurable outcomes– Standard II.2: The module/unit learning objectives describe
outcomes that are measurable and consistent with the course-level objectives.
• Assessment and Measurement– Standard III.1: Measures objectives; consistent with learning
activities
• Learner Interactions and Activities– Standard V.1: Help students achieve the objectives
• Course Materials– Standard IV.1: Deep and comprehensive enough for students to
achieve the objective
• Course Technology– Standard VI.1: Tools and media support the objectives
Rubric ScoringStandards Points Relative Value
14 3 Essential
12 2 Very Important
14 1 Important
TOTALS
40 80
• Team of three (3) reviewers
• One score per standard based on team majority
• Assigned point value; not sliding scale
Assigned Point Values
If the standard
is met …
If the standard
is not met …
Essential
Standards
3 points 0 points
Very Important
Standards
2 points 0 points
Important
Standards
1 point 0 points
Points are NOT assigned on a sliding scale:
Awarding Points
Points are awarded for each standard based on
1. the team majority, and
2. the pre-assigned weighting of each standard
If 2-3 Reviewers believe that a standard is:• met, then the full pre-assigned points are awarded• not met, then zero points are awarded
How to decide….
For EACH standard:1. Read the specific review standard and the
annotation; review the examples, if needed.2. Look for evidence that the standard is met in this
course.3. Ask yourself: Does this course meet the standard
at an 85% or better level?4. Decide Yes or No and enter your answer in the
web-based rubric form.5. Include comments/suggestions as documentation.
To Meet Expectations…
A course must achieve:
• “Yes” on all 14 of the 3-point “essential” standards.
• A minimum of 68 out of 80 points
68/80 = 85%
Online & Hybrid Courses
• Rubric designed for application to fully online and hybrid/blended courses
• Same set of standards apply to both • How we achieve the standards may differ• For hybrids, focus on pedagogical integration
of online and F2F components• Refer to Course Format Definitions document
RecommendationsRecommendations
Writing a Useful Recommendation
Constructive Try to offer solutions, not just identify problems.
Specific Include a specific example of what is being recommended.
Measurable How will the ID/instructor know when the recommendation has been implemented?
Sensitive Avoid negative language. Keep recommendations and comments on a positive note.
Balanced Point out strengths as well as weaknesses.
When MUST you write a recommendation?
You MUST write a recommendation if you decide that the course does not meet the specific standard.
Improve these Recommendations• Assignment instructions weren’t clear.
– I wasn’t sure if assignment #1 required a written paper.
• You didn't tell the students how to find the additional resources.– I had trouble locating the resources; you may
want to put a link on the home page for easy access.
• The text on the page was too hard to read. – The green text on a blue background was
difficult to read. Suggest black on white.
Improve these Recommendations
• The learning objectives aren’t measurable.– Rephrase the learning objectives to include an
active verb (explain, distinguish, compare, etc.)
• The assessments are weak.– The assessments are unrelated to the learning
objectives. Review your learning objectives and make sure that they are reflected in your exams.
Hands-OnHands-OnPracticePractice
Hands-On Practice• Work in Pairs• Review the Anthropology Course
– Go to http://pgcconline.blackboard.com– Username is fipse– Password is fipse
• Focus on 14 essential (3 point) standards• Write 1 recommendation per standard
Steps1. Get your training pair assignment2. Find your computer in lab (2 per pair)3. One person logs into the course 4. Other person logs into QM rubric5. Read Instructor Worksheet for course
Then… follow facilitator’s instructions for first standard
General Standard I:Course Overview and Introduction
1.1: Navigational instructions make the
organization of the course easy to
understand.
1.2: A statement introduces the student to the
course and to the structure of the student
learning, and, in the case of a hybrid
course, clarifies the relationship between
the face-to-face and online components.
General Standard II:Learning Objectives (Competencies)
II.I: The course learning objectives
describe outcomes that are
measurable.
II.2: The module/unit learning objectives
describe outcomes that are measurable and consistent with the
course-level objectives.
General Standard III: Assessment and Measurement
III.1: The types of assessments selected
measure the stated learning objectives
and are consistent with course activities
and resources.
III.2: The course grading policy is stated clearly.
III.3: Specific and descriptive criteria are
provided for the evaluation of students’
work and participation.
General Standard IV: Resources and Materials
IV.I: The instructional materials support
the stated learning objectives.
IV.2: The instructional materials have
sufficient breadth, depth, and
currency for the student to learn the
subject.
General Standard V:Learner Interaction
V.1: The learning activities promote theachievement of stated learning objectives.
V.2: Learning activities foster instructor-student, content-student, and if
appropriate to this course, student-studentinteraction.
V.3: Clear standards are set for instructorresponse and availability (turn-aroundtime for email, grade posting, etc.)
General Standard VI:Course Technology
VI.I: The tools and media support the
learning objectives and are
appropriately chosen to deliver the
content of the course.
General Standard VII: Learner Support
• No 3-point elements
This standard has no “essential” 3-point elements because it’s primarily concerned with academic support services, student support services and technical support services….usually thought to be the primary responsibility of the institution and not the individual instructor.
General Standard VIII:Accessibility
VIII.1: The course acknowledges the importance ofADA requirements.
To meet this standard, the course must include BOTHof these elements:
• The course must be offered using software that is accepted as “ADA compliant.”.
AND• The course should include a brief statement that
clearly tells students how to access ADA services at the institution.
Showcase Your CoursesShowcase Your Courses
Course Showcase
• Think about your own course:– What do you do that meets QM
expectations?– What improvements might you make?
• Share your thoughts with your partner
• Summarize for the group
ScenariosScenarios
Evaluating Scenarios
• Divide into groups.• Discuss the scenarios that focus on
the 14 essential standards. • Take the “quiz” with your group.
Use the material in your books and each other as references.
• Jot down your reasoning and be prepared to discuss your decision.
Peer Course Review Peer Course Review ProcessProcess
About the Course
• QM is designed to review “mature” courses (taught at least two semesters)
• QM logo indicates year course met expectations
• Triggers for subsequent reviews:• Faculty request• More than 3 years since original review• New textbook or instructor• Professional or accreditation review pending
About the Review
• On average, a course review takes 7-10 hours
• Factors affecting review time include• Reviewer familiarity with the discipline• Reviewer familiarity with the CMS• Reviewer familiarity with the QM review process• Organization of the course
• Suggested review methodology:• Read Instructor worksheet• Familiarize yourself with CMS and course • Pre-review team discussion• Proceed through standards, but save Standard III
(Assessment and Measurement) for last• Post-review team discussion• Submit your review
Your Point of View as a QM Peer Course Reviewer
• Take the students’ point of view• Advocate for the student• If you can’t find evidence that the
standard is met, don’t assume it is or isn’t there….. ask the faculty member.
Timeline
• Active review period approximately 3 weeks
• Compiled reports due in approximately 6 weeks
• Teams:– Have pre-review discussion.– Set Team calendar.– Commit to 2-3 week review period.– Have post-review discussion.
Post-Review
• Reviewers – complete exit interview– receive stipends
• Faculty – receives Final Review Report– completes Faculty Response Form
Review Outcome• If meets expectations:
– Recognized by Quality Matters– Notifications distributed– ID support provided if requested
• If does not yet meet expectations:– ID support provided if requested– Team Chair and ID approve revisions– Course meets expectations
Roles and Responsibilities
Faculty Developer
1. Part of the review team.
2. Provides access to the course.
3. Completes Instructor Worksheet
4. Part of the initial team discussion
5. Receives compiled report
6. Returns Faculty Response Form
Peer Reviewers
1. Establish Team Calendar
2. Review the course individually
3. Complete the online web review form
4. Discuss review with Team as needed
5. Complete an Exit Interview
Receive… Recognition as Certified Peer Reviewer. Compensation
Team ChairReviewer Roles & Responsibilities plus….
1. Organizes Team calendar
2. Confirm Instructor Worksheet is used
3. Creates draft report from compiled reviews
4. Convenes Team discussions
5. Reviews, edits and submits Team Report
Receives… Recognition as Certified Peer Reviewer and Chair. Compensation
QM To DateQM To Date
QM to Date
• Overall Participation:– Individuals & programs from 130
institutions across 28 states
• Course Reviews:– 103+ courses reviewed– 18 MD schools; 10 non-MD schools
• Peer Reviewer Rubric Training:– 600+ trained
Multiple Uses of QM
Reported Uses of QM System:• Guidelines for initial online course development
• Quality assurance of existing courses
• Ongoing faculty professional development
• Institutional reaccredidation packages
• Formation of distance learning policies & steering committees
Awards - 2005
• WCET Outstanding Work (WOW) Award, November 2005.
• USDLA 21st Century Best Practice Award,October 2005.
• Maryland Distance Learning Association (MDLA) Best Program Award,March 2005.
QM in Transition
• 2003 – August 2006– QM project funded by FIPSE grant money– materials and some services freely available
• August 2006 and beyond– QM project funded by MarylandOnline– Some limited materials will be freely available– Other materials available to individuals and
institutions at nominal fees– Institutional membership affords full access to
materials and services
Research FindingsResearch Findings
Overall Course Review Results
Upon initial review:
• 53% meet expectations
• 22% do not meet expectations - missing at least one essential 3-point element(s)
• 25% do not meet expectations - missing at least one essential 3 point element(s) and a minimum of 68 points
Summary of Course Reviewsvariety of course management systems
– 50% Bb, 35% WebCT, 16% WebTycho, 2% other
– 65% of respondent Reviewers felt unfamiliarity with CMS did not impede review; 34% somewhat
• variety of disciplines– general studies, information technology,
teacher education, engineering, allied health– 75% of respondent Reviewers felt unfamiliarity
with discipline did not impede review; 25% somewhat
Analysis of Infrastructure
Whether a course meets/does not meet QM
expectations is NOT correlated with:
• Course’s institution of origin
• Course management system– 53% of Bb courses– 54% of WebCT courses– 50% of WebTycho courses
Analysis of Academic Area
Met Expectations Academic Area # Courses
78% information technology 9
60% math & statistics 10
55% allied health, health, nutrition, nursing 11
50% natural sciences 11
50% education 10
50% history, government, politics 6
50% behavioral & social sciences 12
50% english, writing, communications 10
33% arts & humanities 9
Post-Course Review
• Met expectations– Most faculty made suggested
improvements even though their course met expectations!
• Did not meet expectations– Improvements made or in progress – Most improvements made by faculty– Some ask for ID support
Common Themes
• Course reviews revealed 11 common areas for course improvement
• Elements that are missing in 20% or more of the courses reviewed
• These are potential targets for – faculty training– special attention in the initial course
development phase:
Common Areas for Improvement• Instructor self-introduction (I.4) 22%
• Activities that foster interaction (V.2) 22%
• Technology/skills/pre-req knowledge stated (I.6) 24%
• Links to academic support, student services, tutorials/resources (VII.2-VII.4) 24-27%
• Learning objectives at module/unit level (II.5) 27%
• Netiquette expectations (I.3) 32%
• Self-check/practice with quick feedback (III.5) 38%
• B/W alternatives to color content (VIII.4) 54%
• Alternatives to auditory/visual content (VIII.2) 59%
Serving as a Serving as a
QM Peer ReviewerQM Peer Reviewer
Quality Matters Peer Course ReviewerCertification Process
Training Course Review Experience
Quality Matters Certification
Attend QM Training
Trained
Assigned to Peer Review
Team
Submit Report
Certified
+ =
• Name on QM website
• Use of QM Logo
• Eligible for Peer Reviewer PoolDemonstrate
Competencies
Kane 1/15/05
Quality MattersPeer Course Review Process
1. Course Selected
2. Trained Review Team Assigned
3. MOUs and Instructor Worksheet
4. Pre-Review Discussion
5. Individuals complete reviews
7. Final Review Report
6. Team Discussion(s)
Next StepsNext Steps
Next Steps
• Within the next week– Return to “Foundations” Blackboard site– Complete Scenarios Quiz– Grade of 85% or greater required to
pass training and be a Peer Reviewer
Thanks to YOU…Thanks to YOU…Quality Matters! Quality Matters!