Shelley A. Chapman, PhD Texas A & M University February 2013.
-
Upload
tyler-reynard-warren -
Category
Documents
-
view
216 -
download
2
Transcript of Shelley A. Chapman, PhD Texas A & M University February 2013.
![Page 1: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/1.jpg)
Using IDEA for Faculty Evaluation
Shelley A. Chapman, PhD
Texas A & M University February 2013
![Page 2: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/2.jpg)
Plan for this Session
“Teaching Effectiveness” What it is Uniqueness of IDEA Conditions for the Good Use of IDEA 3-Phase Process for Faculty
Evaluation Using Reports to Improve Teaching
![Page 3: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/3.jpg)
Most Surveys
How well do the instructor’s methods resemble those of a “model” teacher?
How well do students rate their progress on the types of learning the instructor targeted?
Teaching Effectiveness
![Page 4: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/4.jpg)
What makes IDEA unique?
1. Focus on Student Learning
2. Focus on Instructor’s Purpose
3. Adjustments for Extraneous Influences
4. Validity and Reliability
5. Comparison Data6. Flexibility
![Page 5: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/5.jpg)
Conditions for Good Use
The instrument o Focuses on learning o Provides suggested
action steps
![Page 6: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/6.jpg)
Conditions for Good Use
The Facultyo Trust the processo Value student
feedbacko Are motivated to
make improvements
![Page 7: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/7.jpg)
Conditions for Good Use
Campus Cultureo Teaching excellence - high
priorityo Resources to improve -
providedo Student ratings -
appropriate weight
![Page 8: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/8.jpg)
Conditions for Good Use
The Evaluation Processo 30-50% of evaluation of teachingo 6-8 classes, more if small (<10)oNot over-interpreted
(3-5 performance categories)
![Page 9: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/9.jpg)
Underlying Philosophy of IDEA
Teaching effectiveness is determined primarily by students’ progress on the types of learning the instructor targets.
![Page 10: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/10.jpg)
Faculty Information Form
![Page 11: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/11.jpg)
Diagnostic Report Overview
1. How did students rate their learning experience?
2. What contextual factors impacted those ratings?
3. How do my scores compare to: IDEA, discipline, and institution?
4. What might I do to facilitate better learning for my students next time?
![Page 12: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/12.jpg)
Wisdom
Knowledge
Information
Data
What the Report Can Provide
Calculation of Scores
Context: Variables and Comparisons
Suggested Action Steps
![Page 13: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/13.jpg)
Using IDEA
As Part of a Faculty Evaluation Process
![Page 14: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/14.jpg)
Multiple forms of Assessment
Student Ratings
External Perspectiv
eArtifacts
BalancedPlan for SummativeEvaluation
![Page 15: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/15.jpg)
Evidence of Good Teaching
Artifacts
• Syllabi• Graphic Organizers• Assignments and project
descriptions• Rubrics• Written Teaching
Philosophy/Reflections• Samples of Student Work• CATs and results
![Page 16: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/16.jpg)
Evidence of Good Teaching
• Classroom Observation• Classroom Visitation• Invited Presentations• Alumni Surveys• Focus Groups of
Graduating Students
External Perspectiv
e
![Page 17: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/17.jpg)
Classroom ObservationsTime What Happened What Was Said
![Page 18: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/18.jpg)
Classroom ObservationsTime What Happened What Was Said
8:05
8:10
8:15
Instructor shut door
Students are shuffling papers, opening books.
Student comes in late
Several students raise hands
Female in first row is called on
Instructor (I): OK, Class. Let’s begin. Make sure you turned in your homework as you came in.
Today we will begin our discussion on the brain. Turn in your textbooks to chapter 5.
Is your brain more like a computer or a jungle? Who would like to respond first?
Student (S) My brain is a jungle! I am so unorganized! (class laughs)…
![Page 19: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/19.jpg)
Flow of Communication Map
Instructor
M
F
F
F
M M M M M
M M
M
F F F F
F
F
F
F F
![Page 20: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/20.jpg)
Evidence of Good Teaching
• Administer Appropriately• Collect 6-8 Reports (more if class size is <10)• 30-50% of Overall Evaluation
Student
Ratings
![Page 21: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/21.jpg)
Evidence of Good Teaching
• Student Comments-formative • Be mindful of standard error of
measurement (±.3)• Use 3-5 Performance Categories
Student
Ratings
![Page 22: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/22.jpg)
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
![Page 23: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/23.jpg)
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
I. Set ExpectationsWhat does this entail regarding IDEA?
![Page 24: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/24.jpg)
Criterion-Referenced Standards
Criteriono Use averages on 5-
point scaleo Recognize that
some objectives are more difficult to achieve
o “Authenticate” objectives
Page 1
![Page 25: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/25.jpg)
Norm-Referenced Standards
Use Converted Averages
o IDEAo Disciplineo Institutio
n
![Page 26: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/26.jpg)
Comparison Information: Converted Averages
![Page 27: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/27.jpg)
T-Score Distribution
40%
Similar
Lower20%10%
Much Lower
10%Much Higher
20%Higher
Gray Band
![Page 28: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/28.jpg)
Comparison Scores
![Page 29: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/29.jpg)
Create Categories of Performance
Below Acceptable Standards
Marginal, Needs Improvement
MeetsExpectations
ExceedsExpectations
Outstanding
Does Not Meet Expectations
Meets Expectations Exceeds Expectations
![Page 30: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/30.jpg)
Performance Categories: EXAMPLE
CriterionAverage Rating
Effectiveness Category
NormativeT-Score
Below 3.0 Below acceptable standards Below 38
3.0-3.4 Marginal, improvement needed
38-44
3.5-3.9 Meets expectations 45-54
4.0-4.4 Exceeds expectations 55-62
4.5 or higher Outstanding 63 or higher
![Page 31: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/31.jpg)
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
II. Collect DataWhat do you look for regarding IDEA?
![Page 32: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/32.jpg)
Improving Online Response Rates
• Create value for student feedback
• Monitor and Communicate through multiple modalities:• Twitter• Facebook• Other
• Prepare Students• Talk about it• Syllabus
![Page 33: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/33.jpg)
Example: Course Syllabus
Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions)
Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomenaStudents will think more critically about the earth and environment
Objective 8: Developing skill in expressing myself orally or in writing
Students will be able to present scientific results in written and oral forms
IDEA Center Learning Objective
Course Learning Outcomes
![Page 34: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/34.jpg)
For Personnel Decisions
Pages 1 and 2 What were students’ perceptions of the
course and their learning?
![Page 35: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/35.jpg)
Things toConsider…
Were the appropriate objectives selected?
• How many?• Do they match the course?
• How might you “authenticate” the objectives selected?
![Page 36: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/36.jpg)
FIF: Selecting Objectives
• 3-5 as “Essential” or “Important”
o Is it a significant part of the course?
o Do you do something specific to help students accomplish the objective?
o Does the student’s progress on the objective influence his or her grade?
Be true to your course.
![Page 37: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/37.jpg)
Things to Consider…
What were the students’ perceptions of their course and their learning?
![Page 38: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/38.jpg)
How Did Students Rate Their Learning?
![Page 39: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/39.jpg)
Your Average (5-point Scale)
Raw Adj.
A. Progress on Relevant Objectives1
Four objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
1If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.
How Did Students Rate Their Learning?
![Page 40: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/40.jpg)
ProgressOnRelevantObjectives
4
4.3 + 4.34.14.23.6 5
![Page 41: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/41.jpg)
Summary Evaluation: Five-Point Scale
Report Page 1
Your Average Score
(5-point scale)
Raw Adj.
A. Progress on Relevant ObjectivesFour objectives were selected as relevant (Important or Essential—see page 2)
4.1 4.3
Overall Ratings B. Excellent Teacher 4.7 4.9
C. Excellent Course 4.1 4.4
D. Average of B & C 4.4 4.7
Summary Evaluation(Average of A & D) 4.3 4.5
50%
25%
25%
![Page 42: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/42.jpg)
Understanding Adjusted Scores
![Page 43: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/43.jpg)
Work Habits
(Item 43)
Student Motivation (Item 39)
HighHigh Avg.
Avg.LowAvg.
Low
High 4.48 4.38 4.28 4.13 4.04
High Avg. 4.38 4.29 4.14 3.96 3.76
Average 4.28 4.14 4.01 3.83 3.64
Low Avg. 4.15 4.05 3.88 3.70 3.51
Low 4.11 3.96 3.78 3.58 3.38
Impact of Extraneous Factors
Gaining Factual Knowledge – Average Progress Ratings
Technical Report 12, page 40
![Page 44: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/44.jpg)
Work Habits
(Item 43)
Student Motivation (Item 39)
HighHigh Avg.
Avg.LowAvg.
Low
High 4.48 4.38
High Avg. 4.38 4.29
Average 4.01
Low Avg. 3.70 3.51
Low 3.58 3.38
Impact of Extraneous Factors
Gaining Factual Knowledge – Average Progress Ratings
Technical Report 12, page 40
![Page 45: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/45.jpg)
Raw or Adjusted Scores
Purpose Raw or Adjusted?
How much did students learn? Raw
What were the instructor’s contributions to learning?
Adjusted
How do faculty compare?
Adjusted
![Page 46: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/46.jpg)
Doraw
scores meet or exceed
Expectations?*
Are adjusted
scores lower or higher than
rawscores?
Use adjusted
scores
Useraw scores
Lower Yes
Higher
When to Use Adjusted Scores
*Expectations defined by your unit.
No
![Page 47: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/47.jpg)
Three-Phase Process for Faculty Evaluation
Set Expectations
Collect Data
Use Data
III. Use DataWhich data will you use and how?
![Page 48: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/48.jpg)
.
IDEA Faculty Worksheet
Keep track of reports Look for longitudinal trends Use for promotion and tenure
Created by Pam Milloy, Grand View University
Available from The IDEA Center Website
![Page 49: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/49.jpg)
Using the DataSummative (pp.1-
2) Criterion or Norm-
referenced Adjusted or raw Categories of
Performance 30-50% of Teaching
Evaluation 6-8 Classes (more if
small)
Formative (p.3) Identify areas to
improve Access applicable
resources from IDEA website
Read and have conversations
Implement new ideas
![Page 50: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/50.jpg)
Reflective Practice
Collect Feedback
Interpret Results
Read & Learn
Reflect & Discuss
Improve
IDEA resources that are keyed to reports
Talk with colleagues
Try new ideasOnline, Paper
What the reports sayand what they mean
![Page 51: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/51.jpg)
Research
Relationship of Learning Objectives to Teaching
Methods
![Page 52: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/52.jpg)
Information Improvement
![Page 53: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/53.jpg)
Information Improvement
![Page 54: Shelley A. Chapman, PhD Texas A & M University February 2013.](https://reader035.fdocuments.net/reader035/viewer/2022070401/56649f1e5503460f94c3554d/html5/thumbnails/54.jpg)
Questions?