Program Level Assessment
description
Transcript of Program Level Assessment
![Page 1: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/1.jpg)
Program Level Assessment
Workshop forCollege of the Redwoods
Fred TrappAugust 18, 2008
![Page 2: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/2.jpg)
Assessment of Program Effectiveness• What the program will do or achieve
• Curriculum• Retention• Graduation• Placement • Satisfaction (graduate and employer)
![Page 3: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/3.jpg)
Assessment of Student Learning Outcomes• What students will do or achieve• Direct methods of assessment
![Page 4: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/4.jpg)
Direct Measures of Learning• Capstone experience• Standardized tests• Performance on national licensure certification or
professional exams • Locally developed tests • Essay questions blind scored by faculty• Juried review of projects• Externally reviewed exhibitions performances• Evaluation of internships based upon program
learning outcomes
![Page 5: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/5.jpg)
Indirect Measures of Learning
• Alumni, employer, and student surveys (including satisfaction surveys)
• Exit interviews of graduates and focus groups graduate follow up studies
• Retention and transfer studies• Length of time to degree• Graduation and transfer rates• Job placement rates
![Page 6: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/6.jpg)
Steps of Implementation• Step One: Developing Assessment
Plans• Step Two: Developing Implementation
Strategies• Step Three: Collecting Data, Analyzing
& Reflecting on its meaning• Step Four: Implementing Change as
the result of assessment
![Page 7: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/7.jpg)
Assessment Language• Program Outcomes/ Topic Areas• Student Learning Outcomes• Means of Assessment/ Evidence• Criteria for Assessment• Standards of Success/ Quality of
Performance
![Page 8: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/8.jpg)
Communication (program outcome topic)
SpeakRelate Listen ParticipateWrite
Program Outcome Topic & Student Learning Outcomes
Student learning outcomes
![Page 9: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/9.jpg)
Means of Assessment/Evidence- Circumstances (learning events)
• Assignments (in class and out of class)• Feedback on practice• Self evaluation• Peer evaluation• Role Play• Pre Tests• Simulation
![Page 10: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/10.jpg)
Goal
OutcomeOutcome OutcomeOutcomeOutcome
Object (deliverable or performance)
Learning events
Means of Assessment/Evidence- Deliverable or Performance
![Page 11: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/11.jpg)
Communication
SpeakRelate Listen ParticipateWrite
Speech
Means of Assessment/Evidence- Deliverable or Performance
InterviewEssay
![Page 12: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/12.jpg)
Communication
SpeakRelate Listen ParticipateWrite
Speech
Criteria for Assessment
Verbal deliveryNonverbal delivery
OrganizationContent
criteria
![Page 13: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/13.jpg)
Communication
SpeakRelate Listen ParticipateWrite
Speech
Standards of Success/ Quality of Performance
Verbal deliveryNonverbal delivery
organizationcontent
Quality of Performance
![Page 14: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/14.jpg)
Student Learning Outcomes
• Students should be able to<<action verb>> <<something>>
• Learner Centered• Specific• Action oriented• Cognitively appropriate for the
program level
![Page 15: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/15.jpg)
KNOWLEDGECOMPREHENSION
APPLICATIONANALYSISSYNTHESISEVALUATION
CiteCountDefineDraw
IdentifyList
NamePointQuoteRead
ReciteRecordRepeatSelectState
TabulateTell
TraceUnderline
AssociateClassifyCompareComputeContrast
DifferentiateDiscuss
DistinguishEstimateExplainExpress
ExtrapolateInterpolate
LocatePredictReportRestateReview
TellTranslate
ApplyCalculateClassify
DemonstrateDetermineDramatize
EmployExamineIllustrateInterpretLocate
OperateOrder
PracticeReport
RestructureScheduleSketchSolve
TranslateUse
Write
AnalyzeAppraiseCalculate
CategorizeClassifyCompareDebate
DiagramDifferentiateDistinguish
ExamineExperiment
InspectInventoryQuestionSeparate
SummarizeTest
ArrangeAssemble
CollectComposeConstruct
CreateDesign
FormulateIntegrateManageOrganize
PlanPrepare
PrescribeProduceProposeSpecify
SynthesizeWrite
AppraiseAssessChoose
CompareCriticize
DetermineEstimateEvaluate
GradeJudge
MeasureRankRate
RecommendReviseScoreSelect
StandardizeTest
Validate
![Page 16: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/16.jpg)
KNOWLEDGECOMPREHENSION
APPLICATIONANALYSISSYNTHESISEVALUATION
CiteCountDefineDraw
IdentifyList
NamePointQuoteRead
ReciteRecordRepeatSelectState
TabulateTell
TraceUnderline
AssociateClassifyCompareComputeContrast
DifferentiateDiscuss
DistinguishEstimateExplainExpress
ExtrapolateInterpolate
LocatePredictReportRestateReview
TellTranslate
ApplyCalculateClassify
DemonstrateDetermineDramatize
EmployExamineIllustrateInterpretLocate
OperateOrder
PracticeReport
RestructureScheduleSketchSolve
TranslateUse
Write
AnalyzeAppraiseCalculate
CategorizeClassifyCompareDebate
DiagramDifferentiateDistinguish
ExamineExperiment
InspectInventoryQuestionSeparate
SummarizeTest
ArrangeAssemble
CollectComposeConstruct
CreateDesign
FormulateIntegrateManageOrganize
PlanPrepare
PrescribeProduceProposeSpecify
SynthesizeWrite
AppraiseAssessChoose
CompareCriticize
DetermineEstimateEvaluate
GradeJudge
MeasureRankRate
RecommendReviseScoreSelect
StandardizeTest
Validate
Course outcomes
![Page 17: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/17.jpg)
KNOWLEDGECOMPREHENSION
APPLICATIONANALYSISSYNTHESISEVALUATION
CiteCountDefineDraw
IdentifyList
NamePointQuoteRead
ReciteRecordRepeatSelectState
TabulateTell
TraceUnderline
AssociateClassifyCompareComputeContrast
DifferentiateDiscuss
DistinguishEstimateExplainExpress
ExtrapolateInterpolate
LocatePredictReportRestateReview
TellTranslate
ApplyCalculateClassify
DemonstrateDetermineDramatize
EmployExamineIllustrateInterpretLocate
OperateOrder
PracticeReport
RestructureScheduleSketchSolve
TranslateUse
Write
AnalyzeAppraiseCalculate
CategorizeClassifyCompareDebate
DiagramDifferentiateDistinguish
ExamineExperiment
InspectInventoryQuestionSeparate
SummarizeTest
ArrangeAssemble
CollectComposeConstruct
CreateDesign
FormulateIntegrateManageOrganize
PlanPrepare
PrescribeProduceProposeSpecify
SynthesizeWrite
AppraiseAssessChoose
CompareCriticize
DetermineEstimateEvaluate
GradeJudge
MeasureRankRate
RecommendReviseScoreSelect
StandardizeTest
Validate
Program outcomes
![Page 18: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/18.jpg)
Possible Outcomes?
• Students will appreciate….• Students will be exposed to….• Students will demonstrate knowledge of
the history, theories and applications of…
• Students will attend….• Faculty will provide students with
opportunities to….
![Page 19: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/19.jpg)
Possible Outcomes
• Students will appreciate…• Students will be exposed to….• Students will demonstrate knowledge of
the history, theories and applications of…• Students will attend….• Faculty will provide students with
opportunities to….
![Page 20: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/20.jpg)
Writing Program Level Student Learning Outcomes• Student Learning Outcomes
• Students will recognize, analyze, and interpret human experience in terms of personal, intellectual, and social contexts.
• Students should be able to critically comprehend, interpret, and evaluate written, visual, and aural material.
Compound outcome statements will be problematic to assess.
![Page 21: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/21.jpg)
Program Level Student Learning Outcomes• Students will be able to:
• Design a solution to a technical problem• Interpret performance data• Demonstrate the functions of key operating
systems• Articulate the ethical standards of the
profession• Produce diagnostic quality radiographic images• Evaluate the advantages of alterative solutions
![Page 22: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/22.jpg)
Course LevelStudent Learning Outcomes• Students will be able to:
• Students will have a basic knowledge of human anatomy
• Students will be able to define 200 medical terms
• Students should know the history of the United States
• Students will be able to translate dollars into pounds and Euros
![Page 23: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/23.jpg)
Course vs. Program Assessment• Single course• Course concepts
• Fewer confounds from outside the course detractors
• Faculty (indiv. & small groups)
• “Micrometer” precision• Less fear generated in faculty• Faculty professional
obligation, less overhead cost• No external certifications
• Multiple courses• Program performance
competencies (post grad)• More confounds from outside
the course detractors• Faculty (small & large groups)
• “Two sticks & a chain” precision
• Potential fear generated in faculty
• Faculty professional obligation, but some overhead cost
• External certifications possible
![Page 24: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/24.jpg)
Identify Criteria for Assessing Program Level Outcomes• Must be shared across faculty• A way to further define the student
learning outcome• Provides a common language for
describing student learning• Must be outcome specific• Number of criteria will vary by outcome
![Page 25: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/25.jpg)
Means of Assessment/Evidence- Deliverable or Performance + Criteria
Communication
SpeakRelate Listen ParticipateWrite
Salespresentation
deliverycontent
organizationOral
Report
Eulogy
![Page 26: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/26.jpg)
volumepoise
conclusion
Speaking
Course Course Course Course Course
eye contactstyle
appearance
gesturesrate
evidence
sourcesexamples
organization
transitionsverbal variety
attention getter
The Reality of Assessing Program LevelStudent Learning Outcomes
![Page 27: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/27.jpg)
volumepoise
conclusion
eye contactstyle
appearance
gesturesrate
evidence
sourcesexamples
organization
transitionsverbal variety
attention getter
Can our students deliver an effective public speech?
The Reality of Assessing Program LevelStudent Learning Outcomes
![Page 28: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/28.jpg)
Program Learning Outcome Example #1
Gather factual information and apply it to a given problem in a manner that is relevant, clear, comprehensive, and conscious of possible bias in the information selected
BETTER: Students will be able to apply factual information to a problem
Criteria: RelevanceClarity
Comprehensiveness Aware of Bias
![Page 29: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/29.jpg)
Program Learning Outcome Example #2
Imagine and seek out a variety of possible goals, assumptions, interpretations, or perspectives which can give alternative meanings or solutions to given situations or problems
BETTER: Students will be able to provide alternative solutions to situations or problems
Criteria: Variety of assumptions, perspectives, interpretationsAnalysis of comparative advantage
![Page 30: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/30.jpg)
Program Learning Outcome Example #3
Formulate and test hypotheses by performing laboratory, simulation, or field experiments in at least two of the natural science disciplines (one of these experimental components should develop, in greater depth, students’ laboratory experience in the collection of data, its statistical and graphical analysis, and an appreciation of its sources of error and uncertainty)
BETTER: Students will be able to test hypotheses.Criteria:Data collectionStatistical AnalysisGraphical AnalysisIdentification of sources of error
![Page 31: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/31.jpg)
Standards for Success/Quality of Performance
• Rubric• Scale or description for assessing each of
the criteria• Two to Five-point scales for each criteria• Anchored with descriptions and supported
by examples
• Portion of the Whole Group Which is Successful
![Page 32: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/32.jpg)
ExceedsExpectations
MeetsExpectations
Does not meetExpectationsCriteria
Performance Anchors:
Descriptionsof
Performance
OUTCOME:
Standards for Success/Quality of Performance Rubric
![Page 33: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/33.jpg)
YES!!!!AlmostNot YetComponents
Performance Anchors:OUTCOME:
Standards for Success/Quality of Performance Rubric
![Page 34: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/34.jpg)
ExemplaryEssentialBasicComponents
Performance Anchors:
Standards for Success/Quality of Performance RubricOUTCOME:
![Page 35: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/35.jpg)
Student LearningOutcome
3
2
1
4
Criteria Test Questions
1 2 3 4
5 6 7 8
10 11 12
1713 14 15 16
9
18
![Page 36: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/36.jpg)
Student LearningOutcome
3
2
1
4
Criteria Test Questions
1 2 3 4
5 6 7 8
10 11 12
1713 14 15 16
9
18
78%
56%
82%
42%
%correct
![Page 37: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/37.jpg)
Studentlearning outcomes Course
1
X
X
X
Course2
X
X
X
X
X
Course3
X
X
X
X
Course4
X
X
X
X
X
Course5
X
X
X
X
7
1
234
56
Identify Assessment Points in the Program
![Page 38: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/38.jpg)
Studentlearning outcomes Course
1
X
X
X
Course2
X
X
X
X
X
Course3
X
X
X
X
Course4
X
X
X
X
X
Course5
X
X
X
X
7
1
234
56
Phase 1
![Page 39: Program Level Assessment](https://reader035.fdocuments.net/reader035/viewer/2022062323/568167e8550346895ddd566a/html5/thumbnails/39.jpg)
Communicate EffectivelyDemonstrate Oral Communication Skills
VerbalDelivery
NonverbalDelivery
Organization
Does notmeet Meets Exceeds
20%
13%
24%
65%
57%
58%
15%
30%
18%
Program-wide Assessment Report
Is there a problem with performance in any of the criteria?