Engaging learning outcomes across a discipline and in Institutions Brian Frank Queen’s University.
-
Upload
posy-cooper -
Category
Documents
-
view
222 -
download
1
Transcript of Engaging learning outcomes across a discipline and in Institutions Brian Frank Queen’s University.
Learning outcomes are not new…E.g. Ontario’s college sector, professional programs in Canada, accreditation requirements in the US
2
…but closing the loop is
Survey: Only 6% of 146 profiles of good practice submitted contained evidence that student learning had improved (Banta & Blaich, 2011).
Baker, G. R., Jankowski, N. A., Provezis, S., & Kinzie, J. (2012). Using Assessment Results: Promising Practices of Institutions That Do It Well. Retrieved from http://www.tamiu.edu/adminis/iep/documents/NILOA-Promising-Practices-Report-July-2012.pdf
using evidence from learning outcomes to improve student learning and inform curriculum
Student self-assessmentFormative evalution to instructor
Explicit objectives and assessmentReciprocal teaching
FeedbackSpaced vs. mass practiceMetacognitive strategies
Creativity programsSelf-questioning
Professional developmentProblem solving teaching
…Teaching quality
Time on taskComputer assisted instruction
-0.6 0.4 1.4
Effect size (performance gain in σ)
800 meta-analyses
50,000+ studies
200+ million students
Hattie, J. (2009). The Black Box of Tertiary Assessment: An Impending Revolution. In Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research (pp.259-275). Wellington, New Zealand: Ako Aotearoa
Role of learning outcomes in delivery:
• Curriculum and assessment planning• University learning space planning• University-wide student services and academic
support planning• Potentially: Competency based credentials
Learningoutcomes
Assessment
Learning & teachingactivities
Level:CourseProgramFacultyInstitution
Canadian Engineering Accreditation Board:
3.1: Demonstrate that graduates of a program possess 12 graduate attributes
3.2: Continual program improvement processes using results of graduate attribute assessment
8
Engineering Graduate Attribute Development (EGAD) Project
WHO
Engineering educators and educational developers across Canada
MANDATE
Collect and develop resources and training
Run annual national workshops, and customized institutional workshops
9
EGAD Workshops
1. Introduction to Continuous Program Improvement Processes
2. Graduate Attribute Assessment as a Course Instructor
3. Creating Useful Learning Outcomes4. What to Look for in an Outcomes-Based
Process5. Leading a program improvement process
10
Program objectives and outcomes
Mapping curriculum and assessment planning
Collecting evidenceAnalyze and interpret
Curriculum & process improvement
What do you want to know about the
program?
1 2
345
Example process
13
Developing or adapting outcomesTool: Learning outcomes collection
Aligning outcomes within a courseTool: Course planning table
Aligning outcomes and curriculumTool: Curriculum map
Scoring performanceTool: Rubrics
Course 1 Course 2 Course 3
Outcome 1 Develop Master/assess
Outcome 2 Assess
Outcome 3 Develop/assess
PHYS101 Course Outcomes: Students will:1. Describe motion of…2. Predict the behaviour…
Teaching Activity Assess
Week 1 … …
Week 2 … …
Week 3 … …
Diploma Bachelor Masters
Knowledge … … …
Critical think … … …
Writing … … …
Interpersonal … … …
Marginal Meets Exceeds
Outcome 1 … …
Outcome 2 … …
Outcome 3 … …
HEQCO Learning outcomes consortium
IssueNo one has effectively closed the loop in OntarioConsortium goalDevelopment of useful learning outcomes assessment techniques and to their wide-scale implementation in their institutionsFocus on generic learning outcomes and cognitive skills (critical thinking, communications, lifelong learning, etc.)
15
17
Developing or adapting outcomesTool: Learning outcomes collection
Aligning outcomes within a courseTool: Course planning table
Aligning outcomes and curriculumTool: Curriculum map
Scoring performanceTool: Rubrics
Course 1 Course 2 Course 3
Outcome 1 Develop Master/assess
Outcome 2 Assess
Outcome 3 Develop/assess
PHYS101 Course Outcomes: Students will:1. Describe motion of…2. Predict the behaviour…
Teaching Activity Assess
Week 1 … …
Week 2 … …
Week 3 … …
Diploma Bachelor Masters
Knowledge … … …
Critical think … … …
Writing … … …
Interpersonal … … …
Marginal Meets Exceeds
Outcome 1 … …
Outcome 2 … …
Outcome 3 … …
College sector
Durham: Is Student Success ePortfolio effective for assessing Essential Employability Skills (EES)George Brown: Tools & rubrics to assess EES (communication and problem solving)Humber: reliable instrument for reading, writing, critical thinking, and problem solving across curriculum
18
University sector
Guelph: Process and tools for mapping & assessment of university-wide learning outcomes using VALUE rubricsToronto: Analytic rubrics for communications, application of knowledge, and teamwork in engineeringQueen’s: Mixed methods assessment of generic learning outcomes across four fields
19
Approaches to direct assessment of learning outcomes across program
①Course-specific criterion-referenced scoring using course deliverables
②Stand-alone standardized instruments③General criterion-referenced scoring using
course deliverables
20
Approaches to direct assessment of learning outcomes across program
①Course specific criterion-referenced scoring using course deliverables • Provide clear guidance to students• Useful for course improvement • Limited ability to assess development over
multiple years
②Stand-alone standardized instruments③General criterion-referenced scoring using
course deliverables21
Example: Leveled outcomes for each year
22
Theme First year Second year Third year Graduating yearCom
munications
Process
Describes typical expectations engineers to communicate effectively.
Generates a traceable and defensible record of a technical project using an appropriate project records system.
Writes and revises documents using appropriate discipline-specific conventions
Written
Summarizes and paraphrases written work accurately with appropriate citations
Composes documents in styles including progress reports, professional career (cover letters, CV, RFP), design reports
Demonstrates conciseness, precision, and clarity of language in technical writing.
Write concise, coherent and grammatically correct materials that reflect critical analysis and synthesis, appropriate to audience needs.
Oral
Delivers clear and organized formal presentation following established guidelines
Delivers effective formal oral presentations including appropriate facial gestures, natural body posture and movement
Demonstrates formal oral presentations with appropriate language, style, timing and flow.
Demonstrates confidence in formal and informal oral communications
Graphical
Creates effective figures, tables, and drawings employing standard conventions to compliment text.
Creates accurate and complete technical graphics.
Uses graphics to explain, interpret, and assess information
Performance by outcome within a course
FEAS - 3.12-FY1 FEAS - 3.12-FY2 FEAS - 3.12-FY5 FEAS - 3.12-FY60
10
20
30
40
50
60
1 - Not Demonstrated 2 - Marginal 3 - Meets Expectations 4 - Outstanding
Attributes
Per
cent
age
(%)
24
3.12-FY1 Uses information effectively, ethically, and legally to accomplish a specific purpose, including clear attribution of Information sources.
3.12-FY2 Identifies a specific learning need or knowledge gap.
3.12-FY5 Identifies appropriate technical literature and other information sources to meet a need
3.12-FY6 Critically evaluates the procured information for authority, currency, and objectivity.
Approaches to direct assessment of learning outcomes across program
①Course specific criterion-referenced scoring using course deliverables
②Stand-alone standardized instruments (CLA, etc.)– Measure development over multiple years,
institutional comparison. Validity & reliability data.– Can be expensive, measure limited set of skills– Low completion rates, poor motivation particularly
fourth year students, so results suspect
③General criterion-referenced scoring using course deliverables
25
Approaches to direct assessment of learning outcomes across program
①Course specific criterion-referenced scoring using course deliverables
②Stand-alone standardized instruments③General criterion-referenced scoring using course
deliverables• Can assess development over multiple years• No additional student work, so no problem with
motivation, completion rates• Encourages alignment between program course
outcomes and course delivery• Requires some additional grading time• Limited availability of validated rubrics
26
Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics
• Meta-rubrics that synthesize the common criteria and performance levels gleaned from numerous individual campus rubrics for 14 Essential Learning Outcomes
• Can be used to mimic approach taken by some critical thinking tests that allow programs to provide their own “artifact” that is scored against a common set of criteria
27
28Rhodes, Terrel, ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities.
29A. Greenhoot, D. Benstein, Using VALUE Rubrics to Evaluate Collaborative Course Design, Peer Review, vol. 13 no. 4, AAC&U
Assessing development using VALUE rubrics
Queen’s approach: piloting general outcomes in 4 fields
30
Physical science Social science
Engineering Humanities
Outcomes assessment plan over three years
31
Outcome 1. Course specific scoring
2. Standard tool (limited cohort)
3. General scoring (VALUE)
4. Think aloud
Critical thinking If available CLA or CAT Critical
thinking Local
Problem solving
If available CLA or CAT Problem solving Local
Written comm.
If available CLA or CAT Written comm
Lifelong learning
If available LASSI, MLSQInfo
lit/lifelong learn
SRLIS
Engaging learning outcomes across a discipline and in Institutions
Brian FrankQueen’s University
If you don’t know where you’re going, you’ll probably end up somewhere else.
OTHER SLIDES (USED ONLY IF THERE ARE QUESTIONS)
Engineering Graduate Attribute Development (EGAD) Project
33
CEAB requirements include:a) Identified learning outcomes that
describe specific abilities expected of students
b) A mapping of where attributes are developed and assessed within the program
c) Description of assessment tools used to measure student performance (reports, exams, oral presentations, …)
d) An evaluation of measured student performance relative to program expectations
e) a description of the program improvement resulting from process
34
Performance by student in a course
Engineering Graduate Attribute Development (EGAD) Project
35
0 1 2 3 4 5 6-10 11-15 16-20 21-25 26-30 31-35 36-40 41-500
50
100
150
200
250
300
350
400
26 4267 65 62 45
228
100
41 25 10 2 2 0
344
187
7346 38
7 200 0 0 0 0 0 0
Below target Below threshold
Number of indicators
Num
ber o
f stu
dent
s
4 approaches to facilitating change
Disseminating(good for knowledge, poor for
long term change)
Supporting individual innovators
Enacting Policy Developing shared vision
36
Henderson, C., Beach, A., & Finkelstein, N. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature. Journal of Research in Science Teaching, 48(8), 952–984. doi:10.1002/tea.20439
Effective strategies: are aligned with or seek to change beliefs, long-term interventions, understand university as a complex system, honest about issues and problems.
prescribed emergent
indi
vidu
alen
viro
nmen
tal
Software packages evaluated
• Canvas• Desire2Learn• eLumen• LiveText• Moodle• Waypoint Outcomes• (No response from Blackboard)
Engineering Graduate Attribute Development (EGAD) Project
38
Want to merge into one tool!
Software summary
• Desire2Learn is the closest to a complete package to manage courses, learning outcomes, rubrics, and reporting; Analytics tool in early stages
• eLumen outstanding at analysis, but poor integration into general LMS
• Waypoint Outcomes/LiveText outstanding at managing outcomes, rubrics, and feedback
Engineering Graduate Attribute Development (EGAD) Project
41
Engineering Graduate Attribute Development (EGAD) Project
42
Student: You are here!(67%)
Norm referenced evaluation
Gra
des
Criterion referenced evaluation
Used for large scale evaluation to comparestudents against each other
Student has marginally met expectations because submitted work mentions social, environmental, and legal factors in design process but no clear evidence of that these factors Impacted on decision making.
Used to evaluate students against statedcriteria