Assessing Intrapersonal and Interpersonal Competence ...
Transcript of Assessing Intrapersonal and Interpersonal Competence ...
Assessing Intrapersonal
and Interpersonal Competence:
Fragments from the Field
George Kuh
Senior Scholar, NILOA
Supporting Students’ College Success Summit
Washington, DC
April 13, 2017
Overview
A word about NILOA
Partial answers to three questions
Discussion
NILOASince 2008, our mission is to discover and disseminate effective use of assessment data to strengthen student learning and advance high quality, actionable assessment work.● SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ●OCCASIONAL PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ●LISTSERV ● PRESENTATIONS ● TRANSPARENCY FRAMEWORK ● FEATUREDWEBSITES ● ACCREDITATION RESOURCES ● ASSESSMENT EVENTCALENDAR ● ASSESSMENT NEWS ● MEASURING QUALITY INVENTORY ●POLICY ANALYSIS ● ENVIRONMENTAL SCAN ● DEGREE QUALIFICATIONSPROFILE/TUNING ● DQP CAMPUS COACHING ● ASSIGNMENT DESIGNAND ONLINE LIBRARY ● EMERGING LEARNING SYSTEM NETWORKING
www.learningoutcomesassessment.org
Three Questions
1. How does assessment of the competencies identified in this report intersect with the current state of assessment in higher education institutions?
2. What processes are involved in turning data from assessments of these competencies into improvement in higher education? What are the challenges, opportunities, or potential feedback loops to make this work consequential?
3. How might this improvement process work, specifically, when considering assessments of the identified competencies in higher education?
Do we measure what we value?
or
Do we value what we measure?
Wise decisions are needed about what and how to measure the proficiencies demanded by the 21st century
1. How does assessment of intra- and interpersonal competencies intersect with the current state of assessment in higher education institutions?
-- NAS report generally accurate – most institutions do very little assessment of these outcomes despite statements valuing such behaviors by institutions and employers
-- AAC&U Essential Learning Outcomes (ELOs) and Lumina Degree Qualifications Profile (DQP) only indirectly address interpersonal and intrapersonal competence
1. How does assessment of intra- and interpersonal competencies intersect with the current state of assessment in higher education institutions?-- NAS report generally accurate – most institutions do very little assessment of these outcomes despite statements valuing such behaviors by institutions and employers
-- AAC&U Essential Learning Outcomes (ELOs) and Lumina Degree Qualifications Profile (DQP)
-- AAC&U VALUE rubrics re teamwork, ethical reasoning and institution-specific rubrics (e.g., Tidewater CC personal development rubric)https://list.aacu.org/sites/default/files/files/VALUE/VALUE_CulturalAndSocialUnderstandingRubric.pdf
Valid Assessment of Learning in Undergraduate Education
(VALUE) Rubrics Inquiry and analysis Critical thinking Creative thinking Written communication Oral communication Reading Quantitative literacy Information literacy Teamwork Problem solving Civic knowledge and engagement Intercultural knowledge and competence Ethical reasoning and action Foundations and skills for lifelong learning Integrative learning
1. How does assessment of intra- and interpersonal competencies intersect with the current state of assessment in higher education institutions?
-- NAS report generally accurate – most institutions do very little assessment of these outcomes despite statements valuing such behaviors by institutions and employers
-- AAC&U Essential Learning Outcomes (ELOs) and Lumina Degree Qualifications Profile (DQP)
-- AAC&U VALUE rubrics re teamwork, ethical reasoning and institution-specific rubrics (e.g., Tidewater CC personal development rubric)https://list.aacu.org/sites/default/files/files/VALUE/VALUE_CulturalAndSocialUnderstandingRubric.pdf
-- Behavioral observations and performance assessments via internship/job evaluations, high-impact practices, etc.
High-Impact Activities
First-Year Seminars and Experiences Common Intellectual Experiences Learning Communities Writing-Intensive Courses Collaborative Assignments and Projects Inquiry & Creative Activity Diversity/Global Learning Service/Community-Based Learning Internships and Field Placements Capstone Courses and Projects
ePortfolio
U of Iowa Student Employment Project“Guided Reflection on Work” (GROW)
Supervisors had two structured conversations with every student employee during spring semester and coded conversations into the following categories:
How the job and academics complement each other (“How is your job fitting in with your academics?”)
Transfer between work and academics (“What are you learning here at work that is helping you in school?”)
Transfer between academics and work (“Are you learning anything in class that you can apply here at work?”)
Transfer between work and future career (“Give me a couple of examples of things that you are learning here at work that you will be using in your future profession?”)
IOWA GROW® participants were more likely (p<.01) to agree/strongly agree that their job was helping them attain the following outcomes of student employment:
1 My supervisor helps me make connections between my work and my life as a student.
6My job has helped me develop more effective time management skills.
2 My job has helped prepare me for
the world of full-time employment.
3 My job has helped me improve
my written communication skills.
4 I can see connections between my job and major/coursework.
7 My job has helped me improve my oral communication skills.
8 My job has helped me develop conflict resolution skills.
9My job has helped me use critical
thinking skills to form opinions and
solve problems.
5 My job has helped me learn about career options. 10
Because of my job, I am able to work effectively with individuals with a variety of backgrounds, experiences, and cultures.
2. What processes are involved in turning data from assessments of these competencies into improvement in higher education? What are the challenges, opportunities, or potential feedback loops to make this work consequential?
-- The classic assessment cycle
The Assessment Cycle
2. What processes are involved in turning data from assessments of these competencies into improvement in higher education? What are the challenges, opportunities, or potential feedback loops to make this work consequential?
-- The classic assessment cycle
-- Outcomes statements aligned with course requirements
-- Curricular mapping -- Assignment design
Design assignments to elicit the student behaviors that demonstrate the desired outcomes.
Questions? Comments? Click
here!
Welcome to NILOA’s Assignment Library!The purpose of this website is to provide a searchable online library of collegiate-level course assignments in a wide variety of academic disciplines that link to one or more proficiencies in the Lumina Foundation’s Degree Qualification Profile (DQP). This page allows you to:
Browse assignments Borrow and adapt an assignment to fit your needsSubmit an assignment to be included in our libraryConsult a resourceComment on and ask questions about assignmentsStart a conversation on your own campusGive us feedback!
Learn about the DQP Submit an assignmentBrowse our library Look for resources
What do you want to do?
3. How might this improvement process work, specifically, when considering assessments of the identified competencies in higher education?
-- Campaign to mobilize outside-in pressure (accreditors, employers, professional societies) to underscore the value of interpersonal and intrapersonal competence and compel institutional leaders, faculty and others to cultivate these outcomes, and collect and make consequential use of assessment results
-- Measurement approaches (e.g., rubrics, performance assessments) must be developed and widely used to document these competencies
Operational Verbs to Assess Teamwork Skills
Verbs that describe executive functions
lead, operate, administer, control, coordinate, manage, plan, navigate
Verbs that describe forms of deliberative activity
argue, challenge, debate, defend, justify, resolve, dispute, advocate, persuade
Verbs related to modes of communication, in groups
collaborate, contribute, negotiate, feedback
Verbs describing how to explain a position or problem
articulate, clarify, illustrate, interpret, translate, elaborate, elucidate
Operational Verbs to Assess Teamwork Skills
Verbs describing the cognitive functions underlying analytical reasoning
compare, contrast, differentiate, distinguish, formulate, map
Verbs describing information processing
calculate, determine, estimate, manipulate, measure
Verbs describing ways to format information
arrange, assemble, collate, organize
Verbs describing rethinking or reconstructing
accommodate, adapt, adjust, improve, modify, refine, reflect, review
Operational Verbs to Assess Teamwork Skills
Verbs describing information processing
calculate, determine, estimate, manipulate, measure
Verbs describing ways to format information
arrange, assemble, collate, organize
Verbs describing rethinking or reconstructing
accommodate, adapt, adjust, improve, modify, refine, reflect, review
3. How might this improvement process work, specifically, when considering assessments of the identified competencies in higher education?
-- Outside-in pressure (accreditors, employers, professional societies) is
needed to compel institutional leaders and faculty to take action and make consequential use of assessment results
-- Measurement approaches (e.g., rubrics, performance assessments) must be developed and widely used to document these competencies
-- Bundling interventions (learning communities, peer mentoring, grit/conscientious assignments)
-- Problem-based and small group assignments that actively involve students in applying what they are learning
3. How might this improvement process work, specifically, when considering assessments of the identified competencies in higher education?
-- Activities such as frequent low-stakes testing, feedback, and autobiographical writing help facilitate self-regulated learning and minimize stereotype threat
(see Singer-Freeman, K. & Bastone, L. (2016). Pedagogical choices make large classes feel small (NILOA Occasional Paper No. 27). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes
Assessment.http://www.learningoutcomesassessment.org/documents/Occasional%20Paper_27.pdf
3. How might this improvement process work, specifically, when considering assessments of the identified competencies in higher education?
-- an essential component is a common or widely shared documentation/record keeping system:
Comprehensive Student Record (AACRAO & NASPA)U of Central Oklahoma Transformative Student
Recordhttp://www.learningoutcomeassessment.org/documents/Hynes_Assessment_in_Practice.pdf
-- ePortfolio, a promising pedagogical innovation and meta framework (see Table 4-1, pp 106-108)
ePortfolios make student learning visible – to students themselves, to their peers and faculty, and to external audiences.
ePortfolio Resources
Batson, T. (2002). The electronic portfolio boom: What's it all about? Campus Technology.https://campustechnology.com/articles/2002/11/the-electronic-portfolio-
boom-whats-it-all- about.aspx
Cambridge, D. (2010). Portfolios for lifelong learning and assessment. (Chichester, United Kingdom: Wiley.
Clark, J.E., & Eynon, B. (2011/2012). Measuring student progress with e-portfolios. Peer Review, 13(4)/14(1), 6–8.
Eynon, B. & Gambino, L. (2017). High impact ePortfolio practice: A catalyst for student, faculty and institutional learning. Sterling, VA: Stylus.
Eynon, B., Gambino, L.M., & Török, J. (2014). What difference can ePortfolio make? A field report from the Connect to Learning Project. International Journal of ePortfolio, 4(1), 95–114.
3. How might this improvement process work, specifically, when considering assessments of the identified competencies in higher education?
-- Faculty and institutional leaders must value interpersonal and intrapersonal competence and address these outcomes explicitly in institutional and program learning goals, curricular mapping, assignments, and out-of-class learning experiences, and in efforts to document student attainment of these competencies
Faculty Engagement in Assessment is Key, But Non-Trivial Challenges Must Be Addressed
Concern that it is a fad
Concern that it is externally imposed
Concern that it is a top-down administrative mandate
Concern over the corporatization of higher education
Challenges to academic freedom / shared governance
Faculty Engagement in Assessment is Key, But Non-Trivial Challenges Must Be Addressed
Misalignment in language / difficulty in communicating warrant and purposes
Fear over misuse of results and data
Fear over lack of use of the data
Skepticism over the ability to assess student learning
Worries that it might affect standards
Concerns related to simplicity / complexity
Concerns over workload of faculty / that it is an add-on
Fundamentals
A sea-change is needed to move interpersonal and intrapersonal competence into the core of desired collegiate learning outcomes for ALL students
Engaging pedagogical approaches used extensively in-class and out-of-class that promote self-regulation and reflection combined with frequent, timely feedback to insure students recognize, practice, and demonstrate these competencies
Alignment of curricular and other learning experiences (e.g., applied, experiential learning, high-impact practices) that require students to practice and apply these competences in multiple contexts
Principles for Consequential Use of Evidence
1. Backward design. That is, begin assessment activity with the end use in mind. What practical questions are of greatest interest to potential partners—faculty, administrators, staff, and intended internal and external end users? How might they use the results? What do partners and end users expect to find?
Principles for Consequential Use of Evidence
2. Identify users of evidence when designing assessments and its anticipated uses at different levels within an institution. What institution-wide or program-level goals and audiences can be influenced by results? Present assessment and the questions it explores in a way that persuades people that the results of the inquiry will have practical value in their work, at their level.
Principles for Consequential Use of Evidence
3. Leverage the accreditation process for meaningful campus action to improve student learning. Ensure that the campus benefits from all the time and energy that accreditation requires by framing this work as more than an act of compliance—as a process to move the campus toward improvement in learning, teaching, and institutional performance.
Principles for Consequential Use of Evidence
4. Connect assessment work to related current national initiatives and projects. Membership associations, national organizations, foundations, and other collective initiatives provide a broader context in which to embed assessment work, to learn from the work of other institutions, and to increase the impact, legitimacy, and value of assessment results.
Principles for Consequential Use of Evidence
5. Link assessment work to campus functions that require evidence of student learning (e.g., program review, center for teaching & learning). Action and improvement are more likely if (a) assessment work is sustained, not episodic: (b) campus structures and processes are in place to report results, discuss implications, and plan needed action, and (c) key end users and partners are engaged in the process.
Principles for Consequential Use of Evidence
6. Effective student learning outcomes assessment is a continuous process. Taking evidence-informed action is important, but it also marks the beginning of a new cycle. What was the impact of the change? Did student performance improve? Follow-through and taking time to evaluate the impact of evidence-based change is essential in fostering a culture that supports the meaningful use of assessment results.
Principles for Consequential Use of Evidence
7. The value of assessment work is a function of the extent to which results are used. Are assessment priorities shaped by questions and concerns for student success that anticipate the productive application of results? Is work too often stalled in the data collection stage? Does the sharing-results phase tend to terminate the cycle? Where does the cycle get stuck, why, and what might free it up?
Questions
&
Discussion
www.learningoutcomeassessment.org