Presentation made to SoTE ConferenceMarch 28, 2014

Susan Larson

Using Assessment of Undergraduate Research to

Improve Programming and Student Outcomes

What is undergraduate research, scholarship & creative activity (UR)? Council on Undergraduate Research: An

inquiry or investigation conducted by an undergraduate student that makes an original intellectual or creative contribution to the discipline. www.cur.org

Concordia College: Undergraduate research, encompassing all forms of scholarship and creative activity, is an investigation or inquiry conducted by a student, under the mentorship of a faculty member, which contributes to a high-level intellectual or creative outcome.

Student, process centered Outcome, product centered

Student initiated Faculty initiated All students Honors students Curriculum based Co-curricular fellowships Collaborative Individual Original to the student Original to the discipline Multi- or interdisciplinary Disciplinary based Campus community audience Professional audience

- Beckman & Hensel, CUR Quarterly, 2009

Defining UR

Value of UR Cognitive and intellectual growth

Gains in knowledge and skills Academic achievement and educational

attainment Professional growth and advancement Personal growth and development

Assessing the Benefits of UR Survey of Undergraduate Research Experiences

(SURE) Includes demographic variables, learning gains, and

evaluation of aspects of summer programs. Students report the greatest learning gains on

items related to the research process, scientific problems, and lab techniques

Students report learning gains on personal development measures (e.g., tolerance for obstacles and working independently)

Summer research sustains an interest in pursuing science

Lopatto (2004), Cell Biol Educ, 270-277. http://www.grinnell.edu/users/lopatto

Assessment of undergraduate research opportunity program (UROP) at Univ. of Michigan Longitudinal assessment evaluating student retention,

academic performance, engagement, pursuit of graduate and professional education

Evaluated UROP students and matched control group: individuals who applied, but did not participate

Significant positive impact of UROP on degree completion rates (most notably for African American students) and likelihood of pursuing post-graduate education

Nagda, Gregerman, et al. (1998), The Review of Higher Education, 55-72 http://www.lsa.umich.edu/urop/aboutus/evaluationassessment

Assessing the Benefits of UR

Ethnographic study of summer undergraduate research (UR) experiences Initial work done at four liberal arts colleges Focus groups with students and faculty

Both groups described gains related to process of "becoming a scientist" Faculty focused on “professional socialization” Students more focused on “personal and

intellectual development”

Hunter, Laursen, Seymour (2007). Science Education, 36-74. http://www.colorado.edu/eer/index.html

Assessing the Benefits of UR

Why Should YOU Assess UR? Evaluate the effectiveness of your program

Determine if your programming is meeting your objectives

Develop your programming For donors, your administrators, grant writing

Understand the benefits of UR for your students To improve student learning in your context To improve faculty mentoring

To broaden participation in UR

But, we don’t have to re-invent the wheel

What Should You Assess? Student (learning) outcomes

What did you learn in this experience? How did this affect your educational experience,

career, etc.? Strategies for developing an assessment program Example of assessment program at my home

institution

Program effectiveness Make use of COEUR

Characteristics of Excellence in Undergraduate Research

Questions to Ask What do you want to know?

Should align with the objectives of your program Student researchers more likely to go to grad school? Self-confidence and self-efficacy? Impact on departments/advisors scholarly activity?

Who is the target of your assessment? Who does the assessing? What is the quality of the instrument? How will you use the results?

Benchmarking Program improvement Understanding student learning outcomes

Assessment of UR and SoTL There are great opportunities for SoTL

projects related to undergraduate research So, consider framing your UR assessment to

answer the questions you need to ask, but also the questions you want to ask. E.g., benefits to faculty: often discussed, less often

assessed E.g., how is UR different/similar to other forms of

engaged learning at UVU?

Designing a Sustainable Assessment Program State expected outcomes, clear and measurable

Research skills, student development goals Program goals

Determine methods and criteria to assess outcomes Direct assessment and/or indirect assessment Qualitative and/or quantitative assessment

Start small, start with successes, minimize the burden

Identify and collect information Set priorities, conduct periodic and timely assessments

Use the data! Share results, use assessment results in planning and

progam evaluation

The Assessment Cycle Assessment of UR is an on-going process of

Establishing clear, measurable expected outcomes for UR programs

Providing participants (students, faculty, institutions, system) with opportunities to achieve those outcomes

Systematically gathering, analyzing, and interpreting evidence to determine how well UR activities match our expectations

Using the resulting information to improve UR programs and initiatives

Maki, Journal of Academic Leadership, 2002

Example of an Assessment Plan for UR: Concordia College The assessment plan for the Office of

Undergraduate Research seeks to accomplish the following goals Provide a count of undergraduate research,

scholarship and creative activities at Concordia, including demographic data

Assess learning achieved in undergraduate research experiences

Evaluate the effectiveness of the Office of Undergraduate Research.

Counting UR

Faculty surveys administered by the Office of Institutional Research

Faculty report on # of students they mentored # of projects they mentored Funding for the projects Outcomes of the projects, e.g., presentations,

conferences Survey also distinguishes classroom-based

and individually-mentored research

Other Ways to Count UR Enrollment in directed research courses # of students participating in funded travel

opportunities # of students presenting at on-campus symposia

or related events Course syllabi which note an extensive research

experience Count of students who engage in off campus

research Make use of department annual reports

OTHERS?

The Challenge and Value of the Count Do you need to count everyone?

Can we even do it if we tried?

Is counting enough? Do you need to know what happens to these

students? Do you need to know their demographic

characteristics?

Assessment of Student Learning Students are expected to achieve some of

the following learning goals/objectives. I. formulate questions in their field II. demonstrate ability to understand literatureIII. explain methodology IV. demonstrate proficiency in the use of the

tools and instruments of the area of studyV. analyze and interpret literature and/or results

as appropriate to their field of studyVI. demonstrate ability to communicate (writing,

presenting) within the area of study, and

Assessment of Summer Research Program Indirect assessment -- student surveys

In-house self-report assessment of learning gains and confidence

Responses to open-ended question about learning outcomes

Evaluation of the degree to which students met the goals in the learning agreement they sign with their mentor at the start of the summer.

Science students complete Lapatto’s SURE survey (http://www.grinnell.edu/academic/csla/assessment/sure).

Faculty mentor survey Faculty rate their student’s learning gains and respond

to open-ended questions about learning gains

Faculty and students rate their satisfaction with the summer program activities.

Assessment of Summer Research Program

How Much Did You Gain as a Result of Your Summer Research Experience?

0= No gain5=Great gain 

No gain

A little

Moderate

Good Great

Formulating research questionsAll students: mean=3.77, SD=1.09

  19% 13% 39% 29%

1st year: mean=3.90, SD=.99

  10% 20% 40% 30%

1st yr mentors: mean= 3.78, SD=.97

  11% 22% 44% 22%

Proficiency in the use of the tools and instruments in your area of study

All students: mean=4.13, SD=.88

    13% 29% 58%

1st year: mean=4.80, SD=.42

      20% 80%

1st yr mentors: mean=3.78, SD=.83

    44% 33% 22%

Analyzing data & interpreting resultsAll students: mean=4.45, SD=.72

  10% 19% 35% 32%

1st year: mean=4.00, SD=1.05

  10% 20% 30% 40%

1st yr mentors: mean=4.56, SD=.53

    33% 33% 33%

Meeting Learning Agreement Goals

Considering your learning agreement, rate your learning gains for the goals specified in the agreement.

Mean S.D.

All Concordia summer research students

3.96 .96

1st year research students 4.40 .521st year research student mentors

4.11 .86

Open Ended Question “What were one or two of the most important things

you learned from your research experience?” Research skills specific to specific projects Understanding of the research process

E.gs., interpreting data; reading scientific literature; the importance of a detailed lab notebook.

Understanding of science E.g., understanding how scientists work together; the

importance of previously published findings; appreciation of the differing perspectives individuals from different disciplines bring to a problem

Personal gains Patience; honesty; flexibility; independence; the

importance of team work.

Summer Program Activities

Allowed them to develop some practical skills Appreciated the chance to interact with other

student researchers Some felt that the workshops targeted first-

time researchers and took too much time away from the lab

Impression of summer activities (very useful/enjoyable = 5)

Mean S.D.

How useful were the summer program activities?

3.65 .63

How enjoyable were the summer program activities?

3.88 .71

Other Ways to Assess SLOs Institutional assessment data

E.g., NSSE and critical thinking measures employed by the institution

Departmental assessment programs Nationally available survey’s

E.g., SURE and others by Lapotto E.g., Undergraduate Research Student Self-Assessment

(URSSA), on salgsite.org Rubrics for scoring posters/talks given by research

students Reflective essays for students who travel and

present UR Focus groups

Other Ways to Assess Student Outcomes Post graduate activities

Alumni survey Career center survey

Publications and presentations generated by involvement in UR

Review CUR’s assessment database for ideas https://www.zotero.org/groups/

curassessmentbibliography/items/

Using the Assessment Data Closing the assessment loop

Lower scores on communicating research Added an oral presentation expectation to the summer

program Lower rating on learning gains related to ethics

Will enhance summer program workshop on ethics Information about mentor availability and its

impact on the summer experience Conversations with mentors about their expectations

Program Evaluation May want to go beyond assessing the

experience of students to understand if the UR program you offer is robust and effective

Could be accomplished with the assistance of COEUR

http://www.cur.org/publications/publication_listings/coeur/

What is COEUR? A summary of best practices that support and

sustain highly effective undergraduate research environments

The purpose of COEUR is to provide a guide for those who wish to build, evaluate, and maintain robust, productive, meaningful and sustainable undergraduate research programs Function as a guide for strategic planning Provide programmatic and institutional benchmarking

against national standards Outline a structure for UR self-study Provide a framework for assessment and evaluation of

UR

The Characteristics Campus mission and culture

e.g., institutional commitment Administrative support

e.g., UR program office Research infrastructure

e.g., space, instrumentation and equipment Professional development opportunities

e.g., research leaves, mentorship training

Recognition e.g., UR in promotion and tenure guidelines

External funding e.g., internal and external funding for research

Dissemination e.g., peer-reviewed publication, exhibition, or

performance; student research conferences Student-centered issues

e.g., community of student scholars

The Characteristics

Curriculum e.g., integration of teaching and research;

student course credit for research Summer research program

e.g., faculty & student compensation Assessment activities

e.g., assessment of student learning Strategic Planning

The Characteristics

Simple Evaluation Rubrics  

CUR’s Characteristic of Excellence

Thinking about your own contextsuccesses at your institution challenges at your

institution

Campus mission and culture Institutional commitmentScholarly facultyFaculty commitmentBroad disciplinary participationAccessible opportunities for undergraduatesIntegration with other engaging and high-impact opportunities

   

What is your marker of excellence?

How do you measure this?

What evidence do you have that this component

is a priority at your institution?

EXAMPLEAdministrative supportStudent travel student funding

EXAMPLEX# of students traveling to professional meetings

EXAMPLEStudent receiving travel funds fill out pre-post survey; reported annually; count %age of students doing UR who travel to meetings

EXAMPLEIncreased travel funds from X$ per year to X$$ per year, student travel award recipients published in alumni magazine

http://www.cur.org/publications/publication_listings/coeur/

Conclusions There are many published benefits of UR

But, local, contextualized assessment of UR are important

When undertaking an assessment program Know what you are assessing and why Keep it simple and sustainable

Don’t assess everything all the time Keep your focus on enhancing student learning

Use assessment data and program evaluation to improve your UR programming COEUR can be a useful tool for improving the

experience you offer students