TRIO Student Support Services Peer Reviewer training:

Relevance Reviews

March 2, 2015

Outline of presentation

• Purpose of evidence priorities

• What is evidence

• Role of relevance review

• Examples

• Overview of three studies cited in the NIA

• Procedure for other studies

• Finding the information you need in the application to do a relevance review

Purpose of evidence priorities

• To maximize likelihood of success of strategies/activities implemented under the competitive priorities 1(a) and 1(b), ensuring that grantees draw on what works.

1(a) Influencing the development of non-cognitive factors (e.g. mindsets, perseverance, cooperation, attention, goal setting, etc.)

1(b) Providing individualized counseling for personal, career and academic matters

What is evidence?

• Applicants must: Cite at least one study

Study has to support the proposed activities

Study has to be found effective in at least one context

Has to be a technical study (journal article or report)

• Quality of evidence is reviewed by a different part of ED (IES)

• Your role isn’t to determine evidence quality, but relevance

The role of the relevance review

• It’s about determining: Does this applicant propose something that’s reasonably similar to what’s described in the study?

• To make that determination, you’ll need an understanding of:

– What we mean by “reasonably similar”– What the applicant is proposing– What the intervention is that’s described in the

study the applicant cites.

What is reasonably similar?

• This is an art, not a science.

• The critical guide: Examining whether the applicant’s proposed activity has the key features of the intervention in the cited study.

• There can be important differences between the applicant’s proposed intervention and the cited intervention, they have to be justified in the application.

Example #1

• A cited study examines an approach in which college juniors & seniors speak on panel discussions designed to give advice to first year students.

• What if an applicant proposes something similar, but uses second year students as panelists instead?

• Is there a reasonable justification?

• Let’s say applicant’s setting is a two-year community college (there are no juniors and seniors) and this is cited as the justification.

What do you think?

Example #1

• A cited study examines an approach in which college juniors & seniors speak on panel discussions designed to give advice to first year students.

• What if an applicant proposes something similar, but uses second year students as panelists instead?

• Is there a reasonable justification?

• Let’s say applicant’s setting is a two-year community college (there are no juniors and seniors) and this is cited as the justification.

• Yes, that seems reasonable.

Example #2

• A cited study examines an approach using individualized counseling with proactive, frequent outreach to students.

• What if an applicant proposes individualized counseling, but a more traditional version of having counselors available to help students when needed?

• The applicant doesn’t mention a justification of differences since they see the two approaches as essentially similar.

What do you think?

Example #2

• A cited study examines an approach using individualized counseling with proactive, frequent outreach to students.

• What if an applicant proposes individualized counseling, but a more traditional version of having counselors available to help students when needed?

• The applicant doesn’t mention a justification of differences since they see the two approaches as essentially similar.

• No, that doesn’t seem reasonable.

Cited studies in the NIA

• The NIA provided three example studies.

• As a result, they will be the most frequently cited by applicants, so we’ll examine them in this training session.

Cited studies in the NIAStrategies To Develop Non-Cognitive Factors

1. Walton, G.M. & Cohen, G.L. (2011). A brief social-belonging intervention improves academic and health outcomes of minority students.

2. Stephens, N.M. et al (2014 ). Closing the social-class achievement gap: A difference-education intervention improves first-generation students' academic performance and all students' college transition.

Strategies To Provide Individualized Counseling

3. Bettinger, E.P., & Baker, R. (2011). The effects of student coaching in college: An evaluation of a randomized experiment in student mentoring.

• Overview: Intervention is a short reading and writing activity that’s about 60 minutes that can be delivered either in person (one on one) or in small groups in online sessions either before matriculation or in the first year of college.

• Goal: Help students see their worries about belonging and about everyday adversities as normal – that everyone worries at first whether they belong and it gets better with time and that it's not evidence that “people like me don't belong in college.”

Walton, G.M. & Cohen, G.L. (2011). A brief social-belonging intervention

• Content: Includes both survey information about students’ experience in the transition to college and carefully written stories from upper year students all of which convey the core message that everybody worries in their first year about whether they belong in college and that it gets better with time.

• Framing: Not represented to students as an intervention, but as college officials wanting to know more about people's experiences in transition to college so that they can help future generations.

Walton, G.M. & Cohen, G.L. (2011). A brief social-belonging intervention

• Approach: Not just passively delivered. Instead, active experience with what we call “seeing is believing” exercises, where students write about how what they've read that is true to their own experience and sometimes even create videos or write letters to future students.

Walton, G.M. & Cohen, G.L. (2011). A brief social-belonging intervention

• Overview: Intervention is an hour long panel discussion about the transition to college for students beginning their first years. (Panel was diverse in race / gender / first generation. Moderator who asked the panelists planned questions.) Panelist responses carefully in advance, based on student's actual experiences in college.

• Goal: Show how social class background matters – for both good and bad. Show how all students can experience college differently.

Stephens, N.M. et al (2014 ). A difference-education intervention

• Approach: Panelists’ share stories that highlight how their social class backgrounds can matter in college, both for good (strengths) and bad (obstacles). Stories also highlight strategies they used, taking their backgrounds into account, to succeed in college.

Stephens, N.M. et al (2014 ). A difference-education intervention

• Overview: Model is individualized (one-on-one) student counseling that authors call student coaching. Similar in some ways to mentoring and advising, but with some unique features.

• Goal: Help students succeed in their higher ed institution, including persistence (staying in school)

Bettinger, E.P., & Baker, R. (2011). Student coaching in college

• Approach:

Students can opt out at any point Calls between coaches and students are

individualized for student’s specific situation Focus of coaching is generally not on academic

subjects, but rather on:• Time management• Goal setting and planning• Using resources and advocating for themselves• Navigating college decisions

Bettinger, E.P., & Baker, R. (2011). Student coaching in college

Coaches meet with students weekly or biweekly

One of key feature: coaching is intrusive and proactive. Coaches contact students via phone or email or social networking or text and they don't wait for the students to contact them.

To the extent possible, coaches have access to school data so they can track things like attendance in class, student performance and activities. They have this as background knowledge in talking with students.

Bettinger, E.P., & Baker, R. (2011). Student coaching in college

Posted on the peer review resources page:

• Transcript of study author’s presentations on the three interventions

• Study author’s PowerPoint presentations

• Links to the studies in the NIA

More info on these studies

• The relevant information to be reviewed will be listed under the Competitive Preference Priorities Section.

• Peer reviewers will receive a panel sheet indicating which CPPs are addressed for each application assigned.

Where in applicant’s proposal to find relevant information for relevance

determination