Research Agenda Workshop

Regional Educational Laboratory at EDCrelnei.org

1

Research Agenda Workshop

[District/State Name]

Session 1: [Date and time]Session 2: [Date and time]

[Facilitator Name]


Welcome and Purpose

Our purpose in this workshop series is to • Engage in a collaborative process to identify district/state

priorities, and • Develop a set of research questions that comprise a

coherent research agenda that will be the work of our district/state for next 3-5 years.

2


Goals for the Session

• Session 1 (today) will focus on reviewing different types of research, including levels of evidence, making causal claims, and linking questions to methods. We will leave today having prioritized initial research topics.

• Next session, we will generate research questions, and refine and prioritize our identified topics and research questions. We will leave with an initial research agenda for our district/state.

3


Agenda for the Session

• Introduce workshop participants• Investigate types of research and levels of evidence,

including reviewing the pre-work and additional research examples

• Discuss examples of researchable questions and research agendas

• Review district/state goal and generate some possible topics

• Prioritize possible research topics• Closing

4


Introductions

• Please share your:– Name– Agency/where you work– Your role there

– [if there are pictures of the participants you can insert an additional slide with the name and pictures while introductions are taking place].

5


6

Primary Types of Research

• Different types of research allow us to answer different types of questions and to make different types of claims.

• What’s Happening– Descriptive

• What’s Known– Descriptive

• Making Connections– Correlational

• Making an Impact– Impact/Causal


7

What’s Happening

• Descriptive studies do not examine cause and effect but look at what is happening: trends; baselines; and experiences of individuals, groups, or programs– What is the trend in science NECAP scores over past 5 years?– What are the certification characteristics of beginning teachers in

the region?– What are the features of district teacher evaluation systems?

• Methods tend to include secondary data analysis, document and records review

• Tend to use descriptive statistics: averages, frequencies, and percentages


8

What’s Known

• Another type of descriptive study focusing on reviews of the literature and other research that has already been conducted.– Literature review– Meta-analysis


9

Making Connections

• Correlational studies look at the relationship between two or more variables but one cannot infer causality. – How do math course taking patterns vary race and gender?– What factors are related to differences in graduation rates across

districts?

• Methods tend to be secondary data analysis of administrative or other state, district, or school datasets

• Statistical methods include: differences in means, ANOVA, multivariate regression models


10

Making an Impact

• Causal/Impact studies usually examine questions about impact, building on descriptive studies– Does this afterschool program increase student engagement?– Does this math intervention lead to improved math achievement?– Do new teacher evaluation policies improve student achievement

outcomes?

• Many “opportunistic” possibilities in implementation• According to IES, an impact study:

– examines the effectiveness of a particular policy, program, or practice– can be small-scale or large-scale– can be used in a formative way (e.g. a short-term pilot study) or a

summative way


11

Making an Impact

• Only certain types of designs can be used to assert causal claims or claims about program impact

• IES-funded impact studies must meet What Works Clearinghouse standards– Randomized Control Trials (RCTs) = “the gold standard”– Randomization of treatment and control groups (e.g. those who

get the intervention and those who don’t)– Other designs, including quasi-experimental, matched

comparisons or others may not yield unbiased estimates of impact


12

Investigating Research Examples

• Refer to the 2 research summaries you read during pre-work

• You will be placed into breakout rooms where small groups will focus on one study and discuss – Research questions– Data sources– What was answered/found– Study design—discuss and add this information– What else there is to learn/ future research questions


13

Discussion: Research Design

• Back in the large group we will share out from our small group discussions

• What were the studies about? • What is and is not answered by research

questions/designs?– What related research questions might you ask?


14

Researchable Questions

• What is a researchable question and is not?– Reasonable– Appropriate– Answerable– Measurable

• How do you develop researchable questions?– Questions, concerns, and values of stakeholders/community– Important issues raised in the field or literature– Professional standards or guidelines– Views and knowledge of experts– One’s own views and judgement


15

Research Agendas

• What is a research agenda? • What does a research agenda look like?

– Linear– Topical– Examples


16

Example: Research agenda that is less coherent

• TOPIC: COLLEGE READINESS– Descriptive study of high school dropout rates for key

subgroups– Descriptive study of policies for assigning students to

college math vs. general math– RCT of the impact of classroom coaching for new high

school English teachers in 5 large urban districts(Source: REL Kickoff Meeting)


17

Example: Coherent, linear research agenda

• TOPIC: HIGH SCHOOL GRADUATION• Action Goal: Increasing the percentage of District X

students who graduate from high school– Descriptive study of high school dropout rates for key

subgroups– Descriptive study of re-enrollment rates in traditional high

schools and education trajectories of re-enrollees– RCT of the impact of alternative programs for degree

completion on high school graduation(Source: REL Kickoff Meeting)


18

Example: Coherent, topical research agenda

• TOPIC: MATHEMATICS LEARNING• Action Goal: Increasing mathematics achievement in the

Northeast region– RCT of the impact of student access to Algebra I

in grade 8– Descriptive study of math education practices for students with

disabilities in six schools in two Northeast states– Descriptive studies of math performance patterns for students

with disabilities in MA and NY


Our Process to Identify Topics

• Work from district/state goal to identify topics and narrow focus to two topics

• Identify subtopics and questions within top two topics• Prioritize questions [or subtopics]

19


Example: Gender in Schools Research district/stateGoal: Provide research that informs and promotes gender equity in schools

20

Topic Subtopic Questions TimingGirls in STEM Tracking achievement

differences

Encouraging interest and enrollment in STEM

Persistence in STEM majors

Attainment for BoysStructured Inequality


Moving from Subtopics to Questions

21

Subtopic Questions Timing

Tracking achievement differences

What are gender differences in STEM achievement in K-12 and how have they changed over time?Do gender differences in STEM achievement vary among districts and schools?


What can teachers do?Are single-sex schools and classrooms better?What programs help promote STEM for girls?



Moving from Subtopics to Questions

22

Subtopic Questions Timing

Tracking achievement differences

What are gender differences in STEM achievement in K-12 and how have they changed over time?Do gender differences in STEM achievement vary among districts and schools?


What can teachers do?Are single-sex schools and classrooms better?What programs help promote STEM for girls?


A fully specified research question

A general question that can be used to develop a set of research questions


23

District/State Goals


24

Goals & Research Topics

• Reflecting on our goals, brainstorm topics that are important to you and that fit under the goals.

• Record in your workbook on page 5.– If you are unsure whether a topic fits under the goal,

include it in the “possible” column of the page.• The ideas don’t need to be refined, but we will be

narrowing the list later in the session, so focus on topics that are most important to you.

• You will have 5 minutes to think and write by yourself.


25

Research Topics-Breakout Rooms

• You will be placed into a virtual breakout room to discuss the topics you brainstormed (15 minutes)

• Share your topics with your small group• On your breakout room whiteboard:

– Write the topics that you would like to bring back to the whole group for consideration as priority topics

• These topics will be brought back to the whole group, when the breakout time ends.


Grouping our topics

• Together, we will engage in a process of grouping the topics.

• Our identified topics are diverse but some are related.• What topics generated in the groups do you think go

together?• After we review the grouping of topics we will name the

topics.– The name becomes the “topic” and the items grouped under that

topic name are “subtopics”

26


27

Narrowing the List of Topics

• The Goal is to narrow the list of topics to 2 priority topics for our work—we will do this using a modified Focusing Four process– Brainstorm: We did earlier and grouped our ideas into topics– Clarify: We will review the groupings.

• We can clarify what any topic is, suggest moving topics or splitting groupings.

– Advocate: Think about narrowing to 2 topics that could comprise a good agenda when taken together.

– Canvass: You will each have 3 votes to use to indicate the topics that you think should be a research priority for our district/state.


28

Clarify

• Review the “groupings” of topics• Do any topics on any list need to be clarified?• Do the groupings make sense? Should we move any

items or split any groupings?• Can anything be combined or removed?


29

Advocate

• Think about which cluster of topics, when taken together, provide the best research direction for the district/state.

• Share your preferred priority topic and provide a few positive reasons why you think the topic grouping should be a priority for the district/state.


Break

5–10 minute break

30


31

Polls

• Vote once in each Poll to indicate your priority research topic for the district/state.– You can vote for the same topic in each poll if you like, or vote

for different topics.

• At the conclusion of the canvassing, we will review the results of our Polling and make some decisions about our 2 topics. – This will be your take-away from Workshop session 1.


32

Prepare for Next Session

• For one or both of the topics, think about:– What research questions interest you under the priority topic

areas? (these do not have to be at the level of researchable questions but they should be questions that are a finer grain of detail than the topics themselves). Email questions to facilitators before Session 2.

– Begin to think about the data sources that are appropriate to answering these questions and where they reside, if they exist.


33

Thank you

• Thank you for your participation and engagement today.• See you [INSERT DATE AND TIME OF NEXT

SESSION]!


34

Research Agenda Workshop Session 2

[District/State Name]

Session 1: [Date and time]Session 2: [Date and time]

[Facilitator Name]


Welcome and Purpose

Our purpose in this workshop series is to • Engage in a collaborative process to identify district/state

priorities, and • Develop a set of research questions that comprise a

coherent research agenda that will be the work of our district/state for next 3-5 years.

35


Goals for the Session

• Session 1 focused on reviewing different types of research, including levels of evidence, making causal claims, and linking questions to methods. We left with some prioritized topics.

• This session, we will:– Generate, refine and prioritize research questions, and

– Leave with an initial coherent research agenda

36


Agenda for the Session

• Review topics from session 1• Generate questions (review questions sent in by email

and generate additional questions)• Prioritize questions

– We will prioritize all the questions generated in our two topic areas.

• Generate preliminary research agenda• Next steps

37


38

Review Research Topics and Begin to Generate Questions

• Highlights of topics from last session and review of research questions sent in by email.

• In virtual breakout rooms, work on the white boards for 30–40 minutes to generate questions related to the priority topic (and any subtopics within it)

• Groups should think comprehensively within their topic/subtopic


39

Share Out Small Group Work/Refine Questions

• Re-group into whole group and share white boards from small group work.

• Discuss questions generated and refine by:– Edit, add, and/or delete questions– Check: any additional questions—is something

missing?


40

Prioritize Research Questions

• Focusing Four process– Brainstorm: Review our list of questions, generated earlier– Clarify: Do any questions on the list need to be clarified? – Advocate: Think about narrowing a few questions in each

topic/subtopic area that could comprise a good agenda

– Last step will be done after the break


Break

5–10 minute break

41


Poll

• Poll: Vote for the research questions. – Vote once in each Poll– You can vote for the same question in each Poll or

different questions

42


43

Preliminary Research Agenda

• Review identified priority topic, subtopics, and questions• How many top rated questions should be on the

agenda? Consider:– Types of work (data alignment, workshops, studies, etc.)– Sequence– Short- and long-term nature of work

• Remember…


44

Research Agenda- continued

• Refinement of the research agenda should continue over time.

• The agenda should: – encompass the work that is most important to the district/state, – Align with the district/state goal and– Be the work that the district/state will do over the next 3-5 years.


45

Next Steps

• Continued refinement of agenda• Involvement in prioritization and study development• Work to conduct research or identify research support• Use agenda to recruit research and funding partners for

this work.


46

Thank you

Thank you for your participation and engagement today.

Research Agenda Workshop

Documents

Transcript of Research Agenda Workshop