Connecting the Dots between Research Priorities and Stakeholder Needs: An Evaluation of Stakeholder...
Transcript of Connecting the Dots between Research Priorities and Stakeholder Needs: An Evaluation of Stakeholder...
Connecting the Dots between Research Priorities and Stakeholder Needs:
An Evaluation of Stakeholder Perceptions of theNSRC Grant Program
Curt Grimm, Ph.D.Barbara Wauchope, Ph.D.
Charlie French, Ph.D.Carsey Institute, UNH
Durham, NH
NSRC Leaders MeetingHubbard Brook Foundation
North Woodstock, NHOctober 11, 2011
Purpose of the Evaluation To assess the work of the NSRC from
the perspective of multiple stakeholders
1. NSRC grantees – Principal Investigators on research grants
2. Practitioners and others who partner with University grantees
3. Other practitioners, policy-makers, and citizens who have interest in/use for the research
Areas of Evaluation1. NSRC grantmaking process
2. Current research needs and priorities re: Northern Forest
3. Relevance and usefulness of the current NSRC research focus
4. Mechanisms grantees use to communicate research results
5. Communication/outreach strategies that might be more effective
Methodology:Data collection
Online Surveys:1. NSRC grantee Principal Investigators
(n=80)
2. Users or beneficiaries of NSRC-funded research – identified by the grantees in their survey (n=9)
3. Actual or potential users or beneficiaries of NSRC-funded research – subscribers to the Northern Forest Center Digest subscriber list (n=69)
Methodology:Resulting samples – PI Survey (n=80)
76.3%
5.0%
6.3%
2.5%5.0%
1.3% 1.3% 2.5% University/college
Research institute
State/Federal agency
Forest-serving org
Community non-profit
Business/industry
Non-affiliated
Other
Methodology:Resulting sample – Stakeholder Surveys combined (n=78)
5.2%
15.6%
13.0%
3.9%
28.6%
2.6%
15.6%
15.6%
Scientist/researcher
State/federal agency
Forest-serving org/inst
Forest-based bus/ind
Comm non-profit, econ dev org
Policy-maker
Citizen
Other
Methodology:Data collection
Telephone interviews:
Actual or potential users or beneficiaries of NSRC-funded research
– list generated by NSRC theme directors and Carsey staff (n=24)
Methodology:Resulting sample – Stakeholder interviews (n=24)
Policy-m
aker
Practitioner
NF resid
ent
Research
er
Advocacy
Philanthro
py
Education
0%
20%
40%
60%
80%
100%
Methodology:Data collection & resulting samples
Focus groups/informal interviews:
Northern Forest Higher Education Resource Network members (n=12) University/college presidents, deans,
faculty
Cooperative Extension county-based educators (n=4) Forest and wildlife staff
New Hampshire Local Government Center employees (n=2)
Key Findings Perceptions of NSRC are generally
positive Scientists and non-scientists have
somewhat different perspectives All agree on the need for both basic
and applied research • Scientists somewhat more supportive of
need for basic research
• Non-scientists overwhelmingly say science should address practical problems
Key Findings All four themes are relevant and
important but #1 is of strongest interest: the economy
Current dissemination strategy is not effective
E-mail-based and colleague-based outreach strategies, either your own or through existing networks, are likely to have more impact
NSRC GrantmakingPI experience with NSRC (n=80)
Number of awards: 0 – 6 Mean: 1.89 Median: 2
0%
20%
40%
60%
80%
100%
Completed = 42In-progress = 38
NSRC GrantmakingPIs: How they hear about Request For Proposal
Colle
ague(
s)
Gra
nt al
ert
Prof m
eeting
NSRC w
ebsite
Acad jo
urnal
, new
slet
ter
Oth
er0%
20%
40%
60%
80%
100%
NSRC GrantmakingPIs: Themes broad enough? (n=76)
Yes No Not sure0%
20%
40%
60%
80%
100%
NSRC GrantmakingPIs: Effectiveness of grantmaking process (n=76)
Publ
icizin
g RFP
Awar
d no
tifica
tion
NSR
C gr
ant a
dmin
RFP p
roce
ss
TA d
urin
g pr
opos
al
TA d
urin
g gr
ant
1.0
1.5
2.0
2.5
3.0
3.5
4.0
NSRC GrantmakingPIs: Effectiveness of mission (n=70-76)
1.0
1.5
2.0
2.5
3.0
3.5
4.0
NSRC GrantmakingPI Perceptions of the program Positive about program overall, including its
efficiency and effectiveness Promotes useful research relevant to the Northern
Forest region Promotes collaboration – between scientists,
across disciplines, with non-scientists Increases research on the Northern Forest Supports a broad scope of projects and disciplines Promotes interdisciplinary research
NSRC GrantmakingPI Perceptions of the program “Without this program there would be very little
research that directly deals with issues important to this region.”
“done well to fund a breadth of projects and expertise across disciplines and regions”
“ exceptionally well-positioned to allow researchers to leverage resources from other sources”
“great collaboration and synthesis within themes. Less opportunity to cross themes.”
NSRC GrantmakingPI Concerns
Grant focus Not relevant enough to Northern Forest residents/too
relevant – important scientific questions not funded Not enough social science
Limitations created by small size of grants On research topics imposed by themes On scope that’s possible and resulting impact On fostering grad students
Grantmaking process: Scientific review process not followed in some cases Poor communication to rejected applicants Bias in awards
NSRC GrantmakingPI Concerns
“projects in isolation fail to convey full impact – needs a vehicle for nesting and connecting to NSRC’s work in the big picture”
“funding pool is so small that it’s hard to develop a good collaborative, cross-disciplinary proposal”
“not all themes are equally "open" to competition”
“the themes divide up the pie and a proposal that crosscuts themes is disadvantaged”
“short proposal precludes effective review by panels who are quite mixed in discipline, depth, and breadth of knowledge”
NSRC GrantmakingStakeholder awareness of NSRC (n=76)
Yes No Not sure0%
20%
40%
60%
80%
100%
NSRC GrantmakingStakeholder understanding of NSRC (n=15)
What is the program focus: Research (14) Funding for research (11) Balance of applied/basic science research (3) Applied research (2) Basic science research (1)
Who NSRC serves: Broad range of stakeholders (8) Targeted audience (5)
NSRC GrantmakingStakeholder collaboration on grants (n=75)
Yes No Not sure0%
20%
40%
60%
80%
100%
Research Needs & Priorities: Perceived benefits of NSRC research
0%
20%
40%
60%
80%
100%
PIs (n=77)
Stakeholders (n=75)
Current NSRC Focus: Themes of present research reported by PIs
0%
20%
40%
60%
80%
100%
Research Needs & Priorities:Stakeholder interest in NSRC themes
1: Sustaining forest com-
munities
2: Sustaining ecosystem
health
3: Forest productivity
4: Biodiversity0%
20%
40%
60%
80%
100%
Current NSRC Focus:PI reported themes vs. Stakeholder interest
4
Theme 1
Theme 2
Theme 3
Theme 4
0%
20%
40%
60%
80%
100%
Stakeholders
Them
e 1
Them
e 2
Them
e 3
Them
e 4
0%
20%
40%
60%
80%
100%PIs
Research Needs & Priorities :Stakeholders: Topics of interest to be addressed:
North Fo
rest eco
nomy
North Fo
rest co
mmunities
Land use
change, c
onflcts
Outdoor recre
ation
Land co
nserva
tion
Forest
management
Forest
products in
dustry
Ecosys
tem servi
ces
Forest
science
Other0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Other PI-Perceived Research Needs and Priorities: Understanding local community need and
demand for biological data (and how to get them what they want)
Effects of forest habitat supply on featured wildlife species
Forest hydrology Forest – stream ecosystem connections
Other Stakeholder Perceived Research Needs and Priorities: Effects/impacts of renewable energy
development Climate change impacts on economy,
community, ecosystems…; and vice-versa Tourism’s potential to turn around the
current economic downturn Local food and agricultural systems impact
on economy and ecology Economic and ecological impact of
community forests
Stakeholders Emphasize Healthy Economy and Communities “What’s true in the rainforest is true in the
Northern Forest – we have to deal with the poverty”
“If we don’t have a healthy community, we won’t have a healthy forest. A healthy forest is a working forest”
“really integrative, holistic collaborative engaged work is a far better way to go. I would really like to see that value really written into the RFP and the grant proposal evaluation process”
Relevance and Usefulness of NSRC Foci: PIs: Relevance of NSRC Research by Audience
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Relevance and Usefulness of NSRC Foci: PIs: Do you use results of other NSRC research
Yes No Not sure0%
20%
40%
60%
80%
100%
Relevance and Usefulness of NSRC Foci: Stakeholder perceptions
“absolutely critical…it’s given us the basis of our economic development program”
“helpful in that it confirms other research that has been conducted “
“the one that I collaborated with was really valuable to my department, and my staff”
“great information for scientists and researchers” “able to expand my own work through channels
of NSRC project PIs”
Relevance and Usefulness of NSRC Foci Stakeholder perceptions (continued) “haven’t found them to be very relevant; too
theoretical and abstract … most of the work is ecologically focused and not relevant for communities and specifically for their economic growth”
“I can’t think of anything that has had a direct impact”
“I don’t feel really connected to the research to be honest … but I can see the potential”
“there doesn't seem to be an emphasis on application of the research in the real world”
“some of it, truthfully, just goes over my head”
Current Outreach & Dissemination:
PIs: Most important products of their research
Peer review jo
urnal
Meeting pre
sentation
Seminar, worksh
op
Collaborative partn
er
Technica
l assi
stance
Inform
ational websit
e
Best practi
ces m
anual
Network cr
eation
Policy docu
ment
Newspaper a
rticle
Trade publication
Testimony
Other0%
20%
40%
60%
80%
100%
Current Outreach & Dissemination:
PIs: Which resources are they using
Conf
pre
sent
atio
ns
NSR
C web
site
Peer
revi
ew jo
urna
l
Sem
inar
, wor
ksho
p, m
tg
Info
rmat
iona
l Web
site
New
spap
er A
rticle
Trad
e pu
blicat
ion
Best p
ract
ices
man
ual
Polic
y do
cum
ent
0%
20%
40%
60%
80%
100%
Current Outreach & Dissemination:
Stakeholders: Where they go for information
0%
20%
40%
60%
80%
100%
Current Outreach & Dissemination:
PIs: Where do they go to find NSRC research
Colleagu
e
Academic
journal
NSRC w
ebsite
Prof m
tg/co
nference
Prof s
eminar/work
shop
Newspaper a
rticle
Best pra
ct/mgm
t manual
Trade public
ationOth
er0%
20%
40%
60%
80%
100%
Current Outreach & Dissemination:
PIs: How effective are resources for communication
Peer review jo
urnal
Seminar, worksh
op, mtg
Conference
prese
ntations
Best practi
ces m
anual
Newspaper A
rticle
Policy docu
ment
Inform
ational Websit
e
Trade publication
NSRC websit
e1.0
1.5
2.0
2.5
3.0
3.5
4.0
Current Outreach & Dissemination:
Stakeholders: Average likelihood of using source
1.0
1.5
2.0
2.5
3.0
3.5
4.0
I rely on my colleagues to stay up to date with the latest research findings. That is where I learned about NSRC.
- Focus Group Participant, Forestry Practitioner
Current Outreach & Dissemination:
PIs: Barriers to dissemination of their research
0%
20%
40%
60%
80%
100%
Current Outreach & Dissemination:
Stakeholders: Barriers to obtaining research
Info doesn't g
et to m
e
Not enough
time
Info difficu
lt to acce
ss
Topics not o
f interest
Other
0%20%40%60%80%
100%
Current Outreach & Dissemination:
Stakeholders: Other barriers to obtaining research
Information is not always summarized well into one or a few sources (it's all over the place)
Restricted travel to meetings Cost to obtain peer reviewed articles “Lack of easy access to information about
the grants and the project findings is a challenge, as is lack of time to sort through lots of information”
Current Outreach & Dissemination:
Stakeholders: Effectiveness of NSRC dissemination Some potential audiences are not aware of
NSRC ‘Forestry practitioners’ and forest-serving
institutions typically hear about NSRC research findings through word-of-mouth.
There is no mechanism to drive people to NSRC’s website.
All 14 of those interviewed who knew of NSRC said dissemination was ineffective
Alternative Dissemination Strategies: PI Suggestions Online access to publications and perhaps
even data collected for each project Issue a NSRC press release / press
conference after each project year, with a bit about each project
More face-to-face contact with research users/stakeholders (but that takes time and resources)
develop an outreach document summarizing project results and management implications
Alternative Dissemination Strategies: PI Suggestions (continued)
Work harder with local stakeholder groups such as environmental managers
More informal outlets (workshops for professionals, articles in general interest magazines or newspapers)
More informal public presentations and seek co-sponsorship with aligned local organizations
A more interactive and content-rich website
Alternative Dissemination StrategiesStakeholder suggestions
NSRC should actively reach out to practitioners and other stakeholders.
Use Northern Forest Center’s News Digest to disseminate key findings and links
E-mail announcements with links to websites
Better use of forest-serving organizations’ websites to share research and grant info
Alternative Dissemination StrategiesStakeholder suggestions (continued)
Ensure that info being shared is up-to-date. Keep summary of findings short and simple
and link to a website for more information. Just getting an email saying, “hey, we
updated the project list and here is what is going on across the region” would be helpful… don’t even need to send the project abstracts… just knowing that they have been updated is helpful.”