Foundation Scheme Pilot 1:What? Who? When? How?

Lori Burrows, PhDCIHR University Delegate

June 12, 2014

What?

First Pilot competition of CIHR’s new Foundation Scheme: 5-7 year grants aimed at scientific LEADERS.

If successful, you cannot hold other CIHR open grants as PI during the span of your Foundation grant. You can be a co-investigator on other Foundation or Project grants. Existing grants will be rolled into your Foundation grant. If unsuccessful, you WILL NOT LOSE EXISTING GRANTS.

Many elements of this competition remain uncertain, because they are still in development and may yet change based on the results of ongoing pilots.

example of ‘still in development’

“The adjudication criteria considered at Stage 1 and 2 are different. A detailed description of these criteria can be found within the CIHR Adjudication and Selection Process Manual for the 2014 Foundation Scheme Competition. (link to come)”

Money available for 2014/15The combined total amount available for 2014-15 Transitional OOGP and Foundation Pilot is ~$500M. The two competitions overlap for the first time…unclear how this will play out in terms of total number funded.

120 to 250 Foundation grants may be funded; ‘most Foundation grant budget requests will fall within a range of $50K to $1.5M per annum’.

The budget request should be consistent with the applicant's previous CIHR open grant research funding history.

TIP: You can point out that your historic request was higher than the amount you actually got, due to ATB cuts.

Who?You must fit CIHR’s ‘independent researcher’ definition &

a) be the PI or co-PI on an open CIHR grant that ends between Oct 1, 2014 and Sept 30, 2015, orb) be an established investigator (>5 yr since 1st appt) who has never held a CIHR open grant as PI or co-PI, orc) be a new investigator (<5 yr since 1st appointment); leaves of absence aren’t included, and part time appointments will be pro-rated up to the maximum of 60 months post-appointment

CIHR will verify your eligibility at registration.

What are defined as ‘open’ grants?

Open Operating Grant Program (OOGP)Partnerships for Health System Improvement (PHSI)Knowledge Synthesis Grant (KRS)Knowledge to Action Grant (KAL)Proof of Principle Program Phase I and II (POP I and POP II)Industry-Partnered Collaborative Research (IPCR)

New names for PI and co-IProgram Leader: individual responsible for setting and overseeing the intellectual and strategic direction of the research. Non-Canadian PLs must have Canadian co-Leader.Program Leader – Administrative Coordinator: must be a PL, at Canadian institution; assumes all responsibility for administrative matters related to the grant including submitting the application and coordinating any required reportingProgram Expert: participates in or contributes to the program of research but does not direct the program; may or may not be involved for the entire length of the grant; may or may not be named on the grant at discretion of PL

TIP: CVs of Experts are not included or assessed in Foundation applications, and they are not named in funding decisions. They can document their involvement in Foundation grants in their CVs, reports, etc., as agreed upon with the Program Leader(s), and they can apply for Project grants.

When? Registration – June 23, 2014

Stage 1 Application – Sept 15, 2014Stage 1 Decision – Dec 1, 2014

Stage 2 Application – Feb 5, 2015Stage 2 Decision – May 15, 2015

Stage 3 Decision – July 2, 2015

Funding Start – July 1, 2015 (yes, before the stage 3 decision…)

How?1. Register via ResearchNet by June 23, 8 pm2. If you pass the eligibility test, CIHR will send a link for the

stage 1 application, due Sept 15, 2014, 8 pm3. If unsuccessful at stage 1 (decision in Dec 2014), you can

apply to the Spring 2015 Transitional OGP, registration deadline Jan 2015.

4. If successful at stage 1, you can apply to stage 2 OR to the Spring TOGP…not both.

TIP: Depending on the comments at stage 1, and projected success rates at stages 2 and 3, you may want to consider whether you go forward, or bail out to the TOGP. If you go forward to stage 2 and are unsuccessful, you will have to wait until Fall 2015 to re-apply, with no money until July 2016 even

if you succeed the second time.

choose to

skip FdnMar/14OOGP

Mar/15transitional

OOGPMar/16Project

Sept/16Project

Sept/14FdnPilot

Sept/15FdnPilot

Sept/16Fdn

Sept/14stage 1

(Dec/14)

Feb/15stage 2

(May/15)

May/15stage 3

(July/15)

Sept/15stage 1

(Dec/15)

Feb/16stage 2

(May/16)

May/16stage 3

(July/16)

YES (13%?)

NO (87%?) NO (87%?)

YES (13%?) YESNO

$ Oct/14 $ July/15

decided not to go forward to stage 2

NO

YES YES

MAYBE MAYBE

$ July/15YES

YES

NO –REAPPLY

$ July/16YES

$ July/15

$ July/16decided not to go forward to stage 2

NO –REAPPLY

NO –REAPPLY

NO –REAPPLY

YES

$ July/16

2014 2015 2016

YES

$ April/17

chose to

renew early

choose to

renew early

choose to

skip Fdn

How do I register?

http://www.cihr-irsc.gc.ca/e/48053.html

You need a ResearchNet account, a CIHR PIN and a Common CV account.

the Common CV

• fill out the CIHR version of the Common CVTIP: this can take significant time!

• to register for the pilot, pick CIHR as the agency and ‘registration’ for the format

• validate it and and click ‘submit’ (it doesn’t go anywhere)

• the system will generate a confirmation number that needs to be entered into the registration application on ResearchNet

• if you change anything, you need to repeat this process, the confirmation number will change

on ResearchNet…

• Under ‘Apply for Funding’, pick ‘Foundation Scheme….’ and follow the instructions.

• The Program Leader(s) must not change between registration and application.

• You must provide a 5 (new investigators) or 7 yr budget estimate. You don’t need to justify at this point, and you can change it at stage 2. TIP: For new people, look at CIHR’s funding database for historic amounts handed out for the type of work you’re proposing to do

• Suggest reviewers. This is critical, as these people will be recruited into the College of Reviewers. Exclude reviewers you don’t want.

What’s my competition like?

• as of June 6, 43 registrations completed, 874 in progress

• 305 are current grant holders; 278 have never held CIHR grants as PI (might be some overlap); 267 are new investigators and 67 are undefined

• includes registrations opened on spec…but CIHR expects more at the last minute

• CIHR is expecting ~1200 bona fide applicants, despite high numbers of registrants (since many will register but not submit). But, they are surprised at the new investigator numbers so far…

Stage 1 application (in development)

Detailed instructions (soon) available on ResearchNet under ‘Opportunities’, ‘Foundation

Scheme’.

“To complete your Stage 1 Application, follow the instructions in the Foundation Scheme: 1st Live Pilot ResearchNet

"Application" Phase Instructions. (link to come)”

Stage 1 application content

• Program Leader(s) will need to generate a Foundation Scheme CV using the Common CV

• SHORT stage 1 application has 3 sections:– Summary (1 page)– Caliber of the Applicant (1.5 pages, 75%)

• -Leadership (1/2 page, 25%)• Significance of contributions (1/2 page, 25%)• Productivity (1/2 page, 25%)

– Vision and Program Direction (1 page, 25%)

Program Leader’s CV

http://www.cihr-irsc.gc.ca/e/48206.htmlSee website for details of what is to be included. Only select contributions, with maximum numbers per section.

e.g. Publications (Most relevant – up to 25)‘Program Leaders will provide information regarding their publications in the form of journal articles, books, book chapters, reports, manuals, clinical care guidelines, and/or conference publications’.

Summary (1 page)

• summarize research experience and vision of the research program:– overarching focus of research career, major questions or

issues explored, and the impact of the research;– expertise and experience;– Important collaborations, within or outside of the research

community; and– vision or direction of the program of research to advance

knowledge and/or its application to health care, health systems, and/or health outcomes.

TIP: the Summary is a MARKETING DOCUMENT – SELL YOURSELF!

Leadership (1/2 page)

• highlight significant and effective leadership experience, reputation in your field and evidence that you’ve led major projects or research programs – point to examples in your CV

• reviewers are asked if you are widely recognized, influential, community-building; if you have established, resourced, directed major efforts including training, infrastructure development, collaborations

Significance of Contributions (1/2 page)

• highlight your contributions and their impact, tangible benefits or positive influences – give examples from your CV

• reviewers are asked if you have significantly advanced knowledge and/or its translation into health care/systems/outcomes; and if you’ve engaged/trained/launched the careers of promising individuals in research or health-related areas

TIP: although the sections may have overlap in terms of what you point to, CIHR thinks that true Leaders will have so many examples of their

awesomeness, they won’t need to repeat any of them

Productivity (1/2 page)

• highlight your productivity – quality and quantity of outputs, with examples from CV

• reviewers are asked if you have demonstrated an outstanding level of research outputs based on prior work; if your previous work generated high quality research outputs

TIP: if there is more than one Program Leader, their joint productivity will be considered

Vision and Program Direction (1 page)

• articulate a compelling vision and direction for the program…highlighting the goal, overall objective(s), expected outputs/contribution(s), and significance of the proposed program. Specific details for each project in the program are not expected, because the grants are meant to be flexible, to allow innovation and new directions.

• reviewers are asked if program is well-defined and well-articulated, logical career progression for the Program Leader(s)? forward-looking, creative, and appropriately ambitious? Does the vision aim to significantly advance knowledge and/or its translation?

Stage 2 application• get your stage 1 reviews and an invitation to go forward (or

not)• the stage 2 application is 12 pages total plus ½ page of

budget justification• will include:

– summary (1 page)– quality of program – 40%

• research concept (3 pages, 20%)…coherence, significance• research approach (2 pages, 20%)…appropriate, flexible

– quality of expertise, experience, resources – 60%• expertise (3 pages, 20%)…who are your Experts?• mentorship and training (2 pages, 20%)…comprehensive plan• quality of support environment (1 page, 20%)…appropriate

– budget justification (1/2 page)

Peer review of your stage 1 applicationThis critical component remains under development.

You will suggest (and exclude) potential reviewers upon registration. CIHR is shooting for 5 reviewers per application; some may be selected from existing peer review committees. Eventually, reviewers will be members of the College of Reviewers.

Each reviewer will have up to 20 applications, and after scoring the criteria (and resolving discrepant scores via online discussion with dissenting reviewers), will rank the applications (no ties allowed). Each application will have a combined rank that is an average of 5 individual rankings, and the combined ranks are used to make decisions at stage 1.

Right now, estimates are that the top ~350 may go forward to stage 2.

Pilot of the Structured Review Forms: Adjudication ScaleStage 1 Reviewers

Stage 2 Reviewers

Stage 1 Reviewers stated that they used the full range of the adjudication scale; however, Stage 2 Reviewers disagreed

Although the majority of Stage 1 Reviewers stated that the adjudication scale allowed the description of meaningful differences, their comments indicated that this was not true

Limitation of Stage 2 Reviewer comments is that they only saw the top 39 (of 77) applications

Distribution of Ratings: Letter Scale vs. Current Number System

  % A’s Given out by Reviewers

% B’s Given out by Reviewers

% C’s Given out by Reviewers

% D’s Given out by Reviewers

% E’s Given out by Reviewers

Quality of Idea 41.15% 35.16% 16.93% 4.69% 2.08%

Importance of Idea 35.68% 39.84% 17.19% 5.99% 1.30%

Approach 24.74% 34.90% 23.44% 13.80% 3.13%

EER 42.45% 39.84% 12.50% 4.43% 0.78%

Average 36.01% 37.44% 17.52% 7.23% 1.82%

*Scientific Merit and Potential Impact based on Spring 2013 Knowledge Synthesis competition consensus score

  4.5 – 4.9(A)

4.0 – 4.49(A)

3.5 – 3.99(B)

3.0 – 3.49(C)

≤ 2.99(D/E)

Scientific Merit 16% 20% 35% 24% 6%

Potential Impact 29% 29% 25% 12% 4%

Final Score 12% 13% 30% 34% 10%

Different scale (ABCDE is too close to letter grades in school)? Could also be a problem for international reviewers as the letters mean different things in different countries.

Add granularity to the scale (e.g., A+, A, A-)? 5 letters is interpreted as a 20% difference between letter grades and this is not sensitive enough to capture subtle differences between applications;

25%

new:A+B73.5%

old:A+B55%

Stage 1 - Preliminary Reviews

28

• A little more than half of the Stage 1 Reviewers read other reviewer’s reviews

• Reading other reviewer’s reviews did not often influence a reviewer’s assessment of the application

• In future surveys, this question will be re-framed to:– Ask reviewers whether the ability to read other reviewers’ reviews was

useful/helpful– Differentiate the responses between reviewers who had divergent

ratings/comments and those who had similar ratings/comments

Stage 1 Asynchronous Online Discussion

75% of reviewers did not participate in online

discussions…

…and not because there was nothing to say. Almost half

were not finished with their reviews in time.

Reviewers felt their opinion was not very influential, but about half thought others could at least

occasionally change their assessment. (Not clear if these were

the 25% that did participate).

Stage 2: Face-to-Face Meeting

All of the applications were listed in rank order on a monitor (the top half

from Stage 1) and a funding cutoff line was indicated, so people had an

idea of which apps needed discussion

Most thought the face-to-face meeting

was necessary

Applications were grouped into A (green zone), B (grey zone) and C

(red zone), and the discussion centered on group B apps.

Overall Satisfaction with Structured Review Process

31

Applicant satisfaction with the structured review process correlated with the state of the application (funded, moved to Stage 2, unsuccessful after Stage 1)

Most Stage 1 Reviewers were satisfied with review process

Most Stage 2 Reviewers were satisfied with review process

Registration – June 23, 2014Stage 1 Application – Sept 15, 2014

Stage 1 Decision – Dec 1, 2014

If you have questions, comments or concerns, email me!burrowl@mcmaster.ca

TOGP registration – Jan 2015TOGP application – Mar 2015

Stage 2 Application – Feb 5, 2015Stage 2 Decision – May 15, 2015Stage 3 Decision – July 2, 2015