2021 Research Trainee Evaluation Criteria

2021 RESEARCH TRAINEE COMPETITION

EVALUATION CRITERIA

1

2021 Research Trainee Evaluation Criteria

All 2021 Research Trainee applications are assessed against a defined set of criteria in three categories: 1) track record, 2) program of research, and 3) environment and support. To be considered for funding, the application must receive a minimum score of 3.8.

Applications are evaluated via a process that incorporates six principles of peer review: integrity, accountability, transparency, balance, confidentiality, and impartiality. For our 2021 competitions MSFHR will also provide peer reviewers with additional guidance on how to take COVID-19 impacts into consideration as part of the review process.

KT science applications will be evaluated against a set of KT science evaluation criteria in addition to standard Research Trainee award evaluation criteria.

Evaluation Criteria

Criterion Weighting

PhD Health Professional (HP)1

Track Record 45% 40%

Research Project 35% 40%

Environment and Support 20% 20%

Rating Scale

Descriptor Range Outcome

Outstanding 4.5 – 4.9 May be funded

(min. score of 3.8) Excellent 4.0 – 4.4

Very good 3.5 – 3.9 Fair 3.0 – 3.4

Not fundable Less than adequate 0 – 2.9

1 Evaluation criteria weightings for health professional applicants have been adjusted to reflect differences in training experiences between health professional and PhD stream researchers, which can affect the content of their track record as it relates to research.


EVALUATION CRITERIA

2

2

Track Record Weighting: 45% (HP 40%)

Assessment Criteria Scientific productivity, originality and potential for impact should be considered within the context of the norms of the research area and the applicant’s career stage. Research Trainees should be making important contributions to their field, and demonstrate exceptional promise for the future. Elements to consider when evaluating the applicant’s track record include:

• The relevance, importance and breadth of the applicant’s significant contributions. • The receipt of special distinctions, awards, presentations, or other acknowledgements

relative to education, training, and work experience. • Demonstration of professional leadership and accomplishments in research and

research-related activities, including engagement in knowledge translation activities appropriate to the research (e.g., conference presentations, workshops, etc.).

• The applicant’s scientific productivity throughout their training, including primary authorship of peer-reviewed publications or other literature, as appropriate for their career stage and experience. Consider the number of co-authors and the position of the applicant’s name in the authors’ list (this can vary across disciplines), and the relative potential impact of the publications.

o Note that journal impact factor can vary across disciplines and may not reflect the relative importance of the publication within the context of the applicant’s field of research.

• Referee comments on the applicant’s academic strengths, research skills, level of independence, and their perceived creativity and critical thinking skills. Consider how long the referees have known the applicant and the nature of their relationship.

Additional criteria for KT science projects only: • The applicant’s record of KT activities, including non-traditional methods of disseminating

and/or taking up research findings (e.g., development and implementation of toolkits, blogs, guidelines, books, tutorials, podcasts, videos, etc).

• Referee comments about the applicant’s ability to work with diverse research user groups.

Assessment Descriptor Score

Outstanding • The applicant’s significant contributions are impressive, impactful and hold substantial

weight within their field of research. • There is evidence of a high number of special distinctions, awards, presentations, or

other acknowledgements. • The applicant provides exceptional examples of professional leadership and

accomplishments in research and research-related activities, including clear engagement in KT.

• The applicant’s publications indicate an extraordinary level of productivity relative to their respective research discipline and career stage.

4.5 – 4.9

May be funded


EVALUATION CRITERIA

2

3

• The referees are unanimous in their support of the applicant and provide detailed assessments of their academic strengths, research skills, level of independence, creativity and critical thinking abilities.

• For KT science research project only: The applicant has an impressive record of traditional and non-traditional KT-related activities, and is exceptionally adept at working with diverse research user groups.

Excellent • The applicant’s significant contributions are important and well-regarded within their

field of research. • There is evidence of several special distinctions, awards, presentations, or other

acknowledgements. • The applicant provides strong examples of professional leadership and

accomplishments in research and research-related activities, including appropriate KT activities.

• The applicant’s publications indicate a high level of productivity in their respective research discipline and career stage.

• The referees provide very strong support for the applicant and clearly describe their academic strengths, research skills, level of independence, creativity and critical thinking abilities.

• For KT science research project only: The applicant has a strong record of traditional and non-traditional KT-related activities, and is experienced in working with diverse research user groups.

4.0 – 4.4

May be funded

Very Good • The applicant’s significant contributions are notable and somewhat influential within

their field of research. • There is evidence of some distinctions, awards, presentations, or other

acknowledgements, but fewer in number or from less competitive sources compared to other applicants.

• The applicant provides some notable examples of professional leadership and accomplishments in research, research-related activities, and KT.

• The applicant’s publications indicate a moderate level of productivity in their respective research discipline and career stage.

• The referees are mostly strong in their support of the applicant and provide a broad assessment of their academic strengths, research skills, level of independence, creativity and critical thinking abilities.

• For KT science research project only: The applicant has a moderate record of traditional and non-traditional KT-related activities, and has experience working with diverse research user groups.

3.5 – 3.9

May be funded (above

3.8)

Fair • The applicant’s significant contributions are relatively modest with limited impact

within their field of research.


EVALUATION CRITERIA

2

4

• There is some evidence of distinctions, awards, presentations, or other acknowledgements, but they are limited, and mostly restricted to local or non-competitive sources.

• The applicant provides few examples of professional leadership and accomplishments in research, research-related activities, or KT activities.

• The applicant’s publications indicate a modest level of productivity in their respective research discipline and career stage.

• The referees are mixed in their support of the applicant, providing only a limited assessment of their academic strengths, research skills, level of independence, and creativity and critical thinking abilities.

• For KT science research project only: The applicant has a limited record of traditional and non-traditional KT-related activities, and some experience working with a few similar research user groups.

3.0 – 3.4

Not fundable

Less than Adequate • The applicant’s significant contributions are relatively commonplace with no real

impact on or influence within their field of research. • There is little-to-no evidence of notable or relevant distinctions, awards, presentations,

or other acknowledgements. • The applicant provides no suitable examples of professional leadership and

accomplishments in research, research-related activities, or KT activities. • The applicant’s publications indicates a lack of suitable productivity relative to their

research discipline and career stage. • The referees are ambiguous in their support of the applicant, providing limited,

impersonal assessments of their academic strengths, research skills, level of independence, and creativity and critical thinking abilities. For KT science research project only: The applicant has no record of traditional and non-traditional KT-related activities, and is relatively inexperienced in working with research user groups.

0 – 2.9

Not fundable

Research Project Weighting: 35% (HP 40%)

Assessment Criteria The applicant’s proposed research project should cover the full duration of the award (up to three years). Elements to consider in assessing the research project include:

• Design, rationale, methods and analyses are adequately developed, well-integrated, innovative, and relevant to the aims of the research project

• The project is feasible and aligns with the proposed timeline. • The project addresses a need or gap in health research and/or the health care system. • The proposed research project as a whole is aligned with the applicant’s skills and abilities. • Knowledge dissemination and exchange activities are relevant and soundly designed, both within and

beyond the academic community, and appropriately engage with relevant research users.


EVALUATION CRITERIA

2

5

Additional criteria for KT science research project: • The project clearly describes how the dissemination or implementation of research findings is being

studied, addresses the study of both the methodology and the outcomes of the KT research, and adds to the body of knowledge on effective KT.


Outstanding • The research project is highly innovative and original, presents leading-edge concepts

and methodology, and provides a very clear description of hypotheses and experimental rationale. Overall, the proposal is of a caliber for an applicant who is at the forefront of their field.

• The feasibility and projected timeline for completing the research project is straightforward and highly realistic.

• The project addresses an extremely important need or gap within health research and/or the healthcare system, with strong potential for significant impact.

• The research project is perfectly aligned with the applicant’s skills and abilities. • The applicant provides ample evidence of appropriate and effective KT activities, both

within and beyond the academic community, and is regularly engaged with relevant research users.

• For KT science research project only: The proposed research project provides a clear plan on how findings will be implemented and/or disseminated, clearly addresses the processes and outcomes through a coherent and effective framework, and will be an invaluable addition to the body of knowledge on effective KT.

4.5 – 4.9

May be funded

Excellent • The research project is high quality, presents concepts and methodology that are

pertinent and highly innovative, and provides a clear description of hypotheses and experimental rationale.

• The feasibility and projected timeline for completing the research project is clear and realistic.

• The project addresses a significant need or gap within health research and/or the healthcare system, with potential for significant impact.

• The research project is strongly aligned with the applicant’s skills and abilities. • The applicant provides strong evidence of appropriate KT activities, both within and

beyond the academic community, and is engaged with relevant research users. • For KT science research project only: The proposed research project provides clear

detail on how findings will be implemented and/or disseminated, addresses the processes and outcomes through a coherent and effective framework, and will be a strong addition to the body of knowledge on effective KT.

4.0 – 4.4

May be funded

Very Good • The research project is good quality, presents pertinent concepts and methodology,

generally supports the goals of the proposed research, and provides a suitable description of hypotheses and experimental rationale.


EVALUATION CRITERIA

2

6

• The proposed research is generally feasible with only minor concerns regarding the projected timeline for completion.

• The project addresses a key need or gap within health research and/or the healthcare system, with potential for some impact.

• The research project is mostly aligned with the applicant’s skills and abilities. • The applicant provides some evidence of appropriate KT activities, both within and

beyond the academic community, and periodically engages with research users. • For KT science research project only: The proposed research project provides evidence

on how findings will be implemented and/or disseminated, addresses most of the processes and outcomes through an appropriate framework, and will be a useful addition to the body of knowledge on effective KT.

3.5 – 3.9

May be funded (above

3.8)

Fair • The research project is adequate quality, presenting predictable concepts and

methodology that only partially support the goals of the proposed research, and provides only a generalized overview of hypotheses and experimental rationale.

• The feasibility and projected timeline of the proposed research is adequate, but could use more detail.

• The project addresses a narrow need or gap within health research and/or the healthcare system, with potential for limited impact.

• The research project is only partially aligned with the applicant’s skills and abilities. • The applicant provides few examples of KT activities, mostly within the academic

community, and has limited engagement with research users. • For KT science research project only: The proposed program of research provides some

detail on how findings will be implemented and/or disseminated, addresses the processes and outcomes through a general framework, and will likely have a limited impact on the field.

3.0 – 3.4

Not fundable

Less than Adequate • The proposed research may be valid but its experimental design and methodologies are

flawed, unclear and underdeveloped, and the hypotheses and experimental rationale are inadequately described.

• The feasibility and projected timeline for completing the research project is inadequate and unrealistic.

• The project does not address a suitable need or gap within health research and/or the healthcare system.

• The research project is generally incompatible with the applicant’s skills and abilities. • The applicant does not provide adequate examples of KT activities, and has little-to-no

engagement with relevant research users. • For KT science research project only: The proposed research proposal provides no

evidence on how findings will be implemented and/or disseminated, fails to address the proposed research outcomes and does not contribute to the body of knowledge on effective KT.

0.0 – 2.9

Not fundable


EVALUATION CRITERIA

2

7

Environment & Support Weighting – 20%

Assessment Criteria Please refer to the Career Development and Training Environment sections and Supervisor(s) Approval Form to assist in rating the appropriateness of the applicant’s environment and support. Elements to consider when evaluating an applicant’s training environment and support include:

• The supervisor(s) research experience and qualifications, research funding and resources available to support trainees, and training history relative to their career stage.

o Note: The expectations of a recently established independent researcher should differ from a long-established one, and should be assessed accordingly.

• The level of support, access to research infrastructure (e.g., technology and resources) and leadership available in the training environment, noting that these will vary depending on institutional characteristics and the applicant’s area of research

• The fit between the applicant’s proposed research project and the supervisor(s)’s overall research program

• The support provided and potential for professional development (e.g., appropriate infrastructure, interlaboratory collaboration, specialized training, participation in conferences, etc.) and mentorship available in the training environment, recognizing that the availability and nature of these opportunities will vary according to institutional characteristics.

• How the applicant’s proposed research project and expected training complements their desired career pathway.


Outstanding • The supervisor(s) demonstrates an impressive research track record, possesses ample

research funding and resources for trainees, and has an exceptionally successful prior track record of researcher training, relative to their career stage.

• The training environment is exceptionally well supported and provides access to the leadership, resources and technologies needed to excel in the applicant’s area of research.

• The applicant’s proposed research project and the supervisor’s overall research program are completely complementary.

• The training environment provides ample opportunity for the trainee to participate in meaningful professional development activities and receive exceptional mentorship.

• The applicant’s proposed research project and the expected training perfectly complements their desired career pathway.

4.5 – 4.9

May be funded

Excellent • The supervisor(s) demonstrates a strong research track record, possesses the

necessary research funding and resources required for trainees, and has a successful prior track record of researcher training, relative to their career stage.

• The training environment is well supported and provides all of the necessary technology, resources and leadership needed for the applicant’s area of research.

4.0 – 4.4

May be funded


EVALUATION CRITERIA

2

8

• The applicant’s proposed research project and the supervisor’s overall research program are almost completely complementary.

• The training environment provides many opportunities for the trainee to participate in professional development activities and receive strong mentorship.

• The applicant’s proposed research project and the expected training significantly complements their desired career pathway.

Very Good • The supervisor(s) has a reputable research track record, possesses suitable research

funding and resources for trainees, and has a mostly successful prior track record of researcher training, relative to their career stage.

• The training environment provides most of the necessary technology, resources and leadership needed for the applicant’s area of research.

• The applicant’s proposed research project and the supervisor’s overall research program are reasonably well-matched.

• The training environment provides some opportunity for the trainee to participate in professional development activities, and general access to relevant mentors.

• The proposed research project and expected training generally complements the applicant’s desired career pathway.

3.5 – 3.9

May be funded

(above 3.8)

Fair • The supervisor(s) possesses a modest research track record, possesses partial

research funding and resources needed for trainees, and a limited history of successful training of trainees, relative to their career stage.

• The training environment provides some of the necessary technology, resources and leadership needed for the applicant’s area of research.

• The applicant’s proposed research project and supervisor’s overall research program are related, but have some incongruences.

• The training environment provides few opportunities for the trainee to participate in professional development activities, with some indirect access to mentors.

• The applicant’s proposed research project and expected training appear to only partially complement the applicant’s desired career pathway.

3.0 – 3.4

Not fundable

Less than Adequate • The supervisor(s) demonstrates a research and funding track record unsuitable for

the applicant, and a poor record of successfully training trainees, relative to their career stage.

• The training environment does not provide the necessary technology, resources or leadership needed for the applicant’s area of research.

• The applicant’s proposed research project and supervisor’s overall research program are generally unrelated.

• The training environment provides no real opportunities for the trainee to participate in professional development activities or receive mentorship.

• The applicant’s proposed research project and expected training does not complement their desired career pathway.

0.0 – 2.9

Not fundable

2021 Research Trainee Evaluation Criteria

Documents

Transcript of 2021 Research Trainee Evaluation Criteria