HORIZON 2020 The Evaluation Procedure in

Practice

L'évaluation des projets H2020 du Défi 5

Paris – 17/10/2016

HORIZON 2020

2

Horizon 2020 - Time to grant - 8 months

Horizon 2020: Time to grant is maximum 8 months

= 8 months between the deadline for submission of proposals and the signature of grants for successful proposals

= evaluations + grant preparation

All the steps are supported by IT tools with a single entry point for the participants and the experts: the Participant portal

HORIZON 2020

3

Horizon 2020 – No negotiation – Impacts on the evaluations

Evaluate each proposal as submitted - not on its potential if certain changes were to be made

• If identified shortcomings (other than minor ones and obvious clerical errors), reflected in a lower score for the relevant criterion

• No recommendations i.e. do not suggest additional partners, additional work packages, resources cut…

• Proposals with significant weaknesses that prevent the project from achieving its objectives or with resources being seriously over-estimated must not receive above-threshold scores

• Any proposal with scores above the thresholds and where there is sufficient budget will be selected as submitted

− Successful applicants are invited to address shortcomings

HORIZON 2020

4

Overview of the Evaluation Process

Receipt of proposals

Individual evaluation

Consensus group

Panel Review Finalisation

Evaluators

Individual Evaluation Reports

(Usually

done remotely)

Consensus Report

(May be done

remotely)

Panel report

Evaluation Summary Report

Panel ranked list

Eligibility check

Allocation of proposals to evaluators

Final ranked list

HORIZON 2020

5

H2020 – Experts' pivotal role

Experts, as peer reviewers, assist in the:

• evaluation of proposals where they play a pivotal role

• monitoring of actions

In addition, experts assist in the :

• preparation, implementation or evaluation of programmes and design of policies. This includes the Horizon 2020 Advisory Groups.

HORIZON 2020

6

Identification of experts

Identification of the experts starts before the closure of the call

We look for

a high level of skill, experience and knowledge in the relevant areas

and, provided the above condition can be satisfied, a balance in terms of:

• skills, experience and knowledge

• geographical diversity

• gender

• where appropriate, the private and public sectors, and

• an appropriate turnover from year to year

HORIZON 2020

7

Identification of experts

• At least 3 experts/regularly 5 experts

• Via the EMI data base (experts register via the Participant portal)

• Through key words

• Check availability

• Preliminary experts lists with main and reserve experts

General comments

• Importance to update the profiles regularly (experience and key words)

• Need to constantly include new expertise

• Lack of experts from - local, regional and national administration, including environmental and civil protection agencies - Industry, business associations and innovation agencies

HORIZON 2020

8

Allocation of the proposals

After the closure of the call – composition of the panels of experts

• Exclude conflict of interest

• Check availability on the basis of the final dates notably for the central phase

• Ensure complementarity: expertise, nationality, gender

• This is only then that we can confirm that the experts will be involved in the evaluations process

HORIZON 2020

9

Conflicts of interest (COI) (1)

You have a COI if you: • were involved in the preparation of the proposal (including pre-

proposal checks)

• stand to benefit directly/indirectly if the proposal is successful

• have a close family/personal relationship with any person representing an applicant legal entity

• are a director/trustee/partner of an applicant or involved in the management of an applicant's organisation

• are employed or contracted by an applicant or a named subcontractor

• are a member of an Advisory Group or Programme Committee in an area related to the call in question

• are a National Contact Point or are directly working for the Enterprise Europe Network

HORIZON 2020

10

Conflicts of interest (COI) (2)

You have a COI if you:

• In the following situations, the Commission/Agency will decide whether a COI exists

− Were employed by an applicant or sub-contractor in the last 3 years

− Were involved in a grant agreement/decision, the membership of management structures or a research collaboration with an applicant in the last 3 years

− Are in any other situation that casts doubt on your impartiality or that could reasonably appear to do so

Example: If you are involved in a competing proposal

COI conditions are spelled out in your contract, and in the Code of Conduct (Annex 1)

HORIZON 2020

11

Conflicts of interest (COI) (3)

• You must inform the Commission/Agency as soon as you become aware of a COI

− Before the signature of the contract

− Upon receipt of proposals, or

− During the course of your work

• If there is a COI for a certain proposal you cannot evaluate it

− Neither individually

− Nor in the consensus group

− Nor in the panel review

− The Commission/Agency will determine if there is a COI on a case-by-case basis and decide the course of action to follow

• If you knowingly hide a COI, you will be excluded from the evaluation and your work declared null and void

− The allowance/expenses you claimed may be reduced, rejected or recovered

− Your contract may be terminated

HORIZON 2020

12

Overview of the Evaluation Process

Receipt of proposals

Individual evaluation

Consensus group

Panel Review Finalisation

Evaluators

Individual Evaluation Reports

(Usually

done remotely)

Consensus Report

(May be done

remotely)

Panel report

Evaluation Summary Report

Panel ranked list

Eligibility check

Allocation of proposals to evaluators

Final ranked list

HORIZON 2020

13

Briefing of the experts – Remote phase

Remote web briefing – before the start of the remote phase for individual evaluation

WHO?

the experts + the topic moderator + the EC topic responsible

WHAT?

1 - a general presentation on the evaluation procedure, the role of the experts – this presentation is based on the general presentation available on the participant portal and which is adapted to the type of action

2 - a topic presentation + we also suggest them to watch the videos from from the Infoday

3 – clarification on the IT tools

HORIZON 2020

14

Evaluation criteria

• There are three evaluation criteria for full proposals:

− Excellence (relevant to the description of the call or topic)

− Impact

− Quality and efficiency of the implementation

You should also check requests for ‘exceptional funding’ from third country

participants not included in the list

• The criteria are adapted to each type of action, as specified in the WP

HORIZON 2020

15

Experts - Guiding principles • Independence

− You are evaluating in a personal capacity − You represent neither your employer, nor your country!

• Impartiality & Consistency

− You must treat all proposals equally and evaluate them impartially on their merits, irrespective of their origin or the identity of the applicants

• Objectivity

− You evaluate each proposal as submitted; meaning on its own merit, not its potential if certain changes were to be made

• Accuracy

− You make your judgment against the official evaluation criteria and the call or topic the proposal addresses, and nothing else

HORIZON 2020

16

Experts - Confidentiality

You must:

• Not discuss evaluation matters, such as the content of proposals, evaluation results or opinions of fellow experts, with anyone, including:

− Other experts or Commission/Agencies staff or any other person (e.g. colleagues, students…) not directly involved in the evaluation of the proposal

− The sole exception: your fellow experts who are evaluating the same proposal in a consensus group or Panel review

• Not contact partners in the consortium, sub-contractors or any third parties

• Not disclose names of your fellow experts

− The Commission publishes the names of the experts annually - as a group, no link can be made between an expert and a proposal

• Maintain confidentiality of documents, paper or electronic, at all

times and wherever you do your evaluation work (on-site or remotely)

− Please take nothing away from the evaluation building (be it paper or electronic) − Return, destroy or delete all confidential documents, paper or electronic, upon

completing your work, as instructed

HORIZON 2020

17

Role of independent experts • You evaluate proposals submitted in response to a given

call

• You are responsible for carrying out the evaluation of the proposals yourself

− You are not allowed to delegate the work to another person!

• You must close reports in the electronic system within a given deadline

− This is part of your contractual obligations!

− The allowance/expenses you claim may be reduced or rejected otherwise

• Significant funding decisions will be made on the basis of your assessment

If you suspect any form of misconduct (e.g. plagiarism, double funding), please report this to EC staff

HORIZON 2020

18

Individual evaluation

• Read the proposal and evaluate it against the evaluation criteria

− As submitted - not on its potential if certain changes were to be made

− Do not penalise applicants that did not provide detailed breakdown costs – they are not required

• Disregard excess pages marked with a watermark

• Complete an Individual Evaluation Report (IER)

− Give comments and scores for all evaluation criteria (scores must match comments)

− Explain shortcomings, but do not make recommendations (e.g. no additional partners, work packages, resource cuts)

− Give your view on operational capacity

• Sign and submit the form in the electronic system

Look at the substance: Some proposals might

be handicapped by language difficulties,

others deceptively well written

HORIZON 2020

19

Evaluation Process

Individual Evaluation

Report

Individual Evaluation

Report Individual Evaluation

Report

Consensus group

Consensus Report

Individual Evaluation

Report

Individual Evaluation

Report

Expert Expert Expert Expert Expert Minimum 3 experts … but can be more

Individual evaluation

Consensus

Proposal Eligible proposal

HORIZON 2020

20

Briefing of the experts – Central

Central briefing in Brussels

WHO?

the experts + the topic moderator + the EC topic responsible

+ (dedicated rapporteur)

WHAT?

Short version of the general presentation and of the topic presentation

+

a focus on the consensus part

HORIZON 2020

21

Consensus

• It involves a discussion on the basis of the individual evaluations

• The discussion is articulated around criteria/subcriteria

• The aim is to find agreement on comments and scores

− Agree comments before scores!

− If an applicant lacks basic operational capacity, you make comments and score the proposal without taking into account this applicant and its associated activity(ies)

• “Outlying” opinions need to be explored

− They might be as valid as others – be open-minded

− It is normal for individual views to change

• Moderated by Commission/Agency staff (or an expert in some cases)

− Neutral and manages the evaluation, protects confidentiality and ensures fairness

− Ensures objectivity and accuracy, all voices heard and points discussed

− Helps the group keep to time and reach consensus

HORIZON 2020

22

Consensus report (CR) • The rapporteur is responsible for drafting the CR

− Including consensus comments and scores

• The quality of the CR is of utmost importance

− It often remains unchanged at the panel stage

• The aim of the CR is to give:

− A clear assessment of the proposal based on its merit, with justification

− Clear feedback on the proposal’s weaknesses and strengths, of an adequate length, and in an appropriate tone

− Explain shortcomings, but not to make recommendations

• Avoid:

− Comments not related to the criterion in question

− Comments too long, or too short and inappropriate language

− Categorical statements that have not been properly verified

− Scores that don’t match the comments

− Marking down a proposal for the same critical aspect under two different criteria

HORIZON 2020

23

Proposal scoring

• You give a score of between 0 and 5 to each criterion based on your

comments

− The whole range of scores should be used; use steps of 0.5 − Scores must pass thresholds if a proposal is to be considered for funding

• Thresholds apply to individual criteria…

The default threshold is 3 (unless specified otherwise in the WP)

• …and to the total score

The default overall threshold is 10 (unless specified otherwise in the WP)

• For Innovation actions and the SME instrument, the criterion Impact

is given a weight of 1.5 to determine the ranking

• For first stage of a two-stage procedure, you only evaluate the

criteria Excellence and (part of) Impact

− In that case, only the aspects of the criteria in bold are considered − Default threshold for individual criteria is 4 (unless specified otherwise in the WP) − The level of overall threshold will be set so that success rate at stage 2 will be as

close as possible to 1:3 (in terms of budget)

HORIZON 2020

24

Interpretation of the scores

The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.

Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.

Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.

Good. The proposal addresses the criterion well, but a number of shortcomings are present.

Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.

Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.

0

1

2

3

4

5

HORIZON 2020

25

The panel review

• Consists of experts from the consensus groups and/or new experts

• Ensures the consistency of comments and scores given at the consensus stage

• Resolves any cases where a minority view is recorded in the CR

• Prioritises proposals with identical total scores

• Endorses the final scores and comments for each proposal

− Any new comments and scores (if necessary) should be carefully justified

• Recommends a list of proposals in priority order

HORIZON 2020

26

Observer(s)

• Appointed by the Commission/Agency may attend any meetings or monitor remote evaluation, to ensure a high quality evaluation

• They do not evaluate proposals and, therefore, do not express any opinion on their quality

• They check the functioning and running of the overall process

• They advise, in their report, on the conduct and fairness of the evaluation sessions (including 1st stage of 2 stage processes) and, if necessary, suggest possible improvements

HORIZON 2020

27

Cross-cutting issues

Cross-cutting issues integrated in the work programme (WP)

• Social Sciences and Humanities (SSH) integrated across all Horizon 2020 activities

• Gender dimension in the content of R&I - question on the relevance of sex/gender analysis is included in proposal templates

• The strategic approach to international cooperation consists of a general opening of the WP and targeted activities across all relevant Horizon 2020 parts

− Information on 'automatic funding' to third country participants is restricted – see list of countries

• Other cross-cutting issues may also be included in the WP Responsible Research and Innovation (RRI) including science education, open access to scientific publications, ethics…; standardisation; climate change and sustainable development …

HORIZON 2020

28

HORIZON2020 SC5 INFO DAY 2016

Videos and presentations available on EASME website

HORIZON 2020

29

More info

• Grants Manual – Submission and evaluation

• FAQ on proposal submission and evaluation

• Self-evaluation forms

• Call for experts and videos

• More info on cross-cutting issues

THANK YOU FOR YOUR ATTENTION

EASME on Twitter

@EU_ECOINNO • @H2020EE • @H2020SME • @EEN_EU

EASME-H2020-SC5-ENQUIRIES@ec.europa.eu