Post on 19-Jun-2020
1 of 51
Future and Emerging Technologies
Features and evaluation process
Briefing for Remote Evaluators 08 November 2017
Antonio LOREDAN Call Coordinator FET-Open Unit
2 of 51
2
You can send your questions to the following address:
REA-FET-OPEN-2017-2-EVALUATIONS@ec.europa.eu
they will be answered at the end of the presentation
3 of 51
1. FETOPEN RIA - Main Features
2. Confidentiality and Conflict of Interest
3. Admissibility and Eligibility Check
4. Evaluation Process
5. Remote phase: IERs, REs and QCs roles
Agenda:
4 of 51
1. FETOPEN RIA - Main Features
2. Confidentiality and Conflict of Interest
3. Admissibility and Eligibility Check
4. Evaluation Process
5. Remote phase: IERs, REs and QCs roles
5 of 51
Excellent Science in Horizon2020
INDUSTRIAL LEADERSHIP € 16,5 billion
SOCIETAL CHALLANGES € 28,6 billion
EXCELLENT SCIENCE
€ 24,2 billion
OTHER * € 6 billion
H2020 budget € 74,8 billion
*OTHER: • Spreading excellence & widening participation • Science with and for society • JRC • EIT
ERC 54%
FET 11%
MSCA 25%
Research infrastructure
s 10%
6 of 51
FET-Open Research and Innovation Actions: supporting early-stages of research to establish a new technological possibility.
Collaborative projects up to € 3 Mio funding (indicative)
Single step submission, '1+15' pages
No thematic restriction, no emphasis on any subject
Bottom-up, but targeted - not blue sky research
Typical FET project result is a proof of a concept
Scope defined by FET gatekeepers
7 of 51
SCOPE: […] Proposals are sought for collaborative research with all of the following characteristics ('FET gatekeepers'):
Gatekeepers
Long-term vision Interdisciplinary
Novelty
Foundational
High-risk
S&T breakthrough
Far beyond… Ambitious
Not incremental
Open & Agile practices
New solutions & synergies
8 of 51
Long-term vision new and radical long-term vision of a science- and
technology-enabled future that is far beyond the state of the art
9 of 51
Breakthrough S&T targeted
Scientifically ambitious and technologically concrete breakthroughs plausibly attainable within the life-time of the project Ambitious
10 of 51
Foundational
Far beyond…
The breakthroughs must be foundational in the sense that they can establish a basis for a new line of technology not currently anticipated
11 of 51
Novelty
Not incremental
New Ideas and concepts, rather than the application or incremental refinement of existing ones
12 of 51
High-risk
Open & Agile practices
Balancing the high risk versus being utterly unrealistic . High-risk is not a synonym with not-doable
13 of 51
Interdisciplinary
New solutions & synergies
The proposed collaborations must go beyond current mainstream collaboration configurations in joint S&T research, and must aim to advance different scientific and technological disciplines together and in synergy towards a breakthrough
14 of 51
FET-Open Research and Innovation Actions
Expected impact
Establish baseline of feasibility and innovation potential
Strengthening European leadership in the early exploration of
visionary, new and emerging technologies
New R&I practices
15 of 51
1. FETOPEN RIA - Main Features
2. Confidentiality and Conflict of Interest
3. Admissibility and Eligibility Check
4. Evaluation Process
5. Remote phase: IERs, REs and QCs roles
16 of 51
The expert must keep his/her work under the Contract strictly confidential, and in particular:
a) not disclose (directly or indirectly) any confidential information relating to proposals or applicants, without prior written approval by the Agency.
b) not discuss proposal(s) with others (including other experts or Agency staff that are not directly involved in the evaluation of the proposal(s), except during evaluation meetings and with prior approval by the responsible Agency staff.
c) not disclose:
- details on the evaluation process or its outcome, without prior written approval by the Agency
- details on his/her position/advice;
- the names of other experts participating in the evaluation.
Social media: Take great care not to post pictures or comments on evaluation matters through social media
17 of 51
The expert must keep his/her work under the Contract strictly confidential, and in particular:
d) not communicate with applicants (including linked third parties or other third parties involved in the actions) during the evaluation or afterwards — except in panel hearings.
e) may not remove from the premises any documents, material or information on the proposal(s) or on the evaluation.
f) is responsible for ensuring adequate protection of electronic documents and information and for returning, erasing or destroying all confidential information after the end of the evaluation.
Social media: Take great care not to post pictures or comments on evaluation matters through social media
18 of 51
An expert has a conflict of interest if : was involved in the preparation of the proposal
would benefit directly or indirectly if the proposal is accepted
has a close family (or other personal) relationship with any person representing an applicant legal entity
is a director, trustee or partner or is in any way involved in the management of an applicant legal entity
is employed or contracted by one of the applicants or a named subcontractor
is a member of an ‘advisory group’ (set up by the Commission to advise on the preparation of EU or Euratom Horizon 2020 work programmes) in an area related to the call
is a ‘national contact point (NCP)’ or is directly working for the ‘Enterprise Europe Network (EEN)’
is a member of the H2020 Programme Committee
19 of 51
The Agency decides on whether there is a conflict of interest (on the basis of circumstances, available information and related risks), if the expert: was employed by one of the applicants in the last three years
is involved in a contract, grant agreement, grant decision, management structure (e.g. member of management or advisory board etc.) or research collaboration with an applicant/fellow (or had been so in the last three years)
is in any other situation that could cast doubt on their ability to participate in the evaluation of the proposal impartially (or that could reasonably appear to do so in the eyes of an external third party).
20 of 51
You must inform the Agency as soon as you become aware of a COI
If there is a COI for a certain proposal you
cannot evaluate the give proposal
might be excluded from the panel review for the given proposal
The Agency will determine if there is a COI on a case-by-case basis and decide the course of action to follow
If you knowingly hide a COI, you will be excluded from the entire evaluation and your work declared null and void
The allowance/expenses you claimed may be reduced, rejected or recovered
Your contract may be terminated
21 of 51
If a conflict becomes apparent at any stage of the evaluation, you must immediately inform us.
If a conflict is confirmed, you must stop evaluating the proposal concerned. Any comments and scores already given by you will be disregarded. If necessary, you will be replaced.
22 of 51
1. FETOPEN RIA - Main Features
2. Confidentiality and Conflict of Interest
3. Admissibility and Eligibility Check
4. Evaluation Process
5. Remote phase: IERs, REs and QCs roles
23 of 51
ADMISSIBILITY
To be considered admissible, a proposal must:
1. Be readable, accessible and printable.
2. Be complete
3. Include a draft plan for the exploitation and dissemination.
24 of 51
A proposal/application will only be considered eligible if: (a) Its content corresponds, wholly or in part, to the topic description for
which it is submitted
(b) At least three legal entities are present in the consortium. And :
Each of the three must be established in a different EU Member State or Horizon 2020 associated country.
All three legal entities must be independent of each other.
ELIGIBILITY
25 of 51
Part A: Administrative part of the proposal
Part B: Technical Annex : Section 1-3 (16 pages maximum)
Cover page
Section 1: S&T Excellence
Section 2: Impact
Section 3: Implementation
Section 4-5
Section 4: Members of the consortium
E.g. legal entity, CV, subcontract, third party
Section 5: Ethics and Security
Ethics self-assessment & supporting documents
Security checklist
Section 4 & 5 are not
covered by the page
limit.
Section 1,2 & 3 are strictly
limited to 15 pages!
WATERMARK!!!
Cover page strictly limited to 1 page!
Completeness and page limits
26 of 51
Admissibility and eligibility of proposals have been checked by the Agency - however, if you spot an issue relating to eligibility, please contact us.
27 of 51
27
You can send your questions to the following address:
REA-FET-OPEN-2017-2-EVALUATIONS@ec.europa.eu
they will be answered at the end of the presentation
28 of 51
1. FETOPEN RIA - Main Features
2. Confidentiality and Conflict of Interest
3. Admissibility and Eligibility Check
4. Evaluation Process
5. Remote phase: IERs, REs and QCs roles
29 of 51
FET-Open RIA in a nutshell
30 of 51
FET-Open RIA in a nutshell
Calls Number of
eligible proposals
Number of grants
Total Requested
(M€) Success rate
Sep-14 639 24 78,1 3,70%
Mar-15 665 11 41 1,65%
Sep-15 800 11 37,8 1,37%
May-16 544 23 87,8 4,22%
Jan-17 365 26 84,8 7,12%
Total 3019 95 329,5
Sep-17 395 26* 84*
*Estimated
31 of 51
Call Schedule
32 of 51
Feedback in 5 months
Ethics screening/
assessment
Panel review Final comment & score
Cross-reading By panel membres
Quality check of
IER with support from Vice Chairs
Remote evaluators write
Individual Evaluation Report
Eligibility check
REA
Proposal submission
Collated report Check by remote
experts
Expert Assignment
Collate IER comments, median score calculated
Conflict of interest check at any stage of evaluation
Iterations if necessary
Each proposal is allocated to 4 remote evaluators
Each evaluator drafts comments and assigns scores
Draft ESR created
Evaluation process (1)
New in Jan 17 cut off
33 of 51
Feedback in 5 months
Ethics screening/
assessment
Panel review Final comment & score
Cross-reading By panel membres
Quality check of
IER with support from Vice Chairs
Remote evaluators write
Individual Evaluation Report
Eligibility check
REA
Proposal submission
Collated report Check by remote
experts
Expert Assignment
Each proposal is allocated to 4 remote evaluators
Each evaluators drafts comments and assigns scores
Evaluation process (2)
• Systematic cross reading of proposals
• Panel members briefed to carefully look at 'diverging' opinions
34 of 51
Feedback in 5 months
Ethics screening/
assessment
Panel review Final comment & score
Cross-reading By panel membres
Quality check of
IER with support from Vice Chairs
Remote evaluators write
Individual Evaluation Report
Eligibility check
REA
Proposal submission
Collated report Check by remote
experts
Expert Assignment
Each proposal is allocated to 4 remote evaluators
Each evaluators drafts comments and assigns scores
Evaluation process (3)
• Discussion of all 'highly scored' proposals
• Low scored proposals are discussed too
• Special attention to 'diverging' opinions
• Final score decision by consensus or vote if necessary
• Panel should add comments on divergences
35 of 51
Feedback in 5 months
Ethics screening/
assessment
Panel review Final comments &
score
Cross-reading By panel membres
Quality check of
IER with support from Vice Chairs
Remote evaluators write
Individual Evaluation Report
Eligibility check
REA
Proposal submission
Collated report Check by remote
experts
Expert Assignment
Each proposal is allocated to 4 remote evaluators
Each evaluators drafts comments and assigns scores
Evaluation process (4)
Evaluation Summary Report (ESR): collation of all (4) evaluators' comments, per sub-criterion, which may be mutually contradicting - full transparency
36 of 51
Main Actors: • Remote Evaluators: ~600
o Write remotely good quality individual evaluation reports and assign corresponding scores
o Submit individual evaluation reports within deadline
• Quality Controllers (QC): ~70 o Fine tuning of proposals' descriptors o Check the quality of individual evaluation reports
• Cross Readers (CR): ~70 o Cross-read proposals o Take part in the panel review in Brussels
• In addition - Independent Observer
37 of 51
Observer
Appointed by the Agency may monitor the remote evaluation, to ensure a high quality evaluation
He checks the functioning and running of the overall process
He advises, in his report, on the conduct and fairness of the evaluation sessions and, if necessary, suggests possible improvements
He does not evaluate proposals and, therefore, does not express any opinion on their quality
38 of 51
1. FETOPEN RIA - Main Features
2. Confidentiality and Conflict of Interest
3. Admissibility and Eligibility Check
4. Evaluation Process
5. Remote phase: IERs, REs and QCs roles
39 of 51
Remote evaluation and quality control (8 November– 7 December 2017)
Remote phase
40 of 51
You can send your questions to the following address:
REA-FET-OPEN-2017-2-EVALUATIONS@ec.europa.eu
they will be answered at the end of the presentation
41 of 51
Remote phase and your role (1):
Your role is to draft Individual Evaluation Reports (IERs) with your assessment of the proposals in accordance with the evaluation criteria.
Disregard excess pages marked with a watermark
Check to what degree the proposal is relevant to the call or topic
42 of 51
• Read the proposal and evaluate it against the evaluation criteria
Without discussing it with anybody else
As submitted - not on its potential if certain changes were to be made
Do not penalise applicants that did not provide detailed breakdown costs – they are not required
• Complete an Individual Evaluation Report (IER)
Give your view on operational capacity
Give comments and scores for all evaluation criteria (scores must match comments)
Explain shortcomings, but do not make recommendations (e.g. no additional partners, work packages, resource cuts)
• Sign and submit the form in the electronic system
Look at the substance: Some proposals might be
handicapped by language difficulties, others
deceptively well written
Remote phase and your role (2):
43 of 51
The quality of the IER is paramount
There will be no consensus meetings! No grant negotiation phase
The IERs remains unchanged at the panel stage and are then sent to applicants, with some additional comments if necessary.
The aim of the IER is to give:
A clear assessment of the proposal based on its merit
Clear feedback on the proposal’s shortcomings/weaknesses and strengths.
Individual Evaluation Report (IER)
44 of 51
Remote evaluators (RE) tasks:
RE evaluates each proposal as submitted, not on its potential
If RE shortcomings (other than minor ones and obvious clerical
errors), reflect those in a lower score for the relevant criterion
RE explains the shortcomings, but cannot make recommendations
Proposals with significant weaknesses that prevent the project
from achieving its objectives must not receive above-threshold
scores
45 of 51
Interpretation of the scores
The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.
Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.
Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.
Good. The proposal addresses the criterion well, but a number of shortcomings are present.
Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.
Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.
0
1
2
3
4
5
46 of 51
As remote evaluator you are requested to:
'Accept to evaluate' each proposal asap but not later than within 2 days!
Submit at least 2 (or all if less than 2 ) proposals assigned to by November 15
Submit at least 4 (or all if less than 4) proposals assigned to you by November 20
47 of 51
IER will be quality checked by a Vice Chair Quality Controller (QC) for:
Relevance and completeness of the comments
Consistency between comments and scores
Inappropriate language
Immediately after an evaluator submits the IER the quality control starts
Quality check of IERs (I)
48 of 51
Quality check of IERs (II)
QC provides feedback to REA in case an IER needs to be re-opened.
If applicable, you should revise a given IER based on REA request, following input from QC.
QC verifies that you have identified strengths and weaknesses of the proposal under the 3 criteria.
Your comments have to be relevant within each sub-criterion.
49 of 51
Feedback to proposers – Evaluation Summary Report (ESR)
Collation of all evaluators' comments , per sub-criterion, which may be
mutually contradicting (no consensus) - full transparency
Proposal score calculation (per criterion) - median of the scores from
individual evaluators
Final score per criteria is decided by the final panel review
Total final score for the proposal is calculated as the weighted sum of the
final scores from the 3 evaluation criteria
Final panel review adds also some additional comments
50 of 51
Remote Evaluators FET-Open RIA, Cut-off date 27 September 2017; Useful Info
https://ec.europa.eu/info/remote-evaluators-fet-open-ria-cut-date-27-
september-2017-useful-info_en
51 of 51
Thank you for your attention!