20201015 Schofield KTH Evaluators View PUBLIC

14.10.2020

1

KTH HE Week 2020-10-15

THE EVALUATOR’S VIEW:HOW TO WRITE A COMPETITIVE

EU PROPOSAL

Monica SchofieldTutech Innovation

[email protected]

KTH Horizon Europe Week 15 October 2020

These slides are for personal use of participants only

tutech.deKTH HE Week - 2020-10-15

Today’s topics

• Some words on transition H2020-HE

• The evaluation criteria and process

• Who evaluates and how

• Getting your message across

• Addressing pathways to impact

• Forming the optimal partnership

• Checking your network

Ask questions whenever

Time limit!


HORIZON 2020 HORIZON EUROPE

14.10.2020

2


Framework 9 – Horizon Europe

STILL ….


plus c'est la même chose

• A Programme

• Work Programme

• Calls

• Submission of proposals

• Principle of partnerships

• Evaluation criteria (largely)

• Funding rates (largely)

Continuity in much of theoperational aspects(as requested by the community)


Change lies in the political context…


The politics of 2019-2020 (COM slide Feb 2020)

Preparation of HE started2016/17.

2019 New Commission,New Parliament

2020 Brexit

14.10.2020

3


Political message: R&I is needed to deliver EU policy objectives

A European Green Deal

An economy that works for people

A Europe fit for the digital age

Protecting our European way of life

A stronger Europe in the world

A new push for European democracy


The politics of 2020


“strengthen the impact of research and innovation in developing, supporting and implementing Union policies and … to address global challenges, including climate change and the Sustainable Development Goals.”


The new priority – TAKE NOTE

14.10.2020

4


Evaluation – principles unchanged

• Continuity with H2020

• Aim to improve feedback toproposers

• Aim to improve transparency androbustness

• Possibility that questions may be put to proposers

• Piloting of anonymous stage 1 proposals

S&T Excellence

ImpactImpact

Implementation

0…5

0…5

0…5

Ref: R&I Day session: Quality, transparency and fairness in Horizon Europe proposal evaluation’ 22.9.2020

tutech.de

Excellence

https://ec.europa.eu/research/participants/data/ref/h2020/call_ptef/ef/2018-2020/h2020-call-ef-ria-ia-csa-2018-20_en.pdf

Award criteria H2020 RIA/IA

Clarity and pertinence of the objectives

Soundness of the concept, and credibility of the proposed methodology

Extent that the proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)

Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge and gender dimension in research and innovation.

WHAT

tutech.de

Impact



The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;

Any substantial impacts not mentioned in the work programme, that would enhance innovation capacity, create new market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society;

Quality of the proposed measures to: Exploit and disseminate the project results (including management of IPR), and to manage research data where relevant.Communicate the project activities to different target audiences

WHY

tutech.de

Implementation



• Quality and effectiveness of the work plan, including extent to which the resources assigned to work packages are in line with their objectives and deliverables;

• Appropriateness of the management structures and procedures, including risk and innovation management;

• Complementarity of the participants and extent to which the consortium as whole brings together the necessary expertise;

• Appropriateness of the allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfil that role.

HOW

14.10.2020

5


Sub-criteria likely tochange and may even varyacross the calls


Horizon EuropeExcellentScience

IndustrialLeadership

LEIT

SocietalChallenges


Evaluation criteria

Continuity with H2020


Preparing a proposal

The Call

Ideas

Partners

The document

that sells the project

14.10.2020

6


Receipt of proposalsReceipt of proposals

IndividualevaluationIndividualevaluation

Consensusgroup

Consensusgroup

Panel ReviewPanel Review

FinalisationFinalisation

Evaluators

IndividualEvaluationReports

(Usually done

remotely)

ConsensusReport

(May be done remotely)

Panel report

Evaluation Summary Report

Panel ranked list

Eligibility check

Allocation of proposals to evaluators

Final ranked list

Overview of the evaluation process

Possibility to ask questions of proposers

Portfolio assessment


Process as seen by the evaluator

• Briefing

• Access to proposals

• Check if conflict of interest

• Briefing meeting by webinar

• Reading time (half a day per (full) IA,RIA proposal)

• Entry of individual scores and comments into evaluation tool

• No visibility on other scores until all evaluators scored• Scores must be justified by comments (“comments first, score second”)

• Consensus meeting

• (Ranking)


Proposal evaluation

Ideas

Partners

The Call

The document that says it

all!


Before you start, look at the criteria for selection

Guide for applicants

Work programme and its context

(Destination)

AND check what has already been funded!

14.10.2020

7


ADDRESSING THE EVALUATORS


Where do evaluators come from?

• Named in the expert database

• Known to Project Officers

• Competence in an aspect of the call (Not just specialists in the science!)

• Gender balance

• Nationality balance

• Industry/academia (other) balance

• Newcomers vs old timers

• Available


What this means in practice

Challenge to recruit people from a range of organisations

Payment of 450 euros per day - filters out those who can earnmore by doing other things…

… unless they are motivated for other reasons

Taken as a group, the evaluators are pretty competent, but theywon‘t all be deep specialists in your field


Stating the obvious

Evaluators are human – this is not an infallible process

14.10.2020

8


http://ec.europa.eu/research/participants/portal/desktop/en/experts

Become an evaluator!


How do evaluators read proposals?

• Not necessarily from beginning to end

• May read a couple of proposals before scoring

• May pick the easiest to judge first!

• May read proposals on the move


Keeping evaluators well disposed

• Make the document easy to read

• Don‘t cram pages and make sure diagram legends can be read on a tablet

• Write so it is interesting and fluid to read

• Avoid “motherhood and apple pie“ statements• Avoid repeating chunks of the call• Don‘t write in policy speak clichés

• Don‘t say you are fantastic, unique etc, but make clear what makes this a special project


Winning proposals

• Original or creative project idea – EXCITING!

• Well articulated (measurable) objectives – CLEAR

• A well-matched partnership of different kinds of skill – GREAT PROJECT COMMUNITY

• Impacts stated in call are fully addressed – RELATES TO NEEDS

• Evidence of a strategy to move knowledge beyond the researchcommunity – SOCIETY WILL REALLY BENEFIT

• The implementation plan is thought through – CAN DELIVER

• The proposal is reasonably priced – SO THAT WE CAN FUND MORE PROJECTS

14.10.2020

9


Section 1What do you want to achieve and how?

Section 1What do you want to achieve and how?


What the evaluators take from section 1

• What is the project trying to achieve and how it relates to the call?

• What makes this project special?

• What is the key result going to be?

• Why should I believe you can do this?

• Where does this project fit into previous work?

• Does the partnership look as though it knows what it wants to achievetogether?

tutech.de

Proposal Section 1 – objectives

OPENING STATEMENT: MAKE IT EASY FOR THE EVALUATOR TO UNDERSTAND THE PROJECT

• WHAT is the project aimed at achieving?

• WHY is this relevant to the ‘bigger picture’?

• WHAT will be the result at the end of the project?

The specific objectives of your project shouldinclude the objective to achieve impact.


Be specific

This project

• Will develop X,Y, Z

• Will provide a means to

• Will achieve a something

NOT

• Will bring together lots of people

• Will analyse lots of things

• Will gather lots of data

14.10.2020

10


Positioning your project

• Be specific about the context of your project

• Make the distinguishing features very clear

• Don‘t confuse the evaluators or give the impression youare repeating work done before

• As a rule, evaluators don‘t appreciate projects that aresimply a continuation of a previous project

What would the world miss of significant value if this project was not funded?


Section 2Why is this worth doing and

what are you going to do to ensure the knowledge is shared and used?

Section 2Why is this worth doing and

what are you going to do to ensure the knowledge is shared and used?


ResearchersResearchers

DevelopersDevelopers

Users, society Users, society

Linking knowledge to impact

PAVING THE PATHWAYS TO IMPACTMONITORING AND EVALUATING IMPACT IN

HORIZON EUROPE

14.10.2020

11

TRACKING IMPACT IN HORIZON EUROPE

Scientific impact Promote scientific excellence, support the creation and diffusion of high-quality new fundamental and applied knowledge, skills, training

and mobility of researchers, attract talent at all levels, and contribute to full engagement of Union's talent pool in actions supported under the Programme.

Societal impactGenerate knowledge, strengthen the impact of R&I in developing, supporting and implementing Union policies, and support the uptake

of innovative solutions in industry, notably in SMEs, and society to address global challenges, inter alia the SDGs

Economic impact Foster all forms of innovation, facilitate technological development, demonstration and knowledge transfer, and strengthen deployment of

innovative solutions

THREE TYPES OF IMPACT BASED ON OBJECTIVES


Software technologies

Communication & networking

Micro, nano& opto

electronics

Micro & nanosystems

InterfacesKnowledge technologies, digital content

Miniaturised, low cost, low power

components and microsystems

Pervasive, mobile, wireless, trustful infrastructures

Natural interactions with “knowledge”

Applied IT for major societal & economic challenges

Trust & security

Complexproblem solving

IT for economic challenges

IT for societal challenges

Addressing pathways – link to societal needs

The target: enabling anywhere, any time natural and enjoyable access to IT services for ALL


DESTINATION: Climate neutral, circular and digitised production

Breakthrough science New technology

Societal and economic factors

Regulation

Putting research into a bigger context

Changing

consumer

behaviour

New systems


HE Work Programme structure

Context

Destination

Expected impact

Scope - topics

Conditions

The overall ambitions

What is ultimately to be achieved

What the project should lead to

Focus of the project

Budget, Partners …

14.10.2020

12


Addressing impact

What are you going to do in the projectto support this happening

- pathways to impact, dissemination

Why should I believe you will actuallyaddress this?

- choice of partners, deliverables

In what way could your project have impact…- narrative, link to call


Think beyond the project

Technological

opportunity -

breakthrough

Formation of

products or

services

Public/Market/

Regulatory

acceptance

Project content Innovation scenario

Influence on

public policyPolitical

/societal

acceptance

Utilisation by society

Development

of a challenge

solution


Think about the “Pathway to Impact”

• What can you do in the project?

• Who should you involve to ensure the outputs of the project are transported beyond the project?

• What is the best way to involve them?


Reading the WP: example outcome

TWIN-TRANSITION-23-2021: The use of BIM for circular construction by valorisation of construction and demolition waste (RIA)

Expected outcomes: Projects are expected to contribute to the following outcomes:

• Reduce the quantity of construction and demolition waste (CDW)

• Demonstrate a more circular and climate neutral construction sector

• Increase the reusability potential of construction products post demolition and reduce the downcycling of construction waste by facilitating modular dismantling of complex construction products

14.10.2020

13


Reading the WP: example scope

Scope: Based on volume, construction and demolition waste (CDW) is the largest waste stream in the EU. Considering that most of the waste quantities is concrete and steel, the embodied energy and CO2 emission in the CDW is significant. By recycling and reusing CDW in new constructions, the sector would come closer to the targets of becoming fully circular and climate neutral.

Quantitative waste estimation is crucial for waste management. This could be achieved by utilising technologies such as Building Information Modelling (BIM). Such a model could provide information of the materials type (e.g. whether there are hazardous materials that require special care) and quantities, and thus an estimation of the duration for removal, the number of pick-up trucks needed, cost, etc. and make waste separation easier and faster, e.g. by combining BIM with robotic arms.


Reading the WP: example scope

Proposals should:

• Develop, test and promote the necessary BIM applications for CDW management in different types of constructions;

• Develop automated solutions for waste separation process;

• Produce all required training material for the proper use of the developed technologies;

• Demonstrate of all developed solutions in at least four construction sites across different Member States;

• Assess the value of the solutions in terms of the additional monetary value/reduction of CO2 emissions produced

• Finally, the projects should provide contributions to relevant standards.


Reading the WP – start with the outcome

Expected outcome Howwill this be achieved? Who has to be involved? How toconnect them to the project? How can you measure/predict theoutcome? PARTNERS NEEDED FOR OUTCOMES

Reduce the quantity of construction and demolition waste

Demonstrate a more circular and climate neutral construction sector

Increase the reusability potential of construction products post demolition and reduce the downcyclingof construction waste by facilitating modular dismantling of complex construction products


Reading the WP – start with the outcome

Scope What competencies are needed to cover scope? WHAT PARTNERS NEEDED TO COVER SCOPE

Develop, test and promote the necessary BIM applications ….

Develop automated solutions for waste separation process

Produce all required training material ….

Demonstrate developed solutions in at least four construction sites …

Assess the value of the solutions in terms of the additional monetary value/reduction of CO2 emissions produced

Contribute to standards

14.10.2020

14

tutech.de

Developing a strategic approach to dissemination


Dissemination – to support impact

Target groups –who to inform

Content – what information to provide

Channels – how best to reach target

ResearchersThe publicPolicy makersRegulatorsBusiness …

ResultsIdeasProblemsPR…

WebinarsNewsletterConferencesSocial media…

Which policy makers, what results, which conferences?Why these?


Who might have an interest in the results?

• Colleagues – researchers in same field

• Researchers from other disciplines (why?)

• Policy makers

• Regulators, standards bodies

• Government

• Local authorities

• General public

• Industry

• NGOs, civil society

• The media, creative industry

• Funding bodies! (e.g. for new programmes)

• ….tutech.deKTH HE Week - 2020-10-15

Involving people

To achieve the specified impacts who needs to be involved?

• Users• Policy makers• Opinion formers

What is the best way to involve them?

• As partners• As advisors• As target for knowledge transfer activities

14.10.2020

15


THE PARTNERSHIP


Align the outputs to what you say you will do

Excellent science

Industrial leadership

Societalchallenges

• Shift frontiers of knowledge• Publish in top journals

• Achieve innovation• Achieve breakthrough products• …

• Contribute to transformations in society

Evaluators will look for evidence of ability to ...Project Focus


Align the partnership with the stated goal

Excellent science

Industrial leadership

Societalchallenges

• Top class complementaryresearch partners

• Industrial commitment

• Partners capable of implementingand duplicating

What evaluators will look for in partners


Form partnerships that support the project logic

Achieve a breakthrough – a completely new way of doing things

Move technology towards widespread application

Address the needs of a specific sector

Provide support for the implementation/acceptance of

aspects of policies or programmes

Bring communities of experts or practitioners together

Do you have the top guys in the project?Can you change the mindset of the research community?

Knowledge of policies?Involvement in key committees?Track record in giving policy advice?

Construction of network?Sustainability of network?

User involvement?How to connect to the status quo?Routes to (industrial) exploitation?

Are the “standard bearers” involved?Do you show you know the sector?

Funding for a research facility Funding in the longtermAccess to the facility?

Funding to support career development of young researchers

How and how many will it help?

14.10.2020

16


The partnership should reflect the TRL

PAGE

61

Full deployment

First of the kind commercial system (commercial level)

Demonstration System (pre-commercial scale)

Technology demonstrated

Technology validated in environment

Technology validated in lab

Experimental proof of concept

Technology concept formulated

Basic principles observed

Claudia Häfner24.09.2013 KIT, Karlsruhe

TRL 1:

TRL 2:

TRL 3:

TRL 4:

TRL 5:

TRL 6:

TRL 7:

TRL 8:

TRL 9:


Balance what partners represent

Partner a

Spec. Gen. Exploit. User SME Region Branch ...

Partner b

Partner d

Partner c

Partner e

Partner f


Do this exercise before you invite too many of your research friends to join the partnership!


14.10.2020

17


How good is your network?

SE EU

Researchers

SMEs

Big industry

Policy advisers

Users

….


Stating the obvious

If the budgets of proposals exceed thefunding budgets, not all proposals can win funding, however good they are!

Success rates:FP7 – 19%H2020 - 12.6%HE ?


Worthwhile anyway?

“Even if the project isn’t funded, this has been worthwhile for the partners I have got to know, and the process has helped me structure ideasfor my future research directions”

Monica Schofield

[email protected]

[email protected]

THANK YOU!

20201015 Schofield KTH Evaluators View PUBLIC

Documents

Transcript of 20201015 Schofield KTH Evaluators View PUBLIC