Www.FITT-for-Innovation.eu Evaluation Criteria FITT (Fostering Interregional Exchange in ICT...

www.FITT-for-Innovation.eu

Evaluation Criteria

FITT

(Fostering Interregional Exchange in ICT Technology Transfer)

2 | 02.2010 Evaluation criteria

Criteria for evaluation of transfer projects

The described practice is designed to assist in the preliminary

assessment of research-grounded technology projects for their

commercialization potential in the realm of technology transfer.

The process of assessing research projects is necessitated by the high

failure rate, and resulting high cost, of technologies either prior to

reaching the market or once in the market.

The covered Evaluation Criteria are intended to provide guidance for

assessing an idea, a technology or a research project, at an early-stage

of technology transfer (thus prior to product development).


The evaluation process

Project evaluation may take place at various stages

• Early-stage (proof-of-concept “maturation” towards technology transfer)

• Pre-incubation Incubation

Our focus is Early Stage Project Evaluation, which may appear

• In a continuous manner (or at regular intervals)

• Based on a CFP (Call For Proposal, typically once per year)

Such early stage evaluation covers :

• Evaluation criteria

• A process for the application of these criteria, including the structure/organization of the evaluation committee

The current practice focuses on recommended Evaluation Criteria


An illustration of the evaluation process

Pre-incubation Incubation

Research Market

Early-stage evalPreincubation

entry eval

Incubation entry eval

Process :

• Description of project to be evaluated (document)

• Evaluation criteria

• Jury (evaluation committee)

Evaluation criteriaAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeBbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee Aaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd EeeeeAaaaaaaaa Bbbbbbbbbb Bbbbbbbbbb Cccccc Ddddd Eeeee

Development Proof-of-concept Licensing


Coverage/definition of evaluation criteria

Evaluation criteria should cover three main aspects of a project

• Technical aspects

• Market assessment

• Team considerations

Evaluation criteria should be defined and published in advance in order to

allow the evaluated teams to adapt to the process

• Evaluation criteria will be used to establish the overall process, evaluation documents and the selection committee

Evaluation criteria may be used by the evaluation committee to

• Allocate funds/resources to selected projects

• Provide consultancy to the project team (for example, to coach the team on aspects considered as “weak”)


Evaluation criteria

Possible evaluation criteria

Originality of the innovation Profile of the inventor Positive ROI/NPV calculations

Scientific return/opportunities for the laboratory Business opportunity Venture value

Project feasibility Market opportunities/threats Regulatory constraints

Potential users IP (protection issues, prior art) Business model

Scientific relevance of the project Lab support Financial return

Team aspects Realism of the announced plan Social & economical impact

Risk management Potential applications Production issues

• Lots of possible evaluation methods/criteria are mentioned in the literature

• Several possible groups of criteria :


Focus on first-stage evaluation criteria

Most important criteria for first-stage evaluation

Originality of the innovation Profile of the inventor Positive ROI/NPV calculations

Scientific return/opportunities for the laboratory

Business opportunity Venture value

Project feasibility Market opportunities/threats Regulatory constraints

Potential users IP (protection issues, prior art) Business model

Scientific relevance of the project

Lab support Financial return

Team aspects Realism of the announced plan Social & economical impact

Risk management Potential applications Production issues

Deemed premature for 1st stage evaluation


The DIGITEO example - Global positioning

The OMTE checklist is used for maturation projects


Timing of the annual call for proposal

Long selection process

→ March : launch of call for proposal/deadline for submissions

→ April : preselection of 10 projects

→ May: coaching by Digiteo’s marketing team

→ September

→ June/July : final application, oral présentation, deliberation, final decision

Digiteo’s CFP (OMTE)


From proposal to selection

~ 10 proposals

Preselection classification performed by Digiteo’s scientific committee and marketing staff

Coaching : work on the three components technology/marketing/IP submit presentation for the final selection

Selection process :

• External experts (technology transfer specialists from : industry cluster, incubator, Paris region, OSEO innovation fund, chamber of commerce, etc.)

• Digiteo’s technology transfer committee

• Formal selection announced by Digiteo’s steering committee

5 projects selected (budget constraints)


Selection steps

DIGITEO’s Steering Committee DIGITEO’s Steering Committee

Scientific Scientific CommitteeCommitteeScientific Scientific

CommitteeCommittee Expert PanelExpert PanelExpert PanelExpert Panel

Technology Technology Transfer Transfer

CommitteeCommittee

Technology Technology Transfer Transfer

CommitteeCommittee

4. Final decision

1. Scientific relevanceTechnical differentiation

2. TT potentialValue creation

3. Recommandations


Digiteo’s evaluation checklist


DIGITEO – Method/criteria

Evaluation method :

• Evaluation of the applications according to the 12 criteria

• Individual evaluators may apply assessment scores from 1 to 3 (3 being the highest)

Evaluation criteria used for the OMTE call for projects

« Product/technology » aspects

Originality/uniqueness and scientific relevance, project feasibility and opportunities created for the laboratory.

« Market » aspects

Ongoing research contracts and IP related to the project, first applications and users considered.

« Team » aspects

Support of the laboratories in the process, project manager identified to manage the project, realism of the planning proposed and evaluation of the risks by the applicants.


DIGITEO – « Product » criteria

1. Originality of the innovation Originality/uniqueness in comparison with state-of-the-art ?

Definition of the future « product » ?

Positioning compared to competitors ?

2. Scientific relevance of the project Compatibility with the research themes covered by Digiteo ?

Scientific excellence in the field?

Degree of scientific maturation ( is the technology close to a « product ») ?

3. Project Feasibility Technical feasibility of the project?

Feasibility of the planning, with regard to a transfer?

Description of the transfer model envisaged (transfer to an industrial partner / creation of start-up) ?

4. Scientific opportunities created for the laboratory Consequences of the development on the scientific activities of the lab ?

Future impact of the project on the lab’s strategy ?

Impact on the external communications of the lab?


DIGITEO – « Market » criteria

5. Ongoing research contracts Ongoing contracts with industrial partners?

Other contracts/ scientific activities ?

Since when? For how long?

6. Intellectual property (patents, know-how) Background knowledge of the teams involved ?

Protection envisaged (foreground) for the new knowledge and software derivating from it;

Is an IP analysis requested by the teams (analysis of the prior art, patent landscape and « freedom to operate ») ?

7. First potential applications Types/examples of applications ?

Value Proposition (solution to which problem) ?

Applications realised by which kind of company (software company, service provider) ?

8. First potential users Existing and potential actors/ partners to target for the transfer?

Example of end-user for the integrated solution ?

Draft definition of the targeted market (size, segmentation, competitors) ?


DIGITEO – « Team » criteria

9. Support of the laboratories Support of the laboratories involved ?

Balance between the teams involved (complementarity, synergy) ?

Common commitment to a real transfer ?

10. Project manager in charge Profile of the project manager and implication in the project ?

Capability of managing all aspects of the project, keeping with the transfer objective?

Motivation to handle the 3 aspects : technical, IP, marketing ?

11. Realism of the planning

Realism of the planning with regards to the 3 aspects:

Technical

IP

Marketing

12. Evaluation/ consideration of the risks

Identification and management of the risks :

Technical

IP

Marketing


DIGITEO - Assessment

Useful tool to be used as a checklist throughout the evaluation process

The final selection has to include the assessment of the presentation made in front of the jury. Grade given by the jury is based for 50% on written application and 50% on the oral presentation.

The jury should include a majority of external experts

Final selection : classification/ranking of the presented projects (top 5 selected)

Some « Digiteo specifics » not to be considered for a generic checklist


Pros & Cons

CONsPROs

• This practice attempts to formalize methods that are already in use (most of the time on an ad hoc basis)

• The methodology and associated tools (call for proposal, criteria, etc.) are readily available and can be adapted to each individual case

• Only a selected number of criteria are highlighted

• Some criteria may need to be further developed


Why ?

• Methodology developed by Digiteo in order to manage the incoming flow of

technology transfer proposals

• Need for a consistent set of criteria for all steps of evaluation process, communicated transparently to all involved partners : project teams, internal Digiteo evaluators, “technology transfer coaches” and external experts

• Without this methodology, involved parties would get the impression that projects might be evaluated/selected based on obscure reasons. This would leave the doors open for debate, accusations for “unfair competition” and backstage lobbying


Why/impact ?

Impact : Why is it a good practice?

• The Digiteo community judges this approach transparent, fair and clearly communicated

• We may recommend this approach based on our own experience


Outcome

What happened after the implementation :

• The approach turned out as expected

Final selection (with external experts) is based on relative ranking among the presented projects

The scoring system is only used for individual evaluation purposes

• However, you also have to manage those projects that were not finally selected

Debrief the teams that were not selected

Clearly communicate the reasons for not being selected

Focus on things to be improved (and how to improve them)

Encourage them to apply again with an enhanced proposal


Outcome – plans for the future

Plans for the future?

• The approach should be further developed/detailed :

Definition of terms

Explanation on how to apply each of the listed criteria (with some examples)


Suggested Readings

Link to code book

• Technology development/maturation• Proof of concept• (Opportunity) assessment

Link to related websites

OMTE call for proposal and projects selected during previous editions:

http://www.digiteo.fr/Digiteo_OMTE

Www.FITT-for-Innovation.eu Evaluation Criteria FITT (Fostering Interregional Exchange in ICT...

Documents

Transcript of Www.FITT-for-Innovation.eu Evaluation Criteria FITT (Fostering Interregional Exchange in ICT...