1 17/3/2009 European Commission Directorate General Information Society & Media Briefing for Remote...
-
Upload
jada-martin -
Category
Documents
-
view
213 -
download
0
Transcript of 1 17/3/2009 European Commission Directorate General Information Society & Media Briefing for Remote...
117/3/2009
European Commission
Directorate General Information Society & Media
Briefing for Remote Reading
How to fill in the (IER)How to fill in the (IER)Individual Evaluation ReportIndividual Evaluation Report
217/3/2009
This tutorial gives guidelines on how to evaluate the proposals, i.e. it describes
• how to assess a proposal;• how to fill in the Individual Evaluation Form (IER);• scoring a proposal.
317/3/2009
When running the slide show, you can either go slide by slide, or navigate following the hyperlinks:
• access page• return to previous menu• Return to Contents page
Tutorial Manual
417/3/2009
1. Evaluator’s responsibilities
2. How to evaluate / how to fill in the IER
3. Scoring
4. The evaluation criteria
5. The evaluation forms (Call 4, Obj.1.1)
Contents
IP STREP
517/3/2009
• You have signed a non-disclosure agreement• Before, during & after the evaluation do not communicate any information
about the proposals
• You are independent• express your personal view (you do not represent your employer, country, …)
• You are the evaluator• do not delegate your responsibilities
• do not discuss the proposals with 3rd parties
• Do not contact the proposers for clarifications• proposals are evaluated on the information that is presented there, any lack of
information will downgrade the score
• Declare any potential conflict of interest • in case you suspect you have a conflict, contact us immediately
(see e.g. Call-4 Evaluation Handbook)
Evaluator’s Responsibilities
617/3/2009
• Give a fair and clear opinion on each proposal
• Give your personal views
• Evaluate proposals against the specific Objectives and Impact as defined in the Workprogramme
• Evaluate the proposal as written
• Consistently apply the same standard of judgment to each proposal
• Respect the code of conduct
How to evaluate - Introduction
717/3/2009
• Use the official evaluation criteria only• Proposals are evaluated on three criteria
• Scientific and technical quality • Implementation• Impact
• Assess the proposals in terms of all 3 evaluation criteria.• Each criterion is more fully defined by descriptive “bullet
points”, adapted to the instrument type. These are shown on the evaluation forms.
• Provide a relevant comment on each of the “bullet points” These sub-criteria are not scored individually or separately.
The Evaluation Criteria
817/3/2009
How to fill in IER - General
• Comments are confined only to the criterion concerned.• Comments describe only the final view of the proposal.• Start with drafting the comments, then adjust the score
accordingly.• Organise and structure your comment:
• Overall assessment • Followed by two or three illustrations (especially if the score is very
low or very high)• Followed by any comments which mitigate/reduce the overall
appreciation• For negative arguments, a distinction should be made between severe
shortcomings and recommendations for improvements.
917/3/2009
• Comments are of adequate length (not just one sentence !) • Do not be afraid to provide detailed comments. It is
valuable to have sufficient information to be able to understand how you reached your conclusions.
• A detailed comment is helpful for the preparation of the Consensus Report and the discussion that follows.
• Poor comments merely echo the score – Good comments explain it:• “This proposal does not adequately advance the state of the art.”• “This proposal fails to advance the state of the art in X or Y, it does
not take Z into account.”
IER- Level of detail
1017/3/2009
• Comments are clear and provide clear justifications.• Quote proposal text if useful.• Poor comments are vague - Good ones are precise and final :
• “We think the consortium management plan is probably inadequate given the duration of the project and the number of partners.”
• “The consortium management plan is inadequate. It does not include clear overall responsibility for the demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners.”
• Poor comments are ambiguous – Good comments are clear:• “The resources for the project are unrealistic.”• “The resources in Workpackages 4 and 6 are seriously
underestimated given the complexity of the activity involved.”
IER – Clear Messages
1117/3/2009
• Comments are substantial.• Comments are facts not opinions
• not “I think that....”• but “This proposal is.. .”
• Do not write generic comments such as “the proposal is not very innovative” without specifying where exactly it fails.
• Poor comments include words like:• “Perhaps, think, seems, assume, probably, …”
• Good comments include words like:• “Because, percent, specifically, for example, …”
IER – Factual Evidence
1217/3/2009
IER – Avoiding Conflicts
• Poor comments provide an opening for a complaint - Good comments close the question:• “There is no discussion of dissemination activities.”• “Dissemination activities are not adequately discussed.”
• “There are only two SMEs in the consortium.”• “The consortium lacks a sufficient SME participation.”
• “The proposal coordinator is not adequately experienced.”• “The proposal coordinator does not demonstrate in the proposal an
adequate level of experience of work in this field.”
1317/3/2009
IER – Varying the Vocabulary
Why say “Poor” when you can say:• Insufficient, minimal, fails to describe, unacceptable, inadequate,
very generic, not evident, unfocused, very weak, bad, does not meet requirements, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile…
Why say “Excellent” when you can say:• Extremely relevant, credible, very clear, precisely specified,
realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average …
1417/3/2009
IER – Final Checks
• Have you fully explained the proposal’s strengths and weaknesses on all criteria ?
• Do scores match comments (high scores = positive comments, low scores = negative comments) ?
• Have you highlighted any points needing special attention ?• Have you double-checked any matters-of-fact which you
have quoted ?• Have you written at adequate length ?• Overall comment = any comment not covered already in
individual criterion.
“If this was my proposal, would I find this report fair, accurate clear and complete?”
1517/3/2009
• Priority should be given to the comments as they will justify the outcome of the evaluation. Scores should be adjusted to reflect the opinion expressed in the comments. The scores are only a mathematical tool allowing the comparison of the relative quality of different proposals.
• A threshold applies to each individual evaluation criterion and there is an overall threshold. A proposal fails if it is below at least one of the thresholds (incl. overall threshold).
• The overall criterion is not scored separately but automatically calculated from the individual ones.
Scoring
1617/3/2009
The Scoring Scale
Use the full scale! Half marks may be given.0 The proposal fails to address the criterion under examination or cannot
be judged due to missing or incomplete information.1 Poor – The criterion is addressed in an inadequate manner, or there are
serious inherent weaknesses.2 Fair – While the proposal broadly addresses the criterion, there are
significant weaknesses. (can not be fixed @ negotiations)3 Good – The proposal addresses the criterion well, although
improvements would be necessary. 4 Very good – The proposal addresses the criterion very well, although
certain improvements are still possible.5 Excellent – The proposal successfully addresses all relevant aspects of
the criterion in question. Any shortcomings are minor.