Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3...
Transcript of Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3...
![Page 1: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/1.jpg)
Evaluation: PracticalGuidelines
A guide for evaluating public engagement activities
![Page 2: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/2.jpg)
Evaluation: Practical Guidelines i
Contents
1 Introduction 1
1.1 Usingthisguide 1
2 WhatisEvaluation? 2
2.1 Evaluation 2
2.2 Monitoring 2
2.3 Thedifferentaspectsofevaluation 2
3 BuildinganEvaluationStrategy 4
3.1 Settingaims 4
3.2 Settingobjectives 5
3.3 Choosingyourevaluationtools 7
3.4 Howmuchevaluation? 12
3.5 In-houseorindependent? 13
3.6 Ifyourunintotrouble 13
4 GatheringData–ToolsandTechniques 15
4.1 Introduction 15
4.2 Quantitativeresearch 15
4.3 Qualitativeresearch 18
4.4 Otheroptions 20
4.5 Confidentiality 21
5 DataHandling–ToolsandTechniques 23
5.1 Quantitativedata 23
5.2 Qualitativedata 24
5.3 Observationaldata 25
6 Reporting 27
6.1 Reporting 27
6.2 Reportstructure 27
![Page 3: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/3.jpg)
ii Evaluation: Practical Guidelines
Annex1EvaluationQuestionnairesandQuestions 30
Questionnairedesign 30
Questiondesign 32
Annex2Questionnairemodules 36
Annex3SummativeEvaluationSchema 40
MonitoringData 40
EvaluationData 40
Annex4Glossary 41
Annex5Furtherreading 44
Findinginformationonevaluation 44
EvaluationMethods 44
WebResources 44
![Page 4: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/4.jpg)
Evaluation: Practical Guidelines 1
1Introduction
Thisguideisforresearchersseekingtoengagegeneralaudienceswiththeirsubjecttoevaluatepublicengagementactivities,regardlessofpriorexperienceofeitherpublicengagementorevaluation.Itisintendedtohelpresearchersfromanydiscipline.Foramoregeneralguidetoundertakingpublicengagementactivities,see:http://www.vitae.ac.uk/CMS/files/upload/The_engaging_researcher_2010.pdf
Evaluationtechniquesarebasedonsocialandmarketresearchmethods,soeventhoughsomematerialwillbefamiliartosomereaders,thequestionsposedandthethoughtprocessessuggestedwillberelevanttoallthoserunningandevaluatingpublicengagementactivities.
Theevaluationapproachescoveredinthisguidearesuitableforarangeofactivitiesincluding,butbynomeanslimitedto:
• Publiclectures,talksanddebates(bothliveandon-lineusingnewmedia).
• Shows.
• Festivals.
• Activitiesforschoolstudentsand/orteachers(realorvirtual).
• Websites,CD-ROMsandothervirtualactivities.
• Exhibitionsandhands-onevents,includingworkingwithmuseumsandgalleries.
• Opendays.
• Activitiesthatinvolvemembersofthepublicinresearch.
Evaluationstrategiesshouldbeintegraltotheactivitydesignprocess.Iftheactivityinvolvessubmittinggrantapplicationstofundit,youmayfindithelpfultotalktopotentialfundersabouttheirevaluationrequirements,whilstpreparingyourapplication.
Thereisnomagicformulaforevaluation.Toconstructanevaluationstrategy,youneedtothinkaboutyourobjectives,thedatayoucancollectandthereportsyouhavetomake.Donewell,evaluationwillimproveyouractivityandgiveyouasenseofachievementasyouwillunderstandmoreabouttheimpactofwhatyouhavedone.
1.1UsingthisguideTheremainderofthisguidehasfivemainsectionsandaseriesofannexes.
Section2Introducesevaluationanddifferentiatesitfrommonitoring.
Section3Looksatbuildinganevaluationstrategy.Developingyourevaluationstrategyasanembeddedpartoftheactivitywillhelpyoutogetabetterunderstandingofwhattheactivitycanachieveandshouldhelpyouinmakingapplicationsforfunding.Itisthisstagewhichtendstocausethemostdifficultyasyoureconcileyourenthusiasmwithresourceconstraintsandestablishrealisticobjectives.
Section4Focusesondatacollection.Thissectiondescribesdifferenttechniques,whattheycandoandhowyoucanmakethemworktogether.
Section5Givesguidanceonhowtoanalysethedatayouhavecollected.
Section6Looksathowtodraftyourreport.
Annex1Providessomedetailedadviceonconstructingquestionnairesandphrasingquestions.
Annex2Providessomequestionnairemoduleswithspecificquestionsyoucoulduseinyourevaluation.
Annex3Providesanoverviewofwhichtechniquestousedependingontheinformationyouwant.
Annex4Isaglossaryofterms.
Annex5Suggestssomefurtherresourcesontheweb.
![Page 5: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/5.jpg)
2 Evaluation: Practical Guidelines
WhatisEvaluation? 22.1EvaluationEvaluationissaidtobe“veryimportant”,yetprojectmanagerstendtobefrightenedofevaluationbecausetheyseeitasatestandathreat.Evaluationisaprocessthattakesplacebefore,duringandafteranactivity.Itincludeslookingatthequalityofthecontent,thedeliveryprocessandtheimpactoftheactivityorprogrammeontheaudience(s)orparticipants.
Takingtheopportunitytounderstandwhetheryouachievedwhatyousetoutto,howwellyoudidit,whatimpactyouractivityhashadandtoreflectcriticallyonboththeactivitiesandprocesseswillbenefityouandyouraudiences.Thisknowledgecanbeusedinternallybyyourteamtodriveimprovementandexternallytodemonstrateachievements.
2.2MonitoringThesamebasictoolsforgatheringandanalysingdatacanbeusedforcollectingbothevaluationandmonitoringinformation.Asaresult,thereisoftenconfusionbetweenmonitoringandevaluation.
Monitoringisaboutcountingthingsandensuringyouractivityisontrack.Thingslikenumbersofevents,audiencenumbersandnumbersofCDROMsdistributed,etc.aremonitoringdata,notevaluationdata.
2.3ThedifferentaspectsofevaluationTherearethreeaimsofevaluation:
• Tosupportthedevelopmentofyouractivity(formativeevaluation).
• Toensureyoumanageitbetternexttime(evaluationofyourprocesses).
• Toassessthefinalimpactofyouractivity(summativeevaluation).
2.3.1Formativeevaluation
Youshoulduseformativeevaluationduringthedevelopmentoftheactivitytotestideas,concepts,venues,timingsandprototypesonrepresentativesoftheaudience.Thiswillhelpyouassesswhatsort
of‘productanddeliverychannel’isgoingtobemosteffectiveatreachingandengagingyourtargetaudiences.Youmayneedtotestthingsoutwithasampleaudienceseveraltimesduringyouractivity’sdevelopment;thissortofresearchisacentralpartofanevaluationstrategy.
Theemphasisatthisstageislikelytobeondiscussion-basedtools(qualitative research)thatwillgiveyouanin-depthunderstandingofyouraudience(s).Ifyouaredesigninganexhibitorawebsite,watchingpeopleinteractingwithaprototypecanprovideusefulinputforfinalisingit.Qualitativeinputatthisstagecanbecrucialtounderstandinghowtochangeyouractivitytoimproveitsappeal.Tomakeimprovements,it’snogoodsimplyknowingpeopledidn’tlikeitunlessyouknowwhattheydidn’tlikeandunderstandwhytheydidn’tlikeit.
However,don’tover-eggthepudding.Youneedtothinkabouthowmuch,ifany,formativeevaluationisreallyneededasitwilladdtimeandcosttoyouractivity.Ifyouareusingatriedandtestedformula,suchasadebateformat,youmaynotneedtoincludeformativeevaluationinyouractivityplan.Ifyouhaveanydoubtabouthowsomethingwillbereceivedbyyouraudience,testit.
2.3.2Evaluationofprocesses
Youshouldevaluatetheprocessofmanaginganddeliveringyouractivity.Thiswillhelpyoutodoitbetternexttime.Thisinformationisusefultopeersandcolleagues,andwherepossibleshouldbesharedwiththewiderpublicengagementcommunitysothattheycanlearnfromyourexperiences.
Formanyactivities,thistypeofevaluationcanbehandledentirelybytheprojectteamandissimilartoconductingaresearchproject,oryoumaywantexternalhelp,seesection 3.5.Ifthereareanumberofpeopleinvolved,makesureyouhavescheduledintotheworkprogrammeopportunitiestotalkthroughprogressandanydifficultiesorissuesthatarearising.Don’tforgettoincludethistimewhenyouarecostinganactivity.Itwillbetimewellspentandwhenpresentedaspartofyouroverallevaluationstrategy,willshow
![Page 6: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/6.jpg)
Evaluation: Practical Guidelines 3
2
Remember,summativeevaluationismainlyaboutimpacts.Thismeansthatthecountingofoutputs,suchasnumberofpeopleatanevent,numberofhitsonawebsite,etc.isnotpartofyourevaluation.Thisdataisimportantandyoushouldbegatheringit,butstrictlyspeakingthisismonitoringdata,notevaluationdata.
2.3.4Benchmarkingandbaselines
Itsoundsobvious,butifyou’retryingtochangesomething,thenyouneedtoknowthestateofaffairsbeforepeopleinteractedwithyouractivity,sothatyoucanseeiftherehasbeenachangeafterwards.
peoplemakingfundingdecisionsthatyouknowwhatisinvolvedinrunninganactivity.
2.3.3Summativeevaluation
Summativeevaluationisthetypeofevaluationwithwhichpeoplearemostfamiliar.Thislooksatoutcomesandmeasureswhetherornotyouhavemetyouraimsandhadanyimpactontheaudience.Youshouldaskquestionssuchas:
• Howmuchdidtheaudienceenjoyyouractivity?
• Diditchangepeople’sunderstanding/knowledgeorattitudes?
• Hasitinfluencedtheiractions/behaviour?
• Howbiganimpactdidithaveonthosewhoengaged?
Sotheemphasisislikelytobeonnumericaldata,butdepthofunderstandingcanbeimportant,especiallyifyou’reinterestedinwhatpeoplelearntorwhatmessagetheytookaway.Qualitativedatacanalsobecrucialinexplainingwhatliesbehindyourquantitative data.Thereisnoreasonwhyquestionnairescannotaskquestionsthatwillhelpyoutounderstand‘why’peoplegivecertainresponses.Thisisespeciallytrueifyouhavequalitativeresearch,perhapsfromyourformativeevaluation,onwhichtobaseyourquestions.
Youneedtothinkverycarefullyabouttheimpactsyou’reaimingtoachieve,becauseinordertoevaluateimpactstheymustbemeasurable.Youneedtothinkabouttherealisticlevelofimpactthatyoucanmakeandthepracticalitiesofidentifyingthatimpact.
![Page 7: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/7.jpg)
4 Evaluation: Practical Guidelines
3BuildinganEvaluationStrategy
Thissectionwillhelpyouplanyourevaluationandsupportanyapplicationsforfunding.
Table3.1Thissectioncovers:
Buildinganevaluationstrategyrequiresmovingthroughhighlevelaims,toobjectivesetting,identifyingindicatorsandmeasures,andselectionofappropriatetoolstogatherdataagainsttheindicatorsandmeasures.Thischapterofferssometoolsandwaysofthinkingthatwillsupportyouinthisprocess.
3.1SettingaimsThefirstthingyouneedtodoistoclarifyyouraim(s).Whatdoyouwanttoachieve?Whatisthepurposeofyouractivity?Thisisbigpicturestuff.Thereareseveralreasonswhypeopleandorganisationswanttodevelopactivitiestocommunicateresearchtoawideraudience.TheScienceforAll1reportpublishedbytheDepartmentforBusiness,InnovationandSkillsinFebruary2010,mapsthemyriadofmotivationsresearchershaveforwantingtoengagewiththepublicandsummarisesthemasfollows,to:
• winsupportfor‘science’,whichincludesallfieldsofacademicresearch,
• maketheworldabetterplace,
• developskillsandinspirelearning,
• enhancemycareer,
• createamoreefficient,dynamicandsustainableeconomy,
• increasethequalityandimpactofmyresearch,
• enhancesocialcohesionanddemocraticparticipation,
• beethical,accountableandtransparent.
TheResearchCouncilsincreasinglyexpectsomelevelofpublicengagementbytheresearcherstheyfund,seethe“ConcordatforEngagingthePublicwithResearch”(http://www.rcuk.ac.uk/per/Pages/Concordat.aspx).YoumightwanttocheckwhetheryourfunderhassignedtheConcordat.
TheResearchCouncilsviewpublicengagementwithresearchasapossiblewayofenabling,and/ordemonstratingthatyouareconsideringtheimpactofyourresearch,seePathwaystoImpact(http://www.rcuk.ac.uk/kei/impacts/Pages/home.aspx).Otherresearchfundersmayalsorequire,encourage,orofferfinancialsupportfor,publicengagementwork.
YourinstitutionmaybecommittedtopublicengagementandbeasignatorytotheNationalCo-ordinatingCentreforPublicEngagement’s“ManifestoforPublicEngagement”.http://www.publicengagement.ac.uk/why-does-it-matter/manifesto
TheHigherEducationFundingCouncilforEnglandisconsideringhowtoincludetheimpactofresearchbeyondacademicimpactintotheResearchExcellenceFramework.Foranup-dateonhowthisisprogressingseehttp://www.hefce.ac.uk/research/ref/
Whateveryourdrivingforce,havingaclearaimwillhelpyoutostarttheprocessofdevelopingboththeactivityandanevaluationstrategy.
1Thedefinitionof‘science’usedintheBISScienceandSocietystrategyis“deliberatelywide,encompassingphysical,biological,engineering,medical,naturalandsocialdisciplines,includingresearchintheartsandhumanities”.http://interactive.bis.gov.uk/scienceandsociety/site/defining-science/
Sub-section Contents
Settingaims
Settingobjectives
Beingclearaboutthedifferencebetweenaimsandobjectivesandensuringthateacharerealistic.
Choosingyourevaluationtools
Identifyingthedatathatyouwillneedtocollect,andthetoolswithwhichtocollectit,todemonstratewhetheryouhaveachievedyourobjectives.
Howmuchevaluation Beingclearaboutwhatyoucanevaluate.
In-houseorindependent
Canyoumanagetheevaluationyouthinkyouractivityneedsandyourfunderswant?
Ifyourunintotrouble Whattodoifyouractivitystartstogowrong.
![Page 8: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/8.jpg)
Evaluation: Practical Guidelines 5
33.2SettingobjectivesNextyouneedtosetobjectives–thesearethethingsyouneedtodotoachieveyouraim.Eachofthemotivatingfactorslistedabovewillyielddifferenttypesofobjectives.Forexample,ifyouwanttoenhancedemocraticparticipationyouwillsetdifferentobjectivesfromsomeonewhowantstoenhancetheircareer(and/orthatoftheircolleagues).Anactivitywiththefirstaimwillbefocusedonmakinganimpactontheexternalaudience,whilstanactivitywiththesecondaimmayfocusonmakinganimpactonpeoplewithinyourownorganisation.However,havingaimsthatrelatetochangingthingswithinyourownorganisationanddevelopingtheskillsofyouandyourcolleaguesarevalidanditisimportanttobearthisinmindasyoustarttosetobjectives.
Settingobjectivesisanart.Attheoutset,funders(whetherinternalorexternalprovidersofresources)willwanttoseethatyou’regoingtogivethemvaluefortheirmoney.However,youractivitywillbeevaluatedagainstyourobjectives,soitisimportanttoberealisticwhensettingthem.Trytokeepobjectivessimple,ideallyasinglekeyresultwillshowwhetheryouhavesucceeded.Ifanobjectiveiscomplicated,askyourselfwhetheritneedstobebrokendownintotwoormoresimplerobjectives.
Figure3.1Linkingyourobjectivestoyouraimandanyotherrequirements
Goodobjectivesettinghelpsyouthinkthroughnotjusttheevaluationstrategy,butthewholeprocessofrunningtheactivity.
MakeyourobjectivesSMART:
• Specific
• Measurable
• Achievable
• Relevant
• Time-bound
SMARThasbecomeawellknownacronym,butworkingthroughthelistinthisordermaynotbethebestwaytosetaboutdevelopingyourobjectives.Whenconstructingevaluationframeworks,trySRATMinstead.
ThetablebelowsetsoutsomeofthequestionsyouneedtothinkaboutwhenconstructingSMARTobjectivesusingtheSRATMorder.
Table3.2SMARTquestions
Your aim
Funder Objective
Stakeholder Objective
Activity Objective 2
Activity Objective 1
Activity Objective 3
SpecificWhatexactlydoyouwanttodo,withor
forwhom?
Relevant
Willachievingthisobjectivecontributeto
thedeliveryofyouroverallaimand
supportyour/yourfunders’/yourinstitution’s
goals?
Achievable
Istheobjectiveachievable?Inparticularcan
yougetitdoneinthetimeyouhave
available,withinyourbudgetandwithinthe
prevailingpolitical/institutionalclimate?
Time-bound
Whendoyouwanttoachievethisobjective
and/orwhendoyouthinkyouwillbe
abletoachievethisobjective?
MeasurableCanyoumeasurewhetherornotyouhave
achievedtheobjective?
![Page 9: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/9.jpg)
6 Evaluation: Practical Guidelines
33.2.1Specific
Aspecificobjectivehasamuchgreaterchanceofbeingaccomplishedthanageneralgoal.Tosetaspecificobjectiveyouneedtoanswerthesix“W”questionsbelow:
• Why:Specificreasons,purposeorbenefitsofaccomplishingtheobjective.
• Who:Whoneedstobeinvolved?
• Where:Identifyalocation,whichmaybevirtual.
• What:WhatarethetasksIneedtoaccomplish?
• Which:Identifyrequirementsandconstraints.
• When:Establishatimeframe.
Anobjectivethatisspecificwillusuallyhaveasingleresult,whichmeanstherewillbeanobservableaction,behaviourorachievementthatcanbedescribed.
3.2.2Relevant
Toberelevanttheobjectivemustbesomethingyoucanactuallydothatwillhelpyoutoachieveyouraim,takingintoaccounttheresourcesyouhaveforyouractivityincludingfinancial,personnelandtimeresources.Atthispointre-considerwhetherornotthespecificsyouhaveidentifiedaregenuinelyrelevanttoyouroverallaim.Iftheyare,fine,ifnotgobackandrefinethemsothattheyare.
So,ifyouraimisto“win support for my research field”thenall ofyourobjectivesmusthelpyoutoachievethisaim.
3.2.3Achievable
Itisnogoodhavingobjectivesthatyoucan’tachievegiventheresourcesyouhaveforyouractivity.Makingsureyourobjectivesareachievablemakesyouthinkabouthowmuchmoneyyouneed,whatskillsyouandyourteamneedandhowlongitwilltaketoplanandruntheactivity.Peopletendtoforgetaboutthetimeinvolvedandresearchhasshownthatresearcherssaytheygrosslyunderestimatedthetimeinvolved2.Youmaywanttolookforapartnerwhocanprovidefinancialor‘inkind’support,forexample,freeroomhireorsomeonewhohasaskillyouoryourteamlacks.
Inconsideringwhethereachobjectiveisachievableyoushouldthinkaboutthebarriersthatstandbetweenyouandachievingyourobjective.Thiswillhelpwithcontingencyplanning.Askyourself:‘whatif?’Onebarriermaybetheamountoftimethatyouhavetodelivertheproject.Aone-offengagementmayhavelessimpactthanasustainedseriesofcontacts,althoughonetrulyinspiringsessionislikelytohaveamorepositiveeffectthanaseriesofratherdullones.
Mostimportantly,don’tsetyourselfuptofail.Objectivesshouldbeambitious,buttheyshouldalsomotivatetheteamandshouldthereforebeachievable.Also,anyfunderwilljudgeyourcapabilitytoorganisetheactivitybywhetherornotyourobjectivesarefeasible.
3.2.4Time
Foreachobjectivetherehastobeatimescaleforachievingit.However,juststatingadateisnotenough,youneedtoaskyourself:
• Haveyouleftenoughtimetogeteverythingdone?
• Willothercompetingdemandsonyourtimeorthatofothersinvolved,causedelay?
Addressingthesequestionshelpstoshowthefundamentallinkbetweentimeandachievability.
3.2.5Measurable
Youneedtoestablishhowyouwillmeasureprogresstowardtheattainmentofeachobjectiveyouset.Youneedtothinkabouthowyouwillknowwhethertheobjectivehasbeen,orisbeing,achieved.Whataretheindicators?Thenyouneedtothinkabouthowyouwillmeasureeachindicator.Todetermineifyourobjectiveismeasurable,askquestionssuchasHowmuch?Howmany?Rememberthough,whatgetsmeasured,getsdone.So,becarefulthatthecapacitytomeasuredoesnotstarttodominateattheexpenseofrelevance.Seesection 3.4.
2Partnership for Public Awareness – Good practice guide,PeopleScience&PolicyLtd,EngineeringandPhysicalSciencesResearchCouncil,June2003www.epsrc.ac.uk/funding/grants/pe/ppe/Pages/goodpractice.aspx
![Page 10: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/10.jpg)
Evaluation: Practical Guidelines 7
3Youmaydecidethatinordertoachieveyouraimthereareintermediarystepsyouneedtoundertake,suchasunderstandingwhatpeoplecurrentlythinkorundertakingsimpleawarenessraisingpriortodevelopingsupportforyourresearch.Thefollowingfourobjectivescouldallsupporttheachievementofthisaim:
• Todevelopanunderstandingofwhatpeoplethinkaboutmyresearchwithinthenextyear.
• Togenerateawarenessofmyresearchinthelocalcommunitywithinthenextyear.
• That75%ofallparticipantsintheactivitiesruninthenextyearreportthattheyenjoyedtheactivities.
• Toincreasethenumberofstudentparticipantsconsideringacareerinresearchattheendoftheproject.
Thenextsectionusesthesefourobjectivestoshowhowthenatureofyourobjectivescanhelpyoutoidentifyappropriatesourcesofevaluationdata.
3.3ChoosingyourevaluationtoolsSection 4ofthisguidedescribesdifferentevaluationtoolsinsomedetail,butitisimportanttogiveearlythoughttothenatureofthetoolsthatyouarelikelytoneed.Carefulsettingofobjectiveswillhavegivenyouideasaboutwhichtoolsyouwillneedtouseasyouhavethoughtabouthowtodemonstratewhetherornotyouachieveeachofyourobjectives.Theexamplesbelowgivearangeofdifferentdatasourcesforsomeindicators–youdon’thavetousethemall.
3.3.1Evaluationtable
Anevaluationtablesupportsyouinbuildingonthethoughtprocessesusedwhenyoudefinedyourobjectivestoidentifythetypesofdatasources,andthustheevaluationtools,thatyouwillneedtousetoassesswhetherornotyouhavebeenabletomeetyourobjectives.Overleafisanevaluationtableforeachoftheobjectivessetoutin3.2.6above.
Onceyouhaveclearobjectives,youhavethemeanstofocusyourefforts,sotimespentatthisstagewillnotonlyhelpyourevaluation,butalsoyourprojectmanagement,byprovidingameanstoprioritiseyourresources.
3.2.6Fromaimtoobjectives–anexample
Fromthevariousaimsinsection3.1wehaveselectedthefollowingasastartingpoint:
Aim:toinspirepeopleaboutmyresearchfield.
Thissub-sectiondevelopssomepossibleobjectivesforthisaim.Indevelopingobjectivesyouneedtothinkabouthowyouwillrecognisethatpeoplehavebeeninspired.Thismaybedemonstratedbywiderpublicappreciationthatyouaredoingsomethingworthwhile(orlessoppositiontoyourresearch)orbyincreasednumbersofstudentsstudyingyoursubject;itmighteveninfluencetheprioritiesofresearchfunders.Withinthisoneaimyoucouldhavemanyactivitiesforverydifferentsetsofpeople,soindevelopingyourSMARTobjectivesyouneedtothinkabout:
• Targetgroup(s).
• Theresourcesyouhaveavailablefortheactivity.
• Yourtimescale.
• Thecapacityyouhavefordatacollectionwithinanevaluation.
![Page 11: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/11.jpg)
8 Evaluation: Practical Guidelines
3assetoutintheexamplebelow,butcouldalsobegroupsdefinedbygeography(wheretheylive),socio-demographicfeaturessuchasage,genderorethnicity,orexistingattitudesforexampleproresearch,scepticalofauthorityetc.
Todevelopanunderstandingofwhatpeoplethinkaboutmyresearchwithinthenextyear.
Youmaybeinterestedintheviewsofthegeneralpublicasawhole,orparticularsub-setsofthepublic.Thesub-setsmaybeprofessionalorstakeholdergroups
Table3.3(a)Fromobjectivestodatasources
Objectives Indicators Measures DataSources
Todevelopanunderstandingofwhatpeoplethinkaboutmyresearchwithinthenextyear
Publicperspectives
Viewsofspecificsetsofpeoplee.g.professionalgroupA
Viewsofspecificsetsofpeoplee.g.patientgroupB
Viewsofspecificsetsofpeoplee.g.historyteachers
Percentageawareofmyresearchfield.
Percentageunderstandingmyfield.
Percentagewithpositive/negativeviewsaboutmyfield.
Factorsunderpinningtheperspectives.
Factorsunderpinningtheperspectives.
Factorsunderpinningtheperspectives.
Factorsunderpinningtheperspectives.
PercentageofgroupAthatareawareof/understand/positivetowards/negativetowardsmyresearch.
PercentageofgroupBthatareawareof/understand/positivetowards/negativetowardsmyresearch.
Percentageofhistoryteacherswhoareawareof/understand/positivetowards/negativetowardsmyresearch.
Nationallyrepresentativesurvey.
RepresentativesurveyofprofessionalgroupA.
RepresentativesurveyofpatientgroupB.
Surveyofhistoryteachers.
Groupdiscussions.
Groupdiscussions.
Groupdiscussions.
Groupdiscussions.
In-depthinterviews.
Interviews.
Interviews.
Interviews.
Socialmedia/discussionlistsusedbyprofessionalgroupA.
Socialmedia/discussionlistsusedbypatientgroupB.
Socialmedia/discussionlistsusedbypatientgroupB.
![Page 12: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/12.jpg)
Evaluation: Practical Guidelines 9
3
Astheabovetableshows,ifyousimplywanttoknowwhetherthereisanyawarenessandwhetheryouractivitieshavecontributedtothisawareness,youcanuseeitherqualitativeorquantitativetechniques.If
however,youwanttoknowhowmuchawarenessthereisyouneedrepresentativequantitativedataandifyouwanttoknowaboutchangesovertimeyouneedrepeatedcollectionsofquantitativedata.
evensomeone),thelessyoulikeit(them).However,youcannotinspireanyoneiftheyarenotawareofyourresearch.
Togenerateawarenessofmyresearchinthelocalcommunitywithinthenextyear.
Buildingawarenessdoesnotnecessarilymean‘inspiring’.Sometimesthemoreyouknowaboutsomething(or
Table3.3(b)Fromobjectivestodatasources
Objectives Indicators Measures DataSources
Togenerateawarenessofmyresearchinlocalcommunitywithinthenextyear
Awarenessofresearch
Awarenessofresearchprocesses
Roleofyouractivityingeneratingawareness
Percentageofpeople,withinthelocalcommunity,awareofyourresearch.
Percentageincreaseinpeople,withinthelocalcommunity,whoareawareofyourresearch.
Percentageincreaseinpeople,withinthelocalcommunity,whounderstandyourresearchobjectivesandprocesses.
People,withinthelocalcommunity,becameawareofyourresearchthroughwiderdisseminationofyouractivity.
Percentageofpeople,withinthelocalcommunity,whounderstandyourresearchobjectivesandprocesses.
People,withinthelocalcommunity,becameawareofyourresearchthroughdirectinteractionwithyouractivity.
Representativesurveyoflocalcommunity.
Representativesurveyoflocalcommunity.
Surveyoflocalcommunity.
Surveyoflocalcommunity.
Representativesurveysoflocalcommunity(timeseries).
Representativesurveysoflocalcommunity(timeseries).
Groupdiscussions.
Groupdiscussions.
Socialmedia.
Socialmedia.
![Page 13: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/13.jpg)
10 Evaluation: Practical Guidelines
3
Youcanmeasureperformanceagainstthisobjectiveonlybyusingquantitative data.However,youmayalsowishtofindoutwhypeopledidordidnotenjoyyouractivitiessothatyoucanimprovethisaspectofyourperformance.Thiscanbecoveredinsurveys,butyoucouldalsouseobservationalorqualitativetoolstobuild
adeeperunderstanding.Thisisthesortofareawhereyoucanmakeyourevaluationworkontwolevels,bothformativeandsummative.
3http://www.epsrc.ac.uk/SiteCollectionDocuments/Publications/reports/EaBWreport.pdf http://www.ytouring.org.uk/productions/archive/mtg2005/evaluation.html
enjoymentalsoactstodriveotherimpacts.Evaluationshavebeenundertakenwhereenjoyment(orthelackofit)stronglycorrelatedwiththedevelopmentofmorepositive(ornegative)attitudesandenjoymenthasbeenlinkedtothedegreetowhichknowledgeisacquired,aswellasthedegreetowhichattitudesareinfluenced3.
Youmayneedtothinkabouthow‘enjoyment’mightmanifestitselfandaskaboutthataswellasadirectquestionaboutlevelofenjoyment.Forexample,enjoymentmightbemanifestedasadesiretorepeattheexperienceortorecommendtheactivitytoafriend.
75%ofallparticipantsintheactivitiesruninthenextyearreportthattheyenjoyedtheactivities.
Enjoymentofactivitiescanbeveryimportant.Ifyou(andyourcolleagues)haveinvestedtimeandquitepossiblyadditionalmoneyinanactivityandtheaudience/participantshavegivenuptheirtimetotakepart,ensuringthatallthesecontributorsenjoytheactivityisnobadthing.Inadditiontothefairlyobviousfactthatenjoymentislikelytoleadtofurtherparticipation(anddon’tforgetthatthisappliesasmuchtoyouandyourcolleaguesastoanyoneelse)
Table3.3(c)Fromobjectivestodatasources
Objectives Indicators Measures DataSources
That75%ofallparticipantsintheactivitiesruninthenextyearreportthattheyenjoyedtheactivities.
Viewsof“lay”participants
Viewsof“expert”orstakeholderparticipants
Viewsofyourteam
Proportionoflayparticipantsreportingenjoyment.
Proportionofexpertorstakeholderparticipantsreportingenjoyment.
Proportionofyourteamwhoenjoyedtakingpart.
Numberandnatureoffactorsaffectingenjoyment.
Numberandnatureoffactorsaffectingenjoyment.
Numberandnatureoffactorsaffectingenjoyment.
Representativesurveyofparticipants.
Representativesurveyofparticipants.
Representativesurveyofparticipants.
Observation.
Observation.
Observation.
Focusgroups.
Focusgroups.
Focusgroups.
Interviews.
Interviews.
Interviews.
![Page 14: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/14.jpg)
Evaluation: Practical Guidelines 11
3along-termcommitmenttotheevaluation(andquitepossiblytotheactivityaswell).Ifyouaresettinganobjectiveofthistype,youneedtobesurethatitispracticaltoassesswhetherornotyouhaveachievedit,aswellaswhetheritisachievableinitsownright.Oftenthepracticaloptionistouseintentiontostudyorthechoicesthatstudentsmakeearlierintheirstudiesasshorter-termproxyindicators,asinthisexample.
Toincreasethenumberofstudentparticipantsconsideringacareerinresearchattheendoftheproject.Thiscouldbeeitheracrossthegeneralpopulationoramongstspecificgroups.
Anactivitywithaspecificobjectiveassociatedwithrecruitmentwouldneedtohavethescopetotrackparticipatingyoungpeopleovertimeinordertoassesswhetherornotthisobjectiveisachieved.Thisimplies
Table3.3(d)Fromobjectivestodatasources
Objectives Indicators Measures DataSources
Toincreasethenumberofstudentparticipantsconsideringacareerinresearchattheendoftheproject.
Intentions
Choices
Numberofstudentsstatingintentionstostudyrelevantsubjects.
Numberofstudentschoosingtostudyrelevantsubjectsatage14.
Factorsaffectingchoicesatage14and16.
Numberofstudentschoosingtostudyrelevantsubjectsatage16.
Factorsaffectingchoicesatage14and16.
Baselinesurveysofparticipants.
Follow-upsurveysofparticipants.
Follow-upsurveysofparticipants.
Follow-upsurveysofparticipants.
Managementdatafromschools.
Managementdatafromschools.
Focusgroups.
Focusgroups.
Baselinesurveysofparticipants.
Baselinesurveysofparticipants.
![Page 15: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/15.jpg)
12 Evaluation: Practical Guidelines
3
Figure3.2providesasimilaroutputtotable3.3(c),butthedifferenceistheprocessofdevelopingtheoutput.Table3.3(c)wasdevelopedusingadirectionalcausalchain,whereasindevelopingthemindmap,thelinksaremorelikelytobeputinplaceafterthevariouscomponentshavebeenidentified.
3.4Howmuchevaluation?Theresourcesavailablewilldeterminewhatsortofevaluationyoucando.Generallyevaluationisonelinewithinanoverallactivitybudget,eachcomponentofwhichyouarelikelytobeunderpressuretoreduce.
tosettingobjectives,identifyingindicatorsandmeasuresandworkingoutthetoolsyouneedtousetocollecttherelevantinformationmaybemoreappropriateforyou.
3.3.2 Mindmapping
Someresearcherswillbemorecomfortablewithamorefluidapproach.Takingamind-mappingapproach
Figure3.2Fromobjectivestodatasources
AimWinsupport
formyresearch
That75%ofallparticipants
intheactivitiesruninthenextyearreportthattheyenjoyedthe
activities.
Feedbackfrom“expert”orstakeholder
participants
Feedbackyourteam
IndicatorA
IndicatorB
IndicatorC
IndicatorD
IndicatorE
IndicatorF
Feedbackfrom“lay”participants
Objective4
Objective3
Objective2
Measure1
FocusGroups
Observations
Survey
Interviews
Measure2
Measure1 Measure2
Factors
Proportion
![Page 16: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/16.jpg)
Evaluation: Practical Guidelines 13
3Evaluationisnotalwaysseenascontributingdirectlytotheactivityandcanthereforeberegardedasalowerpriority.Todefenditsplace,therehavebeencallstosetaspecificportionofbudgetsasideforevaluation,10%oftheactivitybudgetisoftenquoted.Thisisrarelysensible.Inthecaseofsmallactivitiesthiswouldmeanthatyoucandoalmostnothing,yetformuchlargeractivitiesasmallerfractionmayprovideampleresources.
Ratherthansetanarbitraryproportionofthebudget,itismuchbettertothinkaboutwhatinformationyouneed,howitcanbecollectedandanalysedandthentoconsiderwhethertheeffortandcostofthisworkisproportionatetotheactivity.Ifitisnot,areyoubeingeithertooambitiousornotambitiousenoughwithyourevaluation?
Also,familiarityisafactor.Ifthisisthefirsttimeyou,orindeedanyone,hasdonesomethingitmightbeworthinvestingmoreingoodevaluationsothatyoucanimproveyouractivityinthefuture,andthereforerunsmallerevaluations.Equally,ifyouarerunningafairlystandardactivitytheremaybenoneedtoevaluateitatall,especiallyifthebudgetislimited.
Don’tbeafraidtospeakupifitisnotpossibleorpracticaltoevaluatesomeorallofyouractivities.However,youshouldn’tusethisargumenttoavoidundertakingworkthatiseasilyachievableandthatwill,ifdoneproperly,provideyouandyourstakeholderswithmuchusefulinformation.
3.5In-houseorindependent?Afrequentlyaskedquestionis‘DoIneedanindependentevaluation?’Maybe-thereareprosandconstobothoptions.Somepeoplebelievethatifevaluationisatoolbywhichtoimprove,thenthereareclearbenefitstoself-evaluation.Thefeedbackisdirectandyouwillhavearealdepthofunderstandingofyouraudience,althoughyoumayhavetorecognisethattherearelimitationstothedatagatheringandanalysisskillstowhichyouhaveaccess.Youmayneedtobringinanoutsiderwithspecialistskillsifyouwanttoanswer
moredifficultquestionsaboutimpact.Theymightbefromanotherdepartmentwithinyourinstitutionoranexternalorganisation.
Ontheotherhand,somepeoplebelievethatifyouevaluateyourownactivityyouwillcheattomakeitlookgood,maybeevenwithoutrealisingit.Also,youshouldconsiderwhetherpeoplemightgiveyoumorefavourableanswersbecausetheydon’twanttohurtyourfeelingsorbecausetheybelievethattheremaybeanegativeimpactonthemiftheyarecritical.
Ifitiscrucialthattheevaluationisseentobeunbiased,thenanindependentevaluationmaybetheonlyoption.Indeed,fundersmaywellstipulatethatanindependentevaluationisrequired,perhapsbecauseofthelargesumsofmoneyinvolvedorbecausetheyaretryingtocomparethemeritsofdifferentapproaches.Thisiswherehavingclearobjectiveswillreallyhelp.Ifyouhaven’tgotany,externalevaluatorswilltrytosetthemforyouandbringtheirownvaluejudgementstobear.
Theneedforconfidentialitymayalsoinfluenceyourchoiceofinternalorexternalevaluation.
Ifyoufeelyouareill-equippedtoundertakeyourownevaluation,youmaylooktopeoplewithinyourowninstitution.Ifyouhaveapublicengagementoroutreachoffice,theyshouldbeablehelpwithyourevaluation.Otherwise,youmightconsiderapproachingsomeoneinyourresearchofficeorcolleagueswithexpertiseinsocialresearchmethodsordatacollectiontools.However,ifyou’renotasocialscientist,youshouldnotethatnotallsocialscientistsarefamiliarwithevaluationorthedatacollectiontoolsusedinevaluation.
3.6IfyourunintotroubleSomefunderswereinterviewedduringproductionofthefirsteditionofthisguide.Ingeneralfunderswanttheactivitiesthey’resupportingtobesuccessful.Ifyou’renotreachingyourobjectives,butyouron-goingevaluationmeansthatyouknowwhy,andtheevaluationenablesyoutosetoutaplanthatwillhelp
![Page 17: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/17.jpg)
14 Evaluation: Practical Guidelines
3yougetbackontrack,mostfunderswillgiveyouasympathetichearing.Youmightwanttorevisetheobjectives.Theearlierintheactivityyoutellfunderswhatishappening,themoresympatheticthey’relikelytobe.Remember,inmostcasesthefunder’sprogrammemanageristheretohelpyou.
WhensettingobjectivestheSMARTcriteriawilldriveyoutobespecific,butifthisisyourfirstexperienceyoumaynotknowwhatrepresentsanattainableandrealistictarget.Youwillhavetouseyourjudgementandmakesurethatyourformativeandprocessevaluationsgiveyoudatatoexplainwhyyouareoverorunder-achieving,andinthecaseofthelatterwhatyoucandoaboutit.Asevaluationdatabanks4grow,fundersmaybeabletoprovideadvicethatwillgivebenchmarks.Inthefutureyourdatawillhelpotherswiththisconundrum.
4Seeforexamplehttp://www.britishscienceassociation.org/forms/scicomm/evaluation/access/
![Page 18: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/18.jpg)
Evaluation: Practical Guidelines 15
ToolsandTechniquesGatheringData 44.1IntroductionWorkingthroughtheprocessofdevelopingSMARTobjectivesshouldhavestartedtheprocessofthinkingaboutdatacollectiontools.Thebasictoolsandtechniquesarebasedinsocialandmarketresearchmethods.Thosewhoworkinthesefieldsusecertaintermswithacommonunderstandingofwhattheymean.Thislanguageisexplainedinthissectionandinannex 4.
Thissectionfocusesonthetoolsthatresearchersmightusetosupportin-houseevaluations,ratherthanthemorecomplex,andexpensive,optionsthatmightbeusedbyanexternalcontractor.
Ifyouarecommissioningotherstodoyourevaluation,youshouldfocusontheintelligencethatyourequire(theobjectivesoftheevaluation)ratherthantryingtospecifythenitty-grittyoftools.However,youwillbeabettercustomerifyouunderstandthestrengthsandlimitationsofdifferenttechniquesandunderstandthelanguagebeingusedbyyoursub-contractors.
Broadlyspeakingtherearetwomaintypesofdata,quantitativeandqualitative.Forbothtypesofdatatherearewaysofgatheringdatathatwillensuretheyarereliable.
4.2QuantitativeresearchQuantitativeresearchisbestsuitedtoansweringquestionsabouthowmanypeopledidorthoughtsomething.Youcanalsoaskthemhowmuch,towhatextentandother‘measure’typequestions.
Therearetwounderlyingprinciplestoquantitativeresearch:
• everyrespondentshouldbeaskedthesamequestionsinthesameway,sothattheanswerscanbeaddedtogether;and
• theinformationyoucollectisrepresentativeofallthepeoplethattookpartin,orused,youractivity.
Thequestionscanbeaskedbyaninterviewerface-to-faceoroverthetelephone,orpeoplecancompleteaquestionnairethemselveseitheronpaperorviatheInternet.Whateverthemethod,thequestionsarehighlystructuredtoensureconsistency.Youcanusethisstructuredquestionnaireformattocollectfactualandattitudinaldataandtoexplorethereasonsbehindpeople’sinitialanswers.Oftenpeoplecandothesamething,butfordifferentreasonsandbeingabletocomparethosewithdifferentrationalescanbeimportantinmakingdecisionsonhowtoimproveanactivity.
4.2.1Quantitativesampling
Strictlyspeaking,whendrawingasampletoberepresentativeofyour‘users’everyoneyoureachbyyouractivityshouldhaveanequalchanceofbeingaskedtorespond,althoughthereisacost-qualitytrade-offandsomesamplingmethodsarebetteratthisthanothers.Samplingisaboutavoidingbiasandonlygettingresponsesfromcertaintypesofpeopleorpeoplewholikedyouractivity.Itisimportanttonotethatjusthavingalotofrespondentsisnotgoodenoughiftheyarenotrepresentativeofyouraudience.
Thetypesofsamplingtechniquesyou’llmostlikelyusetocollectquantitativedatathatarerepresentativeofyourusersare:
Census–collectinginformationfromeveryonewhoengagedwiththeactivity.
Systematicsampling–takingevery‘nth’personwhopassesaparticularspotoraccessesawebsite,requestsapacketc.
Quotasampling–ifyouknowthat50%ofyouraudiencewillbegirls(perhapsaschoolhastoldyou)thenyousetaquotaof50%ofyoursampletobegirls.Whichgirlsyouaskshouldthenbe‘random’,sothatyouhavenoreasontobelievethatthoseyouinterviewareinanywaydifferentfromthoseyoudonot.
![Page 19: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/19.jpg)
16 Evaluation: Practical Guidelines
4Youcanalsouseverysimplemethodstoselectasample,likethosebornonacertaindateinthemonth.Forexample,takingeveryonebornonthefifth,fifteenthandtwenty-fifthofeverymonthoftheyearhasbeenfoundtoconsistentlyyielda10%sample.
Samplingmethodsareallverywell,butyouwillfindthatnoteveryoneyouasktotakepartwilldoso.Themainissuewillbethatthosewhoenjoyedyouractivitywillbemorelikelytorespondthanthosewhodidnot.Thisiswhygettingahighresponserateisimportant.Thehighertheproportionofthoseyouselectforinterviewwhorespond,themoreconfidentyoucanbethatyourresultsarerepresentativeofyouraudience.Usinginterviewersmeansyoucangetamorerepresentativesamplethanrelyingonself-completionasthereisabitofpressureonpeopletotakepartbecausetheywon’t(generally)wanttoberude.However,ifyoucan’tbesurethatthosewhodidn’trespondarenodifferentfromthosewhodid,makesureyouincludethelimitationsofyourdatainyourreport.Also,youneedtoaskatleast100peoplebeforeyoucanstartstatingpercentages,evenifyouaskeveryonewhotookpart.
SamplingandtheInternet
Withtheadventofeasytouseandfree/cheapInternetsurveytools,suchasSurveymonkey,ZoomerangandSurveyGizmoyoucandesignyourownsurveyandcirculatethelinktothequestionnairebye-mail.
However,simplybeingabletorunasurveydoesnotmeanthatthedatacollectedwillberepresentativeofanyparticulargrouporaudience.Manypeoplethinkthatcirculatingthelinktoeveryonetheyknow,orcanreachviae-maillists,willprovidealargenumberofresponsesthatcanbereportedonquantitatively.However,thisisnotthecase.
Ifthepeoplereachedbyasurveyarenotsystematicallyselectedtoberepresentativeofaparticularaudienceandarelikelytohaveparticularviewsbecause,forexample,theyallbelongtothesamee-maillist,allyouhaveundertakenisaverylargequalitativeresearchproject;evenifyouhaveaveryhighnumberofresponses.
Anobviousanalogywouldbeasurveyrunbyanewspaperthroughitswebsite.No-onewouldsuggestthatthefindingsrepresenttheviewsofthepopulationasawhole;indeedtheymaynotevenrepresenttheviewsofthenewspaperreadershipasawhole.
4.2.2Quantitativedatacollectiontechniques
Havingconstructedyoursample,ordecidedonacensusapproach,youneedtothinkaboutwhichdatacollectiontechniquetouse.Therearefourbasicquantitativedatacollectiontechniques,eachofwhichhasstrengthsandweaknesses.Thefouroptionsare:
• Face-to-faceinterviews.
• Telephoneinterviews.
• Self-completiononpaper.
• Self-completionon-line(e-mailorInternet).
Forsmall,liveevents,themostlikelymethodwillbeself-completiononpaper,wherequestionnairesaredistributedandattendeesareencouragedtocompleteandreturnthemattheendoftheeventorpostthembacklater.Forsomeactivities,face-to-faceinterviewswithasampleofattendeesisanotheroption.Thesearetoolsforgatheringinstantfeedback.
Youmaywantaconsideredresponse,ortogaugeresponsesafterparticipantshavehadtimetoreflectontheirexperienceortakesomeactions.Inthiscasesendingoutself-completionquestionnairesbypostore-mailorgettinganinterviewertocontactpeoplebytelephoneortovisitpeople,areallpossibleoptions.Thetablebelowsetsoutbasicstrengthsandweaknessesofthedifferentoptions.
![Page 20: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/20.jpg)
Evaluation: Practical Guidelines 17
4Surveystyle Strengths Weaknesses
FacetoFace
Telephone
Self-completiononpaper
On-line
Youknowtherightpeoplehaveresponded.
Youcanbemoreconfidentthatintervieweeshaveunderstoodthequestion.
Youcanrepeatquestionsthatintervieweeshavenotunderstood.
Interviewerswithdifferentlanguageskillscanaccessdifferentcommunities.
Highqualitydata.
Youcanrepeatquestionsthatintervieweeshavenotunderstood.
Youhaveagoodchanceofmakingsuretherightpeoplehaveresponded.
Interviewerswithdifferentlanguageskillscanaccessdifferentcommunities.
Moderatelyhighqualitydata.
Relativelycheaptoundertake(butrememberprintingandpostalcostsifapplicable).
Youcanuselongerquestionnairesbutyouwillgetabetterresponseiftheyareshort.
TranslationcanhelpyoutoaccesscommunitieswhosefirstlanguageisnotEnglish.
Moderatequalitydata.
Ifyouhaveadefinedsampleande-mailaddresses,youhaveagoodchanceofmakingsuretherightpeoplehaveresponded.
Pop-upsworkonaskingeverynthpersonwhovisitsthewebsitetocompleteaquestionnaire.
Youcanuseadditionalmaterialsuchaspicturestohelppeoplerespond.
Relativelycheaptoundertake.
Moderate/highqualitydata.
Resourceintensivesomostexpensive.
Thepersondoingtheinterviewcaninadvertentlybiastheresponse.Seeannex1,sectiononavoidingbias.
Quiteresourceintensivesoexpensive,butcheaperthanface-to-face.
Youneedtelephonenumbersforyoursample.
Youcannotbeconfidentthatintervieweeshaveunderstoodthequestion.
Peoplecanlookaheadatthequestions,whichmightbiastheiranswerstosomequestions.
Youcannotbesurethattherightpeoplehaveresponded.
Youcannotbeconfidentthatintervieweeshaveunderstoodthequestion.
Dependingonhowyouset-upthequestionnairepeoplemaybeabletolookaheadatthequestions,whichmightbiastheiranswerstosomequestions.
Ifyoudonothaveadefinedsample,youdonothavequantitativedata.
Youcannotbesurethattherightpeoplehaveresponded.
OnlyrespondentswithInternetaccesscantakepart.
![Page 21: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/21.jpg)
18 Evaluation: Practical Guidelines
4It’sagoodideatothinkaboutdoingexitsurveysaspeopleleaveeventsorfollow-upsurveysbytelephone,postore-mail–dependingonwhatinformationyouhaveaboutpeople.Ifyou’vedevelopedawebsiteyoucanimplantwhatarecalled‘pop-up’questionnairestocollectdatafromevery‘nth’visitororcollecte-mailaddressestosendaquestionnairelater.
Experienceshowsthate-mailssenttospecificindividualsyieldhigherresponseratesthanpop-upquestionnairesandassection4.1.1emphasises,alowerresponseratereducestheconfidencethatyouhavecollecteddatathatrepresentstheviewsofthewholepopulationinwhichyouareinterested.
4.2.3Constructingaquestionnaire
Therearemanyfactorstoconsiderwhendesigningaquestionnaire.Thefirstis‘whatdoyouactuallywanttoknow?’whichtakesyoubacktotheobjectivesofyouractivity.Donotwasteyourtimeandthoseofthepeoplerespondingbyaskingquestionssimplybecauseyoucan,everyquestionshouldjustifyitsinclusion.
Onceyouhaveidentifiedtheinformationyouneed,thelength,structureandlayout(forself-completionquestionnaires)willimpactontheresponserate,whichyouwanttomaximise.Rememberthatthequestionsyouaskwillinfluencepeople’sanswerstolaterquestions,soyouneedtothinkabouttheorderinwhichyouaskquestions.
Questionscanbe‘pre-coded’wheretherespondentselectsananswerfromalistor‘open-ended’whererespondentscanwriteintheirowncomments.Somepre-codedquestionsincludean‘other,specify’categorywherethosewhohavenotfoundapre-codetotickcanwrite-intheirviews.Remember,ifyouuseopen-endedquestions,someonewillhavetoreadalltheresponses.
Annex 1looksatthefactorstoconsiderandquestioningtechniques,suchasattitudestatementsandscales,thatwillgiveyouappropriatedata.AlthoughAnnex1iswrittenforself-completionpaperquestionnaires,thesamebasicprinciplesapplyforall
datacollectionmethods. Annex 2givesyouexamplequestionsandquestionnairemodules.
Itisalwaysagoodideatopilotaquestionnaireonthetargetaudience.Ideallythispilotingshouldincludeconversationsabouttherespondent’sunderstandingofthemeaningofboththequestionsandtheavailableresponses,sothatyouaresurethatthequestionsareelicitingtheinformationyouwant.Youcanusefriends,familyorcolleaguesforthistokeepcostsdown,iftheyareareasonableapproximationofamemberofthetargetaudienceandnotasimmersedintheactivityasyou.
4.3QualitativeresearchQualitativemethodsenableyoutoaddressthedeeperquestions,suchaswhypeopledidordidnotlikeanactivity,whytheyfeltitwasgoodorbad,andwhatyoucouldchangetomakeitbetter.Thenumbersofferedbyquantitativetechniquescanofferaseductivepromiseofcertainty,buttherearemanyinstanceswherethedepthofinsightfromqualitativeworkwillofferthebestvalueformoney.Thisisparticularlytrueforformativeevaluation.
Socialandmarketresearchersusetheterm‘qualitativeresearch’torefertoindividual,one-to-onein-depthinterviewsandgroupdiscussions/focusgroupsconductedbysomeonewhohasbeeninvolvedinthewholeprocessoftheevaluationandwhothereforehasadeepunderstandingoftheobjectivesoftheactivity.Thisislikelytobeeitheryouoroneofyourteamandbecausequalitativemethodsallowyoutointeractdirectlywith‘users’youcantestoutideasthatyouformduringtheevaluationprocess.
![Page 22: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/22.jpg)
Evaluation: Practical Guidelines 19
4
Bothin-depthinterviewsandgroupdiscussionscanbeone-offeventsorre-convenedtoallowtheinterviewees/participantstoofferfurtherviewsaftertimeforreflection.
4.3.1Qualitativesampling
Qualitativeresearchisaboutdepthofunderstanding,sosamplestendtobesmall.Youwillfindthatyoudon’tneedtotalktolargenumbersofpeoplebeforeyoustopgettingnewinformation.Peopletendtobeselectedtogiveyouacross-sectionofyouraudienceratherthanarepresentativesample.Ratherthanusingthestructuredapproachestoselectpeoplethatyouneedforquantitativesurveys,youpickindividualswhomeetyourcriteriaforinclusion.Ifpossible,trynottoincludepeoplewhoknoweachother.Theirviewsmay
besimilarandtheremaybespecificdynamicswithintheirrelationshipsthataffectthoseofthegroupmorewidely.
Thesmallsizeofyoursampleandthewayithasbeenselectedmeansthattheresultsarenotstatisticallyrepresentativeofthegroupyourparticipants‘represent’.Notonlymightyouhavemissedsomemarginalviews,becauseyoursampleissmall,youhavenoideahowprevalenttheirviewsareinthewiderpopulationbecauseofthewayinwhichtheyhavebeenselected.Foryourfindingstobestatisticallyrepresentativeyouneedtouseoneofthemethodsdescribedinthesectiononquantitativemethods.
Toensurethatyouincludetherightsortsofpeople,youshouldthinkaboutthemainvariablesor
Technique Strengths Weaknesses
In-depthinterviews
Groupdiscussionsorfocusgroups
Usefulfortalkingtothosewithbusydiaries.
Goodforsituationswhererepliesmaybesensitive,wherepeoplemightbereluctanttosaythingsinfrontofothers.
Idealwhenyouwanttocollectdetailsthatarelikelytobeveryindividual,suchashistoriesofindividualinvolvementinanactivity.
Canbeface-to-faceorbytelephone.
Interactionbetweenparticipantscanstimulateideas.
Goodforformativeevaluationasthistypeofresearchcangiveyouquickfeedbackonhowpotentialaudiencesviewyouremergingideas.
Flexible-Youmightputsimilarpeopletogethertoencouragethemtoexploretheissuesindepthfromsimilarperspectives.Alternatively,youmightcreatemixedgroupstogivepeopleexposuretodifferentviewpoints.
Canbeface-to-faceoron-line.
Relativelyexpensiveasyouonlyinvolveonepersonatatime.
Stimulusonlycomesfromtheinterviewer,sotheremaybelittlechallengetoviewsandlimitedcapacitytoprovokereflection.
Notgoodforsituationswhereyoumightwanttoexplorepersonalexperiences.
Audioorvideorecordingisrequiredifyouwanttododetailedanalysisafterthefact(youneedtogetsignedpermissionfromparticipantsforaudioandvideorecording).
Strongpersonalitiescandominatesoyouneedaskilledfacilitator.
Mencandominatemixed-genderdiscussions,soyoushouldthinkcarefullyaboutseparatingmenfromwomenindiscussiongroups.
Youngeradultstendtospeaklesswhengroupedwitholderpeoplesoyoushouldthinkcarefullyaboutagemixes.
![Page 23: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/23.jpg)
20 Evaluation: Practical Guidelines
4characteristicsthatyouthinkwillbeimportant,givenyouractivity.Forexample,aremenandwomenlikelytoresponddifferently?Areyoungerpeoplelikelytobedifferentfromolderpeople?Arethoseinurbanareaslikelytobedifferentfromthoseinruralareas?Iftheanswertothesetypesofquestionsisyes,thenyouneedtomakesureyouincludemenandwomen,youngerandolderpeopleandurbanandruraldwellersinyoursample.AndalthoughmorepeopleinBritainliveinurbanareasthanliveinruralareas,youmightincludethesamenumberofeachgroupinyourresearch.However,youdon’tneedtocovereverysegmentyouidentifyexplicitly–soyoudon’tneedoldermeninruralareas,olderwomeninruralareas,andsoon.Youcouldjusthaveoldermeninruralareasandyoungerwomeninruralareaswithyoungermeninurbanareasandolderwomeninurbanareas.Runningfocusgroupscanbeexpensive,especiallyifyouuseanoutsidecontractor,soyoumayneedtoprioritisewhichgroupstoinclude.
Anoptionthatisbecomingincreasinglyavailableison-linediscussionseitherusingvideo/audiostreamsortextualconversations.Forthesediscussionstobeclassedasqualitativeresearchyoumustapplythesamedepthofrigourtoselectionoftheparticipantsandcoverageofallsalientpointsasyouwouldforface-to-facework.Lessstructuredapproachesusingnewmedia,andparticularlysocialnetworkingtoolsarediscussedinsection4.4.
4.3.2Anecdote
Qualitativeresearchisnotanecdotal.Thesampleandthediscussionarestructuredbyresearchersagainstclearobjectivesfortheevaluation.
However,giventhebudgetsthataresometimesavailableforengagementactivities,youshouldthinkofconversationsinthemarginsofaneventasprovidingusefulfeedback.Whileanyfeedbackisbetterthannone,youneedtobecarefulabouthowyouinterpretsuchad-hoccomments.Forexample,theoneparticipantwhomadeacommentmaybeatypicalandhavesought
yououtsimplytomaketheirpoint.Itisusefultoknowwhatsuchanindividualthinksastheremaybeotherswhofeelthesame,butequallyanysingleindividualmayhavetheirownagenda.
Thinkingofanecdotalevidenceasasourceofideastobeinvestigatedmorethoroughlyusingmorerigoroustechniquesisahelpfulwayofattachingtherightweighttothissortoffeedback.
4.4Otheroptions4.4.1Observationalresearch
Observationinvolvestheplannedwatching,recording,andanalysisofbehaviourasitoccursina‘natural’setting.Inmostcasesyouwillbeobservingpeopleinteractingwithyouractivity.Itisparticularlyusefulforunderstandinghowpeopleusewebsitesorflowthroughanexhibition,aswellastoexplorehowtogetmorepeopletoactivelyengagewithtalksanddiscussions.
Observationenablesevaluatorsto:
• understandindividuals’engagementwithspecifictasksandprocesses,
• understandindividuals’attitudesandrelationshipsincontext,
• definekeyissuesthatmaybefollowed-upininterviewsandsurveys,
• formrelationshipswiththeparticipants,whichwillhelpwithanyfollow-upinterviewing,and
• iteliminatesthebiasofself-reportingthatmayoccurininterviewsandsurveys.
Youcanobserveandmakenotesonhowindividualsinteractwithyouractivity,butyoucanalsouseobservationalmethodstocomparegroupdynamicsacrossevents.Thisisdonebyhavingastructureagainstwhichyoucanrecorddetailsyouareinterestedin,forexample:theorderinwhichwebpagesareaccessed,thenumberandtypeofparticipantsatanevent,levelofinputtodiscussions,typesofpeoplewhoactivelyparticipate,mainsubjectsofconcernandsoon.
![Page 24: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/24.jpg)
Evaluation: Practical Guidelines 21
4Thereducingprice,andincreasingquality,ofdigitalphotographyandvideomakevisualrecordingsaviableoption,butrememberthatyoushouldseekthesignedconsentofparticipantstoanyrecordingandparentalconsentforrecordingsofyoungpeople.
Sampling
Observationalresearchismainlyqualitativewhenconductedaspartofformativeevaluation,see4.3.1forguidanceonsampling.
Whenobservingfinalactivitiessuchasexhibitions,youneedtothinkabouttimesofdayanddaysofweek,asdifferentpeoplearelikelytoengageatdifferenttimesandyouneedtoensurethatyoursampleisasrepresentativeaspossibleofallthosewhoengagewithyouractivity.
4.4.2Visitors’book
Avisitors’book(orsimilartoolsuchasa“PostIt”board)isagoodwayofcapturingthethoughtsofvisitorsandgettingfeedback.However,onlythosewhoarehighlymotivatedwillgivecomments.Sothecomments,whilehelpfulinthedevelopmentofyourprojectandsimilar,futureprojects,willnotberepresentativeofyourachievedaudience,oryourinitialtargetaudience.
4.4.3Socialmedia
Socialmediareferstoweb-basedandmobiletechnologiesthatenableinteractionbetweenusers.Thereareliterallyhundredsofsocialmediatools.AmongthebestknownareFacebookandTwitter,whichareillustrativeofthedifferenttypesoftools.
Thesetoolsallenableyoutointeractwiththepeoplewhoengagewithyouractivity.Theyalsoallowthe‘users’tointeractwitheachother.However,feedbackthroughsocialmediaisbestseenasanecdotalevidence,becauseonlythosewhoareverymotivatedarelikelytotakepart.Youshouldregardtheissuesthatarisethroughthesediscussionsastopicstobeexploredwithmoresystematicqualitativeorquantitativetechniquesratherthandefinitiveevidence.
TheResearchInformationNetworkhasproducedausefulguideforsocialresearchersthatdocumentsthedifferenttypesofsocialmediaandhowtheycanbeusedinsocialresearchandmuchofthiscarriesovertoevaluation.See:http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/social-media-guide-researchers
4.4.4Recordkeeping–managementdata
Recordkeepingcanstimulateself-reflectionandmightbethoughtofasself-observation.Atitssimplestyoucouldkeepadiary,whichrecordsyourthoughtsandfeelingsthroughouttheprocessaswellasyourreflectionsontheprocessitself.Thisformsarecordofwhathappenedandwhen,andisausefulresourcewhenlookingathowyoucoulddothingsbetterinthefuture.Youcouldalsoaskusersofyouractivitytokeeprecordsoftheirinteractionsandtheirthoughtsoveraperiodoftime.Thewidespreadavailabilityofdigitalphotographymeansthatkeepingpictorialrecordsisanoptionthatisreadilyavailable.
4.4.5Mediaimpact
Ifyouwanttoraiseawarenessyoumightsetanobjectiveaboutpressormediacoverage.Measuringtheimpactofthiscanbeverydifficult.Somepeoplemeasurecolumninchesandusethesalesorreadershipfiguresofthepublicationtoestimatethenumbersreached.However,noteveryonereadseverypageofanewspaperormagazineandtheimpactonreadersisgenerallyunknown.
Whenitcomestotelevisionandradioprogrammes,viewingandlisteningfiguresmaybeavailable.Nevertheless,evenintelevision,wherethereisaconsiderableamountofprogrammeresearch,dataonimpactratherthanenjoyment,isunlikelytobeavailable.
4.5ConfidentialityMarketandsocialresearchoperatesfromthepremisethatinformationgivenbyrespondentsinresearchprojectsisconfidential.Questionnairesusuallyreassurepeoplethattheywillnotreceivemarketinginformationorsalescallsasaresultoftakingpartandthatnoone
![Page 25: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/25.jpg)
22 Evaluation: Practical Guidelines
4willknowwhattheypersonallyhavesaid,otherthanthepeopleprocessingthedata.
Peoplemaybehappytoputtheirnametotheirviewsandrespondingtoanevaluationofapublicengagementactivityisunlikelytocauseproblemsforrespondents.Nevertheless,theremaybecaseswherepeoplefeelthatyouwillpassontheirviewstoothersandthatthismaybedetrimentaltotheminsomeway.Offeringconfidentialitydoesn’tusuallycauseproblemsinreportingandcanbebeneficial,butitmaymeanyoucan’tconducttheevaluationin-house.Somepeoplemaybemoreforthcomingtosomeoneunrelatedtotheactivity–anothercaseforanindependentevaluator.
Youshouldalsonotethataskingpeopletheirviewsinfrontofothersmayrestrictwhattheyarepreparedtosay,dependingonthetopic.
YoualsoneedtobeawareoftheDataProtectionActsee:www.informationcommissioner.gov.uk
![Page 26: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/26.jpg)
Evaluation: Practical Guidelines 23
ToolsandTechniquesDataHandling 5Theprevioussectionlookedatthevariousmethodsforcollectingdata.Thischapterlooksathowyouanalyseit.Hereitisassumedthatdatacollectionandanalysisisbeinghandledin-houseratherthanthroughspecialistsub-contractors.Ifyouareusingsub-contractors,buttheyarenotprovidingyouwithafinalreport,thenyouwillneedtospecifytheanalysisyouwantandtheformatinwhichyouwantthedata.
5.1Quantitativedata5.1.1Coding
Thefirstthingtodowithyourquestionnairesistocodeanyquestionswhererespondentshaveenteredtheirownanswersratherthantickingabox.Readthrougheachquestiononeatatime,thatis,lookatalltheresponsestoQ1together,allthosetoQ2togetherandsoon.Youshouldbelookingforsimilarresponsessothatyoucandraw-upa‘codeframe’forthequestion.Thisallowsyoutoaddtogethersimilarresponsesfromdifferentpeople.Onceyouhaveyourcodeframeyouwillgiveeachcodeanumber.Thenyouneedtoreadeachquestionnaireandputtheappropriatecodeorcodes(peoplemayhavesaidmorethanonething)bythesideofthequestion.Itisthisnumberthatyouwillenterintoyourdataset,nottheverbatimcomments,theseshouldbekeptseparately.
5.1.2Dataentry
Ifyouareusingpaperquestionnairesyouwillhavetoinputyourdata.Dataentryisatimeconsumingandrelativelyspecialisttask.Youhavetomakeverysureyoudon’tmakemistakes.Forsmallamountsofdatayoumightdoitin-house,forlargeramountsconsiderusingaspecialistdataentryfirm,thespeedandqualityofentryislikelytoyielddividendsandthecostisonlygoingtobeafewhundredpoundsforthesizeofjobyou’relikelytohave.Astrengthofon-linesurveysisthatthedataisautomaticallycollated.Similarlyifyouhavecommissioneddatacollectionitwillbeprovidedintheelectronicformatthatyouspecify.
5.1.3Analysis
Ifyouonlyhaveasmallnumberofrespondents,perhapsfewerthan50,youcoulddoyouranalysisbyhandbyjustcountingthroughthequestionnaires.However,ifyouwanttodoanyanalysisbeyondtotalcountsofhowmanypeoplegaveeachanswer,oryouhavemorerespondents,thenthesimplestwaytoanalysesmalldatasetsistousespreadsheets.Itreallyisworththetimetoenterthedata.
Figure5.1overleafshowsanexamplespreadsheetwithrawdataentered.Eachcolumnrepresentsasinglerespondentandeachrowaquestion.Eachcellrepresentstheanswertoaquestion.So,forexample,forQ1thepossibleanswersareWedandSat.‘1’indicatestheansweris“yes”and‘0’thatitis“no”.Sothepersonwhoseanswersareenteredincolumn‘C’said‘yes’to‘Wed’and‘no’to‘Sat’.Itmayseem,andindeedis,timeconsumingtobreakdatadowntobinarycomponents,butthisisvitalifyouwanttobeabletodomoresophisticatedanalysisofyourdatathansimplecountsofthewholepopulation.
Summingacrosstherowsgivesyouthetotalnumberofresponsesforthatansweracrossthesample.However,you’requitelikelytowanttoanalysethedataagainstkeyvariables,suchasageorgender.Thesevariablesareoftenknownascross-breaksandifyouarecommissioningdatacollectionandpreparationyouwillbeaskedtospecifythecross-breaksthatyouwouldliketohaveincludedinthedatasetthatisprepared.
Itisfairlystraightforwardtosetupsimplecross-breaksinaspreadsheetpackageusingtheSUMIFfunction.Thislogicalfunctionallowsyoutosumtheresponsesinonerow(orcolumn)wheretheresponsesinanothercolumnmatchaparticularcriterion.So,intheexampleabovethefunctioncouldbesettosumrow4(verygood)forallthecaseswhererow3equals1i.e.peoplewhoattendedonSaturday.IfpeoplewhoattendedonSaturdayaremorelikelythanthosewhoattendedonWednesdaytohavethoughtthattheeventwasverygood,thenyouhaveanimportantfinding.
![Page 27: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/27.jpg)
24 Evaluation: Practical Guidelines
5
Thecapacitythatspreadsheetsofferisstraightforward,butimportantanalysisshowswhyitisvitaltobreakdatadownintothebinarydata.Thissimpleformulawouldnotworkifyouhavetwosinglerowsforquestions1and2withSaturdayorWednesdayenteredinthefirstrowandVGood,Good,OK,Poor,VPoor,NoopinionorDKinthesecond.Assoonasadatasetbecomeslargeenoughthatyoumightwanttointerrogatesub-samplestheneedforproperdatapreparationbecomesparamount.
Forlargedatasetsandcomplexsurveysitisbettertousebespokepackagesfordataanalysis.TheindustrystandardisSPSS(StatisticalPackagefortheSocialSciences),whichoffersfargreaterprocessingpower
thanspreadsheetsandaccesstoawiderrangeofstatisticaltestsandmethods.
5.2QualitativedataQualitativedataisgatheredbyrecordingthediscussions.Recordingmaybeliteralthroughaudioorvideo(ineithercasepermissionshouldbesoughtfromtherespondentsbeforerecordingstarts)orvianote-takingtorecordkeypoints.Bearinmindthatifyouchoosetotakenotesratherthantaperecord,youwillloosesomeoftherichnessofthedataandyouwillneverbeabletorecaptureit.Mostsocialandmarketresearchersrecordfocusgroupsandone-to-oneinterviewssotheycanconcentrateonrespondingto,andobserving,theinterviewee(s).
Figure5.1RawData
![Page 28: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/28.jpg)
Evaluation: Practical Guidelines 25
5Youcanuseflipchartsinagroupsituationandthisallowsparticipantstoconfirmthatyouhaveaccuratelyrecordedwhattheymeant.Thisapproachalsomeansthatsomeanalysisisbeingundertakeninsituaskeypointsareidentifiedandrecordedbythegroupsupportedtoagreaterorlesserextentbythefacilitator.
Analysisofrecordedconversationscanbeundertakenbymakingtranscriptsorbylisteningbacktothetapes,makingnotesandrecordingquotes.
Whattolookfor:
• Mainandsub-themesandissues(acrossdifferentgroups/individuals)thatemergefromthediscussion.
• Ideasfromparticipantsthatwillsupportthedevelopmentofyouractivity.
• Trackingindividualviewsthroughthediscussion,exploringhowandwhyviewschange(iftheydo)andanypreconceivedorhyperbolicviews.
• Thecontext,andthustheinterpretation,ofcomments.
• Illustrativequotesforuseinreports.
• Thelanguageused–thiswillhelpwiththedesignofquantitativequestionnaires.
Itisunlikelythatyouwillbeusingqualitativedatatoproveordisproveahypothesis,ratheryouwilllookatdatatoseewhatissuesemergefromthem.Sotheapproachisnot“Was the event boring because the speaker was no good?”ratheritis“How enjoyable was the event?”,“Why was it enjoyable/not enjoyable?”
Inessence,interviewdatacanbetreatedintwoways.Somepeopletakecommentsatfacevalueandcategorisethetextintothemes.Itisimportanttorememberthough,thatqualitativeresearchisaboutmorethanjustwhatpeoplesay.Peopledonotalwaysexpressthemselvesclearly,maycontradictthemselvesandtheirbodylanguagewilladdtoyourunderstandingofwhattheymean.Yourunderstandingofwhattheymeanisimportant,butyouneedtorecognisethatitisyour understanding.
Oneofthesimplestwaystoanalysequalitativedata,thatallowsyoutoincorporatethecontextofthe
discussion,is‘charting’.Listeningbacktothetapesorworkingthroughtranscripts,youidentifythemainissuesorthemesraisedinthediscussions.Youplotwhomadeeach(relevant)comment,leadingyoutobeabletoidentifythetypeofpersonwhoraisedeachissueandthereforeforwhomthiswasanimportantpoint.However,asyouworkthrougheachdiscussionchartingit,youtakeintoaccountcontextandintendedmeaningaswellasthepuretext.
TherearespecialistsoftwarepackagestosupporttheanalysisofqualitativedatatheseincludeNVivo,ATLAS.tiandFrameworkamongstothers.Suchpackagesallowqualitativedatafromdifferentsourcestobelinkedtogetherinasingleworkunit,sothatelectronicannotationscanbeusedtolinkthemes,dataandissuesacrossthedifferentsources,whichcanincludenotesandtranscriptsofdiscussions,pictures,audioandvideofiles.Importantly,usingthesetoolsdoesnotreplacethein-depthreadingoftranscripts/notes.Itonlyprovidesanelectronicmeansofcataloguingandlinkingsourcestosimplifytheprocessofdrawingtogetherevidencefromdifferentinterviewsorsources.
5.3ObservationaldataObservationaldataprovidescontextualinformationagainstwhichtounderstandtheresultsthatemergefromsurveysandfocusgroupsaswellasprovidingdataonhowpeopleengagewithyourexhibitionorwebsite.
Socialresearchersareawarethatwhatpeopledoisnotwhatnecessarilywhattheysaytheydo,andobservationaldata,takenwithotherinformation,canhighlighttheseinconsistencies.Forexample,afacilitator’sobservationsofgroupmembers’behaviourmayrevealwhetheraparticularindividualwhohadnotbeencontributingtothediscussiondidsobecausetheyfeltalienatedbytheprocessorsimplybecausetheywereshy.
Asmentionedinsection4.4,observationaldatacantakeanumberofforms.Hand-writtennotesorfielddiariestakenbytheevaluatorthroughouttheprocess,whichmaybeaugmentedbystillphotographsorvideo
![Page 29: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/29.jpg)
26 Evaluation: Practical Guidelines
5footage,canprovidearecordthatcanbesubjectedtoanalysisinthesamewayasqualitativedatafrominterviewsandfocusgroups.
Atthesametime,systematisedobservationscanrangefromquitesimpletalliesofattendanceataneventandbasicdemographicinformation,tofairlycomplexcategorisationsandcodingofbehaviour.Thesedatacanbefairlystraightforwardlyanalysedinastandardspreadsheetpackage.Astheamountofdatagrowsandtheformsinwhichtheyareheldmultiply,itislikelytobecomemoreappropriatetouseoneofthespecialistanalyticaltoolsforqualitativeresearch.
![Page 30: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/30.jpg)
Evaluation: Practical Guidelines 27
Reporting 66.1ReportingGatheringandanalysingdataisallverywell,butreporting,orsharingdata,iswherethebenefitsofgoodevaluationstarttoberealised.Therearefourmainaudiencesforevaluation:
• Youandyourteam.
• Yourfunder.
• Otherstakeholders.
• Yourpeers.
6.1.1Youandyourteam
Evaluationplaysanimportantroleinhelpingyoutoimproveyourengagementactivities.So,theprimaryaudienceformuchoftheevaluationisyourteam.Whathaveyoulearnt?Howwillyouapplythisinthefuture?Aretherethingsthatyou’velearntthatcarryacross,notjusttootherpublicengagementactivities,butyourotherworkoryourinteractionswithfundersandusers?
Choosethemostsuitableformatforyourreport.Itmaybeinwriting,butitmaybethatapresentationenablestheteamtoreflectbetterontheirexperiences.
6.1.2Yourfunder
Whoeverhasprovidedtheresourcesforyouractivitywillprobablywanttoknowwhatthey’vegotfortheirmoney,sothefirstaimofyourevaluationreportisdemonstratingtheoutcomesandimpact.However,fundersareinterestedinsharinggoodpractice,soifthereareimportantpiecesoflearning,sharethem.
Yourfundermayhavegivenyouastandardformatforyourreport.Youshoulduseit,evenifitisconstrictive.Theyprobablyusethestandardformattoallowthemtosumdataorcollateinformationfromacrossdifferentactivities,sotheycanreporttotheirseniorsandaccountfortheirbudgets.Ifthereareotherthingsthatyouwanttosay,sendanadditionalnoteorreport.Fewfundersarelikelytocomplainaboutgettingtoomuchfeedback.Youneverknow,youmightwanttoapproachthemagain,inwhichcaseyoushouldshowhoweffectivelyyoumanagedtheactivityandhowthoroughlyyouevaluateditandreportedonit.
6.1.3Otherstakeholders
Youmayhavehadpartnerswhocontributedresources,peopleorexpertisetotheevent.Makesurethatyoushareyourevaluationresultswiththem.Itwillhelpthemtoassessthevalueoftheircontributionsandmayinfluencetheirfuturedecisionsongettinginvolvedinfutureactivities,eitherwithyouorothers.
6.1.4Yourpeers
Finally,thereareyourpeerswhoaregettinginvolvedinpublicengagementwork.Youmayknowsomedirectly,butalsotalktoyourfunderaboutanynetworkstheyhaveorareawareof,forsharinggoodpractice.Yourevaluationcouldprovidetheanswersthatsomeoneelsehasbeenlookingfor.Itmightbethatyou’vecrackedaproblem,orthatyoustumbledintoatrapthatyoucanhelpotherstoavoid.Ittakessomecouragetoadmitthelatter,butpeoplewillbeverygratefulandinthefuturetheymayrepayyouwithvaluableadvice.Youmightwanttopostyourevaluationathttp://www.britishscienceassociation.org/forms/scicomm/evaluation/access/ sootherscanlearnfromyourexperiences.
6.2ReportstructureYourreportshouldbestructuredaroundthequestions/objectivesyourevaluationsetouttoaddress.Theseshouldstemfromtheobjectivesyousetforyouractivity.Monitoringinformationisalsolikelytohavearoleinyourfinalreport.Forclarity,youshouldhave:(seetableoverleaf)
![Page 31: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/31.jpg)
28 Evaluation: Practical Guidelines
6
Forthesectionsabouttheobjectives,trytoturntheobjectiveintoaquestionorseveralquestions.Thenassessalltheinformationyouhavefrombothmonitoringandevaluationandallocateeachpieceofinformationtoasectionofthereport.Ifyouhaveinformationthatdoesn’taddanything,don’tincludeit.
Don’tfeelyouhavetoreporteverypieceofdetail.Itcanbeusefultohaveshort‘conclusions’attheendofeachmainsectionthatsumsupthemainpointsfromthesection.
Theuseofchartstoillustratenumericaldata,whichcaneasilybederivedifyouhaveyourdatainaspreadsheet
Section Contents
Executivesummary
Introduction
Annexes
Lessonslearnt
Conclusions
Objective2,3,etc.
Objective1
Unexpectedoutcomes
Somepeople,especiallyseniorpeopleinfundingorganisationsorthoseonassessmentpanels,willonlyreadthissection.Itshouldthereforesetouttheactivity’sobjectivesandgiveashortdescriptionoftheactivity,butimportantly,itmustpulloutthekeypoints.
Therearetwomainstylesforsummaries.Anexecutivesummaryshouldmirrorthestructureofthefullreportsothatanyonewhowantsmoreinformationonacertainsectioncaneasilyfindit.
Ageneralsummaryisshorterandmorenarrativeinstyle.
Thismeansyoushouldthinkaboutwhichstyleisrequiredandwritethissectionlast.
Setsout:• thebackgroundtoyouractivity,• whyyouwantedtodotheactivity,
• whatyouhopedtoachieveandwhy,
• theaimsandobjectivesoftheactivity,
• theaimsandobjectivesoftheevaluation,
• thestructureoftheremainderofthereport.
Include:
• fulldetailsofyourmethodology,
• howyouselectedyoursample,
• copiesofquestionnairesandtopicguides,
• someinformationabouthowyouanalysedyourdata.
Whatyouwoulddodifferentlynexttimeandwhy.
Keylearningpointsforothers.
Adiscussionofunexpectedoutcomesandhowtoensuretheyeitheroccurornotagain,asappropriate.
Asummingupofthekeyachievementsofyouractivity,itsstrengthsandweaknesses.
Asabove.
Theobjectiveanddatarelatingtowhetheritwasmetwithsomediscussionastowhytheactualoutcomeoccurred.
Describeanyunexpectedoutcomesandwhethertheyaregoodorbad.
![Page 32: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/32.jpg)
Evaluation: Practical Guidelines 29
6suchasExcel,willhelpyouandthereadertoidentifyhighsandlowsandtrends.Usebulletpointsundertables,chartsandgraphstohighlightthemainpoints.Quotesfrominterviewsandfocusgroupswillservetobringtolifethespiritoftheactivity.Thinkabouthowyoumightuseimagestoillustrateyourresearchfindings.Somepeopleliketohavethissortofdetailinannexestokeepthemainreportshort.You’llfindthatdifferentpeoplehavedifferentpreferencesabouthowdataispresented,soitissensibletocheckwithkeyreaders,suchasyourfunder,toseewhattheyprefer.
Theconclusionssectionshouldpulltogetherallthedataandgettothepointof‘whatdoesitallmean’?‘Sowhat?’Whatdotheprecedingsectionstellyouinanutshell?Somepeoplewillonlyreadthissection,especiallyifthereisnotanexecutivesummary.
YoushouldusePlainEnglish.Yourreportshouldbeaccessibletoasmanypeopleaspossible.Themainaudienceswillbeotherpeoplelikeyouandyourteamandyourfunder.Remember,manyfunderswillnotbeevaluationexpertseither.
Donotbetemptedtousejargon.Thereareoccasionshowever,whenyou’llwanttouseverypreciseterms.Ifyoudo,makesureyoudefinethemsomewheresoall
yourreadersknowwhatyoumean.Forexample,thisguidehasaglossaryofterms.
Don’tbeafraidtomakevaluejudgementsorgiveyourviews,especiallywhenyou’reconsideringhowyoumightdothingsdifferentlynexttime.Remember,youhavelearntalotfromconductingyouractivityandcanspeakwithsomeauthority,especiallyifyou’vebackeditupwithgoodevaluationdata.
![Page 33: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/33.jpg)
30 Evaluation: Practical Guidelines
EvaluationQuestionnairesandQuestions
Annex1
QuestionnairedesignThissectionfocusesondesigningpaperquestionnairesforselfcompletionsurveys.Theprocessofdesigningon-linequestionnaires,whichwillalsobecompletedbytherespondent,isverysimilar.Youcan‘route’peoplesotheyarenotaskedquestionsthatarenotrelevanttothembasedontheirpreviousanswersinpaperquestionnaires,butwithon-linequestionnairesyoucanprogramtheroutingandsoitcanbemuchmorecomplex,forexample,combiningtheanswerstomorethanonequestion.
Maketherespondent’sexperiencepositive
Makesurethattherespondentfindstheexperiencestraightforward.Theymayevengainsomethingfromtheprocess.
Makesurethatyourlanguageisappropriatetoyouraudience.Thisisespeciallyimportantwithyoungeraudiences,butalsobearinmindthatgeneralliteracylevelsintheUKarenotthesameasthoseofgraduatesandthatforsomepeopleEnglishmaynotbetheirfirstlanguage.
Youcanusecolour,pictures,cartoon“smileyfaces”orotherlighterapproaches,butmakesurethesematchthemoodofyourevent.Youwanttoengagerespondents,butyoudonotwantthemtofeelthatyouarebeingcondescending.
Makesureyourrespondenthasthechancetosaywhatisontheirminde.g.byusingageneralopen-endedquestionattheend.
Ifpossible,pilotthequestionnaireonafewpeoplebeforecirculatingitwidely.Thiswillhelpyouidentifyanydifficultieswithwordingorconcepts.Evenpilotingitonyourcolleagues,friendsorfamilywillprovidesome
usefulfeedbackonhowtoclarifythequestionsandthewayyouaskthem.
Reluctancetogivefeedback
Thiscanbethecurseoffeedbackquestionnaires–peopleeitherdon’twanttohurtyourfeelings,sotonedowntheircomments;orforgetthemselvesandlaunchscathingattacksthatdon’treallyhelpyoutoimprove.Thekeyistoensurethatpeopleunderstandtheirfeedbackisimportantandcanhelpyou.Emphasisethatcriticalfeedbackwillhelpyougetbetterandthatitwon’thurtyourfeelings(eventhoughitmight)andthatpositivefeedbackalsohelps,becauseitshowswhatworkswell.
Theproblemwithleavingquestionnairesfortheaudiencetocompleteonleavinganeventisthatthosewhohadagreattimearemostlikelytofillitin.ThosewhohateditarealsomorelikelytofillitinthanthosewhohadanOKtime.Also,somepeoplewilljustwanttoleaveandnothavethetimeortheinclinationtocompletethequestionnaire,althoughleavingafewminutesattheendoftheactivityforpeopletofillinaquestionnairemayhelp.However,unlessyouhavesomewayofensuringthatahighproportionofthoseyoutargetcompletethequestionnaire,theresultswillnotnecessarilyberepresentativeofyourwholeaudienceandyouwillnotbeabletotellinwhatwaysortowhatextenttheresultsarebiased.
Thisisnottosaythatthedataisuseless,butitneedscarefulinterpretationandifyoucancalculateorestimatetheproportionthatcompletedaquestionnaire,thiswillhelpyouassesstheaccuracyofyourresults.Thehighertheresponserate,themorerepresentativetheresultswillbeandanythingover65%isverygood.Ifeveryonewhoengagedwithyouractivityresponded,thedataisrobust,eveniftherewereonlysixpeople.Justdon’ttrypercentagingonfewerthanabout100people.
![Page 34: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/34.jpg)
Evaluation: Practical Guidelines 31
Maximisingtheresponserate
Ifyouractivityisanevent,distributethequestionnaireatthestartandaskpeopletocompleteitbeforetheyleave.
• Makethequestionnaireshort,simpleandrelevant.
• Usepre-paidenvelopestoincreasetheresponseratewhereyouhavedistributedquestionnairesbypostorwhereyouthinkpeoplewillwanttopostbackthequestionnaire.
Askingpeopletocompletetheirquestionnairesattheendofengagingwithyouractivitywillincreaseyourresponserateandmayimprovethequalityoftheinformationyougather.However,itwillmeanthatyoucanonlyaskrelativelyfewquestionsbecausepeoplewillonlybepreparedtogivealimitedamountoftime.Youalsoneedtothinkaboutthetimepeoplehaveforreflection.Youmaygetamoreconsideredresponsewhenpeoplehavehadachancetothinkabouttheactivitylater.
Remember,trytomakeitfun.Inresponsetoaconsultationonup-datingthisguide,onerespondentsaidthatatatalkaboutaerodynamicstheaudiencewasaskedtomaketheirfeedbackquestionnairesintopaperaeroplanesandtherewasaprizeforthosewhocouldgettheirplaneintotheboxatthefront!
Confidentiality/dataprotection
Youmusttakeallreasonablestepstomakesurethattherespondentsarenotadverselyaffectedbytakingpartintheevaluation.Youmustkeeptheirresponsesconfidential,unlessyouhavetheirpermissiontodootherwise,andyoumustnotdoanythingwiththeirresponsesthatyouhavenotinformedthemabout.So,unlessyoumadeitclearwhenyougavethemthequestionnaireoronthequestionnaireitself,youcannotusetheresultstobuildadatabaseformarketing,forexample.
Therearetwousefulsourcesofinformation:theInformationCommissioner’sOfficewebsite,andtheMarketResearchSocietywebsite,whichhasvariouscodesofconductrelatingtodataprotectionandconfidentialityissues.
FortheInformationCommissioner’sOffice:www.dataprotection.gov.uk
Formarketresearchguidelines: http://www.mrs.org.uk
Ifyouareworkingwithchildrenorvulnerableadults,suchaspeoplewithlearningdifficultiesortheelderly,youandyourteamshouldbecheckedbytheCriminalRecordsBureau.Seehttp://www.direct.gov.uk/en/Employment/Startinganewjob/index.htm?CID=EMP&PLA=url_mon&CRE=crb Thisisveryimportantforanycontactaswellaspartoftheevaluationasforcontactduringyourproject.
Length
Keepitfocused,simpletocomplete,andasshortaspossible–onetotwopages.Thiswillmaximisethenumberofresponsesandminimisethetimeitwilltakeyoutoanalyseit.
Donotaskforinformationthatyoudonotplantouse:itwasteseveryone’stime.However,donotworryifyouaskquestionswhichlaterprovenottobeuseful;justdon’taskthemnexttime.
Structure
Designthequestionnaireasafunnel,movingfromsimple,unthreateningandnon-sensitivequestions,tothosethatrequiremorethoughtandmaybemorepersonalinformation.
Mostquestionnaireswillbenefitfromamixof‘closed’or‘pre-coded’and‘open’questions,wherepeopleentertheirresponseintheirownwords,asthishelpstokeeppeopleinterested.
Itisusuallybesttoplacesensitivequestionstowardstheendandalwaysgiveareasonwhyyouwantpersonalinformationsuchasdemographicdata(e.g.age,sex,education,andethnicity).
Avoidlongbatteriesofagree/disagreequestions,asrespondentswilldriftintogivingautomaticanswers-breakupquestionsvisuallyifthequestionnaireislong.
![Page 35: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/35.jpg)
32 Evaluation: Practical Guidelines
Avoidingbias
Therearemanydifferentsourcesofpotentialbiasinresearch.Theseinclude:
• Questionnairebias-leadingquestions.
• Methodologicalbias-forexample,on-linesurveysexcludepeoplewhodonothaveinternetaccess.
• Samplingbias-forexample,askingforfeedbackonlyfromthosewhoaskedquestionsataneventmaybeaverypoorindicationofhowtheaudienceasawholefelt.
• Responsebias-thosewhocompletequestionnairesmaybeverydifferentfromthosewhodon’t.Thisisamajorissueforallsurveyresearch.
• Questionbias-mostpeopleliketoputthemselvesandtheirbehaviourinagoodlight.Ifyouaskaquestionwhichembarrassestherespondentormakeshim/herfeelbad,theymaymassagethetruth.So,forexample,asking“How often do you eat chips?”ratherthan“Do you eat chips?”givespeoplepermissiontosaythattheydowithoutfeelingthattheyshouldn’t,andthosethatdon’t,justsaynever!
• Interviewerbias–thenatureorbehaviouroftheinterviewermeansthatpeoplearereluctanttogivesomepossibleanswers.Thewaythatyouframediscussionsandwhoispresent,aswellasthewayyoulookandspeak–canhaveasignificanteffectonresponses.It’simpossibletoentirelyneutralisethesethings,butyoucanatleastbeawareoftheeffectyoumayhave.
• Biasedscales–thathave,forexample,lotsofoptionsfor‘good’,butonlyonefor‘bad’;forexample,excellent,verygood,fairlygood,notverygood.
QuestiondesignThemostimportantthingsinquestiondesignare:
• avoidleadingquestions.
• avoidbiasedscales.
• neverasktwoquestionsinone,youdon’tknowwhichpartpeoplehaveanswered;forexample,‘wouldyousaythatyouunderstoodthespeakerandthediscussant?’
• neveraskhypotheticalquestions,forexample,‘willyougobacktoworkafteryourbabyisborn?’Youcanask:‘doyouplantogobacktoworkafteryourbabyisborn?’
• makesuremeasurementbandsdon’toverlap.Sodon’task‘howoldareyou?’withasetoftickboxanswersthatrun15-40,40-60,60andover.Theanswersmustrun:15-39,40-59,60andover,otherwisethoseaged40and60don’tknowwhichboxtotick.
Thereareanumberofdifferenttypesofquestionsyoucanuseandbelowisanintroductiontoeach.
Pre-codedquestions
Thesearetheeasiestquestionsforpeopletoanswerandtheeasiesttoanalyse,buttheycanbedifficulttodraft.Pre-codedquestionstaketheformofashortquestionthatencapsulatesonethought,followedbyalistofpossibleanswers.Peopleareaskedtotickoneormoreoftheanswers.Atypicalquestionwouldbegender–peoplejustticktheirgenderfromthe‘male’,‘female’options.Anothertypicalquestionwouldbe:‘Whichsectionsoftheexhibitiondidyouenjoy?Pleasetickallthosethatapply.’Thequestionwouldbefollowedbyalistoftheexhibitionsectionsandpeoplewouldtickallthesectionsthattheyenjoyed.
Thesetypesofquestionsareeasytoanalysebecauseyoujustadd-upthenumberofpeoplegivingeachresponse.
Theycanbedifficulttodraftbecausetheyneedtobeveryclearandtheanswerlistmustberelevanttothequestionandunderstoodbytherespondent.
Ifyouareusing‘pre-coded’questions,youneedtobeconfidentthatthecategorieschosenreflectthespectrumofactualexperience.Ifyouaskpeoplewhattheirfavouritesubjectswereatschoolandoffer“maths”,“science”,“designandtechnology”and“other”asoptions,butyouractivitywasbasedinanartgallery,you’renotlikelytogetaveryaccuratepictureofpeople’srealfavourites.Youcanalwaysusean“other,writein”categorytocaptureanythingyou’vemissed,
![Page 36: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/36.jpg)
Evaluation: Practical Guidelines 33
butrememberyou’llhavetoreaditbeforeyoucananalyseit.Oryoucouldjusthave‘other’anddon’tforgettoadda‘don’tknow’option.However,ifthelististoolongsomepeoplewillonlytickthetopfewandnotreadthefulllist.Pilotingthequestionnairewillhelpyoutoensurethatyouhaveasfullalistaspossible.
Usingscales
The1-5Likertscaleisthemostcommonlyusedformofrating.Itissimpletounderstandandrelativelydiscriminating.Thescaleiscommonlyanchoreddescriptivelye.g.5=Agreestrongly,4=agree,3=neitheragreenordisagree,2=disagree,1=disagreestrongly.Youshouldalsoadda“don’tknow”category.Otherscalesthatareusedincludescoringonalineofonetotenorapercentagescore.
Anotherwayofdifferentiatingbetweenpeople’sviewistopresentthemwithstatementsthattherespondentchoosesbetween.Theseareoftenorderedonanimplicitscale,butyouareaskingtherespondenttoticktheonethatbestfitstheirviewwheninfacttheymaynotagreewithanyofthosepresented.Anexampleofthistypeofquestionis:
Whichofthefollowingstatementsbestreflectsyourfeelingsaboutsciencetoday?
a) It’scontinuallymakingourlivessaferandbetter.
b) It’schangedmanythingsforgood,butIwonderhowmuchmorethereisthatcanbeachieve.
c) It’sproducinglotsofnewthings,butI’mnotsureweneedthemall.
d) It’soutofcontrolanddamagingourlivesandenvironment.
Whenitcomestotheanalysisallyoucanreallydoispresentthepercentageofrespondentswhoagreewitheachstatement.
Ranking
Rankorderingisbestavoided–manyrespondentswon’tdoitproperly,unlessyousticktoaskingforfirst,secondandthirdchoices.Otherwisepeoplegetconfusedandgetpushedintodeclaringpreferences
theydon’treallyhaveortheymayjustgiveup.Youcangetroundthisbyaskinghowimportantanissueis,usingaLikertscale.
Attitudinalquestions
AttitudinalquestionsareusuallyaskedusingLikertscalesanditisusualtohaveasetofattitudinalquestionsthattogethergiveyouanoverallimpressionoftheattitudesoftherespondents.
Ifyouhaveasetofattitudestatements,someshouldbepositiveaboutyouractivityandanequalnumbershouldbenegative.Ingeneralpeoplearemorelikelytoagreewithstatementsthantodisagreeandyouneedtobeawareofthisinyouranalysis.Becarefulnottohavestatementssuchas“There is no point in studying history”.It’sverydifficultforpeopletoknowwhethertheyshouldagreeordisagree.Peoplewhoagreemightsay,‘no,thereisn’t’,ratherthan‘yes,Iagree’.Ifyou’reusingnegativestatementskeepitshort–forexample,“Studying history is pointless”.Thenpeoplecansay‘yes,itis’.
True/falsestatements
Agoodwaytotestwhetherpeoplehavelearntanything,orwhethertheyknewanythinginthefirstplace,istoasktrue/falsestatements,butyouneedtobesurethatthereiscertaintyontheissue.
Open-endedquestions
Open-endedquestionsallowrespondentstoanswerquestionsintheirownwords,butyouhavetoreadthemall.Also,somepeoplewillbeverybrieforsaynothing,soyoumaynotbeveryenlightenedbytheirresponses.However,anopen-endedquestionattheendofaquestionnaireactsasacatch-all.Ifyouwanttouseopen-endedquestionsherearesomeguidelines.
• Usebroadopeners:Who,what,where,when,and(especially)why,toencouragepeopletoexpressthemselves.
• Balancedopenquestions(whydidyoulike[it],whydidyoudislike[it])helptherespondentstructureananswerwithoutfeelingpressuredtogiveaparticularreply.
![Page 37: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/37.jpg)
34 Evaluation: Practical Guidelines
• Inface-to-faceinterviewsconductedbyaninterviewer,usingopenphrasessuchas‘Tellmeabout’,‘Tellmemore,’or‘Whydoyousaythat?’encouragespeopletotalk.
• Avoidquestionsthatcanbeansweredwithasimpleyesorno.
• Avoidaskingmorethanonequestionatthesametime.
Analysis
Thinkabouthowyouwillusethedatacollectedwhendraftingyourquestionnaire.
Ifyouwanttocalculatetheaverageageofvisitors,usingbands16-25,26-40,41-60,over60willnotallowyoutodoso.Youwillneedtoaskactualage.
Questionnairechecklist
Belowisachecklistthatwillhelptoensureyouhaveaclear,easytocompleteandanalysequestionnaire.Youcanjustinitialtherowwhenyou’vedonethattaskandaddthedatetokeeptrackofwhereyouare.Itmightbehelpfultoalsokeepnotesofwhyyoumadecertaindecisionsaboutthequestionnaire.
Section Initials Date Notes
Questionformat&wording
Doallquestionshaveoneunambiguousfocus(i.e.thereareno‘twoinone’questions)?
Doquestionshave‘don’tknow’codes(formostquestionsthiswillbeavalidresponse)?
Domultipleresponselistshave‘noneofthese’options(wherethiscouldbeavalidresponse)?
Dosingle-responsequestionshaveaninstructionaskingtherespondentto‘tickoneonly’?
Domultiple-responsequestionshaveaninstructionaskingtherespondentto‘tickallthatapply’?
Whereanswersaresoughtasnumericvalues,arethereinstructionstellingrespondentswhatformattoentertheirdatain(e.g.forfivepleasewrite/typeas5)andifnecessary,whatunit(e.g.UKpounds).Thisismoreimportantforon-linesurveysbutwillhelpwithdataentryofpaperquestionnaires.
Topandtail
Doestheintroductioncontainallnecessaryinformationaboutconfidentiality,purpose,lengthandcontactsforfurtherinformation?
Hasare-contactquestionbeenasked?(Follow-upresearchwillbedifficultwithoutthisastheInformationCommissioner’sOfficedoesn’treallylikeitanditcontravenestheMarketResearchSocietyCodeofConduct).
![Page 38: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/38.jpg)
Evaluation: Practical Guidelines 35
Section Initials Date Notes
Overview
Hasthequestionnairebeenpilotedorcheckedforsensebycolleagues?
Havequestionsbeenmappedagainsttheinformationyouneed?
Isthenumberofopenquestionsappropriateorisitlikelytocauseanunreasonableamountofcoding?
Isthenumberofopenquestionsappropriateorisitlikelytoputanunreasonableburdenontherespondents?
![Page 39: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/39.jpg)
36 Evaluation: Practical Guidelines
QuestionnairemodulesAnnex2
Thissectionprovidessomequestionnairemodulesthatcanbecopiedintoyourquestionnaire.Thesemoduleswillalsoenableyoutodesignsimilarquestionsonotherissuesthatyouwanttoaskabout.
Whowasthere?
Togetbasicsocio-demographicinformationusesimpletickboxes.
Thereareatremendousnumberofcharacteristicsthatcanbeexplored.Youneedtofocusonwhatmattersinrelationtoyourobjectives.Isitpeoplefromageographiclocation,ofaparticularageoracertainmindsetthatwereimportanttoyou?Doyouwantamixofgenderand/orethnicgroupsorareyouspecificallytargetingonegroup?
Peoplecanbeverysensitiveaboutgivingthistypeofpersonaldata,soyoushouldexplainwhyyouwantit.That’swhyitisusualtoleavethesetypesofquestionsuntiltheendofthequestionnaire.Ifpeoplereallydon’twanttogiveyouthisinformation,theywillstillhavegivenyoufeedback.Ifyouputthisfirstandtheydon’twanttocompleteit,theywon’tanswertheotherquestionseither.
Q Sothatwecanseethetypesofpeoplewhocametoourevent,foreachofthequestionsbelow,pleaseticktheboxthatbestfitsyou.
OR
Q Sowecananalysethefindingsfromthissurveyfordifferentgroups,foreachofthequestionsbelow,pleaseticktheboxthatbestfitsyou.
Areyou
Male
Female
Areyou
Infull-timeeducation
Employed–full-time
Employed–part-time
Notcurrentlyemployedbutlookingforwork
Retired
Other
Whichofthegroupslistedbelowdoyouconsideryourselftobelongto
White
Black-African
Black-Caribbean
Black-Other
Pakistani
Indian
Bangladeshi
Chinese
Other(pleasewritein)
Whatwasyouragelastbirthday?
Lessthan16
16-30
31-45
46-50
51-65
Over65
Approximatelyhowfarfromheredoyoulive?
Lessthan1mile
1-5miles
6-15miles
Morethan15miles
Don’tknow
Ifyouwanttocalculatetheaverageageofvisitors,youwillneedtoaskpeopletowriteintheiractualage.
![Page 40: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/40.jpg)
Evaluation: Practical Guidelines 37
Whatweretheirattitudes?
ForattitudinalinformationitisprobablymostappropriatetouseLikertscales.AgoodsourceofattitudequestionsisthePublicAttitudestoScienceseries.TheDepartmentforBusiness,InnovationandSkills(BIS)defines‘science’asfollows:
By ‘science’ we mean all-encompassing knowledge based on scholarship and research undertaken in the physical, biological, engineering, medical, natural and social disciplines, including the arts and humanities, which is underpinned by methodologies that build up and test understanding about our world and beyond. http://www.bis.gov.uk/policies/science/science-and-society
Themostrecentreportandquestionnairecanbeaccessedhttp://interactive.bis.gov.uk/scienceandsociety/site/
Theabovereportalsogivesthefiguresfortheresponsesofanationallyrepresentativesampleofadultstothesestatements.Thismeansthatyoucandevelopapictureofhowtypicalyouraudienceisofthewiderpublic.
Belowisaseriesofstatementsthatpeoplehavesaidaboutscience.Foreachpleasetickwhetheryouagreestrongly,agree,neitheragreenordisagree,disagree,disagreestronglyordonotknow.
Statement Agree Agree Neither Disagree Disagree Don’tKnow Strongly Strongly
ThespeedofdevelopmentinscienceandtechnologymeansthatitcannotbeproperlycontrolledbyGovernment.
Scienceisgettingoutofcontrolandthereisnothingwecandotostopit.
Itisimportantthatyoungpeoplehaveagraspofscienceandtechnology.
Thebenefitsofsciencearegreaterthantheharmfuleffects.
Scienceandtechnologyaremakingourliveshealthier,easierandmorecomfortable.
![Page 41: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/41.jpg)
38 Evaluation: Practical Guidelines
UKlawstatesthatallmedicinesmustbetestedonanimalsbeforebeingusedonpeople
PleaseputXintheappropriatebox
True
False
Don’tknow
AnyresearcherintheUKcancarryoutexperimentsthatinvolvepeoplewithoutalicence
PleaseputXintheappropriatebox
True
False
Don’tknow
ThetestingofnewdrugsonpeopleisnotregulatedintheUK
PleaseputXintheappropriatebox
True
False
Don’tknow
Youneedalicencebeforeyoucanplantgenetically-modified(GM)cropsintheUK
PleaseputXintheappropriatebox
True
False
Don’tknow
TheGovernmentdoesnotregulatethedisposalofradioactivewasteintheUK
PleaseputXintheappropriatebox
True
False
Don’tknow
Therearestrictlawsgoverningtheuseinresearchofgovernmentpapers
PleaseputXintheappropriatebox
True
False
Don’tknow
Didtheeventwork?
Atonelevelyoumightwanttoknowifpeoplesimplyenjoyedengagingwiththeactivity.Givethemthechancetotellyou,butyoucangetmorevaluebyfollowing-upthequestionandaskingthemtosaywhy.Youcandothisbyhavingalistofpossiblereasonsandaskingthemtotickthosethatapplytothem.Oryoucanaskthemtowrite-inusingtheirownwords.This“writein”approachworksbestforsmallnumbersofrespondents,becauseyou’llneedtoreadwhatpeoplehavewritten.Forlargernumbersofpeople“pre-codes”willbemoremanageable,butleaveopenthe“other”optionsothatyoucancaptureanswersyouhadn’tthoughtof.
Youmightbelookingformoresophisticatedfeedback.Iftheprimaryfunctionoftheactivitywastogiveparticipantsthechancetocontributetheirviewsandcomments,itisimportanttoseewhetherthishasbeenachievedandwhatfactorshaveenabled,orhindered,effectiveparticipation.
Ifinformationprovisionwaspartoftheprocesswasitaccessibleanduseful?Similarly,ifyouwereusing“experts”howwastheircontributionrated?
Whatdidtheylearn?
Herearesomeexamplesoftrue/falsequestions.Youwillbeabletodevelopyourown,dependingonyourfield.
Foreachofthestatementsbelow,pleasetickwhetheryouthinkitistrueorfalseorwhetheryoudon’tknow.
![Page 42: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/42.jpg)
Evaluation: Practical Guidelines 39
Youmightalsothinkaboutwhattheexpertsexperienced.Didtheyenjoytheprocess,whathavetheylearned,havetheirattitudeschanged?
Theexamplesbelowshowsomeofthequestionsthatyoumightconsiderasking.Forthese“diditworkforyou?”questions,themostvaluablebitoffeedbackcanbethewhyorwhynotthatunderpinstheyesornoanswer,soitisalwaysworthleavingsomespaceforthis.
1.Ienjoyedtheevent
Pleaseticktheboxthatbestdescribesyourview
StronglyAgree Agree
Disagree StronglyDisagree
2.Pleasewriteinwhichpartoftheeventyouenjoyedthemost
3.Pleasewriteinwhichpartoftheeventyouenjoyedtheleast
4.WhatdoyouthinkIshouldchangeabouttheevent?
Wereyouabletoexpressyourviewsfreelyandopenly?
PleaseputXintheappropriatebox
Yescompletely
Yes,butsometimesIfeltnervous
NotasmuchasIwouldhaveliked(ifyoutickthispleasesaywhyintheboxbelow)
Notatall(ifyoutickthispleasesaywhyintheboxbelow)
Whywasthis?Pleasewrite-in
Don’tKnow
Didyouunderstandthetalk?
PleaseputXintheappropriatebox
Yes,easily
Yes,butonlyafterthediscussion
Notverywell(ifyoutickthispleasesaywhatmighthavehelpedintheboxbelow)
Notatall(ifyoutickthispleasesaywhatmighthavehelpedintheboxbelow)
Whatmighthavehelpedyouunderstandthesciencemoreeasily?
Pleasewrite-in
Don’tKnow
Didyoufindtheexperts:
PleaseputXinanyboxyouagreewith
Helpful?
Confusing?
Abletoansweryourquestions?
Selfimportant?
Didnotwanttolistentoyouropinions?
Abletoexplainthemselvesclearly?
Eagertolisten?
Ifyou’dliketosayanythingelseabouttheexperts,pleasewriteitintheboxbelow
![Page 43: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/43.jpg)
40 Evaluation: Practical Guidelines
Annex3SummativeEvaluationSchema
Belo
wis
as
chem
ato
hel
pyo
uth
ink
abou
tth
ety
peo
finf
orm
atio
nyo
uw
ant
and
how
you
mig
htc
olle
ctit
,dep
endi
ngo
nth
ena
ture
ofy
our
activ
itya
ndy
our
obje
ctiv
es.R
emem
ber:
Tom
easu
rec
hang
eyo
une
edt
oha
vea
bas
elin
efro
mb
efor
eth
eau
dien
cee
ngag
edw
ithy
our
activ
itya
nda
noth
ers
eto
fdat
ata
ken
afte
rth
eyt
ook
part
.You
will
need
to
ask
exac
tlyt
hes
ame
ques
tions
bef
ore
and
afte
r.
Nat
ure
ofd
ata
Dis
cuss
ion/
M
eeti
ng/T
alk
Web
site
Pro
duct
se.
g.p
ost
er/
CD
RO
M/v
ideo
Exh
ibit
ion/
Ope
nda
yS
how
/Pla
yC
om
peti
tio
n
Num
ber
ofp
eopl
e.C
ount
peo
ple
on
entr
y.C
ount
hits
.N
umbe
rdi
strib
uted
.C
ount
peo
ple
one
ntry
.C
ount
aud
ienc
eon
en
try.
Cou
nte
ntrie
s.
Type
sof
peo
ple.
Cat
egor
isep
eopl
eat
reg
istra
tion
or
byo
bser
vatio
nor
qu
estio
nnai
re
Pop-
upq
uest
ionn
aire
son
the
site
or
regi
stra
tion
proc
edur
es.
Use
ofo
rder
/req
uest
fo
rms
and
ques
tionn
aire
s.C
ateg
orise
peo
ple
one
ntry
by
regi
stra
tion
orq
uest
ionn
aire
.U
set
icke
tsa
les
or
book
ing
mec
hani
sms
to
gath
erin
form
atio
n.
Cou
nte
ntrie
s.
Mo
nito
ring
Dat
a
Eva
luat
ion
Dat
a
Nat
ure
ofd
ata
Dis
cuss
ion/
M
eeti
ng/T
alk
Web
site
Pro
duct
se.
g.p
ost
er/
CD
RO
M/v
ideo
Exh
ibit
ion/
Ope
nda
yS
how
/Pla
yC
om
peti
tio
n
Cha
nge
Cha
nge
view
s/at
titud
es.
Cha
nge
beha
viou
r.In
crea
sein
tere
st.
Incr
ease
kno
wle
dge.
Ask
peo
ple
for
base
line
view
son
a
pape
rqu
estio
nnai
re
whi
let
hey
wai
tfo
rth
eev
ent
tos
tart
or
whe
nth
eyr
egist
ert
oco
me.
Regi
stra
tion
ques
tionn
aire
on
the
site
tog
athe
rin
form
atio
n.
Dist
ribut
ion
met
hods
w
illaf
fect
the
abi
lity
to
colle
ctin
itial
dat
a.U
sing
ano
rder
ing
mec
hani
sm
allo
ws
data
to
be
gath
ered
.
Ask
for
base
line
view
son
a
pape
ror
e-m
ailq
uest
ionn
aire
w
hen
peop
ler
egist
ert
oco
me
orb
uyt
icke
ts.
Ask
for
base
line
view
son
ap
aper
or
e-m
ail
ques
tionn
aire
whe
npe
ople
boo
kor
buy
tic
kets
.
Build
ing
ina
nin
itial
da
tag
athe
ring
exer
cise
to
the
com
petit
ion
proc
ess
will
allo
w
base
line
data
to
be
gath
ered
.
Qua
lity/
fit
for
purp
ose
Stre
ngth
s.W
eakn
esse
s.
Obs
erve
the
ev
ent.
Use
exi
tqu
estio
nnai
res
and/
orfo
llow
-up
focu
sgr
oups
or
ques
tionn
aire
s.
Incl
ude
ques
tions
on
this
ina
que
stio
nnai
re
host
edo
nth
esit
e.
Reco
rdd
wel
ltim
epe
rpa
gea
ndp
age
requ
ests
.
Follo
w-u
pqu
estio
nnai
res
and
focu
sgr
oups
.Ex
itor
follo
w-u
pqu
estio
nnai
res.
Shor
tfa
ce-
to-fa
cein
terv
iew
sdu
ring
the
even
t.O
bser
vatio
n.
Follo
w-u
pqu
estio
nnai
res.
Gro
up
disc
ussio
ns.
Use
ent
rym
echa
nism
to
gat
her
feed
back
.
Inte
ract
ion
wit
hac
tivi
ty.
Obs
erva
tion
of
dyna
mic
sw
illhe
lpy
ou
plan
bet
ter
even
tsin
th
efu
ture
.
Reco
rdt
heo
rder
in
whi
chp
ages
are
ac
cess
eda
ndd
wel
ltim
epe
rpa
ge.
Obs
erva
tion
ofu
sers
and
qu
estio
nnai
res.
Obs
erva
tion.
In-d
epth
inte
rvie
ws
orfo
cus
grou
psa
ndq
uest
ionn
aire
s.Fe
edba
ckfr
oms
taff/
colle
ague
s.
Obs
erva
tion.
Q
uest
ionn
aire
sIm
plic
itin
tak
ing
part
,us
een
try
num
bers
as
am
easu
re.
Lev
elo
fdi
scus
sio
nO
btai
nvi
ews
onis
sue.
List
enin
gto
the
co
nver
satio
ns,r
ecor
dke
ypo
ints
.
An
inte
ract
ive
e-m
ail
faci
lity
will
allo
wt
his.
Not
ag
ood
med
ium
for
gett
ing
peop
le’s
view
s.C
an
use
thes
eas
as
timul
usa
nd
then
use
gro
upd
iscus
sions
an
dqu
estio
nnai
res
Com
men
tbo
oks
and
exit
ques
tionn
aire
s.Bu
ildin
op
port
uniti
esfo
rst
aff/
colle
ague
sto
eng
age
with
vi
sitor
s.
Not
usu
ally
des
igne
dfo
rgi
ving
feed
back
.C
anu
sed
ebat
eaf
ter
the
perfo
rman
ce.
Can
bui
ldt
his
int
oen
try
proc
ess,
but
not
ano
rmal
mec
hani
sm
for
gett
ing
peop
le’s
view
s.
![Page 44: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/44.jpg)
Evaluation: Practical Guidelines 41
GlossaryAnnex4
ActivityUnlessaspecificactivityorapproachisbeingdiscussed,thisguideusesthetermactivityasanall-encompassingphrasefortalks,shows,teachers’packs,hands-onevents,websitesandthemanyotherwaysthatresearchersareusingtoengagegeneralaudiences.
AimTheaimofyouractivityiswhatyouultimatelywanttoachieve.Theaimissupportedbyanumberofobjectivesthatwillhelpyoutorealisetheoverallgoal.
AudienceTheaudienceisthepeoplewithwhomyouaretryingtoengage.
BaselineAmeasureatthebeginningofanactivitythatenablesdeterminationofchange,ifany.
CensusAsurveythatcollectsdatafromeveryone.
ChartingAmethodforanalysingqualitativeresearchdata.
DataInformationcollectedthroughmonitoringandresearch.
EvaluationEvaluationhelpsyoutoseewhetherornotyouhaveachievedyourobjectivesandtoidentifywaystoimprovewhatyoudoduringandafteryouractivity.
EvaluationstrategyTheplanthroughwhichyouwilldeterminewhetherornotyouhaveachievedyourobjectives.
ExitsurveyAsurveyofpeopleundertakenastheyleaveanevent,exhibition,etc.Usuallyconductedbyaninterviewerratherthanaresearcher.
Face-to-faceinterviewsUsedinmarketandsocialresearchtomeanstructuredquantitativesurveysconductedface-to-face.
FocusgroupAresearchmethodthatinvolvesagroupofusually6-8peopleconvenedtodiscussaparticulartopic.
FormativeevaluationResearchthattakesplaceduringthedevelopmentofanactivitytoensureitmeetstheaudiences’needs.
FundersThefundersarethepeoplewhoprovidetheresourcesthatallowyoutoundertakeyouractivity.
In-depthinterviewAninterviewconductedbyaresearcherusingatopicguide,whichallowsrespondentstoexpressthemselvesintheirownwayandraiseissuestheresearcherhasnotconsidered.
InterviewerApersonwhoconductsinterviewsfollowingapredeterminedquestionnairedesignedbyaresearcher.
MilestonesMilestonesareinterimmeasuresthatallowyoutomonitorwhetherornotyouareontracktomeetyourobjectives.
ObservationFormalisedobservationofbehaviour,eitherdirectlyorfromarecording.
ObjectivesObjectivesarethetangiblethingsthroughwhichyouwillachieveyouroverallaim.
OutcomesOutcomesaremeasuresoftheimpactyouhavehadonpeople.
![Page 45: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/45.jpg)
42 Evaluation: Practical Guidelines
OutputsOutputsarethethingsyouproduceaspartoftheactivitye.g.awebsite,aleaflet.
Pop-upquestionnaireAquestionnairethatliterally‘pops-up’onenteringawebsitetocollectinformationaboutusersandusageofthesite.
ProgrammeThereisnoblueprintforaprogramme,itislikelytocontainoneormoreofthefollowingfeatures:
• Afundingmechanismtowhichotherpeopleororganisationscanapply.
• Abudgettocommissionspecificpiecesofwork.
• Resourcestoundertakein-houseactivities.
• Areportingprocess,throughwhichtheprogrammemanagerbidsforresourcesandaccountsfortheiruse.
ProgrammemanagerSomeonewhohasoverallresponsibilityfordeliveringagainstasetofpre-determinedobjectives,andusesavarietyofactivitiesandactionstoachievetheseobjectives.Oftenworksforafunder.
ProjectmanagementProjectmanagementinthiscontextissimplytheproceduresthroughwhichyouensureyoudeliveryouractivity.
ProjectmanagerTheprojectmanageristhepersonultimatelyresponsiblefortheactivity.
QualitativeresearchTechniquesthatallowpeopletoexpressthemselvesintheirownwordsandtoraisetheirconcerns,usuallyviain-depthinterviewsandfocusgroupsconductedbyresearchers,helpsyoutounderstandwhypeopledoorsaywhattheydoorsay.
QuantitativeresearchTechniquesthataskpeoplethesamequestionsinsuchawayastoenabletheanswerstobeaddedtogetherforasamplethatisrepresentativeofthetargetgroup,thusprovidingnumericaldataonthepercentageofpeoplewithparticularviewsorbehaviour.
QuestionnaireAstructuredsetofquestionscallingforapreciseresponsethatallowsanswersfromallthosewhocompleteittobeaddedtogether.
QuotasamplingSetting‘quotas’toensureasamplehasthesamepercentageofpeoplewithspecificcharacteristicsasthepopulationofinterest.Requiresotherdatagivingtheinformationonthepopulation.
ResearcherApersonwhoisinvolvedindesigningandoverseeingaresearchproject.
SamplingAwayofselectingpeopletotakepartinresearchthatensurestheyarechosentoberepresentativeofthepopulationofinterest,althoughnotalwaysinastatisticalsense.
SMARTAllobjectivesshouldbeSMART,whichstandsfor:
• Specific;
• Measurable;
• Achievable;
• Relevant;and
• Time-bound.
StakeholdersThosewhohavealegitimateinterestinyouractivity,e.g.audiencesandfunders.
![Page 46: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/46.jpg)
Evaluation: Practical Guidelines 43
Sub-contractorsSub-contractorsarepeopleororganisationsemployedbytheprojectmanagertodeliverspecificallydefinedproductsorservices.
SummativeevaluationEvaluationattheendofanactivitythatdetermineswhetherornottheobjectiveshavebeenmet.
SystematicsamplingTakingevery‘nth’personwhoengageswithanactivity.Producesastatisticallyrepresentativesample.
TopicguideAlistofquestionsandissuesaresearcherwantstocoverduringanin-depthintervieworfocusgroup.
UserForthepurposesofthisguide,someonewhoengageswithanactivity.
![Page 47: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/47.jpg)
44 Evaluation: Practical Guidelines
FurtherreadingAnnex5
FindinginformationonevaluationTheterm‘evaluation’isusedwidelyineducation,socialpolicy,andtraininganditisintheseareasthatyoucanfindpapersandbooksdevotedtoevaluationasadistincttradition.Elsewhere,toolsandtechniquesusedinevaluationaresimplytheapplicationofresearchmethodologies.Thereislittlewrittenthatisspecifictopublicengagementorcommunicatingaboutresearch–itisoftenacaseofborrowingandadaptingmethodsthathavebeenusedsuccessfullyinotherfields.
EvaluationMethodsBreakwell, G. and Millward, L, (1995) Basic evaluation methods BPS Books, Leicester. 145 pp
Agoodgeneralintroductiontoevaluation,whichcanbeappliedinarangeofsettings.Containscasestudiesandexamplesthatarerelevanttocommunicatingresearchfindings,e.g.evaluationofamuseumexhibition,andcoversawiderangeofresearchmethods,includingquestionnaireconstructionandtimeseries.Containsaninterestingsectiononthepoliticsofcommunicatingevaluationresultstoaudiences.
WebResourcesThemainwebresourcesthatwillhelpyouwithyourevaluationarethewebsitesofvariousevaluationsocieties(usuallyspecialisingineducationalorsocialpolicyresearch).Therearealsosomeuseful‘hintsandtips’sites,mostlyUS-based,writtenbyacademicsandgeneralenthusiasts,allyouneedtodoisentersearchtermsinasearchengine.Thereisanincreasingsetofworkthataimstohelpthosewhowanttoevaluatetheirpublicengagementactivities.
GeneralEvaluationResources
http://www.evaluation.org.uk/resources/online-resources.aspxWebsiteoftheUKEvaluationsociety.
www.eval.orgTheAmericanEvaluationAssociation.
http://www.reviewing.co.uk/evaluation/index.htmGeneralarticlesongettingthebestoutofcourseevaluation,withtipsandlinks.
http://gsociology.icaap.org/methods/SetoflinkstoUS-basedevaluationresources,withabasicguideforthenon-specialist.
http://www.mapnp.org/library/evaluatn/fnl_eval.htmAUS-basedbasicguidetoprogramevaluationincludingoutcomesevaluation.Usefuloverviewofmethodsandissues.
www.mrs.org.ukTheUKMarketResearchSociety’swebsitehasup-to-dateguidanceonethicsandconfidentialityininterviewingthepublic.
www.dfes.gov.ukDepartmentforEducationhasdataonindividualschoolexaminationresultsatvariousKeyStages,GCSEand‘A’level.
http://www.cdc.gov/eval/resources.htmDigitalCreativityspecialissueonevaluation,2009.Widerangeofprofessionalon-lineevaluationresources.Comprehensivelistingofevaluationresources,materials,guidelinesetc.(Americanbased).
![Page 48: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/48.jpg)
Evaluation: Practical Guidelines 45
Evaluationresourcesaimedatthoseengagingthepublicintheirresearch
http://www.raeng.org.uk/societygov/public_engagement/ingenious/evaluation.htmTheIngeniousEvaluationFAQsgiveexamplesofgoodpracticeinevaluationandideasonusingcreativeapproaches.
http://caise.insci.org/resourcesExcellentoverviewofAmerican(NSFfunded)evaluationresources,includingtoolkitsplusworkacrossvariousinformalenvironmentsincludingzoos&aquaria.
http://caise.insci.org/resources/Eval_Framework.pdfNSFevaluationframework.
http://practice.ie/blogs/heather/ideas-and-examples-creative-evaluation-kidsCAISEFrameworkforEvaluatingImpactsofInformalScienceEducationProjects.
http://www.scidev.net/en/practical-guides/evaluating-science-communication-projects-1.htmlJoubert,M.(2007)Evaluatingsciencecommunicationprojects.SciDevNet,8thJanuary2007.
http://www.impact.arts.gla.ac.uk/TheImpactDatabaseisabibliographicaldatabaserelatingtoresearchonthesocialandeconomiceffectsofarts,cultureandmajorevents.
http://www.publicengagement.ac.uk/TheNationalCo-ordinatingCentreforPublicEngagementisfundedbytheUKHigherEducationFundingCouncils,ResearchCouncilsUKandtheWellcomeTrusttohelpacademicsengagewiththepublicintheirresearch.Thewebsitehasahostofresourcestosupportuniversitystaffandstudentstodeveloppublicengagementactivities.
http://www.ucl.ac.uk/public-engagement/research/toolkits/MethodsAseriesofdocumentscollatedbyUniversityCollegeLondontosupportacademicsintheevaluationoftheirpublicengagementactivities.
http://www.ucl.ac.uk/public-engagement/research/frameworkAframeworkforevaluationputtogetherbytheUCLBeaconforPublicEngagementforacademicswhowanttoevaluatetheirpublicengagementactivities.
http://www.manchesterbeacon.org/files/manchester-beacon-pe-evaluation-guide.pdfAguidetoevaluationputtogetherbytheManchesterBeaconforPublicEngagementforacademicswhowanttoevaluatetheirpublicengagementactivities.
http://www.britishscienceassociation.org/forms/scicomm/evaluation/access/ThisdatabaseismanagedbytheBritishScienceAssociationandisaresourceforthoseinterestedinengagingpeoplewithscience&engineering.Theobjectiveofthedatabaseistosharelearning.Itisfreetouploadyourevaluationandtoaccessthedatabase.
Ifnoneofthesesuit,tryenteringkeywordsintoasearchengine.
Forhelponplanningyouractivitysee:http://www.vitae.ac.uk/CMS/files/upload/The_engaging_researcher_2010.pdf
![Page 49: Evaluation: Practical Guidelines - UK Research and Innovation...4.2 Quantitative research 15 4.3 Qualitative research 18 4.4 Other options 20 4.5 Confidentiality 21. 5 Data Handling](https://reader033.fdocuments.net/reader033/viewer/2022060900/609dd89d842cef61f234b7c4/html5/thumbnails/49.jpg)
Research Councils UK Polaris House, North Star Avenue Swindon Wiltshire SN2 1ET United Kingdom
www.rcuk.ac.uk
April 2011. All cartoons © Noel Ford
Produced by RCUK’s in-house service provider