11

Data Collection and Data Collection and AnalysisAnalysis

Dr. Fred Mugambi MwirigiDr. Fred Mugambi Mwirigi

JKUATJKUAT

Introduction Introduction

Data collection is perhaps the most Data collection is perhaps the most important component of Monitoring and important component of Monitoring and EvaluationEvaluation

Without good data we can not tell if or not Without good data we can not tell if or not the project is beneficial as expectedthe project is beneficial as expected

Mechanisms must be put in place to Mechanisms must be put in place to collect adequate, timely and relevant data collect adequate, timely and relevant data on a continuous basison a continuous basis

22

33

Qualities of good dataQualities of good data

AdequateAdequate Timely Timely Relevant Relevant Consistent Consistent Reliable Reliable

44

Data Collection methods Data Collection methods QuestionnaireQuestionnaire ObservationObservation Laboratory processesLaboratory processes Desk review (books, records, other Desk review (books, records, other

documents)documents)

55

QuestionnaireQuestionnaire Quality and adequacy of data collected Quality and adequacy of data collected

depends on the nature of questionnaire depends on the nature of questionnaire usedused

The questionnaire must address all the The questionnaire must address all the variables under studyvariables under study

Length and complication (language) of Length and complication (language) of the questionnaire can affect data the questionnaire can affect data collectioncollection

66

Types of QuestionnairesTypes of Questionnaires

Questionnaires may be:Questionnaires may be:

1.1. Open-endedOpen-ended

2.2. Closed-endedClosed-ended

77

Open-ended QuestionnairesOpen-ended Questionnaires Allow respondent to answer the question in his or her Allow respondent to answer the question in his or her

own wordsown words Example: What benefits have you drawn from the Example: What benefits have you drawn from the

project?project? Open-ended questions allow the interviewer to ask or Open-ended questions allow the interviewer to ask or

probe the respondent further if he/she feels that probe the respondent further if he/she feels that clarification of a point, or additional information, is clarification of a point, or additional information, is needed. needed.

Good for exploratory research. Good for exploratory research. Disadvantage are the difficulties of analyzing the data Disadvantage are the difficulties of analyzing the data

and in categorizing and summarizing answers because and in categorizing and summarizing answers because of the unique responses. of the unique responses.

Also, there is the possibility of interviewer bias and bias Also, there is the possibility of interviewer bias and bias caused by the different education levels of the caused by the different education levels of the respondents respondents

88

Closed-ended QuestionnairesClosed-ended Questionnaires

Respondent answers the question in Respondent answers the question in in very specific pre-determined choice in very specific pre-determined choice of wordsof words

Example: Would you study participte Example: Would you study participte in the project if you were to make the in the project if you were to make the choice a 2choice a 2ndnd time? (tick yes or no) time? (tick yes or no)

They are easy to statistically analyse They are easy to statistically analyse

99

Types of Questionnaire Types of Questionnaire AdministrationAdministration

Self administered Self administered Personally administered Personally administered Group administered Group administered

1010

Key questions when designing a Key questions when designing a questionnairequestionnaire

What should the respondents be asked?What should the respondents be asked? How should each question be phrased?How should each question be phrased? In what sequence should the questions be arranged?In what sequence should the questions be arranged? What is the best questionnaire layout for the research What is the best questionnaire layout for the research

problem in question?problem in question? What communication medium should be utilized What communication medium should be utilized

(personal interview, telephone interview etc.)(personal interview, telephone interview etc.) Should the questionnaire be pretested?Should the questionnaire be pretested? How should the questionnaire be pretested?How should the questionnaire be pretested? Does the questionnaire require a revision?Does the questionnaire require a revision?

1111

Relevance and AccuracyRelevance and Accuracy Questionnaires must collect data that is both accurate Questionnaires must collect data that is both accurate

and relevant and relevant

Relevance means:Relevance means:1. No unnecessary information is obained from the 1. No unnecessary information is obained from the

questionnaire questionnaire 2. All the information that is needed for the purpose of 2. All the information that is needed for the purpose of

the evaluation is collected the evaluation is collected 3. No important information is omitted3. No important information is omitted

Accuracy means that the questions are worded in a Accuracy means that the questions are worded in a manner which ensures the collection of correct manner which ensures the collection of correct information from respondentsinformation from respondents

1212

Guidelines for phrasing Guidelines for phrasing questionnairesquestionnaires

Avoidance of Complexity / Use Simple LanguageAvoidance of Complexity / Use Simple Language Avoidance of leading and Loaded questionsAvoidance of leading and Loaded questions Counterbiasing Statements (leading respondent with Counterbiasing Statements (leading respondent with

an introductory remark before the questionan introductory remark before the question Avoidance of AmbiguityAvoidance of Ambiguity Avoidance of Double-Barreled Questions – Avoidance of Double-Barreled Questions –

Questions which adress two or more issues Questions which adress two or more issues simultaneouslysimultaneously

Avoidance of AssumptionsAvoidance of Assumptions Avoidance of Burdensome (and Memory Taxing) Avoidance of Burdensome (and Memory Taxing)

QuestionsQuestions

1313

Sequencing QuestionsSequencing Questions

The order in which questions are put in a The order in which questions are put in a questionnaire may significantly affect the questionnaire may significantly affect the response rate response rate

Usually, interviewers prefer to ask general Usually, interviewers prefer to ask general questions from respondents before moving questions from respondents before moving on to specific questions (funnel technique)on to specific questions (funnel technique)

1414

Key Factors to consider in Key Factors to consider in questionnaire layoutquestionnaire layout

Don‘t overcrowd the questionnairesDon‘t overcrowd the questionnaires Keep questionnaires as brief as possibleKeep questionnaires as brief as possible Ensure that the title and subtitles of the Ensure that the title and subtitles of the

questionnaire and questionnaire sections are questionnaire and questionnaire sections are carefully phrased and captures the carefully phrased and captures the respondents attentionrespondents attention

Use appealing languageUse appealing language

1515

Pretesting QuestionnairesPretesting Questionnaires

Pretesting is a very useful method for determining whether Pretesting is a very useful method for determining whether respondents have any difficulty understanding the respondents have any difficulty understanding the questionnaire and whether the questions are ambiguous or questionnaire and whether the questions are ambiguous or can lead possibly to biased answerscan lead possibly to biased answers

Pretesting ensures that costly errors in questionnaires Pretesting ensures that costly errors in questionnaires which are given to a large number of respondents is which are given to a large number of respondents is avoidedavoided

The respondents involved in a pretest should be similar in The respondents involved in a pretest should be similar in essence to the target respondents of the researchessence to the target respondents of the research

1616

Pretesting provide answers to important Pretesting provide answers to important questions for the, such as:questions for the, such as:

Can the questionnaire format be followed by the Can the questionnaire format be followed by the interviewers?interviewers?

Is the questionnaires ambiguous?Is the questionnaires ambiguous? Does the questionnaire flow naturally and Does the questionnaire flow naturally and

conversationally?conversationally? Can respondents answer the questions easily?Can respondents answer the questions easily? Which alternative forms of questions work best?Which alternative forms of questions work best?

Selecting Populations Selecting Populations and Samplesand Samples

IntroductionIntroduction

A sample is a subset of a larger population of A sample is a subset of a larger population of objects individuals, households, businesses, objects individuals, households, businesses, organizations and so forthorganizations and so forth

Sampling enables interviewers to make Sampling enables interviewers to make estimates of some unknown characteristics of estimates of some unknown characteristics of the population in questionthe population in question

A finite group is called population whereas a A finite group is called population whereas a non-finite (infinite) group is called universenon-finite (infinite) group is called universe

A census is a investigation of all the individual A census is a investigation of all the individual elements of a population elements of a population

Population Vs SamplePopulation Vs Sample

Reasons for SamplingReasons for Sampling Budget and time Constraints (in case of large populations)Budget and time Constraints (in case of large populations)

High degree of accuracy and reliability (if sample is High degree of accuracy and reliability (if sample is representative of population) representative of population)

Sampling may sometimes produce more accurate results Sampling may sometimes produce more accurate results than taking a census as in the latter, there are more risks than taking a census as in the latter, there are more risks for making interviewer and other errors due to the high for making interviewer and other errors due to the high volume of persons contacted and the number of census volume of persons contacted and the number of census takers, some of whom may not be well-trained takers, some of whom may not be well-trained

The Sampling ProcessThe Sampling Process

Define the TargetPopulation

Define the TargetPopulation

Select a Sampling Frame

Select a Sampling Frame

Determine if a probability or non-probability sampling

method will be chosen

Determine if a probability or non-probability sampling

method will be chosen

Plan procedure for selecting sampling units

Plan procedure for selecting sampling units

Determine sample sizeDetermine sample size

Select actual sampling unitsSelect actual sampling units

Conduct fieldworkConduct fieldwork11

22

33

44

55

66

77

Defining the Target PopulationDefining the Target Population

The target population is that complete group The target population is that complete group whose relevant characteristics are to be whose relevant characteristics are to be determined through the samplingdetermined through the sampling

A target population may be, for example, all A target population may be, for example, all beneficiaries of a project during the life of the beneficiaries of a project during the life of the projectproject

The Sampling FrameThe Sampling Frame The sampling frame is a list of all those population The sampling frame is a list of all those population

elements that will be used in the sampleelements that will be used in the sample

An example a of sampling frames is the list of direct An example a of sampling frames is the list of direct beneficiaries of a projectbeneficiaries of a project

Often, the list does not include the entire population. The Often, the list does not include the entire population. The discrepancy is often a source of error associated with the discrepancy is often a source of error associated with the selection of the sample (sampling frame error)selection of the sample (sampling frame error)

Sampling UnitsSampling Units

The sampling unit is a single element – or The sampling unit is a single element – or group of elements – subject to selection in group of elements – subject to selection in a sample. Examples:a sample. Examples: Every project beneficiary whose name begins Every project beneficiary whose name begins

with “F”with “F” All project beneficiaries under 18 years of ageAll project beneficiaries under 18 years of age

Probability and Probability and Non-Probability SamplingNon-Probability Sampling

Probability Sampling – Every element in the Probability Sampling – Every element in the population under study has a non-zero population under study has a non-zero probability of selection to a sample, and every probability of selection to a sample, and every member of the population has an equal member of the population has an equal probability of being selectedprobability of being selected

Non-Probability Sampling – An arbitrary means Non-Probability Sampling – An arbitrary means of selecting sampling units based on subjective of selecting sampling units based on subjective considerations, such as personal judgment or considerations, such as personal judgment or convenience. convenience.

Non-Probability SamplingNon-Probability Sampling

1.1. Convenience SamplingConvenience Sampling

2.2. Judgment (purposive) SamplingJudgment (purposive) Sampling

3.3. Quota SamplingQuota Sampling

4.4. Snowball SamplingSnowball Sampling

Convenience SamplingConvenience Sampling This is a sampling technique which selects those This is a sampling technique which selects those

sampling units most conveniently available at a sampling units most conveniently available at a certain point in, or over a period, of time certain point in, or over a period, of time

Major advantages of convenience sampling is that it Major advantages of convenience sampling is that it is quick, convenient and economical; a major is quick, convenient and economical; a major disadvantage is that the sample may not be disadvantage is that the sample may not be representativerepresentative

Convenience sampling is best used for the purpose Convenience sampling is best used for the purpose of exploratory research and supplemented of exploratory research and supplemented subsequently with probability samplingsubsequently with probability sampling

Judgment (purposive) SamplingJudgment (purposive) Sampling

A sampling technique in which the interviewer A sampling technique in which the interviewer selects the sample based on judgment about selects the sample based on judgment about some appropriate characteristic of the sample some appropriate characteristic of the sample membersmembers

Quota Sampling Quota Sampling A sampling technique in which the interviewer ensures A sampling technique in which the interviewer ensures

that certain characteristics of a population are that certain characteristics of a population are represented in the sample to an extent which he or she represented in the sample to an extent which he or she desiresdesires Example: An interviewer wants to determine through Example: An interviewer wants to determine through

interview, the demand for Product X in a district which interview, the demand for Product X in a district which is very diverse in terms of its ethnic composition. If is very diverse in terms of its ethnic composition. If the sample size is to consist of 100 units, the number the sample size is to consist of 100 units, the number of individuals from each ethnic group interviewed of individuals from each ethnic group interviewed should correspond to the group’s percentage should correspond to the group’s percentage composition of the total population of that district composition of the total population of that district

Contd.Contd. Quota Sampling has several advantages and Quota Sampling has several advantages and

disadvantages:disadvantages:

Advantages include the speed of data collection, less Advantages include the speed of data collection, less

cost, the element of convenience, and cost, the element of convenience, and

representativeness (if the subgroups in the sample representativeness (if the subgroups in the sample

are selected properly) are selected properly)

Disadvantages include the element of subjectivity Disadvantages include the element of subjectivity

(convenience sampling rather than probability-based (convenience sampling rather than probability-based

which leads to improper selection of sampling units) which leads to improper selection of sampling units)

Snowball SamplingSnowball Sampling A sampling technique in which individuals or organizations are A sampling technique in which individuals or organizations are

selected first by probability methods, and then additional selected first by probability methods, and then additional respondents are identified based on information provided by the first respondents are identified based on information provided by the first group of respondentsgroup of respondents

Example: 20 project beneficiaries are identified which, in turn, Example: 20 project beneficiaries are identified which, in turn, identify a number of other project beneficiariesidentify a number of other project beneficiaries

The advantage of snowball sampling is that smaller sample The advantage of snowball sampling is that smaller sample sizes and costs are necessary; a major disadvantage is that the sizes and costs are necessary; a major disadvantage is that the second group of respondents suggested by the first group may second group of respondents suggested by the first group may be very similar and not representative of the population with that be very similar and not representative of the population with that characteristic characteristic

Probability SamplingProbability Sampling

1.1. Simple Random SamplingSimple Random Sampling

2.2. Systematic SamplingSystematic Sampling3.3. Stratified SamplingStratified Sampling

Simple Random SamplingSimple Random Sampling This is a technique which ensures that each This is a technique which ensures that each

element in the population has an equal chance element in the population has an equal chance of being selected for the sampleof being selected for the sample

Example: Choosing raffle tickets from a drum, Example: Choosing raffle tickets from a drum, computer-generated selections, random-digit computer-generated selections, random-digit telephone dialingtelephone dialing

The major advantage of simple random sampling is The major advantage of simple random sampling is its simplicityits simplicity

Systematic SamplingSystematic Sampling This is a technique in which an initial starting point is This is a technique in which an initial starting point is

selected by a random process, after which every selected by a random process, after which every nthnth number on the list is selected to constitute part of the number on the list is selected to constitute part of the samplesample

Example: From a list of 1500 name entries, a name on the list is Example: From a list of 1500 name entries, a name on the list is randomly selected and then (say) every 25randomly selected and then (say) every 25 thth name thereafter. name thereafter. The sampling interval in this case would equal 25.The sampling interval in this case would equal 25.

For systematic sampling to work best, the list should be random For systematic sampling to work best, the list should be random in nature and not have some underlying systematic pattern in nature and not have some underlying systematic pattern

Stratified SamplingStratified Sampling A technique which in which simple random sub samples are drawn A technique which in which simple random sub samples are drawn

from within different strata that share some common characteristic from within different strata that share some common characteristic

Example: project members are divided into two groups Example: project members are divided into two groups (beneficiaries and staff) and from each group, respondents are (beneficiaries and staff) and from each group, respondents are selected for a sample using simple random sampling in each of selected for a sample using simple random sampling in each of the two groups, whereby the size of the sample for each group is the two groups, whereby the size of the sample for each group is determined by that group’s overall strength determined by that group’s overall strength

Stratified Sampling has the advantage of giving more Stratified Sampling has the advantage of giving more representative samples and less random sampling error. representative samples and less random sampling error. However, it is more complex and information on the strata may However, it is more complex and information on the strata may be difficult to obtainbe difficult to obtain

OthersOthers

1.1. Cluster Sampling Cluster Sampling

2.2. Multistage Area Sampling Multistage Area Sampling

3.3. Internet SamplingInternet Sampling

Issues in Sample Design and SelectionIssues in Sample Design and Selection

Accuracy – Samples should be Accuracy – Samples should be representative of the target populationrepresentative of the target population

Resources – Time, money and individual Resources – Time, money and individual or institutional capacity are very important or institutional capacity are very important considerations due to the limitation on considerations due to the limitation on them. Often, these resources must be them. Often, these resources must be “traded” against accuracy“traded” against accuracy

Issues in Sample Design and SelectionIssues in Sample Design and Selection

Availability of Information – Often information on sample Availability of Information – Often information on sample participants in the form of lists, directories etc. is participants in the form of lists, directories etc. is unavailable which makes some sampling techniques unavailable which makes some sampling techniques (e.g. systematic sampling) impossible to undertake(e.g. systematic sampling) impossible to undertake

Geographical Considerations – The number and Geographical Considerations – The number and dispersion of population elements may determine the dispersion of population elements may determine the sampling technique used (e.g. cluster sampling)sampling technique used (e.g. cluster sampling)

Data Analysis Data Analysis

3939

Methods of Data AnalysisMethods of Data Analysis

1.1. Quantitative analysisQuantitative analysis

2.2. Qualitative analysisQualitative analysis

Both methods can be applied manually or by Both methods can be applied manually or by computer software (e.g. SPSS or MS computer software (e.g. SPSS or MS Excel)Excel)

4040

Quantitative AnalysisQuantitative Analysis

Analyses done through use of statistical Analyses done through use of statistical processes that may include derivation of processes that may include derivation of percentages, ratios, correlations and percentages, ratios, correlations and trends among others as may be requiredtrends among others as may be required

Statistics give a more accurate picture of Statistics give a more accurate picture of the situation at handthe situation at hand

Statistics must be selected carefully- they Statistics must be selected carefully- they should be able to give expected resultsshould be able to give expected results

4141

Qualitative AnalysisQualitative Analysis

Not all data can be analyzed quantitativelyNot all data can be analyzed quantitatively Some data is too qualitative and so has to Some data is too qualitative and so has to

be analyzed by applying perceptional be analyzed by applying perceptional meaningmeaning

Thematic analysis is one of the most Thematic analysis is one of the most popular methods of qualitative analysespopular methods of qualitative analyses

In this method data is classified into In this method data is classified into themes based on its relatednessthemes based on its relatedness

4242

Choice of MethodChoice of Method

Depends on:Depends on:

1.1. Nature of data to be analyzedNature of data to be analyzed

2.2. Nature of expected findingsNature of expected findings

3.3. Expected use of findingsExpected use of findings

4.4. Audience of the findingsAudience of the findings

A mixed method is preferred in most cases A mixed method is preferred in most cases

4343

4444

End End