Improving market access of farmer groups in Uganda: evaluating the role of working capital

Impact evaluation An application to farmers groups in Uganda

Improving market access of farmer groups inUganda: evaluating the role of working capital

Ruth Vargas Hill and Eduardo Maruyama

May 9, 2012


Outline

Impact evaluationIntroductionRCTs

An application to farmers groups in UgandaIntroductionImplementationResultsConcluding remarks


Why evaluate?

• Evaluating interventions (policies or programs) helps:• Understand the actual rather than the anticipated effects of

programs.• Determine how to design new programs.• Determine the most cost-effective approach to achieve a

desired goal.


Estimating impact: introduction

• When we conduct impact evaluation we assess how a programaffects the well-being or welfare of individuals, households orcommunities:

• Profitability of agricultural production• Increased income or consumption (or other measures of

welfare) of rural households• Poverty levels or growth rates at the community level


Impact evaluation versus other M&E tools

• Impact evaluation is different from other M&E tools in that itfocuses on discerning the impact of the program from allother confounding effects.

• The focus of impact evaluation is providing evidence of thecausal link between an intervention and an outcome.

• This is why impact evaluation is a powerful too, but also whatmakes it difficult to implement in practice.


Impact evaluation versus other M&E tools

Inputs Outputs Outcomes Impacts

HighDIFFICULTY OF SHOWING CAUSALITYLow

Visits by extensions agents,

physical inputs (such as seeds)

Knowledge of the new

technology

Use of the new

technology

Increased yields, higher farm profits, improved

consumption

Example: A program of providing advice on a new technology to farmers


Essential component: counterfactual

• Difficulty is determining what would have happened to theindividuals or communities of interest in absence of theproject.

• We are interested in the difference in an outcome for anindividual with and without the intervention.

• Problem: can only observe people in one state of the world atone time

• The key component to an impact evaluation is to construct asuitable comparison group to proxy for the counterfactual.


Before and after comparisons

• Why not collect data on individuals before and afterintervention (the Reflexive)? Difference in income, etc, wouldbe due to project

• Problem: many things change over time, including the project• The country is growing and profits are rising. Is this due to the

program or would have occurred in absence of program?• This is particularly a problem for agricultural interventions:

many factors affect yield (weather, availability of inputs) andprices in a given year.


Comparison groups

• Instead of using before/after comparisons, we need to usecomparison groups to proxy for the counterfactual

• Two core problems in finding suitable groups:• Programs are targeted

• Recipients receive intervention for particular reason

• Participation is voluntary• Individuals who participate differ in observable and

unobservable ways (selection bias)

• Hence, a comparison of participants and an arbitrary group ofnon-participants can lead to misleading or incorrect results


Randomizing to create a true comparison group

• We need a comparison group that is as identical in observableand unobservable dimensions as possible, to those receivingthe program, and a comparison group that will not receivespillover benefits.

• Number of techniques:• Randomized control trials (the gold standard)• Careful matching techniques: IV, propensity score matching,

regression discontinuity design


Randomized Control Trials (RCTs)

• In RCTs, participation in a policy (or usually eligibility toparticipate in a policy) is randomly assigned.

• This is done to ensure that the only difference between thosein and out of an intervention, is their participation, and as aresult any difference between participants andnon-participants can be attributed to the program alone.

• Because participation (treatment) is randomized, thenon-treatment outcomes between those that are not treatedand those that are treated is equal.


Households or groups of households


Households or groups of households

TT

T

T

TT

T

T

TT

T

T

C

C

C

C

CC

C

C

C

C

CC

C

C

C


Does randomization create a true comparison group?

• We can test that they are equal by collecting data on the twogroups before the intervention and checking that the averagecharacteristics of the two groups are the same.

• For the treatment and control groups to be statistically equalyou need a large number of each. Cannot have one treatedhousehold and one control household.

• Means that you cannot use this method to answer questionsabout country policy changes (e.g. fiscal policy changes).

• There are stragegies that can be used to ensure that thetreatment and control groups are equal (e.g. stratification).


How do we estimate impact by randomizing?

• Identify the outcome we are interested in (e.g. yields, amountof output marketed, price received)

• Estimate the average of the outcome in the treatment group.

• Estimate the average of the outcome in the control group.

• Calculate the difference of these averages and test to see ifthe two averages are significantly different from each other.

• Average Treatment Effect

• Note: it is just differences in the AVERAGE outcome that areestimated.


Challenges to estimating impact

• Sometimes the effect of the program is small.

• Or there are many other factors affecting the outcome ofinterest that it is hard to see if a difference is statisticallydifferent between two groups.

• We try and control for this in two ways:• Include a large number of households in treatment and control.

This increases our power to detect a small effect.• Collect data on characteristics of the household that may

influence the outcome variable at baseline (including thepre-intervention outcome of interest)


Other challenges to estimating impact

• Are we sure that the intervention had no impact on thecontrol group? Are there no spillover effects? (E.g. on prices)

• Was there any attrition as a result of the program that meanswe miss capturing some of the impact? For example didpeople migrate as a result of the program? If so, we will misscapturing the effect of the program on these people.

• We randomized to avoid selection bias, but some of it stillmay remain:

• Did everyone in the treatment group participate as expected?• Did anyone in the control group participate even if they were

not meant to?


Selection bias

36

Target Population

Target Population

Not in evaluation

Not in evaluation

Evaluation Sample

Evaluation Sample

Random Assignment

Treatment group

Treatment group

Participants

No-ShowsNo-Shows

Control groupControl groupNon-

ParticipantsNon-

Participants

Cross-oversCross-overs


Handling selection bias

• Intent to treat (ITT):• Average impact of program in practice: treats all noncompliars

as treated, and treats all crossovers as remaining in the control• Problem: power is reduced by noncompliance and does not

provide an idea of what the average impact of the program onthe treated is.

• Treatment on the treated (ToT):• Instruments for take-up with assignment: gives an idea of the

average impact of the program for a specific group


Summary of advantages and disadvantages

• Powerful method to identify causal impact of a policy orprogram.

• Careful design is needed to ensure you are able to detectchanges

• Can be expensive: baseline and follow-up, a large number ofparticipants (especially if it is to be representative)

• Only valid in some circumstances: randomization over anumber of units needs to be possible.

• Provides information on the average outcome.

• Internally valid, repetition and a theory of change needed tomake predictions from results (external validity).


External validity

• Tells us whether something worked in a specific context,understanding whether an intervention would work again in adifferent setting, external validity, is very difficult to know.

• Great benefit of few assumptions, comes with great cost”narrowness of scope” (Cartwright 2007).

• Any external validity involves some assumptions about theproject working in different conditions.

• Yet replicating a project is almost impossible, replicatingtriggers of mechanisms that produce the change is often morepossible.

• Result: we need to repeat impact evaluations and also have atheory of change to know what will work in the future, i.e. toreally learn.

• But better than an approach that is not internally valid.Predictions cannot be made from this either.


Other concerns in learning from results

• General equilibrium effects.

• Corruption in implementing a large scale.

• Capacity to implement at a large scale.

• Overlap between new environment and old (example ofmedicine)

• Not automatic to go from experiments to learning and policyadvice.


Another approach

• We need a theory of change that guides us in going fromone-off impact evaluation to general lessons. This meansmake assumptions and, ideally, use experiments to test andrefine these assumptions.

• When designing impact evaluation for this purpose, it oftenlooks quite different:

• Theory of change influences the design of the impactevaluation.

• Often identifying the differential impact of different treatments,rather than the impact of one treatment against baseline.


Outline

Impact evaluationIntroductionRCTs

An application to farmers groups in UgandaIntroductionImplementationResultsConcluding remarks


Introduction

• Smallholder agriculture in Sub-Saharan Africa is largelyexposed to pervasive market failures, translating into missedopportunities and sub-optimal economic behavior.

• These failures are often rooted in the importance ofeconomies of scale in procuring inputs and marketing produce.

• By engaging in markets collectively through a farmers group,smallholders can overcome economies of scale.

• Despite the renewed interest from governments and donoragencies in farmers groups as a means to overcome thesemarket failures, evidence shows that they have so far hadlimited success.


Ugandan context

• The majority of Ugandan farmers sell their (unprocessed)produce at harvest time to itinerant traders at the farm-gate.

• Survey of farmers groups engaged in some form of outputmarketing revealed that:

• Farmers get a higher price when they sell collectively.• Yet few farmers sell through the marketing group of which

they are a member (only 47% make sales through group)• Farmers are less likely to sell collectively when they are

liquidity constrained and in need of emergency money.• Groups that offer cash on delivery of produce (rather than

payment some days later) have a higher proportion ofmembers selling through the group.


Key impact question

• Would providing working capital loans to farmers groups sothat they can provide cash on delivery, improve marketingoutcomes for farmers?

• We cannot infer this from the baseline data: good groups maybe better at collective sales and better able to access financewhich allows payment on delivery.

• We would like to compare groups of similar quality and see ifworking capital loans increase sales amongst those thatreceived them.


Testing a theory of change

• Farmer groups can offer higher prices but because of thewaiting times involved in receiving payment, farmers find itcostly to sell though the group.

• Farmers are liquidity constrained and often sell coffee to meeturgent financial needs, so even small delays in payment can beproblematic.

• Waiting for payment involves a high level of trust in the abilityof the group to market and transparency. There is a risk if thegroups cannot be trusted.

• Enabling groups to make payment on delivery through aworking capital loan will reduce the cost of selling through thegroup.

• More farmers will sell through the group and receive higherprices as a result.


The impact of working capital credit• Randomized provision of working capital credit to farmers

groups that had already been engaged in output marketing:• Provide selected groups with a fund to make partial cash

payments to farmers upon delivery of produce. Once the groupmakes a sale the fund is replenished and farmers are given theremaining balance. “Cash on Delivery” (CoD)

• Assess the impact of this credit on the proportion of producesold through the group and on the price farmers received.

• Understand why this worked?• Did this work for farmers likely to face liquidity constraints, or

only in groups where trust was already high?• Implement an intervention on improved transparency to

randomly selected groups to improve trust in some groups. Isthe working capital intervention just as effective in thosegroups with the transparency intervention?

• Information on Sales (IoS): SMS system to provide memberswith specific information about transactions made by thegroup (final sale price, fees deducted, etc.), plus reinforcedtraining on book-keeping.


Coffee/maize group marketing structure

• Farmers groups (“PO”s for producer organizations) aretypically grouped under associations (DCs for districtcommittees).

• The PO handles bulking and coordination of transport withmembers at the village level.

• The DC take care of collection and in some cases valueaddition to the next stage of marketing.

• In most cases, a service organization offers support to DCsand POs through lobbying, access to extension and additionalmarketing services.


Implementation

• The study was carried out in 9 DCs marketing coffee andmaize, containing 165 POs under them.

• March 2010, Baseline survey:• A 3-tiered survey which collected detailed information on DCs,

POs, and member households.• Full roster of members for each PO, and a complete household

survey for at least 2 members of each group.

• November 2010 September 2011, implemented interventionin randomly selected groups:

• Provided working capital credit to randomly selected POs.• Provided SMS information on deliveries to randomly selected

POs.

• October 2011, Follow-up survey.• Collected detailed information on POs and member households.• Collected administrative data from the DC records to obtain

more reliable delivery data.


Randomization strategy

• We randomized the interventions at the PO level stratifyingthe sample by DC, since the sample size is not large enoughat the DC level and the risk of spill-overs is too high at thehousehold level.

• POs in each DC are randomly assigned into 4 groups: (1)CoD, (2) IoS, (3) CoD + IoS, and (4) none.

• The fund for the CoD was managed by the DC, and voucherswere given to treated POs so their members could requestimmediate partial payments for output deliveries.

• For the IoS intervention, a DC staff member was selected tosend the messages to key farmers in the treated POs.


Are control and treatment groups equal?

Control CoD IoS Both(mean)

Members 24.256 2.194 4.558 1.597(3.106)∗∗∗ (4.365) (4.289) (4.339)

Years since foundation 4.400 1.039 0.460 1.014(0.526)∗∗∗ (0.739) (0.731) (0.739)

Marketing services 0.825 -0.020 -0.081 -0.093(0.066)∗∗∗ (0.093) (0.092) (0.093)

Output bulked (kgs.) 854.025 -240.708 -192.862 -325.440(236.818)∗∗∗ (332.863) (329.018) (332.863)

Female leader 0.250 -0.006 -0.064 -0.030(0.067)∗∗∗ (0.094) (0.093) (0.094)

Leader’s age 52.200 -4.639 -2.153 0.190(1.874)∗∗∗ (2.634)∗ (2.604) (2.634)

Leader’s schooling 8.025 0.073 0.208 -0.562(0.460)∗∗∗ (0.647) (0.639) (0.647)

POs 40 41 43 41


• Implementing the interventions represented a major challenge:

1. The POs in our study are spread over many regions in thecountry.

2. Implementation needed to be done by a 3rd party, to avoidservice organizations and DCs contaminating the PO-levelrandomization strategy.

3. In order to avoid undesired heterogeneity in implementation,training, and monitoring of the interventions, a singleimplementing agency was favored over several regionalorganizations.

4. Training and distribution of vouchers within the PO wasdelegated to PO leaders in some DCs.

• Our own monitoring activities as well as the follow-up surveyindicate implementation was problematic.

• Some cross-over and no-shows for CoD intervention• Overall implementation of IoS intervention.


Empirical strategy

McKenzie (2011) shows that using baseline data on the outcomevariable of interest, allows more power to detect impact.Therefore, for our analysis we estimate:

Yi ,1 = α +∑j

γjDi ,j + θYi ,0 + εi ,1


Results

Table 2.1: Impact of interventions on producedeliveries

PO Household

Kgs. P(Delivery) Kgs.

CoD only 747.826 0.186 162.700(325.294)∗∗ (0.079)∗∗ (88.400)∗

IoS only 355.764 0.089 62.660(320.386) (0.078) (87.060)

Both -584.566 0.101 122.000(455.073) (0.077) (86.280)

Observations 165 244 243R2 0.422 0.269 0.084


Results

Table 2.2: Impact of selling through POon transaction features

Price Daysbetweensale andpayment

Sold through PO 0.858 -6.540(instrumented) (0.477)∗ (21.170)

Observations 193 192R2 0.704 0.210


Concluding remarks

• Despite implementation problems, the CoD intervention has asignificant impact on group marketing.

• CoD increases the probability a household will sell through thegroup, how much each household will sell, and the totalamount sold by the group.

• By encouraging farmers to sell through the group, CoD has aneffect on increasing the price they receive.

•

Improving market access of farmer groups in Uganda: evaluating the role of working capital

Business

Transcript of Improving market access of farmer groups in Uganda: evaluating the role of working capital