MIXING METHODS IN EVALUATION OF DEVELOPMENT …
Transcript of MIXING METHODS IN EVALUATION OF DEVELOPMENT …
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
MIXING METHODS IN EVALUATION OF DEVELOPMENT PROJECTS:
PCIP CASE STUDY IN LURIGANCHO-CHOSICA (PERU)
José Luis Yagüe
Miguel Salvo
Research Group in Planning and Sustainable Management of the Rural-Local Development (GESPLAN). Technical University of Madrid
Gordon Prain
Nieves Gonzales
International Potato Center. Urban Harvest Program
Abstract
Evaluation of development projects always implies some difficulty to measure socioeconomic impacts. However, It is generally accepted that the sustainability of processes is a key for success, and so processes themselves should be the object for the evaluation. In this sense, evaluation can be considered as a tool to promote or to reinforce those processes.
This article summarizes a case study in Lima outskirts carried out by researchers from the Technical University of Madrid and the International Potato Center. Mixing methods has become an approach to program evaluation, and so different evaluation methods and tools have been combined here: key questions, process mapping and empowerment evaluation.
This methodology has been useful to identify ongoing processes and their links with project achievements. This is the base to involve stakeholders in consolidating those processes.
Keywords: evaluation, empowerment evaluation, process mapping, development projects
1. Introduction
The way we choose for the execution of a development project is the key element of the final quality, since the supported or impelled development process is the main goal. (Ferrero, 2003; Varela, 2007). Consequently, if we try to evaluate the quality and the sustainability, it is very important to determine the flexibility of the project to be adapted to the changes of the environment and the capacity to strengthen the processes.
In this sense, requirements of the financial agencies are focus on the efficacy/efficiency in order to achievement of results. However there is a growing demand to measure sustainability and long-term impacts.
This approach demands to put the attention more in the processes that are started across the Project, and also their learnings than in the direct and short-term results. This implies an adjustment of the systems of monitoring and evaluation.
Evaluation culture has been gaining a growing worldwide recognition (Diaz-Puente et al., 2007), and an increasing influence to foster change processes (Kirkhart, 2000), capacity building and learning (Taut, 2007).
This paper shows how this process-centered approach has been carried out in the ex –post evaluation of a cooperation Project located in the east cone of Lima (Peru). For this purpose, several methods and evaluation tools had been mixed up. Mixing-methods has become a common approach to program evaluation(Green and Caracelli, 1997; Hishigsuren, 2007;
315
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
Lawrenz and Huffman, 2002), in the belief that ―a combination of methods may bear the
potential to produce a study that is superior to that which can be produced by any single-method approach‖ (Waysman and Savaya, 1997).
2. Methodology
Figure 1 shows the methodology and tools that have been used for the evaluation, organized through key questions. This is the most generalized approach for the evaluation of the programs of the EU because it is very adequate to define evaluation contents (Commission of the EU, 1994). The methodology for answering key questions follows this sequence (Díaz-Puente, 2003):
To structure the information.
To formulate key questions.
To describe the purpose of the question.
To define criteria and indicators.
To analyze quantitative and qualitative information.
To answer the question: conclusions and recommendations.
To describe limitations to the answer.
The task of documenting, previous to the analysis, as in any other evaluation work, includes the review of the different reports of the project, data bases, publications, etc. Another tools are used in a complementary way such as economic analysis of case studies, and mainly participative tools in order to obtain qualitative information: interviews with key informants, focal groups, questionnaires5.
Figure 1. Methodological approach and evaluation tools applied to Project of Integral Productive
Cooperation (PCIP)
5 There are many references about these tools. A traditional one is: UD-NORAD, Evaluación de proyectos de ayuda
al desarrollo. Manual para evaluadores y gestores. IUDC-CEDEAL, Madrid, 1997.
EMPOWERMENT
EVALUATION
WORKSHOPS
TRADITIONAL
EVALUATION TOOLS
PROCESS MAPPING
WORKSHOP
PCIP + URBAN HARVESTORIENTATIONS FOR THE
EVALUATION
GESPLAN-UPM
METHODOLOGICAL APPRACH
PARTICIPATORY EVALUATION
PARTICIPACIÓN
SOCIAL
INTERVIEWS, WORKSHOPS, ,
SURVEYS
ANALYSIS AND RESULTS
ANSWER TO THE EVALUATION QUESTIONS
QUANTITATIVE
INFORMATION
REPORTS, STUDIES, DATA
BASES
PROCCESS MAPPING
EVALUATION QUESTIONS
CONCLUSSIONS AND RECOMENDATIONS Phases with joint learning
KEY
QUESTIONS
METHODOLOGY
316
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
However, there have been two main methodological complements: process mapping and empowerment evaluation.
Process mapping is oriented to identify processes and sub-processes, conscious or underlying, that have been involved in the Project. On the base of these processes is easier to understand the execution, to structure it and then to define the evaluation questions according to each process.
Empowerment evaluation is oriented to involve the main Project stakeholders into the evaluation: groups that have acted and also had enough entity and structure to continue some aspect of the Project. In this sense Empowerment Evaluation not only directs to participation –as a tool to obtain qualitative information– but also to the perception of the different processes by the most important stakeholders that can contribute to its sustainability.
The mix of these two methodological complements leads to the learning of the involved stakeholders (team Project, beneficiaries, local institutions). This learning is achieved through self-evaluation about how processes have been started, how have being carried out, and also how can continue after the Project is finished.
Key questions approach and the rest of the traditional tools are well known, so only process mapping and empowerment evaluation will be briefly described now.
Process Mapping
This methodological approach is usually linked to disciplines related to business and industrial organization. It has been known as an useful tool for understanding complex processes (Pojasek, 2005) and how an organization works (Matsumoto et al., 2005), but it is also very useful to legitimate and spread strategic changes. Consequently Process Mapping has a reforming and analytical capacity (Fenton, 2007).
Because of these characteristics Process Mapping is useful in processes-centered and learning-oriented evaluation. Learning arises during the production of the map that evaluator conducts.
There are many ways to draw the processes map, but basically it is necessary to take stock of all the activities carried out in the Project and the relations among them –direct and indirect. The basic sequence that has been applied in our case study was as follows:
Review of monitoring reports: the evaluator draws every activity in the draft map, looking for precedence relations among them. He could take the original strategy of the project as framework, but it is not necessarily the better option, rather it suits to rethink if the relations among activities fit better in another scheme that facilitates the comprehension of the project and the coherence.
Interviews: the evaluator supports individual meetings with each of the members of the project team. to penetrate into each of the activities, to discuss the relations among them, and also to identify activities that have been carried out but not reflected adequately in the reports.
Validation workshop: the first joint learning process takes place in it, since it is the first time that all the project team have a global vision of every activity and process that the evaluator has detected. The debate is orientated to change activities, add new ones, identify relationships, sub-processes, etc. A very important task is to agree on the name that it is given to the processes.
317
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
Drawing the final map: a complete process map is produced that includes every activity, and then a simplified scheme only with processes and sub-processes is drawn. This map is contrasted with the original strategy of the project to see what areas have been covered, what new contributions to the strategy are, and to look for explanations of the complex process that has been developed. Finally both, the evaluator and the project team, select the main areas for being evaluated and the corresponding evaluation questions.
Empowerment Evaluation.
This approach has become a worldwide phenomenon since it was developed by Dr. David Fetterman and was made known in 1993 through the American Evaluation Society (Fetterman, 2001; Díaz-Puente, 2007). Empowerment evaluation and traditional evaluation are not mutually exclusive and can be mutually reinforcing (Fetterman and Wanderman 2007). Empowerment evaluation has a clear orientation and is designed to help the people to help themselves and improve their projects and programs by using self-evaluation. The evaluator has the role of coach or facilitator depending on the internal capacities of the program.
It is the learning base for key actors who are organized under any form of formal or informal association. In this sense it is necessary to work only with organizations that have vocation of permanency and could use the learning to improve. Empowerment evaluation it is not adapted for groups constituted by individuals joined around any singular activity, e.g. credit beneficiaries, attendants to workshops or training courses, members of different kind of platforms without organic entail (round tables, institutional forums, etc).
As described by Miller and Lennie (2005), there are three steps in conducting an empowerment evaluation, through a series of workshops
Developing a mission and vision: This involves developing statements that capture the mission and vision or unifying purpose of the program. The aim is to reach consensus on key phrases in the statements, which are seen as representing the values of stakeholders and the agreed purpose and long-term outcomes of the program. This process is undertaken even when an existing mission and vision statement exists.
Taking stock: This step involves brainstorming, then rating the most important program activities to be evaluated and discussing the ratings in groups. This provides an opportunity for baseline data on the program and its strengths and weaknesses to be assembled.
Planning for the future: Realistic goals for each of the key activities are identified, together with strategies that will help reach these goals and the forms of documentation or evidence that will enable progress towards these goals to be monitored.
The philosophy of this approach seeks to strengthen the organizations through self-learning in an on-going evaluation basis; so it is strongly recommended to carried out empowerment workshops from time to time in order to consolidate achievements and re-plan the following year.
3. Application to the Project of Integral Productive Cooperation (PCIP) in Lurigancho-Chosica.
Lurigancho-Chosica district is located in the East Cone of Lima, and it is one of the widest and less built-up of this province. A wide variety of agricultural, industrial and commercial activities coexist in an area covering half of the useful surface. The Urban Harvest program6 and the 6 The Urban Harvest program is a system-wide initiative of the Consultative Group on International Agricultural Research (CGIAR), that it is coordinated by the International Potato Center (CIP). This program tries to promote
318
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
Technical University of Madrid proposed a development Project in this place focused on urban agriculture.
The main goal of the project was to reduce urban poverty and to increase food security through empowering local governments and planning for sustainable local development. a Program of Integral Cooperation Productive, was designed and carried out to achieve it. This PCIP was based on the integration of the urban and peri-urban agriculture in the sustainable development of the local and regional governments, and aligned with the national plan of fight against the poverty with perspectives of long-term sustainable development.
The conceptual framework (Prain, 2006) is based on an integral model of urban agricultural production, with a systemic, planned and participative approach, which facilitates the social learning among stakeholders like producers, local governments, research organizations and any other local group. Likewise it promotes social organization, especially among farmers.
The project was executed from 2006 to 2008. and is characterized for presenting a territorial approach, and for promoting the links with local governments to guarantee the sustainability (Salvo, 2006). The activities and results were expected in the following areas:
Training: both urban farmers and municipal managers.
Strengthening institutions: defining a system of co partnership in the urban management among all the entities (local Governments, farmers, irrigation associations, owners' associations, etc).
Social development as a key element of the economic development: improving human development and environment.
Micro investment fund: stimulating new economic activities.
Transfer of results and expansion to other municipalities.
Evaluation tasks
The application of the methodology described above needed 1 workshop of process mapping and 4 workshops of empowerment evaluation, besides a round of interviews to responsible people of the involved institutions, 3 economic analysis of case studies and a survey to beneficiaries of microcredit.
Identification of learning processes and key areas for evaluation purposes
Process mapping was used to get the map that is shown in figure No. 2. This map includes all the realized activities that have been carried out and also the relations among them. It is not the object of this paper to describe all of them in detail, for which a major extension would be necessary, but to show how this methodology allows to offer an image of the complexity of the project, and especially how it allows us to identify processes and sub-processes.
An horizontal reading shows the processes aligned with four big areas of work. Process have been drawn in white color when is considered to be processes and sub-processes that shape the main task. In the other hand, they had been drawn in red if they are considered as derivative processes, normally answering to a vertical reading that shows how they interact in an interesting and sometimes unexpected way.
Figure No.3 outlines the processes -without activities- and shows how they are organized in methodological blocks and how they complement each other mutually. This analysis gives a great more depth to the evaluation that considering the different work areas in a separated way, because it shows how many processes are interdependent. For example, the Microcredit, far from being an independent component, is a crucial element to consolidate the advances
urban and periurban agriculture as a key component to improve sustainability in towns of Africa, Asia and Latin America.
319
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
obtained in training and marketing, offering the beneficiaries the possibility of putting into practice what they have learned. In other area, the map shows how all the activities and achievements of the institutional strengthening process drive finally to actions of transfer, so both process are not separated but strongly linked.
These processes and their interrelationships shape the real strategy of intervention, which can be compared with the initial strategy of the project.
320
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
Figure 2. Detailed processes map of the PCIP Lurigancho-Chosica.
YE
AR
20
06
YE
AR
20
08
YE
AR
20
07
SIS
TEM
A D
E
CO
NTR
OL
INTE
RN
O
EC
A C
UY
ES
(1 c
iclo
)
DIS
EÑ
O C
EN
TRO
P
RO
CE
SA
MIE
NTO
CO
NS
OR
CIO
O
PC
EL&
CC
ER
TIFI
CA
CIÓ
N
EC
OLO
ÓG
ICA
DE
LA
S P
AR
CE
LAS
CU
RS
O E
N
MA
NE
JO
INTE
GR
AD
O D
E
CU
LTIV
OS
CU
RS
O
FOR
MA
CIÓ
N
LÍD
ER
ES
CU
RS
O
GE
STI
ÓN
E
MP
RE
SA
RIA
L
FER
IAS
CO
ME
RC
IO
EC
OLÓ
GIC
O E
N
LIM
A
OP
ER
AC
IÓN
FO
ND
O
RO
TATO
RIO
CE
RTI
FIC
AC
IÓN
O
RG
ÁN
ICA
IMP
ULS
O D
E
RE
LAC
ION
ES
C
OM
ER
CIA
LES
Y
PR
OFE
SIO
NA
LES
ES
CU
ELA
S D
E
AG
RIC
ULT
OR
ES
CO
ME
RC
IALI
Z. D
E
PR
OD
UC
TOS
AU
FER
IAS
M
UN
ICIP
ALE
S
TRA
DIC
ION
ALE
S
FER
IAS
M
UN
ICIP
ALE
S E
N
LA E
AU
(A)
VE
NTA
A
DO
MIC
ILIO
EN
LI
MA
(A)
EX
PLO
RA
CIÓ
N
CA
NA
LES
C
OM
ER
CIA
LES
PA
RTI
CIP
AC
IÓN
E
N E
VE
NTO
S D
E
PR
OM
OC
IÓN
(A)
EC
OLÓ
GIC
A-P
ER
Ú
(A)
VE
NTA
DIR
EC
TA
EA
U
(A)
VE
NTA
A
RE
STA
UR
AN
TES
LO
CA
LES
(A)
RE
PLI
CA
CIÓ
N
SE
MIN
AR
IOS
PO
R
LOS
B
EN
EFI
CIA
RIO
S
FOR
MA
LIZA
CIÓ
N
GR
UP
OS
P
RO
DU
CTO
RE
S
CO
NS
TITU
CIÓ
N
EQ
UIP
O D
E
TRA
BA
JO
FOR
MU
LAC
IÓN
D
E E
STR
ATE
GIA
IN
ICIA
L
VA
LID
AC
IÓN
DE
LO
S G
ER
EN
TES
TALL
ER
ES
P
AR
TIC
IPA
TIV
OS
C
ON
LA
P
OB
LAC
IÓN
TALL
ER
DE
A
LCA
LDE
S 2
006
ES
TRA
TEG
IA D
E
AG
RIC
ULT
UR
A
UR
BA
NA
M
UN
ICIP
AL
AU
EN
LA
S
UB
GE
RE
NC
IA D
E
LA D
.AG
RA
RIA
DE
LI
MA
SE
NS
IBIL
IZA
CIÓ
N
EN
ÁM
BIT
OS
S
UP
RA
MU
NIC
IP.
EQ
UIP
O L
OC
AL
DE
AG
RIC
ULT
UR
A
UR
BA
NA
PR
OG
RA
MA
DE
D
ES
AR
RO
LLO
P
OR
CIN
O
ES
TRA
TEG
IA D
E
AG
RIC
ULT
UR
A
UR
BA
NA
OR
DE
NA
NZA
S
MU
NIC
IPA
LES
EN
A
GR
ICU
LTU
RA
U
RB
AN
A
RE
GLA
ME
NTO
DE
B
UE
NA
S
PR
ÁC
TIC
AS
A
GR
ÍCO
LAS
AC
TUA
CIO
NE
S D
E
OR
DE
NA
CIÓ
N D
E
EX
PLO
TAC
IÓN
ES
EN
TRE
GA
DE
M
ÓD
ULO
S A
C
AP
AC
ITA
DO
S
PROGRAMA ―MI
GRANJITA‖
CU
RS
OS
PA
RA
C
RIA
DO
RE
S
CE
RD
OS
(CE
SA
L)
FOR
MA
CIÓ
N D
E
OR
GA
NIZ
AC
ION
ES
D
E P
RO
DU
CTO
RES
(3
org
aniz
acio
nes)
PR
OY
EC
TOS
D
EM
OS
TRA
TIV
OS
D
E C
RIA
DE
RO
S
INS
EM
INA
CIÓ
N
CO
ND
ICIO
NA
DA
A
INV
ER
SIÓ
NC
AP
AC
ITA
CIÓ
N
TALL
ER
DE
A
LCA
LDE
S 2
006
CO
NV
EN
IO M
AR
CO
U
RB
AN
HA
RV
ES
T-M
UN
ICIP
ALI
DA
D
ME
TRO
. LIM
A
RE
PLI
CA
CIÓ
N
ES
CU
ELA
S D
E
AG
RIC
ULT
UR
A
UR
BA
NA
EN
O
TRO
S D
ISTR
ITO
S
TALL
ER
ES
P
RE
PA
RA
TOR
IOS
C
ON
LA
S
GE
RE
NC
IAS
CU
RS
O M
AN
EJO
O
RG
ÁN
ICO
(2 c
iclo
s)
FOR
MA
CIÓ
N
CO
NTÍ
NU
A E
N
LID
ER
AZG
O,
GE
STI
ÓN
, ETC
.
CE
NTR
O
PR
OC
ES
AM
IEN
TO
DE
HO
RTA
LIZA
S
TRA
NS
FER
EN
CIA
FO
ND
O
RO
TATO
RIO
Y
SE
GU
IMIE
NTO
1ª
CO
NV
OC
ATO
RIA
FO
ND
O
RO
TATO
RIO
2ª
CO
NV
OC
ATO
RIA
FO
ND
O
RO
TATO
RIO
3ª
CO
NV
OC
ATO
RIA
FO
ND
O
RO
TATO
RIO
4ª
CO
NV
OC
ATO
RIA
FO
ND
O
RO
TATO
RIO
5ª
CO
NV
OC
ATO
RIA
FO
ND
O
RO
TATO
RIO
6ª
CO
NV
OC
ATO
RIA
FO
ND
O
RO
TATO
RIO
7ª
CO
NV
OC
ATO
RIA
FO
ND
O
RO
TATO
RIO
INFO
RM
E F
INA
L P
CIP
INFO
RM
E
SE
GU
IMIE
NTO
20
06
PR
OG
AM
AC
IÓN
20
06P
RO
GR
AM
AC
IÓN
20
07IN
FOR
ME
S
EG
UIM
IEN
TO
2007
INFO
RM
E
SE
GU
IMIE
NTO
20
08
EV
ALU
AC
IÓN
P
AR
TIC
IPA
TIV
AP
RO
GR
AM
AC
IÓN
20
08
PR
OP
UE
STA
P
RO
YE
CTO
C
HIM
BO
TE
EA
U/S
ALU
D
TRA
NS
FER
EN
CIA
D
E
ME
TOD
OLO
GÍA
S A
P
RO
YE
CTO
S E
N
OTR
AS
RE
GIO
NE
S
DE
PE
RÚ
ES
CU
ELA
S D
E
AG
RIC
ULT
OR
ES
U
RB
AN
OS
(4 n
uevo
s di
strit
os)
CU
RS
O E
N
MA
NE
JO
INTE
GR
AD
O D
E
CU
LTIV
OS
CU
RS
O M
AN
EJO
IN
TEG
RA
DO
DE
C
ULT
IVO
S Y
C
RIA
NZA
TALL
ER
DE
FO
RM
AC
IÓN
DE
TÉ
CN
ICO
S
MU
NIC
IPA
LES
TALL
ER
DE
D
IAG
NÓ
STI
CO
P
AR
TIC
IPA
TIV
O
CU
RS
O D
E
FOR
MA
CIÓ
N D
E
FAC
ILIT
AD
OR
ES
D
E E
CA
S
TALL
ER
E.E
. CO
N
PR
OD
UC
TOR
ES
: P
LAN
TR
AB
AJO
A
NU
AL
RE
CE
RTI
FIC
AC
IÓN
CU
RS
O D
E
CA
PA
CIT
AC
IÓN
EN
M
AN
EJO
PO
RC
INO
TALL
ER
MA
NE
JO
BIO
HU
ER
TO
INTE
GR
AD
O
ES
CU
ELA
S D
E
CA
PA
CIT
AC
IÓN
A
GR
AR
IA
RE
SE
RV
OR
IOS
Y
AC
UIC
ULT
UR
A
NU
TRIC
IÓN
IN
FAN
TIL
AU
INC
LUID
A E
N
LAS
OR
DE
NA
NZA
S
DE
LIC
EN
CIA
DE
A
CTI
VID
AD
ES
FOR
MU
LAC
IÓN
P
CIP
200
5
PR
OY
EC
TO
HO
RTI
CU
LTU
RA
E
N A
ND
ES
C
EN
TRA
LES
TALL
ER
ES
P
AR
TIC
IPA
TIV
OS
D
E R
ED
AC
CIÓ
N
PR
ES
EN
TAC
IÓN
, V
ALI
DA
CIÓ
N Y
C
OR
RE
CC
IÓN
SE
NS
IBIL
IZA
CIÓ
N
A R
EG
IDO
RE
S
TALL
ER
ES
P
AR
TIC
IPA
TIV
OS
D
E R
ED
AC
CIÓ
N
RE
GLA
ME
NTO
DE
C
RIA
NZA
PO
RC
INA
MA
NU
ALE
S D
E
AG
RIC
ULT
UR
A
UR
BA
NA
RE
UN
ION
ES
FO
RM
ULA
CIÓ
N
PR
OY
EC
TO A
U
CO
N L
A M
ML
CO
NS
TITU
CIÓ
N
DE
L FO
ND
O
RO
TATO
RIO
RE
GLA
ME
NTO
FR
AC
MA
NU
AL
DE
C
AM
PO
VA
LID
AD
OO
RG
AN
IZA
CIÓ
N D
E
LA G
ES
TIÓ
NP
RO
MO
CIÓ
N Y
S
OLI
CIT
UD
ES
P
ILO
TO
INTE
RM
ED
IAR
IOS
P
RO
VE
ED
OR
ES
S
UP
ER
MA
RK
ETS
PR
OD
UC
TOS
BE
BÉ
(A)
AS
ES
OR
AM
IEN
TO
CO
NTI
NU
O
RE
ALI
ZAD
O P
OR
E
CO
-LO
GIC
A
VE
NTA
A
EN
TID
AD
ES
DE
C
ON
SU
MID
OR
ES
O
RG
ÁN
ICO
S (A
)
PÁ
GIN
A W
EB
C
OM
ER
CIA
LIZA
C.
SE
GU
IMIE
NTO
S
EM
AN
AL
(pro
ducc
ión,
té
cnic
as, i
ngre
sos,
)
SE
GU
IMIE
NTO
Q
UIN
CE
NA
L (o
rgan
izac
ión,
sis
t. in
tern
o co
ntro
l)
SE
GU
IMIE
NTO
Q
UIN
CE
NA
L (P
lan
de tr
abaj
o)
CU
RS
O
AC
TUA
LIZA
CIÓ
N Y
N
IVE
LAC
IÓN
AS
ES
OR
ÍA P
AR
A
OR
GA
NIZ
AR
C
UR
SO
S D
E
CA
PA
CIT
AC
IÓN
VIS
ITA
S A
LA
S
EA
U
(vec
inos
, ON
G, e
tc)
PA
PA
CO
CTE
L
(CIP
)
INV
ES
TIG
AC
IÓN
P
AR
TIC
IPA
TIV
A
CA
PA
CIT
AC
IÓN
E
N M
AN
EJO
O
RG
ÁN
ICO
(2 c
iclo
s)
TALL
ER
SO
BR
E
CU
ALI
DA
DE
S
NU
TRIC
ION
ALE
S
DE
L P
RO
DU
CTO
TRA
TAM
IEN
TO
AG
UA
S R
IEG
O
FER
OM
ON
AS
TO
MA
TE
PR
EV
IOU
S
AC
TIV
ITIE
S
BUSINESS AND
MARKETING
TRAININGINSTITUTIONAL
STRENGHENTENPR
OC
ES
SE
S M
AP
OF
UR
BA
N H
AR
VE
ST
(P
CIP
)D
r. Jo
sé L
uis
Yagü
e -
Nov
embe
r, 20
08
SE
MIN
AR
IOS
Y
TALL
ER
ES
AV
ES
(o
rgan
iza
subg
eren
cia)
SE
GU
IMIE
NTO
C
ON
TIN
UO
DE
A
CTI
VID
AD
ES
DE
LA
SU
BG
ER
EN
CIA
BIO
HU
ER
TO E
N
ES
PA
CIO
R
ED
UC
IDO
(a
utoc
onsu
mo)
SE
MIN
AR
IOS
C
OR
TOS
AD
HO
C
AS
OC
IAC
IÓN
A
AN
PE
FER
IA
DE
MO
STR
ATI
VA
E
N E
L C
IP
Insti
tuti
onals
en
sit
izatio
n
Rep
licati
on
14
2In
teg
rati
nt
urb
an
farm
ing
in th
em
un
icip
al d
evelo
pm
en
tstr
ate
gy
Tra
inin
g in
Urb
an
Farm
erF
ield
Sch
oo
ls
6T
rain
ing
in b
usin
ess
cap
acit
ies
7A
co
mp
an
yin
g
9O
pen
ing
new
co
mm
erc
ial
ch
an
nels
15
MA
NA
GE
ME
NT
AN
D
MO
NIT
OR
ING
13
Mic
rocre
dit
10
Mark
eti
ng
Reg
ula
tio
ns
develo
pm
en
t3
Org
an
ic
cert
ific
ati
on
11
Co
nso
lid
ati
on o
f
the p
rod
ucers
'
org
an
izati
ons
12
4T
rain
ing
in b
reed
ing
CU
RS
O
CA
PA
CIT
AC
IÓN
C
RIA
NZA
DE
P
OR
CIN
O
MO
DU
LOS
Y
CU
RS
OS
DE
C
RIA
NZA
DE
A
NIM
ALE
S
1
5
CR
EA
CIÓ
N D
E
MA
RC
AS
Y P
LAN
D
E C
OM
ER
CIA
LIZ.
1.2
.0.0
1.2.
1.0
1.2
.2.1
1.2
.2.2
1.2
.3.1
1.2
.3.2
14
.1.0
.0
14.1
.1.1
14.1
.1.2
14
.1.2
.01
4.1
.2.3
14
.1.2
.21
4.1
.2.1
1.3
.0.0
2.2
.0.0
2.2
.1.1
2.2
.2.1
2.2
.2.2
2.2
.2.3
2.3
.0.0
2.2
.1.2
2.3
.1.0
2.3
.2.1
2.3
.2.2
2.3
.2.0
2.3
.3.0
2.3
.4.0
3.0
.0.0
3.1
.1.0
3.3
.1.0
3.2
.1.0
3.3
.2.0
3.3
.3.0
3.3
.4.0
3.2
.2.0
4.4
.0.0
4.1
.0.0
4.2
.1.0
4.2
.2.2
4.4
.1.1
4.4
.2.0
4.4
.3.0
4.3
.1.0
4.4
.1.2
4.4
.1.3
14
.3.0
.01
4.3
.1.0
14
.3.2
.0
15
.1.0
.01
5.1
.1.1
15
.1.1
.21
5.1
.2.1
15
.1.2
.21
5.1
.3.1
15
.1.3
.11
5.2
.0.0
15
.3.0
.0
13
.2.6
.01
3.2
.7.0
13
.3.0
.01
3.2
.4.0
13
.2.3
.01
3.2
.5.0
13
.2.0
.01
3.2
.1.0
13
.2.2
.01
3.1
.0.0
13
.1.3
.01
3.1
.1.0
13
.1.2
.01
3.1
.4.0
8.2
.0.0
8.2
.1.0
8.2
.2.0
8.2
.3.0
8.2
.4.1
8.1
.1.0
8.1
.2.0
5.2.
1.2
5.2.
1.1
5.2.
2.0
5.1
.0.0
5.2.
0.0
5.2.
1.3
5.2
.3.0
5.2
.1.4
6.1
.0.0
6.2
.1.0
6.2
.2.0
6.3
.0.0
6.4
.1.0
6.4
.2.0
6.5
.0.0
7.3
.0.0
7.4
.0.0
7.1
.0.0
7.2
.0.0
5.4
.1.0
5.4.
2.0
12
.1.0
.0
12
.2.0
.0
12
.3.0
.0
12
.4.0
.0
12
.5.1
.0
12
.5.2
.0
11
.1.1
.01
1.1
.2.0
11
.1.3
.01
1.1
.0.0
11
.1.4
.0
9.1
.0.0
11
.1.5
.0
9.1
.1.1
10
.1.1
.0
9.1
.1.2
9.1
.2.1
9.1
.2.2
9.1
.3.1
9.1
.4.1
9.1
.4.0
9.1
.4.2
9.1
.4.3
9.2
.0.0
9.0
.0.0
9.2
.0.0
10
.2.0
.01
0.1
.2.0
10
.3.0
.01
0.1
.3.0
10
.4.0
.0
11
.1.6
.11
1.1
.6.2
0.1
.0.0
0.2
.0.0
2.4.
0.0
4.4
.4.0
5.3
.1.0
5.3
.2.0
CO
NV
EN
IO C
ON
H
UA
CH
YP
A P
AR
A
RE
ALI
ZAR
AC
TIV
. C
ON
LA
OR
GA
NIZ
.
14
.1.3
.114
.2.0
.0
14
.1.3
.2
TALL
ER
C
AP
AC
ITA
CIO
N E
N
ME
TOD
OLO
GÍA
E
AU
TRA
NS
FER
EN
CIA
M
ETO
DO
LOG
ÍA
EA
U
14
.1.3
.3
8.2
.4.2
CU
BIE
RTA
S
CU
LTIV
O:
PLÁ
STI
CO
CU
BIE
RTA
S
CU
LTIV
O: M
ALL
A
AN
TIÁ
FID
OS
EV
ALU
AC
IÓN
DE
M
ICR
OO
RG
AN
ISM
. E
FEC
TIV
OS
(CIP
)
Part
icip
ato
ry
resarc
h8
8.2
.5.0
CO
NV
EN
IOS
CO
N
4 N
UE
VO
S
MU
NIC
IPIO
S
AS
ES
OR
ÍA P
AR
A
CR
EA
CIÓ
N
SU
BG
ER
EN
CIA
P
AC
HA
CA
MA
C
8ª
CO
NV
OC
ATO
RIA
FO
ND
O
RO
TATO
RIO
13
.2.8
.0
SE
MIN
AR
IOS
Y
TALL
ER
ES
P
OR
CIN
O
(o
rgan
iza
subg
eren
cia)
4.2
.2.1
CR
EA
CIÓ
N D
E L
A
SU
BG
ER
EN
CIA
DE
A
GR
ICU
LTU
RA
U
RB
AN
A
RE
UN
ION
ES
IN
TER
INS
TITU
C.
2003
Y 2
004
1.1.
0.0
2.1
.0.0
321
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
Figure 3. Schematic processes map of the PCIP Lurigancho-Chosica.
OP
ER
AC
IÓN
FO
ND
O
RO
TATO
RIO
CE
RTI
FIC
AC
IÓN
O
RG
ÁN
ICA
ES
CU
ELA
S D
E
AG
RIC
ULT
OR
ES
CO
ME
RC
IALI
Z. D
E
PR
OD
UC
TOS
AU
ES
TRA
TEG
IA D
E
AG
RIC
ULT
UR
A
UR
BA
NA
M
UN
ICIP
AL
SE
NS
IBIL
IZA
CIÓ
N
EN
ÁM
BIT
OS
S
UP
RA
MU
NIC
IP.
PR
OG
RA
MA
DE
D
ES
AR
RO
LLO
P
OR
CIN
O
OR
DE
NA
NZA
S
MU
NIC
IPA
LES
EN
A
GR
ICU
LTU
RA
U
RB
AN
A
CA
PA
CIT
AC
IÓN
RE
PLI
CA
CIÓ
N
ES
CU
ELA
S D
E
AG
RIC
ULT
UR
A
UR
BA
NA
EN
O
TRO
S D
ISTR
ITO
S
TRA
NS
FER
EN
CIA
D
E
ME
TOD
OLO
GÍA
S A
P
RO
YE
CTO
S E
N
OTR
AS
RE
GIO
NE
S
DE
PE
RÚ
ES
CU
ELA
S D
E
AG
RIC
ULT
OR
ES
U
RB
AN
OS
(4 n
uevo
s di
strit
os)
CO
NS
TITU
CIÓ
N
DE
L FO
ND
O
RO
TATO
RIO
INV
ES
TIG
AC
IÓN
P
AR
TIC
IPA
TIV
A
14
2
6
7
9
15
13
10
3
11
12
4
1
5
12
.3.0.0
8
MA
IN
ME
TH
OD
OL
OG
ICA
L
BL
OC
K
CO
MP
LE
ME
NTA
RY
ME
TH
OD
OL
OG
ICA
L
BL
OC
K
SU
PP
OR
T E
LE
ME
NT
LO
BB
Y &
AC
CO
MP
AN
YIN
G
YE
AR
20
06
YE
AR
20
08
YE
AR
20
07
BUSINESS AND
MARKETING
TRAININGINSTITUTIONAL
STRENGHENTENPR
OC
ES
SE
S M
AP
OF
UR
BA
N H
AR
VE
ST
(P
CIP
)D
r. Jo
sé L
uis Y
agüe
-N
ovem
ber,
2008
Insti
tuti
onals
en
sit
izatio
n
Rep
licati
on
Inte
gra
tin
tu
rban
farm
ing
in th
em
un
icip
al d
evelo
pm
en
tstr
ate
gy
Tra
inin
g in
Urb
an
Farm
erF
ield
Sch
oo
ls
Tra
inin
g in
bu
sin
ess
cap
acit
ies
Aco
mp
an
yin
g
Op
en
ing
new
co
mm
erc
ial
ch
an
nels
MA
NA
GE
ME
NT
& M
ON
ITO
RIN
G
Mic
rocre
dit
Mark
eti
ng
Reg
ula
tio
ns
develo
pm
en
t
Org
an
ic
cert
ific
ati
on
Co
nso
lid
ati
on o
f
the p
rod
ucers
'
org
an
izati
ons
Tra
inin
g in
bre
ed
ing
Part
icip
ato
ry
resarc
h
322
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
From these blocks of processes, key questions of evaluation can be established with more concretion, sense and direction that those who would correspond to the areas of the initial strategy:
In what measure has the Urban Farmers Field School been an instrument adapted for the training in opposite to other conventional means? Basic indicators of global participation, level of permanency in the courses, intensive participants, level of satisfaction, links to other activities of the project, etc. are used to compare between these Schools and conventional training courses.
In what measure has PCIP helped to reinforce social organization and producers organization by means of marketing? Several indicators are used to answer this question, related to the constitution and consolidation of new companies and organizations, case studies of profitability of some activities, and also qualitative indicators about perception of the level of participation in the community.
In what measure has the Urban Farmers' Rotating Fund –microcredit– worked as an effective instrument for the promotion of sustainable productive activities? Basic indicators are used, mainly related to execution (returns, number of delinquent debtors, number of financed projects, etc), but also to the level of satisfaction of the beneficiaries, and the analysis of the procedure of management and monitoring.
In what measure has PCIP helped to include Urban Agriculture in the municipal agenda as a strategic element for the development? Answering from the analysis of institutional changes that has involved (municipal flowchart, schedule), direct effects in policies (regulations, programming), and new activities carried out in other municipalities.
Finally, two transverse questions are added as in any other project:
In what measure has PCIP been managed in an effective way for the fulfillment of the goals? The answer focus on levels of execution of budget, procedures of management, control of the activities, profile of the project team, transfer of results and interaction with other institutions and departments, etc.
In what measure has PCIP taken account of questions about gender and environment? Qualitative indicators of perception of the actors have been used.
The project team works jointly with the evaluator both in the construction of the map and in the identification of areas and questions of evaluation, therefore a first learning is reached, especially about the changes that are necessary to improve the methodologies, and also in relation to the possibilities of applying these methodologies to another places.
Empowering stakeholders to enhance sustainable processes
Empowerment evaluation workshops were directed to formalized groups that were able to continue their activity, and consequently were able to introduce changes thanks to this instrument. In this sense we emphasize how evaluation can be considered as one of the last activities in the project, thanks to which the beneficiaries take consciousness about their own organization and be able to think about the future.
In the case of the Urban Agriculture Office, the workshop gave a greater visibility and support to this office inside the municipality, by offering an overview of the achievements of the PCIP to the rest of people responsible of municipal management, and also offering the possibility to discuss among them about the implications in the municipal policy.
323
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
One of the best achievements of the project has been to organize, to create and to consolidate two producers' organizations with access to new markets, with demonstrative effect in organic management, and with support and municipal recognition. Both of them constitute a permanent shop window of the effects that PCIP have achieved, and so workshops were directed to each one of these organizations. Another workshop was carried out with an organization that had not been shaped across the project –though many members had been benefited of the activities–, as a control group.
To give an idea of the usefulness that this approach provided, figure no. 3 shows how the sense of organization, and therefore its continuity, was identified in the Mission of the new organizations, which in addition was connected with many of the carried out processes. On the contrary, the Mission of the control group showed a more individual point of view and scarcely incorporated links to a couple of processes.
Figure 4. Comparison of mission statements developed by producers' organizations through Empowerment Evaluation workshops and their relation with the identified processes
The orientation of these workshops was not mainly directed to obtain information for the evaluation, but to encourage participants to thing about the future of their organization after the project is finished. There is a double learning effect: learning obtained from all the activities that
PRODUCE
HEALTHY FOOD
GOOD PRACTICES
TO IMPROVE
ENVIRONMENT
IMPROVE
TECHNOLOGIES
OF PRODUCTION
ACCESS TO A
GROWING
MARKET
INCREASE
ECONOMIC
BENEFITS
MISION
GENERATE
ENVIRONMENTAL
CONSCIENCE
QUALIFY OTHER
PRODUCERS
IMPROVE SELF-
CONSUMPTION
AND NUTRITION
TRAINING IN
BUSINESS
MANAGEMENT
CONSOLIDATE
ORGANIZATION
TRAINING IN
ORGANIC
PRODUCTION
Cosecha Sana CaparapongoSAC (COSANACA)
6
9
10
5
12
11
8
14
13
IMPROVE PRODUCER´S QUALITY OF LIFE
OFFER AN ALTERNATIVE
OF JOB
PROFESSIONALIZATION
OF THE PRODUCER
ACCESS TO NEW
MARKETS
INVESTMENTS IN
GENETICS AND
INFRAESTRUCTURE
WASTE MANAGEMENT
AND PROPER
TECHNIQUES
ACCESS TO
CONTINUOUS
TRAINING
BETTER PRICES TO
IMPROVE REVENUES
TO PROTECT THE
ENVIRONMENT OF
WORK
MISION
Asociación de Productores de Saracoto
4 13
324
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
have been carried out, and learning obtained from the reflection towards the future. This is the base of the philosophy of empowerment, and for that reason this approach allows participants to chose which elements in the strategy are relevant –and so considered in the analysis– and which are not.
Process mapping had helped the project team to reinforce the overall view of strategy as an well integrated package. For that reason the last empowerment workshop was directed to this group. It has been very important in order to help them to define the last activities of the project, which also affect to several of the final recommendations in the evaluation report: the systematizing and publication of the developed experience, the systematizing and publication of the applied methodologies, the possibility to apply them in new project or places, etc. Definitively a reinforcement of the methodologies applied in the project has been achieved through this evaluation approach.
5. Conclusions
The evaluation presented as case study appoints to a methodological approach that is based on direct and joint participation of the actors involved in the project. The aim is to provide a judgment of value from the perspective of local learning generated across the activated processes, and not from the traditional perspective of accountability
Mixing methods and tools of evaluation, and especially process mapping and empowerment evaluation in a standard methodology of key questions, has proved to be specially useful.
Process mapping offers a great potential in the previous stage of the evaluation. Firstly it helps the project team to self-learning, offering an overview of all the activities out of the frame of the foreseen strategy, and also how processes take place as a logical succession of activities. Secondly it supports the following steps of the evaluation, because it helps to formulate the evaluation questions and to centre the criteria and indicators that are needed. But in addition process mapping is crucial in an learning-centered evaluation, since it allows to see the routes and processes that new projects can follow in the future.
Development processes always imply to manage communities, and consequently unpredictable reactions and also changes due to the local context. The final map shows the result of adapting the original strategy to that changing reality. Consequently it must be considered to be an input for learning that reflects how sustainable results can be obtained. This map is the base to elaborate new proposals, serves as contrast for other evaluation tools, and gives useful information about how to copy those processes into other projects.
Empowerment evaluation approach complements to other participatory tools as describes how processes are perceived by the stakeholders. But in addition this approach helps to promote the sustainability of these processes by working with organizations able to continue them, re-adapt them or apply them in any other places. In this respect empowerment evaluation contributes to the sustainability of the project.
It will be necessary to carry out new case studies in which this methodology is carried out, and also to evaluate the adjustment to other types of development projects, and to evaluate the adequacy to the requirements of the financial agencies. Nevertheless this experience shows how this mix of evaluation tools can contribute to enhance impacts of development projects.
References
325
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
Commission of the EU, The MEANS collection: Evaluating socio-economic programmes.
Luxembourg: Office for Official Publications of the European Communties, 1994.
Díaz-Puente, J.M., Diseño y aplicación de un modelo para el seguimiento y evaluación del
desarrollo rural en la UE. PhD diss., Universidad Politécnica de Madrid, España, 2003.
Díaz-Puente, J.M., Cazorla, A., Dorrego, A., ―Crossing national, continental and linguistic
boundaries: Toward a worldwide evaluation research community in journals of evaluation‖,
American Journal of Evaluation 28(4), 2007. pp. 299-415.
Díaz-Puente, J.M., ―La evaluación en planes y programa de desarrollo: Estado actual y
principales tendencias a nivel internacional‖, in Empowerment Evaluation: Metodología para el
seguimiento de programas y proyectos, Fetterman, D. y Cazorla, A., Madrid, 2007. pp. 21-82.
Díaz Puente, J.M.; Yagüe, J.L.; Afonso, A., ―Building Evaluation Capacity in Spain: A Case
Study of Rural Development and Empowerment in the European Union‖, Evaluation Review Vol 32 (5), 2008. pp. 478-506.
Fenton, E., ―Visualising Strategic Change: The role and impact of process maps as boundary
objects in reorganization‖, European Management Journal, Vol. 25(2), 2007. Pp. 104-117.
Ferrero, G., De los proyectos de cooperación a los procesos de desarrollo. Hacia una gestión
orientada al proceso. PhD diss., Universidad Politécnica de Valencia, España, 2003.
Fetterman, D., Foundations of Empowerment Evaluation, Thousand Oaks, CA: Sage, 2001.
Fetterman, D. and Wandersman, A., ―Empowerment evaluation: Yesterday, today, and
tomorrow‖, American Journal of Evaluation 28(2), 2007. Pp. 179-198.
Green, J. and Caracelli, V., ―Defining and describing the paradigm issue in mixed-method
evaluation‖, in Advances in mixed-method evaluation: The challenges and benefits of
integrating diverse paradigms, New Directions for Evaluation, vol. 74, San Franciso: Jossey-Bass, 1997. Pp 5-17.
Hishigsuren, G., ―Evaluation misión drift in microfinance: Lessons for programs with social
mission‖, Evaluation Review 31(3), 2007. Pp. 203-260.
Kirkhart, K., ―Reconceptualizing evaluation use: An integrated theory of influence‖, in The
expanding scope of evaluation use, New Directions for Evaluation, Vol. 88, Caracelli, V and Prskill, H. San Francisco, 2000. Pp. 5-24.
Lawrenz, F. and Huffman, D., ―The archipelago approach to mixed method evaluation‖,
American Journal of Evaluation 23(3), 2002. Pp. 331-338.
Matsumoto, I. et al., ―Use of process maps to develop a management briefing sheet for a
design consultancy‖, Engineering Construction & Arquitectural Management, Vol.12(5), 2005. Pp. 458-469.
Millers, W. and Lennie, J., ―Empowerment evaluation. A practical method for evaluating a
national school breakfast program‖, Evaluation Journal of Australasia, Vol. 5(2), 2005. Pp. 18-26.
Pojasek, R.B., ―Understanding processes with hierarchical process mapping‖, Environmental
Queality Management, Vol. 15(2), 2005. Pp. 79-86.
Prain, G., ―Urban Harvest: a CGIAR global program on urban and periurban agriculture‖, Food
& Fertilizer Technology Center. Extension Bulletin nº575, 2006, pp.14-36.
326
Area: Rural Development and Cooperation Projects
―Selected Proceedings from the 13th International Congress on Project Engineering‖. (Badajoz, July 2009)
Salvo M., Arce B. y De los Ríos I., ―Integración de las entidades locales en la promoción del
desarrollo rural local consideraciones para Latinoamérica desde la experiencia del Cono Este
de Lima (Perú)‖, Proceedings of the X International Congress on Project Ingeneering, Valencia, 2006.
Taut, S., ―Studying sefl-evaluation capacity building in a large international developmet
organization‖, American Journal of Evaluation, Vol. 28(1), 2007. Pp. 45-59.
Tuggle, F. and Golfinger, W., ―A methodology for mining embedded knowledge from process
maps‖, Human Systems Management, Vol.23, 2004. Pp. 1-13.
Varela, F. et al., La Calidad de las Intervenciones de Desarrollo. CIDEAL-EPTISA (con la colaboración de la Agencia Española de Cooperación Internacional ), Madrid, 2007.
Waysman, M. and Savaya, R., ―Mixed method evaluation: A case study‖, Evaluation practice, 18(1), 1997. Pp. 227-.237.
Acknowledge
The authors gratefully acknowledge the collaboration of the staff of Urban Harvest program and people responsible of the Lurigancho-Chosica Urban Agriculture Office (Subgerencia de Agricultura Urbana) and the Association of Irrigation Users (Junta de Usuarios de Riego). In a special manner they acknowledge every beneficiaries of the PCIP that have unselfishly participated in evaluation tasks that constitute the basis of this paper.
Contact:
José Luis Yagüe Blanco Research Group in Planning and Sustainable Management of the Rural-Local Development (GESPLAN) Technological University of Madrid Dpto. Proyectos y Planificación Rural. E.T.S.I.Agrónomos Avda. Complutense, s/n 28040 Madrid Phone: +34 91 336 38 37 E-mail : [email protected] URL : www.grupogesplan.es
327