TECHNICAL APPENDIX
description
Transcript of TECHNICAL APPENDIX
TECHNICAL APPENDIX
METHODOLOGY SUMMARY FOR CPM RESEARCH
Missions International PO Box 681299 – Franklin, TN 37068 – www.missions.com
© 2004 Missions International
Human Resourcing
• Design of CPM assessment instrument and analysis support
• Hired Statistician– Ph.D. from Vanderbilt in Psychology– VP of Methodology for a Market Research Firm– Over 15 years corporate research experience
with Fortune 500 companies
• Previous analytic work for MI on (SG)2
© 2004 Missions International
OBJECTIVES
• Discover elements (DNA) that drive conversion and planting growth– Measure conversions and planting growth– Measure comprehensive set of potential
“drivers” (DNA)– Build predictive models to uncover which
elements from comprehensive set are most “important” in driving conversions/planting
– Substantive CPM interpretations across predictive models
© 2004 Missions International
Assessment Design
• Qualitative content inputs from key expert leaders in Church Planting Movement - Watson, Sergeant, Sanchez, etc.
• What are “DNA” elements of church planting movement?
• Developed initial battery of items to capture that domain
• Subjected to review by experts• Iterated to list of 31 items designed to capture
domain broadly• All elements measured on five-point frequency-
of-occurrence scale: “Very Often” to “Almost Never”
© 2004 Missions International
Other Variables Measured
• Also included several descriptive / demographic measures– age, gender, education, size of community,
leader/member, formal training, how long ago church started, how many members attend, etc.
• Also included “Dependent” Variables– Self-report on frequency of starting new churches
• On same “Very Often” to “Almost Never” scale– Raw numeric dependent variables:
• How many converted through your church?• How many converted through your church have
been baptized?• How many churches branched off from your
church?
© 2004 Missions International
Samples• Sample sizes by contributing countries:
– (n=542)– (n=507)– (n=858)– (n=61)
• One sample showed extremely limited variability (interpreted as most elements from expert inputs fully in operation in this movement, thus “topping out” the scale – i.e., all aspects tend to be happening very often in this movement).
• Analysis conducted on other countries where more variability existed - pooling of other nations (starting n=1110)
© 2004 Missions International
More About Dependent Variables
• Self-report of Church planting activity– “Our church starts new churches…”
– “Very Often” – “Often” – “Sometimes” – “Rarely”– “Almost Never”
• Coded as quasi-interval 1-5 ratings
© 2004 Missions International
Numeric Dependent Variables
• Number of conversions, baptisms, new churches started
• All influenced by (a) length of existence– 24 conversions from 2 year old church, vs. 24
conversions from 2 month old church– 1 conversion per month vs. 12 per month
• All influenced by (b) number of members– 50 conversions from church of 10, vs. 50 conversions
from church of 25– 5 conversion per member vs. 2 per member
• Must control for length of existence and number of members to make raw numbers sensible and comparable
© 2004 Missions International
“Per Member Per Month”- pmpm
• Implemented per member per month approach• 100 conversions from 10 member church in existence for
1 month =10 conversions per member per month
• 200 conversions from 20 members in existence 5 months =
2 conversion per member per month• Raw numbers would have made 200 look “better” than
100, when clearly in this scenario 200 was less productive relative conversion growth than 100
• Thus created metric – per member per month (pmpm) – to take care of this issue
• Approach taken with number of conversions, number of baptisms, number of churches planted
© 2004 Missions International
Further Treatment of Dependent Variables - indexing
• Baptism often associated with conversion, thus reported numbers of baptisms highly correlated with numbers of conversions
• Created a single index of conversions and baptisms
• This index is a simple average of the two quantities, expressed in pmpm units
© 2004 Missions International
Further Treatment of Numeric Dependent Variables- normalization
• Distributions of variables in original and pmpm forms “positively” / “right” skewed
• Multivariate procedure - assumptions call for transformations to normality
• Cube root transformation chosen
© 2004 Missions International
Further Treatment of Dependent Variables - outliers
• Examined distributions on numeric pmpm variables in terms of “standardized” / “z-scores”
• This procedure helps identify “outliers” – cases that are quite different from most other cases in the sample
• A typical cut-off would be z-scores greater than 3.0 in absolute value (+/- 3.0)
• After inspection, allowing retention of more original sample, the cut-off for this study was set at +/- 3.5
• Typically less than 1% of cases eliminated
© 2004 Missions International
Final Variables For Predictive Models after all Treatments
• Dependent Variables– Index of conversions and baptisms, pmpm
(cube-root transformed)– Number of churches planted, pmpm (cube-
root transformed)– Self-reported frequency of church starts
• Independent Variables– 31 items capturing domain of “DNA of CPM”
© 2004 Missions International
Model Building - regression
• Many available approaches to predictive model building
• Typical statistical tool is multiple regression• Predict a dependent variable from a set of
independent variables• Find a weight for each independent variable
such that the weighted combination of independent variable values produces an output that is “as close as possible” to the actual/observed dependent variable values
© 2004 Missions International
Model Building - Regression
• Weights combine independent variables to produce an output – a predicted dv value
• (w1*iv1 + w2*iv2 + w3*iv3…) = output / predicted dv
• Output/predicted dv as close as possible to observed dependent variable scores
• Technically, squared errors between predicted dv and observed dv are minimized, hence the name “least squares” regression
• When standardized, regression weights can be interpreted as “relative impact” – how much unique impact each iv has on the dv
© 2004 Missions International
Model Building Procedure
• There are different approaches to regression depending on the research objectives
• Many options for variable selection in building a predictive multiple regression equation– Backward entry– Forward entry– Stepwise entry– Simultaneous/ forced entry
• A hybrid “procedure” was developed here to arrive at the final models
© 2004 Missions International
Hybrid Model Building Procedure
• Not logical to accept a variable as a significant driver if it shows no meaningful bivariate correlation
• Thus, screened independent variable set to include only starting variables with meaningful correlations
• Organized meaningfully correlated independent variables into “tiers”, e.g.,– .4s and above– .3s– .2s– .1s
© 2004 Missions International
Hybrid Model Building Procedure
• Used forward entry within each tier, starting with strongest correlation tier first
• Forward entry added variables one at a time based on biggest improvement in R-square
• Forward entry thus identified subset of significant predictors in strongest correlational tier
• That subset became starting basis for consideration of next tier
© 2004 Missions International
Hybrid Model Building Procedure• Forward entry within next tier added additional significant
predictors• Significant predictors from first two tiers became starting
basis, in moving to the next correlational tier• Tested that next tier for any additional variables adding
predictive power• Procedure continued until all tiers were exhausted• (note – not all models had all four tiers of correlational
strengths. If only .3s, .2s, .1s existed, general procedure was still same, progressing from highest strength tier to lowest strength tier)
• After no additional variables added significant predictive power, tried to re-enter any previously eliminated variables
• When no additional variables add anything, final predictive set has been discovered
© 2004 Missions International
Hybrid Model Building Procedure• Traditional regression diagnostics checked at each step
along the way (e.g., outliers, collinearity, normality of errors of prediction, quality of prediction as evidenced in adjusted R-squared, checks for sign reversal/ suppressor variables)
• Note 1 – for treatment of “missing” data, large total sample size allowed use of “listwise” procedure – modeling used only those individual respondents with complete data on all variables in any given model
• Note 2 – a few negative regression coefficients were accepted as valid. This was an acceptable finding only when:– Original bivariate correlation was negative, thus indicating not a
sign-reversal / suppressor problem– Interpretation of negative association was plausible and logical
given understanding of domain of study
© 2004 Missions International
Overall Model Summaries
Dependent Variable R R SquareAdjusted R Square F
df Reg
df Res signif
Self-Report Planting 0.614 0.377 0.372 70.877 9 1052 0.000Conversion/Baptism (cube root) 0.625 0.391 0.385 66.130 9 928 0.000Actual Planting (cube root) 0.467 0.218 0.213 45.064 6 971 0.000
© 2004 Missions International
Self-Report Planting Drivers
= negative association
Item Beta t signifOur church prays passionately at our meetings… 0.21 6.27 0.000A new convert starting a new church immediately after conversion is something that happens… 0.19 6.89 0.000When I share the Gospel with others I try to invite their relatives and friends… 0.18 6.03 0.000In our church we see God perform miracles… 0.09 3.07 0.002
Our church looks for places that have not heard the Gospel, then we do our best to win converts and plant churches there. We intentionally do that… 0.09 2.72 0.007I pray passionately… 0.08 2.39 0.017In our church we assist new converts when they start new churches… 0.07 2.15 0.032When someone from our church starts a new church, we watch over them just long enough for them to get established. Then we leave them to function on their own as soon as possible. That is the way we do it… 0.06 2.02 0.044
We intentionally spread the Gospel to other people groups outside of our community. We do that … -0.24 -7.25 0.000
© 2004 Missions International
Conversion/ Baptism Drivers
= negative association
Item Beta t signifEncouraging new believers to start new churches immediately, is something we do … 0.14 4.10 0.000Think about members of your church who are not your relatives. How often is it the case that they were already your friends before you both became believers… 0.12 4.09 0.000In our church we model for new converts how to start new churches… 0.10 3.40 0.001A new convert starting a new church immediately after conversion is something that happens… 0.09 3.27 0.001
We intentionally spread the Gospel to other people groups outside of our community. We do that … 0.09 2.46 0.014In our church, based on what we are learning from God's word and each other, we make changes in the way we act … 0.07 2.16 0.031It is more important to learn from the Word of God than from any human teacher. We emphasize that principle in our church… 0.06 2.03 0.042Our church receives financial support from outside sources … -0.21 -6.04 0.000People of all backgrounds, classes, occupations, ages, and genders mix together in the churches of our community. That happens… -0.34 -9.27 0.000
© 2004 Missions International
Numeric Planting Drivers
= negative association
Item Beta t signif
When I share the Gospel with others I try to invite their relatives and friends… 0.19 5.45 0.000A new convert starting a new church immediately after conversion is something that happens… 0.18 5.89 0.000From the time I became a Christian until now, I have been sharing the Gospel with unbelievers… 0.12 3.38 0.001My church holds me accountable to put into practice what we are learning. They do that… 0.08 2.61 0.009
We intentionally spread the Gospel to other people groups outside of our community. We do that … -0.15 -4.28 0.000People of all backgrounds, classes, occupations, ages, and genders mix together in the churches of our community. That happens… -0.15 -5.00 0.000
© 2004 Missions International
Interpretative Note • Substantive interpretation is a synthesis across
predictive models, also made in light of original qualitative research with experts
• This synthesis draws out key areas of focus found to be predictive of Church Planting success
• If generalizability holds beyond the measured samples, based on the data collected for this study, deeper implementation of positively related DNA elements (and avoidance of negatively related ones) should lead to more conversion growth and baptisms, and increased levels of church planting activity