Training, Evaluation and Developmental Change in

Learning Systems

Katharine Briar-Lawson, PhDUniversity at Albany

Journey of Enormous Impacts

• Partnerships began in 1985,over two decades of innovations

• Evolved into robust, increasingly data driven training, education initiatives

• Learning communities formed within/ across states, universities

• Survived threats to cuts in IV-E, 426 and related funding streams

Stocktaking: New Opportunities

• Ten years of collaboration and research exchanges

• Clearinghouse functions

• Many fugitive studies (Barbee,2003)

• New Public Child Welfare Journal

• National Partnership Center: research repository for cross state learning

Challenges

• Over $275 million in IV-E training and $80 million in IV-E education dollars spent each year

• Training transfer generally about 10-15% • Turnover high in many states and counties• Aggregate impact of our work not

synthesized on annual basis, nor collective education or training efforts across nation studied

Training Versus Learning Systems

• Lectures, traditional

training models

• Training transfer problem

• Curriculum is trainer “proof”

• Participatory, collaborative action research and learning

• Collaborative team members jointly construct learning

• Team members adapt to context

Work Units as Learning Systems

• Work units as incubators of innovative practice models and service improvements

• Front line units as design teams with inventive, self correcting, data driven learning systems with consensus based helping practices

• Supervisors as capacity builders and case consultants rather than solely administrative managers

Design Teams

• Utah, Nevada, New Mexico, Colorado

• New York (high turnover counties and borough)

• Front line staff create logic models for retention strategies addressing root causes

• Design teams as learning and action systems

Design Teams and Change

• Design teams effective in retention with team effects saturating entire district office(Strollin, 2006)

• Training is embedded and localized, co-production of the training so ownership

• Design team concept congruent with teaming in CPS so new models of practice being tested

Next Round of Challenges

• Consumer perspectives, cultural competence

• Trauma informed training and evaluation• Value, attitude assessment• Competency vs. quality of treatment

issues• Mixed workforce issues• Every training and evaluation act- linked to

re-professionalization, PIP, EBP

Research challenges

• Randomization

• Quasi-experimental designs

• Evidence based knowledge building, dissemination

• Cost of failure, ROI analyses

Evidence Based Movements

• Disconnect between training evaluators and evidence based initiatives in SSW

• Expert knowledge among training evaluators on organizational barriers to “evidence based” practices

• Connect SSW dialogue and capacity building on evidence based practice and field placements to training evaluators

Traditional vs. Developmental Research (Springer & Phillips)

Traditional model:• Intervention controlled,

stable• Hypotheses do not

change• Interventions imposed• Outcome measures

standardized," given”• Single research design

matched to need

Developmental model:• Intervention evolves,

changes• Hypotheses change,

iterative research • Stakeholders are co-

designers• Developed in context,

refined iteratively• Multiplism in methods

Model comparison and CQI fit:

Traditional model:• Experimental and

quasi-experimental logic

• RCT as gold standard

Developmental model:• Develop and/or seize

on naturally occurring quasi-experimental conditions

• Theory driven evaluation models (Weiss)

CQI

• Developmental model and CQI: turning each barrier into the next goal to be addressed with successive new interventions, including training, data

• Journey charting for lessons learned• Training (however defined) linked to PIP,

workforce development, practice improvements, client and system outcomes