EXPERT WORKSHOP “METHODS AND INSTRUMENTS FOR THE EVALUATION AND MENTORING OF VET SYSTEMS”

The “Learning Transfer System Inventory” (LTSI) as a diagnostic tool of training effectiveness in Spain

Pilar Pineda

Carla Quesada

Victoria Moreno

Autonomous University of Barcelona, Spain

1. INTRODUCTION (1)

In 2005 the average investment in training was 2,99% of payroll (Cranfield Study Executive Report. ESADE, 2005).

Only 10% to 30% of learning is applied in the workplace (Holton, Bates & Ruona, 2000).

Evaluation of Training Transfer : analysis of the extent to which the knowledge acquired through training is applied to the workplace, and generates improvements in the workers’ performance

3

1. INTRODUCTION (2): REASONS TO EVALUATE TRANSFER FACTORS

4

1. INTRODUCTION (3): RESEARCH AIMS

Targeted problem: to diagnose the efficacy of continuing training in Spain in terms of the training outcomes and factors that may cause a barrier, or can be a facilitator for training efficacy.

Research aims: To validate the LTSI questionnaire in the Spanish context To diagnose the transfer of training system To evaluate the transfer factors To predict transfer training level through an alternative

measurement of transfer

5

2. THEORETICAL FRAMEWORK (1): The LTSI bases

5

Noe (1988): Motivational influences of training effectiveness

Discuss the influences of trainees’ motivation on training effectiveness.

It is the first model that includes attitudinal and environmental variables involved in the workplace.

The ‘Learning Transfer System Inventory’ (LTSI) is based on:

Roullier & Goldstein (1988): Relationship between training learning, organizational transfer system and the action to the workplace context

Includes the transfer system of the organization as a factor which determines the transfer process.

LTSI

2. THEORETICAL FRAMEWORK (2)

6

Is the only existing questionnaire that is globally validated for measuring transfer factors.

Is based on the idea that performance improvement is affected by 16 factors classified into three types: ability, motivation and work environment.

New factor has been included: ‘intent to transfer’ inspired by Theory of Planned Behavior (Ajzen 1991).

The LTSI from Holton & Bates (1998, 2000):

Factors

ABILITY

1.Content validity2. Transfer Design3. Personal Capacity4. Opportunity to Use

MOTIVATION

5. Motivation to Transfer6. Learner Readiness7. Performance Self-Efficacy8. Transfer Effort-Performance Expectations9. Performance-Outcome Expectations

WORK ENVIRONMENT

10. Supervisor Support11. Supervisor Sanctions12. Peer Support13. Personal Outcomes – Positive 14. Personal Outcomes – Negative 15. Performance Coaching16. Resistance to Change

Intent to transfer: the intent of a person shows the willingness to act.

3. METHODOLOGY

QUANTITATIVE DATA

1,042 Surveys

11 Spanish companies (5 different sectors)

103 training programs (types)

Most common trainees’ occupations: technical

staff, middle managers and unqualified personnel

Exploratory Factor Analysis

Descriptive Analysis

Multivariate Analysis (MANOVA)

Multiple Regression Analysis

Correlation Analysis

SPSS

- The SLTSI (Spanish version) used 55 items with 5-point Likert scale.

- We conducted a cross type reverse translation Spanish-English and English-Spanish

RESULTS

4. RESULTS (1): General & Specific Factors

Strong facilitators: supervisor sanctions.

Barriers: performance coaching, personal outcomes-negative, supervisor support, personal capacity.

Barrier to transfer

Weak facilitator to transfer

Strong facilitator to transfer

9

4. RESULTS (2): General & Specific Factors

Obtain results between 3 and 4 points on a 1 to 5 scale.

Performance coaching is a barrier (2.74 to 5) which means that the supervisor should provide more guidance on the worker's performance following the training carried out.

General Factors

Specific Factors

10

4. RESULTS (3): Factors by type of Training Program

4. RESULTS (4): Factors by type of Training Program

Training programs on safety and workplace hazards present a situation not conducive for the transfer. The barriers are: high resistance to change, high supervisor sanctions, and low motivation to transfer.

Training programs on technical skills are well planned and meet the needs of the workers. The facilitators are: high transfer design and content validity.

Training programs on computer based applications indicate that the chances of a real transfer are high. The facilitators are: less resistance to change, more learner readiness and motivation to transfer.

Training programs related to personal and relational skills must improve ‘performance coaching’ which is too low.

12

4. RESULTS (6): Factors by type of Position of Trainee

13

4. RESULTS (5): Factors by type of Position of Trainee

14

4. RESULTS (7): Intent to transfer

The last version of LTSI includes a new construct ‘intent to transfer’. This construct is inspired by the Theory of Planned Behavior (Ajzen, 1991) that indicates that the intent of a person shows their willingness to act.

Last researches have included the new construct to predict training transfer, and they confirm that the intent to transfer, measured at the end of training, is a good predictor of the actual transfer of learning in the workplace. (Ajzen & Driver 1992; Combs & Luthans 2007; Griffeth, Hom & Gaertner 2000; Kirschenbaum & Weisberg 2002; Bogdan 2009).

4. RESULTS (8): Intent to transfer

‘Intent to transfer’ is positive in general (3.85 to 5).

The 16 factors model explains the 55% of ‘intent to transfer’. It has predictive validity.

Predictors

Correlation between each

predictor ant the intent to transfer

Correlation between each predictor and

the intent to transfer controlling the rest

of predictorContent Validity ,405** ,006Transfer Design ,517** ,166**Personal Capacity ,290** ,090**to Use ,522** ,201**Motivation to Transfer ,516** ,136**Learner Readiness ,272** -,062*Performance Self-Efficacy ,464** ,168**Transfer Effort-Performance Expectations ,589** ,254**Performance-Outcome Expectations ,297** -,117**Supervisor Support ,296** ,060Supervisor Sanctions -,164** -,051Peer Support ,484** ,140**Personal outcomes – Positive ,426** ,060Personal outcomes – Negative ,004 ,029Performance Coaching ,168** -,012Resistance to Change -,227** -,040

It’s necessary to consider the SLTSI factors globally in order to count on all its predictive potential.

16

4. RESULTS (9): Prediction of Transfer

The research has another aim, to explore the predictive ability of the intent to transfer and the LTSI in the Spanish context, seeking an alternative measure of the transfer to facilitate and simplify the evaluation of training outcomes.

This stage includes a second questionnaire, three months after training, to measure the trainee transfer with 8 items. These items measure the 'transfer to the job' and the 'performance'.

The results are from the Catalan context, on training for Public Administration workers

17

4. RESULTS (10): The LTSI in Catalan Public Administration

18

4. RESULTS (12): The LTSI in Catalan Public Administration

19

4. RESULTS (11): Transfer System in private and public sector

A comparative between transfer systems:

Spanish Validation (SLTSI) Catalan Validation (CLTSI)

Facilitators Supervisor sanctions

Supervisor Sanctions Motivation to transfer Transfer effort - Performance

Barriers

Performance coaching Personal outcomes – Negative Supervisor support Personal capacity

Performance coaching Personal outcomes – Negative Supervisor support Personal capacity Performance – Outcome

expectations

Intent to transfer 3.85 point to 5 4.04 points to 5

5. CONCLUSIONS (1)

Validation of the LTSI questionnaire in the Spanish context. The result is the Spanish Learning Transfer System Inventory or SLTSI.

Diagnosis of the current factors situation that determine the transfer of training in the workplace.

Evaluation of the transfer factors in companies from different sectors, which helps us to understand the differences in transfer by type of training program and trainee’s occupation.

Prediction of transfer training using an alternative measurement of transfer.

Aims achieved:

21

5. CONCLUSIONS (2)

Main conclusions:

5. CONCLUSIONS (3)

There are three factors which obtained high scores acting almost as facilitators of the transfer: performance self-efficacy (3.91 to 5), transfer effort – performance expectations (3.91 to 5) and transfer design (3.90 to 5).

We can use ‘training programs on management & administration and training programs addresses to commercial salesmen’ as examples to improve supervisor support throughout training.

The predictive capacity of SLTSI and the intent to transfer in the Spanish context is limited.

23

5. CONCLUSIONS (4)

Future research goal:

24

6. OUR RESEARCH: RESEARCH TOPICS

To analyze the continuous training policies in Catalonia, Spain & Europe.

To advice companies on how to improve training management in their organization

To explore, create and apply models and techniques for evaluating continuous training effectiveness.

To design tools and materials to develop training programs highly effective.

To identify alternative systems to evaluate the training results, operational and viable.

25

6. OUR RESEARCH: LAST RESEARCHES

Evaluation of Continuing Training for Preschool Teachers in Spain (Fundación Tripartita para la Formación en el Empleo, 2005-2006)

Evaluation of the Training Plan for Reflective Practice in Mathematics Teaching (Departament d’Educació de Catalunya, 2005-2008)

Evaluation of Continuing Training for Teachers in Navarre (Departamento de Educación de la Comunidad Foral de Navarra, 2006-2008)

Assessment of Continuing Training for public managers and leaders. (Escola d’Administració Pública de Catalunya, 2007-2008).

Evaluation of a Training Program on the Ration Use of Medicines. (Unió Consorci de Formació d’Hospitals de Catalunya, 2007-2008).

Diagnosis of Transfer of Training in Spain with the application of LTSI (EFI-GAPEF, UAB 2009)

Effectiveness of Training in Catalan Public Administration (Escola d’Administració Pública de Catalunya, 2008-2009).

Thanks for your attention

pilar.pineda@uab.catcarla.quesada@gmail.org