Title: Measuring Transfer of Training-Employing the Levels of Use Inventory

Name of author(s)/Organisation affiliation/position(s):

Marijke Thamm Kehrhahn, Associate Professor, University of Connecticut, USA

Alexandra A. Bell, Associate Professor, University of Connecticut, USA

Address: Department of Educational Leadership

249 Glenbrook Rd Unit 3093

University of Connecticut

Storrs, CT 06269 USA

Corresponding Author Email address: Marijke.Kehrhahn@uconn.edu

Stream: Assessment, measurement, and evaluation of HRD

Submission type: Working paper

1

Abstract

Purpose: HRD practitioners need valid and reliable methods to assess learners’ transfer of

training, while researchers need transfer measures to generate accurate models of the transfer

process. This paper critiques current transfer research, and describes the Levels of Use (LoU)

Inventory as a viable measure of transfer for researchers and practitioners.

Method: By examining analytical reviews of transfer research conducted between 1992-2008

we identified transfer measurement issues and recommendations. We critiqued the ways and

extent to which research published since 2008 has addressed the recommendations.

Findings: We identified four transfer measurement issues. Current researchers have not

addressed these issues adequately. Contrary to recommendations, many researchers continue

to conceptualize transfer as an outcome of training and gather transfer data at one point in

time, from one source, in one way. One measure, the LoU Inventory, shows promise in

conceptualizing and assessing transfer in ways consistent with recommendations.

Implications for Research: Research is needed to establish validity and reliability of

different LoU Inventory formats, and to assess how the Inventory can promote learner

metacognitive knowledge and self-regulation in transfer.

Implications for Practice: The Inventory can provide HRD professionals, supervisors,

learners, and peers with rich information about the nuances of transfer over time at both

individual and group levels.

Significance: HRD practitioners have very few options for transfer measures that enable

gathering data from multiple stakeholders, provide reliable data on transfer efforts, utilize

employee time economically, and effectively inform transfer support efforts. The LoU

Inventory has the potential to fill this gap.

Keywords: Transfer of Training, Assessment

2

Measuring Transfer of Training: Employing the Levels of Use Inventory

Transfer of training, the application of newly acquired knowledge and skills to the job

in ways that enhance work performance, remains a central measure for evaluating the

effectiveness of HRD. Because HRD practitioners focus the vast majority of workplace

training and development activities on improving employee and organizational performance,

they need valid and reliable methods to assess learning transfer. To support practitioner

efforts, researchers need meaningful measures of transfer that generate accurate models of the

transfer process and its relationship to work performance. In this paper we identify and

critique transfer of training measures currently used by researchers and describe in detail the

Levels of Use Inventory as a viable measure of transfer for use by both researchers and

practitioners.

Problem Statement

In several current reviews of transfer of training research (e.g., Blume et al, 2010;

Burke & Hutchins, 2007; Salas et al, 2012), researchers have identified the need for further

development of transfer measures. Blume et al (2010), in their meta-analysis of transfer

research, noted that researchers continue to operationalize and measure transfer in a variety of

ways. Citing literature reviews by Baldwin and Ford (1988) and Ford and Weissbein (1997),

the authors acknowledged improvements in transfer research design, but also noted continued

need to refine transfer measures. To add to the challenge of assessing transfer, some studies

of learning transfer systems focused on relationships among variables purported to influence

transfer, but did not include measures of transfer itself (Burke & Hutchins, 2007).

Although researchers and practitioners recognize transfer as a multidimensional and

dynamic process, many continue to assess transfer as a one-time dichotomous (transfer/no

transfer) event. In conclusion to their reviews of transfer research, Salas et al (2012)

3

suggested multiple measures of training-related changes in knowledge and performance to

capture the influences of after-training variables on transfer, while Burke and Hutchins

(2007) recommended a shift toward capturing a variety of transfer indicators. Watkins, Lysø

and deMarrais (2011) noted the challenge of capturing transfer when training is focused on

general development of open-ended skills, such as leadership development, and

recommended a critical incident interview approach to provide more detailed data on how

participants’ post-training behavior is influenced by participation in training. Overall, HRD

researchers agree that much can be done to improve measurement of training transfer.

In addition to these issues, other weaknesses in transfer measurement exist. Most

notably, transfer measures may provide a snapshot of the extent to which employees are

using a new skill, but rarely provide information on the practices embedded in the transfer

process or the ways in which employees engage in the process of integrating new knowledge

and practices into their work. Overall, HRD practitioners have very few options for transfer

measures that enable gathering data from multiple stakeholders in the transfer process,

provide reliable data on transfer efforts, utilize employee time economically, and effectively

inform transfer support efforts.

Measurements of Transfer of Training: Current Literature

A number of analytical reviews of transfer research were published between 2007 and

2011 (Aguinis & Kraiger, 2009; Blume et al, 2010; Burke & Hutchins, 2007; Gegenfurtner,

2011; Grossman & Salas, 2011), and Educational Psychologist and Educational Research

Review published special issues on transfer of training in 2012 and 2013. Authors in these

reviews and special issues analyzed transfer research (1992-2008), provided critiques of

transfer measures (Blume, et al, 2010; Gegenfurtner, 2011), and made recommendations for

transfer research going forward (e.g., Grossman & Salas, 2011; deGrip & Sauermann, 2012;

Volet, 2013). A number of reviewers stated that future research requires a more explicit

4

discussion and focus on transfer measures, while others recommended a change in direction

to focus research more on illuminating the transfer process.

Integrative Critique of Transfer Measures

Through a review of the literature, we surfaced four transfer measurement issues.

First, generally, researchers have measured transfer in terms of newly acquired knowledge,

skills, and attitudes (KSAs), frequency of use, or the perceived effectiveness of using new

KSAs (Blume et al, 2010; Gegenfurtner, 2011); however, they often described transfer as

some variation of a “transfer/no transfer” or “high transfer, low transfer” dichotomy. These

measures and categorizations offer no insight into the actual process of transfer.

Second, in the vast majority of studies researchers measure transfer once, following

completion of the training—a method inconsistent with the understanding that sustaining the

use of a new skill over time is a critical transfer condition. Blume et al (2010) found only 6 of

93 studies reviewed in which a transfer measure was taken more than once. The single

measure can capture a transfer “snapshot,” but cannot account for transfer initiation,

persistence, or maintenance that may occur outside the timeframe of the single point of

measure.

Third, transfer research is predominantly focused on identifying various systems

variables associated with transfer. Individuals in these systems-focused studies are depicted

as elements in a system that can be influenced by manipulating other elements in the system

to elicit specific transfer outcomes, with little attention to individual self-determination or

agency (Lobato, 2013). We found few studies that explicate the ways in which individuals

participate in the cognitive, behavioral, and metacognitive activities used to adapt learning to

action in the workplace.

Lastly, researchers frequently measure transfer as an outcome variable to measure

training effectiveness; we found few studies that used measures to illuminate the process of

5

transfer. This approach to transfer measurement leaves scholars with information about

whether or not individuals transferred the training, but with little insight into how transfer

occurred. As Baartmann and deBruijn (2013) suggested, “the learning processes toward

integration of KSAs largely remain a black box” (p.126). In summary, the majority of

researchers continue to measure transfer of training as a one-point-in-time outcome measure

of training effectiveness that provides little insight into the individual transfer process.

Trends in 2008-2015 Transfer Studies

Because the analytical reviews discussed above examined published research from

1992 to 2008, we reviewed studies published between 2009 and 2015, and examined

specifically the degree to which they replicated prior transfer measurement approaches or

implemented recommendations for advancing transfer measurement provided in the

analytical reviews.

We located and reviewed 20 studies of transfer of training conducted in actual

workplaces and published in English between 2009 and 2015. (See bold font entries in

References list.) The studies represent the work of researchers internationally. This body of

research reflects many of the same conceptual and methodological approaches used by

researchers prior to 2009. Progress in implementing recommendations for future research

offered by authors in analytical reviews has been slow. For example, among the 20 transfer

studies published since 2009, only 8 studies gathered transfer data from more than one source

and 2 studies used more than one measure of transfer—consistent with recommendations.

Although the frequency of use of newly acquired skills continues to be the predominant unit

of transfer measurement (8 studies), five studies reported data on the effectiveness of using

the new skills, and six studies used both types of measures.

Researchers have made modest progress in the area of extending the time frame for

transfer assessment, recognizing that transfer involves maintenance as well as initiation.

6

Among research conducted over the past 7 years, five studies focused on initiation measures

of transfer, assessing transfer immediately following the training or within the first 4 weeks,

while the large majority of studies (17) measured transfer after some time had passed (1

month to 1 year). Because so little is known about the transfer of training process, the point at

which initiation becomes maintenance is unclear.

Unfortunately, current researchers have not implemented many of the

recommendations for transfer research offered in the comprehensive analytical reviews.

Overwhelmingly, researchers continue to gather transfer data at only one point in time (18

studies). In the two studies where transfer data were collected at multiple points, Lau and

McLean (2013) used the same survey at 1 month, 6 months, and 1 year following training in

Malaysia, and Canadian researchers Taylor et al (2009) used a case study approach to gather

transfer data from multiple sources over several months. Researchers continue to

conceptualize transfer as a measure of training effectiveness (10 studies), and to use transfer

data to create a systems view of variables associated with transfer (10 studies).

We located three studies published between 2009 and 2015 that utilized transfer

measures designed in response to ongoing efforts to improve transfer research. A study of the

effectiveness of diversity training for university research assistants in the U.S. by Roberson et

al (2009) required participants to develop transfer plans and gathered data 4 weeks after

training completion to determine the extent to which participants were using the transfer

strategies they designed. Although the results do not provide details about participants’

experiences of implementing transfer, the conceptual framework highlights the importance of

planning, monitoring, and evaluating the transfer process, in addition to the application of

newly acquired KSAs.

Watkins et al (2011) used a more dynamic and developmental approach to training

evaluation through the use of critical incident interviews with participants, peers,

7

subordinates, and supervisors to identify individual and organizational change associated

with participation in leadership development programs in the U.S and Norway. The resulting

case studies provided dynamic illustrations of ways participants applied and adapted

leadership concepts to their practice over time, and insights into how participants engaged

with others to translate what they had learned into appropriate action.

Lastly, Taylor et al (2009) conducted interviews and focus groups with program

participants, instructors, and workplace supervisors and generated field notes to develop

multi-site case studies to uncover characteristics of the transfer process of low-literacy adults

participating in an employment preparation program in Canada that included classroom

instruction, on-the-job internships, and employment. The researchers concluded that transfer

of learning efforts and success were linked to individual perceptions of the extent to which

skills learned could be useful across multiple life roles and the degree to which skills learned

were essential to work and life success. Participant efforts to transfer were linked also to

program instructors’ understanding that learning would happen over time and that

participants’ development of self-regulated learning strategies were essential for successful

transfer. Overall, Taylor et al provided an in-depth view of a learning transfer system over

time, with an emphasis on the experiences of the learners.

Recommendations for Future Transfer Research

Scholars currently engaged with analyzing transfer research make several

recommendations for improving transfer research. Grossman and Salas (2011) called for

more in-depth research that would provide the next layer of understanding of the transfer

phenomenon, while Blume et al (2010) identified the need for a focus on how different forms

and types of transfer measurement contribute to overall understanding of transfer. Burke and

Hutchins (2007) suggested that future research should “assess training transfer as a

multidimensional phenomenon with multilevel influences” (p.287), taking into account the

8

role of individual meta-cognition and self-regulation. Volet (2013) provided a number of

strategies for improving transfer measurement including determining what KSAs transfer,

how, when, and under what conditions, and examining person-environment dynamics in

transfer scenarios. Several researchers (e.g., Blume et al, 2010; Gegenfurtner, 2011; Volet,

2013) recommended using multiple data collection strategies and sources to triangulate

research findings. The challenge appears to be designing measures to capture transfer efforts

and outcomes over time without fatiguing participants while supporting strong response rates,

particularly in actual workplaces (Burke & Hutchins, 2007; deGrip & Sauermann, 2012;

Volet 2013). Optimally, measures of transfer provide information that can inform those

accountable for transfer—learners, managers, and HRD practitioners—about the design and

effectiveness of transfer interventions and supports (Aguinis & Kraiger, 2009; Grossman &

Salas, 2011) and inform learners themselves about their transfer processes and outcomes.

Editors of recent special issues focused on transfer of training suggested that

researchers consider new perspectives and models for understanding of transfer; one oft-

repeated recommendation was to examine the transfer process and the individual’s

engagement in transfer in more depth. Current transfer research fails to illuminate what

actually happens in the transfer process that results in improved performance; survey studies

provide generalized inputs/outputs data and performance outcomes measures can be used as

indicators of training effectiveness, but neither give a glimpse into the “black box” (deGrip &

Sauermann, 2012, p.29).

Recent work of Billett (2013), Perkins and Salomon (2012), and others highlight the

importance of building an evidence-based understanding of cognitive, meta-cognitive, and

socio-cognitive engagement in the transfer process, aside from motivational, supervisory,

peer, training, and environmental influences. Researchers studying transfer in work settings

conclude that self-regulation and metacognitive knowledge are essential elements in

9

successful transfer, particularly in the absence of favorable transfer environments (e.g., Enos,

Kehrhahn & Bell, 2003).

Based on our extensive review of the literature, we propose that transfer of training be

more broadly researched; not only as the successful application of newly acquired knowledge

and skills, but also as the process through which employees plan, initiate, implement, and

adapt the knowledge and skills to their work. The following section of the paper provides

detailed information on a valid method to measure both.

Levels of Use Inventory

The Levels of Use (LoU) framework (Hall & Loucks, 1977; Hall & Hord, 2011) is

part of a larger learning and change model called the Concerns-Based Adoption Model

(CBAM). The CBAM model was initially developed to assist school leaders in supporting

educators’ use of innovative instructional methods following their participation in a

professional development program. Based on the premise that training does not automatically

lead to high-fidelity implementation of newly acquired knowledge and skills, the CBAM

model includes three essential assumptions. First, initiation and integration of new practices

into a pre-existing complex set of work behaviors is a process and not an event; movement in

the process can be captured as Levels of Use (LoU) of the new practices. Second, progress in

the transfer process depends on addressing employee concerns about the impact of transfer

efforts on their personal work life, concerns about how to use the skills, and concerns about

impact on organizational outcomes. Hall, George, and Rutherford (1977) called this part of

the model, Stages of Concern. And third, newly acquired knowledge and skills are adapted

and configured to best fit the local context, therefore transferred skills in practice may look

very different from one another and very different from what training program developers

intended. In their initial research (n = 800), Hall and Loucks (1977) found that no two

10

individuals were using the same form of the innovation, nor did they agree on operational

definitions. In this review, we focus on the Levels of Use element of the CBAM model.

The Levels of Use framework offers a view of transfer as a process, not an event. Hall

and Loucks (1978) described the transfer process as cumulative, uneven, gradual, and

complex and warned that single measures of transfer can miss the phenomenon altogether,

leading to under-estimation of training effectiveness. The LoU framework presents

implementation of new knowledge and skills as a result of a series of individual decisions

that help move the employee-learner from early stages of planning to transfer, through

mechanical integration of new skills into a pre-existing work repertoire, to routine

implementation, adaptation, and refinement. Specifically, the LoU Inventory (Hall & Hord,

2011) provides a set of behavioral profiles that distinguish different levels of transfer,

including three non-transfer profiles and five transfer profiles (see Table).

Table

Levels of Use Inventory

Categories of Levels of Use Descriptions of Levels of Use Categories

Non-Transfer 0

Non-use

The learner has little or no knowledge of the

innovation*, no involvement with the innovation, and is

doing nothing to become involved.

Decision Point Decides to take action to learn more about the

innovation.

I

Orientation

The learner has acquired or is acquiring information

about the innovation and/or has explored or is exploring

its value orientation and its demands upon learner and

learner system.

Decision Point Decides to use the innovation by establishing a time to

begin.

II The learner is preparing for first use of the innovation.

11

Preparation

Decision Point Decides to go ahead with implementation with

perception that personal needs/concerns have been/will

be addressed.

Transfer

III

Mechanical Use

The learner focuses most effort on the short-term, day-

to-day use of the innovation with little time for

reflection. Changes in use are made more to meet

learner needs than client needs. The learner is primarily

engaged in a stepwise attempt to master the tasks

required to use the innovation, often resulting in

disjointed and superficial use.

Decision Point Decides that innovation should become part of routine

work practices.

IV A

Routine

Use

Use of the innovation is stabilized. Few if any changes

are being made in ongoing use. Little preparation or

thought is being given to improving innovation use or its

consequences.

Decision Point Decides to modify the innovation to achieve better client

outcomes.

IV B

Refinement

The learner varies the use of the innovation to increase

the impact on clients within immediate sphere of

influence. Variations are based on knowledge of both

short- and long-term consequences for clients.

Decision Point Decides to modify innovation based on input of and

coordination with colleagues.

V

Integration

The learner is combining own efforts to use the

innovation with related activities of colleagues to

achieve a collective impact on clients within their

common sphere of influence.

Decision Point Decides to explore alternatives or major modifications

of the innovation to substantially elevate outcomes.

VI The learner reevaluates the quality of use of the

innovation, seeks major modifications or alterations to

12

Renewal present innovation to achieve increased impact on

clients, examines new developments in the field, and

explores new goals for self and the system.

Note: Adapted from G. E. Hall and S. F. Loucks (1977). A developmental model for determining whether the treatment is actually implemented. American Education Research Journal, 14 (3), 263-276.

*Hall and Loucks (1978) defined innovation as a practice that is perceived as new to the

individual and that is most often learned about through participation in formal training.

As shown in the table, transition from one Level of Use to the next depends on the

learner making a decision to move forward with transfer. For example, an employee at Level

0 (Non-Use) makes a decision to learn more about the new skills, perhaps by registering for

training or by discussing with colleagues, moving herself to Level I (Orientation). Likewise,

an employee at Level III (Mechanical Use) decides to persist with transfer efforts beyond

initiation, making a commitment to permanently change his practice, and moves to Level

IVA (Routine Use). According to Hall and Hord (2011), while the decision making process is

individual, HRD practitioners and supervisors, supplied with knowledge of current Level of

Use and Stages of Concern, can help employees move forward with the transfer process by

addressing concerns, encouraging goal setting, and facilitating decision making.

Administration of the LoU Inventory

Hall and colleagues developed the Levels of Use Inventory as a 30-minute interview

protocol with the learner conducted by a trained administrator of the tool. The administrator

codes interviewee responses using a framework that delineates behavioral elements at each

level and places the interviewee at a specific Level of Use (Hall & Hord, 2011). Inter-rater

reliability of 1381 cases was .87 to .96, based on agreement on assigned level of use by two

coders listening to recorded interviews. A validity study was conducted comparing

individuals’ (n = 45) interview scores with ethnographer/observers scores based on one full

13

day of observation (r = .98) (Hall & Loucks, 1977). Further, Hall and Loucks (1978) reported

substantial variation across the eight levels with data collected 2-3 years after introduction of

the innovation (0 = 7%, I = 9%, II = 3%, III = 19%, IVA = 52%, IVB = 6 %, V = 3%, VI =

2%), demonstrating the Inventory’s usefulness in detecting variation in transfer efforts among

learners. Other study samples were similar in their distributions, with the largest percentage

of users consistently at Level III (Mechanical Implementation) and Level IVA (Routine

Implementation). Across studies, LoU researchers found that novice professionals tend to

stay at the Mechanical level of implementation for extended periods of time and that

individuals are most likely to abandon transfer efforts at this stage (Hall & Hord, 2011)

The education community continues to maintain high interest in the CBAM model 40

years after its initial development. The CBAM principles are central elements of the U.S.

standards for educator professional development, revised in 2011 (Learning Forward, 2015).

Hall (2013), in a Legacy Paper published by the Journal of Education Administration,

highlighted the continued relevance of the LoU as a tool for HRD practitioners and

administrators supporting individual transfer efforts. He identified a gap in the research that

calls for longitudinal studies of transfer to provide a better in-depth understanding of

individual processes of change associated with learning and implementing new knowledge

and skills.

LoU as a Transfer Measure

In practice, administration of the LoU Inventory involves either a “branching

interview” or a more formalized “focused interview” (Hall & Hord, 2011) to obtain a detailed

description of an individual’s level of use of an innovation or “innovation bundle” across

seven different dimensions: Knowledge, Acquiring Information, Sharing, Assessing,

Planning, Status Reporting, and Performing. Researchers using this method frequently

include observation and review of documents to corroborate interview findings, as well as

14

methods to establish reliability and internal validity of LoU assessments. Repeating the

interview overtime among many learners in an organization affords a nuanced assessment of

changes in use of innovations at an individual and system-wide level. The majority of

researchers using this method have assessed LoU among faculty in either school settings

(e.g., Hollingshead, 2009; Kong & Shi, 2009, Tunks & Weller, 2009) or higher education

settings (e.g., Folger & Williams, 2009; Hodges, 2014)

Other researchers have conducted either branching or formalized interviews, with or

without corroborating data and validation efforts, focused on the performance dimension of

use to obtain an overall profile of an individual’s LoU. While this method has the advantage

of being less time consuming than the comprehensive method, it provides a less detailed

assessment of ways in which learners use different dimensions of an innovation and transfer

different aspects of training. This method is common also in studies conducted in school

(e.g., Saylor, 1998; Rout et al, 2010; Wang, 2014) and higher education settings (Olafson et

al, 2005).

A study by Saylor (1998), in which the LoU Inventory interview was modified to a

written open-ended format, demonstrated that learners are able to self-assess their LoU with

the same level of accuracy as expert evaluators. In her study of 68 middle school teachers

who completed training in educational technologies at the beginning of the school year,

participants were given a week to respond on their own to a written version of the branching

interview near the end of the school year. Teachers’ assessments were then evaluated by the

researcher and two expert reviewers, and corroborated by a district technology expert’s rating

of each teacher’s proficiency using technology at the end of the school year. The teachers’

self-ratings and evaluators’ classifications as users or nonusers were perfectly consistent.

Although Hall and Hord (2011) state, “it is not possible to measure LoU with

questionnaires and online surveys” (p. 287), many researchers have used quantitative

15

methods to assess LoU. These efforts reflect researchers’ appreciation for the significance of

the LoU construct, mitigated by methodological constraints, such as restricted access to

learners, large sample size, limited funding, and a desire to use statistical approaches to

explore multivariate relationships. The body of studies in which researchers have quantified

LoU is a testament to the English proverb, “Necessity is the mother of invention.” Most

researchers (e.g., Fitzgerald, 2002; LaRocco & Wilken, 2013; Myers, 2009; Weber, 2013)

quantified LoU using an 8-point ordinal scale, with one value on the scale for each of the

eight levels of use. Unfortunately, very few researchers (e.g., Roberts et al, 1997) reported

using methods to assess the reliability and internal validity of responses using these scales.

Saylor (1998) highlighted how quantifying LoU can illuminate trends and

relationships that qualitative methods cannot. Saylor used a discriminate function analysis to

predict variance in Use/Nonuse of technology among middle school teachers based on

individual and environmental support variables 5 months after completing training. Four

factors (teacher efficacy, social support, motivation to transfer, and age) explained 29% of the

variance in Use/Nonuse, and the model classified 87% of participants correctly.

Given the efficiency in assessing LoU quantitatively, researchers’ use of this approach

to assess transfer in settings outside of education it is not surprising. Fitzgerald (2002)

assessed transfer of training and transfer climate factors among 33 direct service staff at a

U.S. state mental health organization engaged in training on ethical decision-making. At 4-

months post training, the LoU change scores provided a detailed profile of significant

increases in transfer among members of the intervention group, a reflection of procedural

knowledge gains, whereas knowledge gain scores did not significantly increase. Similar

findings indicating the LoU was a more sensitive assessment of changes in transfer behaviors

than declarative knowledge scores was demonstrated by Myers (2009) in a study of 53

16

personnel in a U.S. heath care organization participating in training on managing a

harassment free workplace.

Researchers using either qualitative or quantitative assessment of LoU consistently

demonstrate that the LoU framework is sensitive in describing variability in use across

learners who participate in the same training (e.g., Folger & Williams, 2009; LaRocco &

Wilken, 2013; Olafson et al, 2005; Rout et al. 2010), and in describing changes in use over

time (e.g., Hodges, 2014; Kong & Shi, 2009; Olafson et al, 2005; Tunks & Weller, 2009). In

many studies, HRD administrators or school leaders used LoU outcomes to inform training

design and interventions for individuals or groups. However, the LoU framework shows great

promise as a resource for learners to directly plan, monitor, and self-assess their own

learning, and for development of professional learning communities. In an innovative

application of the LoU framework, Orr and Mrazek (2009) developed an online survey

whereby graduate students enrolled in an educational technology course assessed their level

of adoption for 20 different educational/instructional technologies. Learners completed the

survey three times—before the semester, semester’s end, and 4-months after semester’s end.

Individual and aggregate data were available to all learners and various visual displays

portrayed individual and group adoption patterns. Learners used the data to promote

reflection on use of technologies, personalize learning goals, plan learning, and self-assess

learning processes and outcomes. The data also became a focal point for establishing a

supportive community of learners.

Our review of studies using the LoU Inventory indicates that as a measure of transfer

it has the potential to addresses many of the recommendations for transfer research identified

in recent analytical reviews. When administered via interview and customized to a specific

innovation configuration (Hall & Hord, 2011), the LoU provides a detailed profile of

individual transfer behaviors across multiple dimensions of use, including knowledge,

17

planning, assessment, and performance. When repeated over time, it provides a nuanced

description of how individuals change behaviors. In tandem with assessment of

environmental factors, it contributes to understanding the person-environment dynamics in a

transfer scenario. The LoU also shows promise as a means to assess and support learner

metacognitive knowledge and self-regulation by providing feedback about transfer efforts

and outcomes and serving as a guide for planning learning.

Implications for Future Research with the LoU Inventory

The affordances of the LoU Inventory as an assessment of transfer make it a viable

tool for researchers engaged in efforts to enhance HRD practice through scholarly

examination in field settings. Based on our critique of studies using the LoU, research efforts

targeting the following questions will expand its utility as a viable measure of transfer

processes and outcomes over time, and contribute to evidence-based practice by HRD

professionals:

How do level of use outcomes compare across different LoU formats (e.g., branching

interview, focused interview, quantitative scale survey, and self-administered open-

ended questions)? What are the psychometric advantages and disadvantages of each

format for researchers and practitioners?

How can multiple stakeholders (e.g., learners, peers, supervisors, and HRD

evaluators) assess levels of use in survey format? How can inter-rater reliability

among multiple stakeholders be established?

How does the LoU Inventory promote learner metacognitive knowledge and self-

regulation in transfer? How can the Inventory promote professional learning among

peers?

18

What cultural and social factors need to be considered in using the LoU Inventory?

What is the relevance and utility of the Inventory as a research instrument and tool for

HRD practitioners internationally?

Implications for Practice

The Levels of Use framework provides an actionable conceptualization of transfer of

training, and the instrument provides relevant data practitioners can use to measure and

support transfer. Our work with the LoU Inventory has shown that the concepts resonate with

employees, and particularly with supervisors, managers, and HRD professionals and can be

used productively in work settings. Specifically, we recommend the following applications:

Present the LoU framework to learners as part of a training program to support

transfer planning and implementation.

Include a module on the LoU framework in supervisory/management training to build

supervisors’ understandings of and capacity for supporting transfer.

Include a module on the LoU framework in HRD professional development and

degree programs to build comprehensive understandings and skills for designing,

measuring, and supporting transfer.

Because of the time-consuming nature of data collection, we do not recommend using

the LoU Inventory interview in each and every workplace training scenario. We believe,

however, that it has value for use by HRD practitioners and managers in the following

ways:

Use the LoU Inventory interview as tool to support HRD practitioners and

managers to become more familiar with the process, variations, and viewpoints on

transfer in their settings.

19

Use the LoU Inventory interview as an action research tool to illuminate transfer

efforts, provide insight into the design of interventions to enhance transfer, and to

gather follow-along data as interventions are implemented.

Use the LoU Inventory interview to measure transfer progress in circumstances in

which transfer is paramount.

20

References

Aguinis, H. & Kraiger, K. (2009) Benefits of training and development for individuals and teams, organizations, and society. Annual Review in Psychology, 60 (1), 451-474.

Baartman, L. K. J. & de Bruijn. E. (2011) Integrating knowledge, skills and attitudes: conceptualising learning processes towards vocational competence. Educational Research Review, 6 (2), 125–134

Baldwin, T. & Ford, J. (1988) Transfer of training: A review and directions for future research. Personnel Psychology, 41 (1), 63-105.

Billett, S. (2013). Recasting transfer as a socio-personal process of adaptable learning. Educational Research Review, 8, 5–13.

Blume, B., Ford, J., Baldwin, T. & Huang, J. (2009) Transfer of training: a meta-analytic review. Journal of Management, 36 (4), 1065-1105.

Brennan, P. C., Madhavan, P., Gonzalez, C. & Lacson, F. C. (2009) The impact of performance incentives during training on transfer of learning. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53 (26), 1979-1983.

Brown, T. C. & Warren, A. W. (2014) Evaluation of transfer of training in a sample of union and management participants: a comparison of two self-management techniques. Human Resources Development International, 17 (3), 277-297.

Brown, T. C., McCracken, M. & Hillier, T. (2013) Using evidence-based practices to enhance transfer of training: assessing the effectiveness of goal-setting and behavioral observation scales. Human Resources Development International, 16 (4), 374-389.

Brown, T., McCracken, M. and O'Kane, P. (2011) ‘Don't forget to write’: how reflective learning journals can help to facilitate, assess and evaluate training transfer. Human Resource Development International, 14 (4), 465-481.

Burke, L. & Hutchins, H. (2007) Training transfer: an integrative literature review. Human Resource Development Review, 6 (3), 263-296.

Chen, G., Thomas, B. & Wallace, J. (2005) A Multilevel examination of the relationships among training outcomes, mediating regulatory processes, and adaptive performance. Journal of Applied Psychology, 90 (5), 827-841.

Davidson, M. L. (2014) A criminal justice system-wide response to mental illness: evaluating the effectiveness of the Memphis Crisis Intervention Team training curriculum among law enforcement and correctional officers. To be published in Criminal Justice Policy Review. [Preprint] Available from: http://cjp.sagepub.com/content/early/2014/10/23/0887403414554997.full.pdf+html [Accessed: 3rd February 2015].

De Grip, A. & Sauermann, J. (2013) The effect of training on productivity: the transfer of on-the-job training from the perspective of economics. Educational Research Review, 8, 28-36.

21

Enos, M., Kehrhahn, M. & Bell, A. (2003) Informal learning and the transfer of learning: how managers develop proficiency. Human Resource Development Quarterly, 14 (4), 369-387.

Fitzgerald, C. G. (2002) Transfer of training and transfer climate: The relationship to the use of transfer maintenance strategies in an autonomous job context. Ph.D Thesis, University of Connecticut, USA.

Folger, T. S. & Williams, M. K. (2009) Filling the gap with technology innovations: Standards, curriculum, collaboration, success. Journal of Computing in Teacher Education, 23 (3), 107-115.

Ford, J. & Weissbein, D. (1997) Transfer of training: an updated review and analysis. Performance Improvement Quarterly, 10 (2), 22-41.

Frash, R., Antun, J., Kline, S. & Almanza, B. (2010) Like It! Learn It! Use It?: a field study of hotel training. Cornell Hospitality Quarterly, 51 (3), 398-414.

Gegenfurtner, A. (2011) Motivation and transfer in professional training: a meta-analysis of the moderating effects of knowledge type, instruction, and assessment conditions. Educational Research Review, 6 (3), 153-168.

Grossman, R. & Salas, E. (2011) The transfer of training: what really matters. International Journal of Training and Development, 15 (2), 103-120.

Hall, G. E., George, A. A. & Rutherford, W. L. (1979) Measuring Stages of Concern about the innovation: a manual for use of the SoC Questionnaire (Report No. 3032). Austin, Texas, The Research and Development Center for Teacher Education, University of Texas.

Hall, G. E. & Hord, S. M. (2011) Implementing change: patterns, principles, and potholes. 3rd ed. Upper Saddle River, NJ, Pearson.

Hall, G. E. & Loucks, S. F. (1977) A developmental model for determining whether the treatment is actually implemented. American Educational Research Journal, 14 (3), 263-276.

Hall, G. & Loucks, S. (1978) Innovation configurations: analyzing the adaptations of innovations. Austin, Texas, The Research and Development Center for Teacher Education, University of Texas.

Hodges, J. M. O. (2014) An examination of Stage of Concern, Levels of Use, and tutorials on faculty members’ implementation of a learning management platform. Ph.D Thesis, University of Southern Alabama, USA.

Hollingshead, B. (2009) The concerns-based adoption model: A framework for examining implementation of a character education program. NASPA Bulletin, 9 (3), 166-183.

Hanover, J. & Cellar, D. (1998) Environmental factors and the effectiveness of workforce diversity training. Human Resource Development Quarterly, 9 (2), 105-124.

22

Kazbour, R., McGee, H., Mooney, T., Masica, L. & Brinkerhoff, R. (2013) Evaluating the Impact of a Performance-Based Methodology on Transfer of Training. Performance Improvement Quarterly, 26 (1), 5-33.

Kong, F. & Shi, N. (2009) Process analysis and level measurement of textbook use by teachers. Frontiers in Education In China, 4 (2), 268-285.

Kupritz, V. & Hillsman, T. (2011) The impact of the physical environment on supervisory communication skills transfer. Journal of Business Communication, 48 (2), 148-185.

Ladyshewsky, R. & Flavell, H. (2011) Transfer of training in an academic leadership development program for program coordinators. Educational Management Administration & Leadership, 40 (1), 127-147.

LaRocco, D. J., & Wilken, D. S. (2013) Universal design for learning: university faculty Stages of Concern and Levels of Use—A faculty action-research project. Current Issues in Education, 16 (1). Available from: http://cie.asu.edu/ojs/index.php/cieatasu/article/view/1132 [Accessed 2nd February 2015].

Lau, P. & McLean, G. (2013) Factors influencing perceived learning transfer of an outdoor management development programme in Malaysia. Human Resource Development International, 16 (2), 186-204.

Learning Forward. (2015) Standards for professional learning. [Online] Available from: http://learningforward.org/standards-for-professional-learning [Accessed 20th February 2015]

Lobato, J. (2012) The actor-oriented transfer perspective and its contributions to educationalresearch and practice. Educational Psychologist, 47 (3), 232-247.

Millar, P. & Stevens, J. (2012) Management training and national sport organization managers: examining the impact of training on individual and organizational performances. Sport Management Review, 15 (3), 288-303.

Myers, M. J. M. (2009) Transfer of learning rom training program to the workplace in a university healthcare organization setting. Ph.D Thesis, University of Connecticut, USA.

Nielsen, K., Randall, R. & Christensen, K. B. (2010) Does training managers enhance the effects of implementing team-working? A longitudinal, mixed methods field study. Human Relations, 63 (11), 1719-1741,

Nikandrou, I., Brinia, V. & Bereri, E. (2009) Trainee perceptions of training transfer: an empirical analysis. Journal of European Industrial Training, 33 (3), 255-270.

Olafson, L., Quinn, L. F. & Hall, G. E. (2005) Accumulating gains and diminishing risks during the implementation of best practices in a teacher education course. Teacher Education Quarterly, 32 (3), 93-106.

23

Orr, D. & Mrazek, R. (2009) Developing the Level of Adoption survey to inform collaborative discussion regarding educational innovation. Canadian Journal of Learning and Technology, 35 (2). Available from: http://www.cjlt.ca/index.php/cjlt/article/view/522/255 [Accessed 2nd February 2012].

Perkins, D. & Salomon, G. (2012) Knowledge to go: a motivational and dispositional view of transfer. Educational Psychologist, 47 (3), 248-258.

Perry, E., Kulik, C. & Bustamante, J. (2012) Factors impacting the knowing-doing gap in sexual harassment training. Human Resource Development International, 15 (5), 589-608.

Roberson, L., Kulik, C. & Pepper, M. (2009) Individual and environmental factors influencing the use of transfer strategies after diversity training. Group & Organization Management, 34 (1), 67-89.

Roberts, G., Becker, H. & Seay, P. (1997) A process for measuring adoption of innovation within the supports paradigm. Journal of the Association for Persons with Severe Handicaps, 22 (2), 109-119.

Rodriguez, B. C. P. & Armellini, A. (2013) Interaction and effectiveness of corporate e-learning programmes. Human Resources Development International, 16 (4), 480-489.

Rout, K., Priyadarshani, N., Hussin, Z., Pritinanda, A., Mamat, W. H. B. & Zea, G. L. (2010) Implementation of new sixth form geography curriculum: concerns and levels of use of teachers in Malayasia. International Journal of Educational Administration, 2 (1), 63-72.

Salas, E., Tannenbaum, S., Kraiger, K. & Smith-Jentsch, K. (2012) The science of training and development in organizations: what matters in practice. Psychological Science in the Public Interest, 13 (2), 74-101.

Saylor, P. R. (1998) Transfer management interventions: Environmental influences and individual characteristics that affect implementation of staff development initiatives. Ph.D University of Connecticut, USA.

Schindler, L. & Burkholder, G. (2014) A mixed methods examination of the influence of dimensions of support on training transfer. Journal of Mixed Methods Research. [Preprint] Available from: http://mmr.sagepub.com/content/early/2014/11/06/1558689814557132.full.pdf+html[Accessed: 3rd February 2015].

Strickland, D., Coles, C. & Southern, L. (2013) JobTIPS: a transition to employment program for individuals with autism spectrum disorders. Journal of Autism Developmental Disorders, 43 (10), 2472-2483.

Taylor, M., Ayala, G. & Pinsent‐Johnson, C. (2009) Understanding learning transfer in employment preparation programmes for adults with low skills. Journal of Vocational Education & Training, 61 (1), 1-13.

Tews, M. & Tracey, J. (2009) Helping managers help themselves: the use and utility of on-the-job interventions to improve the impact of interpersonal skills training. Cornell Hospitality Quarterly, 50 (2), 245-258.

24

Tunks, J. & Weller, K. (2009) Changing practice, changing minds, from arithmetical to algebraic thinking: an application of the concerns-based adoption model (CBAM). Educational Studies in Mathematics, 72 (2), 161-183. Available from: http://www.jstor.org/stable/40284616 [Accessed 2nd February 2015].

Turcotte, D., Lamonde, G. & Beaudoin, A. (2013) Evaluation of an in-service training program for child welfare practitioners. Research on Social Work Practice, 19 (1), 31-41.

Volet, S. (2013) Extending, broadening and rethinking existing research on transfer of training. Educational Research Review, 8, 90-95.

Wang. W. 2014 Teachers’ Stages of Concern and Levels of Use of a curriculum innovation in China. International Journal of English Language Teaching, 1 (1), 22-31.

Watkins, K., Lyso, I. & deMarrais, K. (2011) Evaluating executive leadership programs: a theory of change approach. Advances in Developing Human Resources, 13 (2), 208-239.

Weber, K. E. (2013) An analysis of faculty development levels of use outcomes at one higher education institution. Ph.D Thesis, University of Dayton, USA.

25