Linking Learning to Performance - ECTA Centerectacenter.org › ~pdfs › meetings › outcomes2006...

Linking Learning to Performance 1

Linking Learning to Performance Training and Technical Assistance Strategies to Support Child Outcomes Measurement Systems

Presented by Larry Edelman Department of Pediatrics, University of Colorado at Denver and Health Sciences Center NECTAC National TA Meeting: Measuring Child and Family Outcomes April 26, 2006 Albuquerque, New Mexico


Introduction The Individuals with Disabilities Improvement Act of 2004 (IDEA 2004) requires that all state Part C and Section 619 of Part B programs increase their rigor in collecting and reporting data on child outcomes. As we design and implement child outcomes measurement systems, a large and diverse group of stakeholders will require new knowledge and skills in order to set our measurement systems into motion. To meet this need, national, regional, state, and local agencies will need to put in place a range of information dissemination, training, and ongoing technical assistance activities.

We have an opportunity to use this mandate as a catalyst to revitalize our early childhood training and technical assistance capacities. No agency has sufficient human resources, time, or funding to do all of the things that ought to be done or that they would like to do. But even though we can’t do everything at once, we can make useful incremental changes that will result in more efficient and effective training and TA resources.

The best way to get a good idea is to get a lot of ideas. Linus Pauling

The following sections describe a range of strategies and techniques. Some of the technologies are still emerging. Others are in wide use but might be new to some agencies. The purpose of this document is NOT to outline everything that we all must do right away, but, rather, to stimulate our thinking about how we might incrementally and feasibly adopt new strategies and techniques for conducting front-end needs assessment, delivering instruction, supporting performance, providing ongoing technical assistance activities, and evaluating our efforts.

I hope that these resources are useful.

Larry Edelman April, 2006

Many thanks to Nan Vendegna and Christina Kasprzak for commenting on earlier drafts of this document.

Contents Front-End Needs Assessment ················································································ 3

Strategies for Delivering Instruction ········································································ 7

Promising Training Delivery Technologies······························································ 13 Avoid One Shot Training························································································· 16

Strategies for Delivering Ongoing Technical Assistance ········································ 18

The Costs of Providing Training·············································································· 20

Strategies for Evaluating Training··········································································· 23

Larry Edelman (303) 864-5264 [email protected]


Front-End Needs Assessment

“Before beginning a long journey, it is wise to know where you are going…that way you will

know that you’ve arrived once you get there.” Pooh Bear

Teachers, early interventionists, program administrators, supervisors, parents, training and technical assistant specialists, state agency staff, and others will need to learn new things in order to meet the challenges of new and more rigorous accountability requirements. It is reasonable to assume that a one-size-fits-all training approach will not suffice because diverse stakeholder groups will need to learn different things (e.g. knowledge and skills) and will require different training methodologies based on geography, communication exchanging preferences, learning styles, functional roles, and a host of other factors.

Beware of ready-fire-aim training development. The decisions regarding which training and information dissemination methodologies to provide need are best made when they are based on accurate and detailed front-end needs analyses. We need to identify the knowledge and skills that each stakeholder group needs to attain in order to competently participate in the child outcomes measurement system. But we also need to learn many other things. For instance, we need to know if there are motivational, environmental, or procedural obstacles preventing the performance that we want to see. We need to know what training methods various segments of stakeholders prefer, or at least find acceptable.

Spend more time up front and focus on performance. Spending more time up front will increase the likelihood of developing training systems that are more effective and less costly than other options. If we base our analysis on the on-the-job performances that we want to see, we will be better able to build systems to support those performances. The following section describes some strategies for conducting front-end needs assessment, including what information to collect and some ways to collect it.


Front-End Analysis: Focus on Performance

Types of Analysis

Examples of Questions to Answer

Examples of Methods for Collecting Information

Goal Analysis

• What are our specific, measurable targets for performance related to practitioners’ roles in the outcomes measurement system?

• Delphi technique

• Priority matrix

• Nominal group technique

• Brainstorming

• Community forum

Job Analysis

• What tasks do practitioners need to do and to what standards do they need to them to meet the expectations of the outcomes measurement system?

• Interview administrators, supervisors, subject matter experts (SME), and “star performers”

• Conduct focus groups

• Observe: typical work days, exemplary practitioners collecting and reporting data; providers using the Child Summary Form

Task Analysis

• What steps and decisions do practitioners need to follow in order to collect, summarize, and report data?

• How can we assure that training is based on consistent, efficient best practices?

• Observe “star performers”

• Review recommended practices

• Conduct interviews and focus groups with supervisors, subject matter experts (SME), and “star performers”

• Collect critical incident reports

• Develop flow charts

• Write “How-To” guides

Performance Analysis

• Why aren’t practitioners doing what we need them to do?

• Can they do it sometimes? Could they do it if their lives depended on it?

• What’s causing problems? Are there motivational, environmental, or procedural obstacles?

• Is this a problem that training can address?

• Are there other solutions?

Use Problem-Solving Strategies:

• Identify problems

• Analyze the causes of the problems

• Generate possible solutions

• Select solutions to try

• Implement solutions

• Evaluate solutions


What else might we need to learn about in order to design effective and efficient training?

Things that we might need to learn about the learners: What are reasonable ways to segment the audience (e.g. by where they live, by roles, etc.)?

How many people are there in each segment (create a density map)?

Where do they live and work?

Which training methods do various segments of the expected training audience prefer, or at least find acceptable?

How accessible is technology to the various segments?

How much turn over is there; how often will waves of new personnel need to be trained?

Which segments of the audience have participated in previous training related to the topics?

What is on the participants’ minds in relation to expectations for their performance?

Things that we need to learn about each training methods being considered: How effective is the method in achieving specific kinds of learner outcomes?

What are the costs for development and delivery?

Which segment(s) of stakeholders prefer or at least finds the method acceptable (hence, the segment of the audience for which it should be developed)?

What conditions need to exist in order to expect that the method will be effective?

Things we need to learn about organizational support and change: What organizational characteristics might likely support or impede desired performance (the

application of new knowledge and skills)?

At the state level: what kinds of leadership, support, advocacy, accommodations, facilitation, and recognition need to be applied to support stakeholders’ learning and application of new knowledge and skills?

At the local program level: what kinds of leadership, support, advocacy, accommodations, facilitation, and recognition need to be applied to support stakeholders’ learning and application of new knowledge and skills?

References on Front-End Analysis Techniques: Asher, P. (2006). Conducting Best-Practice Front-End Analysis. Webinar Presentation, 3/30/06

VanGundy, A. (1988). Techniques of structured problem solving (second edition). New York: Van Nostrand Reinhold.

Wolfe, B.L. & Snyder, P. (1997). Needs Assessment and Evaluation In Early Intervention Personnel Preparation: Opportunities and Challenges in Winton, P.J., McCollum, J.A., Catlett, C. Reforming personnel preparation in early intervention: Issues, models, and practical strategies. Baltimore: Paul Brookes publishing Co.


To learn more about Analyzing Performance Problems: Mager, R.F. & Pipe, P. (1970). Analyzing performance problems or you really oughta wanna (second edition). Belmont, California: David S. Lake Publishers.

To learn more about Delphi Technique: http://www.carolla.com/wp-delph.htm To learn about Nominal Group Technique: http://www.ryerson.ca/~mjoppe/ResearchProcess/841TheNominalGroupTechnique.htm http://instruction.bus.wisc.edu/obdemo/readings/ngt.html To learn more about Priority Matrix: http://www.mindtools.com/pages/article/newHTE_95.htm To learn more about Focus Groups: http://www.stcsig.org/usability/topics/focusgroups.html http://www-marketing.wharton.upenn.edu/forecast/paperpdf/FocusGroup.pdf http://www.extension.iastate.edu/Publications/PM1969A.pdf To learn more about Critical Incident Reporting: http://www.apa.org/psycINFO/special/cit-intro.pdf http://www.ucc.ie/hfrg/emmus/methods/cit.html To learn more about Brainstorming: http://www.hci.com.au/hcisite3/toolkit/brainsto.htm http://www.businessballs.com/brainstorming.htm To learn more about Flow Charts: http://deming.eng.clemson.edu/pub/tutorials/qctools/flowm.htm http://www.hci.com.au/hcisite2/toolkit/flowchar.htm

http://home.att.net/~dexter.a.hansen/flowchart/flowchart.htm

A Few Low Cost and Quick Needs Assessment Ideas Web surveying is a fast, inexpensive method for obtaining survey information. If

designed well, surveys can yield very useful data. No doubt you’ve been asked to respond to surveys through online survey companies such as Survey Money or Zoomerang. You can find links to 34 online survey companies, many of which offer free services, at: http://www.surveymonkey.com/Pricing.asp

Observe (or video tape) and then analyze star performers collecting and reporting assessment data using the instruments that your outcomes measurement system has selected.

Focus groups are an excellent source of information on training needs and evaluation. Such groups do not always need to be conducted face-to-face; they can be used effectively, and inexpensively, via telephone conference calling and skilled facilitation.


Strategies for Delivering Instruction States are likely to need to develop a range of options to support stakeholders’ knowledge and skills required to competently collect and report data. Face-to-face classroom training (F2F) is a traditional and time-honored method that has become the default modality for wide scale training in early care and education. F2F training will most likely be depended upon to prepare practitioners to participate in states’ outcomes measurement systems. And it is likely that F2F will be effective for some stakeholders in some locations at some times. However, for a variety of reasons (e.g. logistic, economic) many states will need to consider other training methods to augment, or in some cases supplant, F2F training.

The following section describes commonly used instructional formats and discusses some of their relative advantages and disadvantages. Please note that these formats are not mutually exclusive. On the contrary, they can, and in most cases should be blended together with marketing strategies and ongoing technical assistance to form holistic learning systems.

Face-to Face Formats

Technology-Based Synchronous Formats

Asynchronous Formats

Blended Systems

• Training Institute

• Regional/Local Training

• Train-the-Trainer

• Conference Presentations

• Long and Short Courses

• Demonstration Sites

• Peer Teaching

• Audio Conference

• Video Teleconference

• Webcast

• Self-Paced Instruction (Low- and High-Tech)

• Online Course

• Online Learning

• Computer-Assisted Instruction (CAI)

• Blended Learning

• Learning Communities

Note: Synchronous refers to training formats in which interaction between participants is simultaneous

Asynchronous refers to training formats in which interaction between parties does not take place simultaneously.


Instructional Formats Advantages and Disadvantages

Face-to-Face Formats Training Institutes Intensive training sessions designed to help implement broad based, system-wide changes. They might last two or more days and cover a great deal of information.

Advantages: • Effective way to focus participants’ attention. • In-person and interactive, allowing for participants to ask

questions and hear consistent answers. • Can address individual needs and concerns. • Can be an effective format for train-the-trainer programs. Disadvantages: • Training practitioners to collect and report data using one

assessment instrument might not require such an intensive, multi-day training.

• Institute’s can be very costly (multiple trainers, travel, facility, meals, etc.).

• Can accommodate only limited numbers of participants. Regional/Local Training Synchronous, instructor-led session with a group of participants in a classroom. Similar to Training Institutes they are on-site and F2F, but rather than seeking to affect broad system-level change, these sessions are typically half- or full-day sessions focused on providing knowledge and/or skills related to a circumscribed topic, e.g. using a specific assessment instrument. Participants can be provided information and tools, walked-through procedures, and given the opportunity to ask questions and address concerns. Might be offered as in-service sessions at job locations or as stand-alone workshops.

Advantages: • Training is in-person and interactive. • Can focus on a single topic, e.g. use of a specific

assessment instrument. • “Traveling road shows” can offer consistent information,

presented in consistent ways in various regions to support uniform practices.

• A variety of training methodologies may be used. Disadvantages: • On-site training sessions can be costly (multiple trainers,

travel, training facility, audio-visual equipment, participant travel, etc.).

• One time exposure to materials yields limited results (however, follow-up activities can be used, e.g. online resources).

Train-the-Trainer Participants are prepared to present training to others (locally, regionally, statewide, or nationally). Trainers can receive information on both training content and delivery. Trainers are provided with everything that they will need to replicate a training program, including curriculum, AV materials, handout materials, evaluation strategies, etc.

Advantages: • Facilitates local dissemination of information, minimizing cost

to lead agency. • Enhances the capacity of local programs and systems. Disadvantages: • Without control over the selection of trainers and the local

delivery of information, it is difficult to assure consistency, accuracy, and/or quality of training.

• If trainers do not have explicit performance expectations or support, they might not have sufficient time or resources back on the job to disseminate information.

Conference Presentations Information can be shared at conferences through plenary or breakout sessions.

Advantages: • Can reach specific disciplinary audiences through a

conference organized by a professional organization. Disadvantages: • Attendance is voluntary; does not easily allow targeted

sharing of information. • Difficult to track attendees, although participant names and

email addresses can be collected for follow up.


Instructional Formats Advantages and Disadvantages Long and Short Courses F2F courses can be developed and/or information can be infused into existing F2F courses.

Advantages: • Information can be delivered to those being prepared to enter

the field. • Can be used in both pre-service and continuing education

courses. Disadvantages: • Courses usually are limited to a relatively small number of

participants. • Can be used as credit-bearing courses and in conjunction

with degree-granting and credentialing programs.

Demonstration Sites Demonstration Sites are provider settings (e.g. classrooms, early intervention programs) that are recruited for their expertise in a specific practice (e.g. using assessment data to inform intervention) or for the ability to illustrate a specific process (e.g. data reporting). Staff from neighboring programs visit to see specific practices and processes in action and plan how to adopt/adapt such activities in their own program.

Advantages: • Can be offered as one-time events or training over time. • Can be integrated with coaching, face-to-face training, and

other methods. Disadvantages: • Can be time consuming to recruit and expensive to support

demonstration sites.

Peer Teaching Practitioners with advanced experienced, abilities, or training are designated as “Master Practitioners” and provide one-to-one or small group training and support.

Advantages: • Enhances the capacity of local programs and systems. • Practitioners who serve as trainers are easily accessible for

follow-up, coaching, problem-solving, etc. • Supports the professional development of the peer teacher. Disadvantages: • Difficult to assure consistency of information when learning

facilitators are locally based and preparation is loosely defined.

Technology-Assisted Synchronous Formats

Audio Conference Voice-only connection of more than two sites using standard telephone lines; uses a device called an "audio bridge." The call might be one-way, two-way, or a combination of both. Can enhance the audio presentation with print documents and/or audiovisual visual materials accessed through a web site, email, or U.S. mail. Groups of trainees can listen together, and then participate in group discussions and activities.

Advantages: • Trainers can assemble in one location to offer statewide

training. • Can be delivered at relatively low cost. • Easily accessible for participants, who listen in from their own

locations. • Can potentially reach a wide audience. • Can be archived and made accessible to future listeners. • Those without computers and/or high-speed internet access

can participate. Disadvantages: • Largely inactive, passive structure can be boring and difficult

for some participants. • Most effective for relatively short sessions, e.g. one to one

and a half hours long.


Instructional Formats Advantages and Disadvantages Videoconference A synchronous session that broadcasts a live training workshop from one central location to multiple downlink sites across a state, region, or nationally. Broadcast can be a one-way or two way exchange. When conducted as a live session, remote sites can call, fax, or email into the broadcast hub to ask questions or contribute ideas, enabling real-time interactions. Video teleconferences might be broadcast using microwave, satellite, or fiber-optic technologies. Can combine the video exchange with classroom facilitation. Teleconferences can be recorded and archived for future viewings.

Advantages: • A cost-effective way to assemble multiple trainers to offer

statewide training. • Practical for participants, as they can attend a convenient

location with minimal travel expenses. • Remote sites can hold follow-up activities to increase

interaction and address individual questions or concerns. • Can augment the video teleconference with print material or

information via fax, snail mail, web site, email, CD-ROM or DVD.

Disadvantages: • Limited opportunity for interaction with presenters, so

participants cannot always have their questions or concerns addressed.

• Largely inactive, passive structure can be boring and difficult for some participants.

• Frequent technical difficulties can impact effectiveness. • Certain technologies, e.g. satellite, can be costly. • Can only be viewed at sites that are equipped and prepared

to receive the transmission.

Webcast A presentation, workshop, or conference broadcast over the web; participants watch and listen over computers with internet access. The training can be presented in real-time and archived so that participants can access and view it at any time. This technique can also be used in conjunction with the regional training model; groups of trainees watch together, and then participate in group discussions and activities, with or without a training facilitator. Webcast can be archived so that future viewers can watch at any time. A related term is Webinar, a small synchronous online learning event in which a presenter and audience members communicate via text, chat, or audio channels about concepts often illustrated via online slides and/or an electronic whiteboard. Webinars are often archived for asynchronous, on-demand access.

Advantages: • Trainers can assemble in one location to offer statewide

training. • Can be delivered at relatively low cost. • Easily accessible for participants, who watch from their own

locations. • Can potentially reach a wide audience. Disadvantages: • Difficult to allow for interaction; often, participants can only

have their questions answered or concerns addressed toward the end of the program.

• Potential for technical difficulties. • Largely inactive, passive structure can be boring and difficult

for some participants. • Those without access to computers and high-speed internet

access cannot participate.



Asynchronous Formats Self-Paced Instruction Asynchronous instruction in which learners participate individually at their own pace at their own chosen time. Participants may be asked to complete assignments and/or assessments to support learning and verify course completion. Self-paced training can be delivered using low-tech media, such as print documents and VHS or higher-tech media such as computers, CD-Rom, DVD, the Internet.

Advantages: • Can be used flexibly to address many different learning

styles. • Can be presented using low-tech delivery, e.g. a printed

manual in conjunction with a videotape. • Can be presented using high-tech delivery, e.g. a web site,

interactive CD-ROM, or DVD. Disadvantages: • Does not provide opportunities for interaction unless

integrated with other methods.

Online Course Participants complete a course that is posted on a web site. Might use a variety of instructional methods including: links to other web resources: assignments (group or individual); interaction among those using the course using threaded discussions, email listservs, chat rooms, etc.; learning assessments.

Advantages: • Learners can participate at their own chosen time. • A variety of training methodologies may be used. • Can be used as credit-bearing courses and in conjunction

with degree-granting, credentialing, and continuing education programs.

Disadvantages: • Those without ready access to a computer with high speed

internet access or those with low computer literacy may find participation difficult and frustrating.

• Potential for technical difficulties that impede access. • Difficult for those who learn best in F2F social situations.

Online Learning Sometimes referred to as e-learning or web-based learning; learners gain access to resources through a web site. This method can be integrated with other strategies in many ways: can be used to deliver simple text documents; can host viewing and downloading of multimedia presentations; can deliver an entire course including the delivery of virtual lectures, readings, assignments, testing, student to student interaction, teacher to student interaction, and course evaluation.

Advantages: • Easily accessible for participants, who watch from their own

locations. • Can potentially reach a wide audience. • Existing and emerging technologies are making it easier and

affordable to create multimedia presentations. Disadvantages: • Potential for technical difficulties that impede access. • Those without access to computers and high-speed internet

access cannot participate. • Unless it is used as an ongoing course, it is difficult to

provide for interaction. • Requires that trainers learn new delivery skills. • Requires a lot of trainer time for preparation and delivery.

Computer-assisted instruction (CAI) Computers are used as a medium of instruction. Can be used for both initial and follow-up training. Delivers content through a CD-ROM or DVD and does not require that a computer be connected to a network or links to outside learning resources.

Advantages: • Easily accessible for participants, who participate via a

personal computer. • Potential to reach a wide audience. • Existing and emerging technologies are making it easier and

affordable to create multimedia presentations. Disadvantages: • Potential for technical difficulties that impede access. • Those without ready access to a computer or those with low

computer literacy may find participation difficult and frustrating.

• Difficult to provide opportunities for interaction.



Blended Systems Blended Learning A sequence of learning activities that combine the use of F2F classroom training with online information and instruction. For example, a pre-service or continuing education course meets F2F for discussions, interactive presentations, and demonstrations of practices. Following the F2F session, the participants use the Internet to download readings and materials, receive e-newsletters, and participate in online discussions, practice skills via online integrated media. Sometimes referred to as a “hybrid course.”

Advantages: • Follow-up activities can be both synchronous and

asynchronous learning opportunities. • Supports learning over time. Disadvantages: • Such experiences are limited to a relatively small number of

participants. • Follow up activities require internet access and ability.

Learning Communities Learning communities are groups of people with common needs and interests who gather together to provide mutual support for learning and performance. Rather than being controlled by an instructor, learners "self-organize" into functioning communities to support each other’s learning. Learning communities are characterized by: distributed control; a commitment to the generation and sharing of new knowledge; flexible and negotiated learning activities; autonomous community members; high levels of dialogue, interaction, and collaboration; a shared goal, problem, or project that brings a common focus and incentive to work together. From: http://carbon.cudenver.edu/~mryder/dlc.html

Advantages • Capacity to adapt to local conditions. • Encourages learning over time. • Fosters creativity and innovation. • Crosses traditional disciplinary and conceptual boundaries. • Appreciation of diversity and multiple perspectives. • Encourages practitioners to be responsible and skilled at

diagnosing and addressing their learning needs. Disadvantages • Not an efficient process for short-term learning or

performance support. • There is a lack of central control, hence a lack of consistency,

standards, uniformity of message. • Difficult to predict results.


Promising Training Delivery Technologies Compared to other fields, early intervention and early care and education are often slow to adopt new technologies. This section describes a handful of training delivery technologies that have been used successfully in a variety of fields and have promise for being very useful for training in early childhood.

For those of us who are not well versed in newer technologies, it might be useful to review one or more of the useful glossaries of elearning terminology that are available. Three such glossaries are: • American Society for Training and Development (ASTD) Glossary:

http://www.learningcircuits.org/glossary.html • Learnframe Glossary of e-Learning Terms:

http://www.learnframe.com/aboutelearning/glossary.asp • ELearning Guru:

http://www.e-learningguru.com/gloss.htm For those of us who are a bit technophobic, or who would like to desensitize others who fear technology, it might be fun to read: 12 Learning Interventions That Combat Technophobia by Linda Puetz http://www.learningcircuits.org/2000/mar2000/mar2000_elearn.html Podcasts

The word” podcast” is a combination of the words iPod and broadcast. It refers to the delivery of educational content via the Apple iPod or related audio player. Podcasting is a way of publishing podcasts to the web, allowing people to subscribe to a podcast and receive new “episodes” automatically.

Resources: To listen to the free podcast “Podcast on How to Podcast: By Elliott Masie,” visit: http://www.learning2006.com/university/

To read a case study of a company that is using Podcasts for auido learning, visit: http://www.gronstedtgroup.com/pdf/T_D_September_05.pdf

Computer-Based Multimedia Computer-based multimedia blends text, images, sound, and color to create easily accessible, interactive information delivery systems. Multimedia can be anything from a simple PowerPoint slide slow to a complex interactive simulation.

Resources: There are many examples of multimedia applications in Early Childhood.

• See a demonstration of Videatives’ integrated professional development solutions for in-service and pre-service teacher education at: http://www.videatives.com/content/videatives/

• Read and Watch “Wondering With Children: The Importance Of Observation In Early Education By George Forman & Ellen Hall at: http://ecrp.uiuc.edu/v7n2/forman.html

• Enspire (http://www.enspire.com) is a company that develops elearning systems. Watch the projects that they created for Texas (Part C) Early Childhood Intervention (ECI). The web-based on-line training modules can be accessed anytime from anywhere at: http://www.dars.state.tx.us/ecis/providers/trainingmodules.shtml


Multiple Formatting of Information Many organizations provide identical sets of information in two or more media in order to accommodate different learning needs and styles.

Resource: To visit a web site that packages information in three ways: streaming video, audio download, and transcript text download, visit: http://www.lms2006.com/podcasts/

Online Meetings Spaces

Many companies offer integrated online meeting facilities that can be used for online collaboration and the presentation and/or, demonstration of information. Although different companies offer different features, one can usually expect: visuals, one and two way calls, real-time question and answer capabilities, downloads, technical assistance, facilitation services, and live chat. Resources: Both “GoToMeeting” and “WebEx” provide free trials and webinars. Visit: http://www.webex.com/ and https://www.gotomeeting.com/

Performance Support Systems

Performance support systems (PSS) are designed to help users do things during, rather than as a precursor, to the performance of a task. For instance, a very simple PSS might involve a job aide such as a sign in a restaurant reminding employees to wash their hands. Generally, electronic performance support systems (EPSS) do so using computers and related technology to give workers information or resources to help them accomplish a task or achieve performance requirements. EPSS represent a shift from acquiring knowledge to performing tasks.

“While there remains an important role for traditional education and training, the shift to user-centered, performance-based models is both inevitable and imminent (Hannafin, 1993, 1995). The delivery model has shifted from courses that teach decontextualized knowledge and skill to modules that support performance involving relevant knowledge and skill. This shift has affected all forms of education and training (IETI, 1995). EPSS design practices represent a convergence among several related fields and specialties, including human performance technology, computer-supported collaborative work, technical communications, electronic publishing, instructional design, and workplace training (McGraw, 1994; Sherry & Wilson, 1996; Witt & Wager, 1994).” (Hannafin, Hill, and McCarthy, see below)

Resource: Read Designing Resource-Based Learning and Performance Support Systems by Michael J. Hannafin, Janette R. Hill, and James E. McCarthy, Sonalysts, Inc. at: http://reusability.org/read/chapters/hannafin.doc


Blogs Short for "Web logs," online journals, personal commentaries posted by the author, which usually are updated on a regular basis. It has been estimated that 6 million people posted their blogs on the Internet in 2004.

Resource: Read an article at Tech Learning that describes how blogs can be used in education.

http://www3.essdack.org/socialstudies/blogs.htm

Integrated Learning: We're in the midst of an e-learning revolution, which includes rapid change, a myriad of emerging technologies, and great opportunities to redeploy training investments. Following are a few related concepts, on which we should keep a sharp eye.

Learning Object: A reusable, media-independent collection of information used as a modular building block for e-learning content. LCMS (learning content management system): A software application (or set of applications) that manages the creation, storage, use, and reuse of learning content. LCMSs often store content in granular forms such as Learning Objects.

Learning Management System A software system that provides the platform for the enterprise’s online learning environment by enabling the management, delivery and tracking of blended learning (i.e., online and traditional classroom) for employees, stakeholders and customers: http://www.clomedia.com/content/templates/clo_feature.asp?articleid=91

ILS (integrated learning system): A complete software, hardware, and network system used for instruction. In addition to providing curriculum and lessons organized by level, an ILS usually includes a number of tools such as assessments, record keeping, report writing, and user information files that help to identify learning needs, monitor progress, and maintain student records.

Resources: Catch up on what’s happening with learning content management systems and related concepts through the American Society for Training and Development’s free “Learning Circuits.” Among the great articles are:

http://www.learningcircuits.org/2002/apr2002/robbins.html

http://www.learningcircuits.org/2001/aug2001/ttools.html

http://www.learningcircuits.org/2002/apr2002/mortimer.htm


Avoid One-Shot Training Learning is a process that occurs over time – a single training ‘event’ is only part of the picture. Even though we know this to be true, many programs end up providing only one-time training programs. Training delivery should be a well-thought out systemic process that capitalizes on a number of strategies, in addition to training events, over time. Practitioners learn through a variety of experiences, some planned and others serendipitous: Participating in list serves, chat groups, bulletin boards

Participating in learning communities

Surfing web sites

Reading books, journals, newsletters, reports

Watching videos, CD-ROMs, DVDs

Continuing education courses: Web and F2F

Workshops, conferences

Participating in pilot projects

Talking to colleagues, families

In-service training: F2F, web, audio, video

Mentoring relationships

Coaching relationships

Consultation

Visiting demonstration sites

Observation, shadowing

Trying out new things

Job aids

On-the-job

Pre-service academic preparation


Delivering Training and TA Support over Time using Multiple Strategies

We can plan systems of personnel preparation that blend key experiences, over time, to support learning AND performance. The contrived example below illustrates how the staff of an Infant/Toddler program might learn data collection and reporting skills over time.

Week 1: Practitioners receive initial email correspondence that introduces the initiative and offers “coming attractions” of what to expect in the future related to the outcomes work.

Week 2: Practitioners complete an online survey designed to elicit information about their expectations and concerns related to the outcomes measurement system and their experiences with different assessment systems.

Week 3: Practitioners watch an introductory video tape, read a Frequently Asked Questions (FAQ) document, and/or have a discussion at a team meeting to share expectations. The FAQ document lists a web site that the staff can visit for more information.

Week 4: Practitioners attend a F2F or teleconference training event focused on using a particular assessment instrument and reporting data.

Weeks 5 - 8: Practitioners practice the use of the instrument without having to report data. They have access to a web-based knowledge bank, a technical assistance hotline, an online refresher course, and are offered onsite and/or telephone technical assistance to follow up on questions they might have.

Week 9: The team discusses how the use of the assessment system is working and plans ways to integrate the data collection and reporting process into the program’s typical work days.

Week 10: Data collection begins.

Week 11: Exemplary (“star”) performers are identified before and during training and during the “practice” period. These practitioners are recruited to provide ongoing consultation, technical assistance to other local practitioners on collecting and reporting data. They are also invited to participate in a train-the-trainer program to build local capacity to orient and train new staff members.

Week 14: Program administrators are contacted to determine what additional activities might need to take place and plan for ongoing training of new staff.


Strategies for Delivering Ongoing Technical Assistance Technical assistance (TA) is a process within a dynamic context that enables a goal-focused, strategy-oriented, accountable organization to transfer knowledge to clients for the purpose of their growth, change, and improvement” (Trohanis, 2001). TA systems might be organized around centralized, regional, and/or locally based systems. Technical assistance in early care and education programs is often delivered through Technical Assistance Specialists who work for or on behalf of national, state, regional, or local agencies. The activities that a TA system employs should be based on knowledge derived from planning, monitoring, and self-assessment processes. Conventionally, TA Specialists offer such services as planning, training, information dissemination, and consultation. This section offers a glimpse of a few TA strategies and activities that might be used, in addition to the conventional TA strategies, to support the performance of practitioners and programs to implement statewide outcomes measurement systems. Hotline or Help Desk: “Help” services might be offered through phone, email, fax, or the online request lines. Responses to requests might be offered in “real” time or queued for follow-up. Coaching: In general, coaching refers to a close relationship between a learner and a coach. The coach listens, offers support, probing questions, information, feedback, perspective, etc., to assist the learner in achieving goals. Coaching might be used to support long-term development, give feedback, and/or provide knowledge and skills. Often delivered one-on-one, coaching can also be provided in small groups or as a combination of both. Coaching might be delivered face-to-face or over a distance through telephone, video, email, or web-based tools. A useful reference is “The Heart of Coaching,” at: http://www.craneconsulting.com/

Mentoring: Mentoring generally refers to a relationship in which someone with more experience (a mentor) helps someone with less experience learn something that he or she would have learned less well, more slowly, or, if left alone, not at all.

Web-Based TA: The web offers a number of effective ways to provide knowledge to support practitioners and programs. TA Support pages might offer: • Frequently Asked Questions (FAQ) documents • Searchable Knowledge Bases • Links to resources • Downloads of helpful documents • Online support request • Online communities (open to all or by membership only; moderated or self-organizing) • Chat rooms • Bulletin boards • Newsgroups (on online discussions hosted on the Usenet network; also called a forum).

Networking: TA can be provided by encouraging interaction among local or regional providers and programs. The use of model demonstration sites is one example of networking. Listservs: Listservs are based on e-mail programs that allow multiple computer users to connect onto a single system, creating an on-line discussion.


Resources Related to Technical Assistance

NECTAC State Early Childhood Technical Assistance Systems Page: http://www.nectas.unc.edu/topics/personnel/statetasys.asp

State-level Survey – Delivering Technical Assistance in Special Education: http://www.ccsso.org/content/PDFs/DRAFT%20REPORT_SE%20TA%20Survey%202005.doc

How to Improve the Design and Delivery of High Quality Technical Assistance: http://www.centerforcsri.org/files/TheCenter_NL_Nov05_B.pdf

Special Education Technical Assistance and Dissemination Network: http://www.ed.gov/parents/needs/speced/resources.html#early

Federally Funded Early Childhood Technical Assistance Projects: http://www.acf.hhs.gov/programs/ccb/ta/fta/federalta.htm

Child Care Technical Assistance Network: http://www.nccic.org/cctan/index.html

A Descriptive Analysis of State Early Childhood Technical Assistance Related to the Implementation of the IDEA (Black)

http://www.nectas.unc.edu/~pdfs/StateTAsysReport09-2002.pdf

Design Considerations for State TA Systems” (Trohanis): http://www.nectas.unc.edu/pubs/titlelist.asp

State Technical Assistance Initiatives for IDEA Part B Programs (Hanft): http://www.nasdse.org/publications/state_TA_initiatives_Part%20B.pdf


The Costs of Providing Training Return on investment (ROI) refers to the ratio of the benefit received from a given investment to the cost of the investment. In training, ROI is often calculated by comparing the tangible results of training (for example, increased quality, productivity, compliance, or decrease in errors) to the cost of providing the training. In order to estimate ROI, we must be able to estimate the cost of providing training and measure the results of training (see the following section of this document). The table below suggests some of the line items that need to be factored in when budgeting for training development and delivery. This table is not intended to be comprehensive, but rather to provide a starting place for calculating the cost of training. Please note that the table does not account for the enormous amount of time required for state lead agency staff and key stakeholders who are engaged in various stages of the development of training. Also note that it is not clear which budget items will be shouldered by state agencies and which will be the responsibility of local communities. Such decisions will vary from state to state and year to year. The table has columns to indicate both statewide and local responsibility for covering costs.

Costs Items State Local

Notes

Print Materials

Research and Write

Produce graphics

Review

Print and bind

Distribute

Video Materials

Research and Script

Shoot

Edit

Review

Duplicate (e.g. VHS, CD, DVD)

Package

Distribute

It is difficult to generalize the per-minute-cost of producing a video. The cost will depend on many factors such as the number of locations and shooting days, desired quality, video format, use of paid narrators, captioning, translating, etc.

A budget might range from $200 to $5000 per finished minute depending on the specifications of the production and the producer (e.g. state agency media department versus a private vendor). The more one tries to satisfy complex needs, the more expensive the production. One might expect no less than ten hours of editing time for each minute of finished video. Video production is made less expensive by narrowing the parameters of the end product.



Notes

Build and Host a Web Site

Research, Design and Develop

Produce

Review

Hosting

Maintenance

Budgets are very difficult to generalize. Costs will depend on whether pages for a new topic are added to an existing site or if a new site is created, whether external consultants are required (for writing, graphics, research, programming), special functionality required of the site (e.g. training registration), who will host it, etc.

Develop Training

Instructor-led F2F training development hours

Rough estimate: 40-100 hours per finished training hour depending on complexity of the training program

Computer-based training development hours


Self-study course development hours


Purchase of hardware

Purchase of software

Deliver F2F Training

Trainer salary/benefits and/or consulting fees

Trainer travel, lodging, per diem

Hiring substitutes to cover classrooms while trainees attend TTT events

Trainee travel, lodging, per diem

Trainee stipend Some providers may need to have someone buy out their time so they can attend training

Marketing the program

Registration

Clerical and administrative support

Off-site training facility costs



Notes

Meal and break costs

Audio-visual equipment rental

Duplicating or purchasing handouts and other print materials

Shipping handouts and other print materials

Administrative staff to coordinate events

Extra staff time needed to participate in training if required training exceeds allotted staff development days

CEUs/academic units Could necessitate an extra person at training site for registration

Other

Other

Other

Other

Other

Other

Other

Other

Interesting article on the costs of e-learning: Many Happy Returns: Calculating E-Learning ROI by John Setaro http://www.learningcircuits.org/2001/jun2001/elearn.html


Strategies for Evaluating Training In 1959, at the University of Wisconsin, Donald Kirkpatrick developed a straightforward model for evaluating training programs that is still widely used today. Below is a brief overview of Kirkpatrick’s model, which describes four levels of evaluation, each increasingly more difficult to accomplish than the previous one.

Level 1 – Reaction: Kirkpatrick’s first level focuses on the participants’ reactions to the training program. This level is analogous to measuring consumer satisfaction. Evaluating reaction tells us how to improve the program, gives trainees input in their training programs, gives feedback to trainers, provides information to managers and others concerned about the program, and can help to establish standards of performance for future programs. In evaluating reaction to training, we typically look at trainees’ perceptions related to: program content; relevance/usefulness; confusion; missing areas; instructor competence; level of engagement; training environment; and overall reaction. Reaction is typically measured immediately at the end of a training intervention.

Level 2 – Learning: This second level of evaluation determines learners’ performance relative to the learning objectives. This level of evaluation presumes that no change in behavior can be expected unless one or more of the learning objectives are accomplished. This evaluation of what has been learned looks at what knowledge was gained, what skills were developed or improved, and what attitudes were changed as a result of participating in the training program. As with level one, the measurement of learning takes place at the training program.

Level 3 – Behavior: The evaluation of behavior relates to changes in work behaviors that occur because people attend a training program. The evaluation of behavior looks at changes in performance on-the-job, as determined by what we believe our programs and services are all about. Measurement must allow for time between the end of the program and observations on-the-job.

Level 4 – Results: The fourth level of evaluation determines the results that occur due to participation in a training program. The evaluation of results looks at the effect training has on such things of importance to the organization as: quality of services; productivity; skill of workers; costs of delivering services; consumer satisfaction; and outcomes for consumers. Data for this level of evaluation must be collected before, during, and after a program, must allow time for results to be achieved, and must be repeated at several sequential times. As in level three evaluation, the kinds of data collected is driven by what we think we know about what our training program does for people and what effects it should have down the road.

NEW!! Level 5 – Organizational Support and Change: Thomas Guskey added a fifth level of evaluation (organizational support and change) that he places in the middle of Kirkpatrick’s model. Guskey bases this new level on the observation that even though a training program might effectively teach the right skills, those skills might not be used back on the job because organizations do not support or actually impede the use of what learners were taught during training (Guskey, 2006).

In short, Kirkpatrick’s model, with the addition of Guskey’s Level 5, informs how we can measure learners’ reaction to the training; the skills, information, and attitudes that were learned; organizational support of learning and change; changes in workplace behavior; and the results that occurred. Unfortunately, many organizations conduct only Level 1 evaluations of training programs. Level 1 evaluation data is very limited; it does not provide enough information to help us develop more effective personnel development programs or be better able to facilitate planned change efforts. The table on the following page, adapted from Guskey’s application of Kirkpatrick’s evaluation levels (Guskey, 2006), offers some ideas for information that might be collected at each of the five levels of evaluation along with some examples of ways to collect it.


Information to Collect at Each of the Five Levels of Evaluation Evaluation

Level Information to Collect Collection Techniques How might the information be used?

Level 1: Reaction How did the participants react to the program?

Participants’ reactions to: • Program content • Relevance/usefulness • Confusion • Missing areas • Instructor competence • Level of engagement • Overall reaction

• Questionnaires using both closed and open ended responses

• Focus groups • Interviews

• To document and continually improve training programs

Level 2: Learning Did the participants acquire the intended knowledge and skills?

• Demonstration of participants’ behavior, understanding, values, and knowledge specified in established learning objectives and desired performances

• Pre- and post-tests: performance and/or paper and pencil (or online) Note: tests should be designed when training objectives are developed

• Observations at training sessions

• To document and continually improve training programs

Level 3: Organizational Support and Change How has the organization supported change and learning?

• Evidence of how the organization supported and facilitated participants’ new performance.

• Interviews with administrators and training participants

• Questionnaires • Focus groups • Review of meeting minutes

and policy and procedures documents

• To document and continually improve organizational support and inform future change efforts

Level 4: Behavior Did the participants effectively use the desired knowledge and skills?

• Performance of desired behavior on-the-job

• Direct observation (allow time between end of program and observations)

• Engage managers early on • Recruit consumer feedback • Contracting, action planning

• To document and continually improve training programs and organizational support

• To document and continually improve implementation of desired performances

Level 5: Results What are the results for the organization?

• Cost of developing and delivering the training program

• The effect training has on measures that are important to the organization: • Improved services • Efficiency • Higher job satisfaction • More highly skilled workers • Costs • Outcomes for consumers

• Review data before, during, and after training for: • quality • timeliness • comprehensiveness

• Interviews • Questionnaires • Focus groups

• To document and continually improve all aspects of program design, implementation, and follow up

• To demonstrate the overall impact of professional development activities

• To monitor compliance

References: Guskey, T.R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press.

Kirkpatrick, D. (1996). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler Publishers.

Kreider, H. & Bouffard (2006). A conversation with Thomas R. Guskey. The Evaluation Exchange, Vol. XI, No. 4 pp. 12-14. Boston: Harvard Family Research Project. http://www.gse.harvard.edu/hfrp/eval/issue32/qanda.html

Linking Learning to Performance - ECTA Centerectacenter.org › ~pdfs › meetings › outcomes2006...

Documents

Transcript of Linking Learning to Performance - ECTA Centerectacenter.org › ~pdfs › meetings › outcomes2006...