1

School of Information Technology and Communication Design

CALIFORNIA STATE UNIVERSITYMONTEREY BAY

Interpretation Learning Modules

CAPSTONE REPORT

Submitted in partial satisfaction of requirements of the degree of

MASTER OF

Instructional Science and Technology

Sieun Lee

December 13, 2011

Capstone Approvals: (At least one advisor and capstone instructor should approve)

___________________________ ___________________________ _____________Advisor Name Signature Date

___________________________ ___________________________ _____________Capstone Instructor Name Signature Date

2

Table of Contents1. Executive Summary-------------------------------------------------------------------------3

2. Introduction----------------------------------------------------------------------------------4

a. Background------------------------------------------------------------------------------4

b. Problem Description--------------------------------------------------------------------5

c. Target Audience------------------------------------------------------------------------6

d. Literature Review-----------------------------------------------------------------------7

3. Solution Description------------------------------------------------------------------------8

a. Goals--------------------------------------------------------------------------------------8

b. Solution to Fill the Gap----------------------------------------------------------------9

c. Instructional Strategy-----------------------------------------------------------------11

d. Media Components-------------------------------------------------------------------13

4. Methods/Procedure------------------------------------------------------------------------14

a. Design and Development-------------------------------------------------------------14

b. Challenges and Solutions-------------------------------------------------------------15

5. Resources-----------------------------------------------------------------------------------16

6. Timeline-------------------------------------------------------------------------------------18

7. Evaluation/Testing Results---------------------------------------------------------------19

8. Conclusion----------------------------------------------------------------------------------24

9. References ---------------------------------------------------------------------------------27

10. Appendices---------------------------------------------------------------------------------28

3

1. Executive Summary

The Capstone project is intended to serve as a self-paced learning module

for military linguists at the Defense Language Institute (DLI). The project will

replace the existing Translation and Interpretation (T&I) training curriculum that

was designed two years ago for instructor-led traditional classroom environment.

As a self-paced learning module, the project is expected to solve the issues

including limited budget and resources, which have delayed the implementation

of T&I training at the DLI.

The project comprises four mini-modules which cover basic concepts of

interpreting, memory retention, paraphrasing, and note-taking skills. The modules

are created mainly by Adobe Captivate, and actively incorporate various multi-

media elements to keep learners engaged.

The goals of the projects are to provide learners with basic knowledge of

interpreting, assist DLI instructors in teaching a course with contents unfamiliar to

them, and to provide flexibility in learning and an alternative method to a human

interpreting practice partner. By using the modules, learners do not have to

depend entirely on instructors, which in turn lowers psychological burden on

instructors. Shy learners, in particular, are expected to benefit greatly from these

modules as they can build their skills by exercising repeatedly on their own until

they feel confident about performing verbal tasks in front of other people.

Capstone prototype was tried out by a group of prospective learners. Pre-

and post-tests were administered as a part of the tryout to measure learning gains.

Since interpreting performance could not be measured on a written test format,

pre- and post-tests were administered to each individual tester by the instructional

designer, during which learners performed interpreting live for evaluation. A

grading rubric that laid out scoring categories and criteria in detail was used by

4

the evaluator to ensure consistency and reliability. Tryout also included individual

observation to understand how learners interact with the module. Online survey

was used to measure learners’ reaction to the instruction. Based on findings from

formative assessment, revisions were made to the prototype as well as to the

subsequent modules.

Analysis of pre-& post-tests, observations and surveys support that not

only learners demonstrated significant learning gains, but they also self-reported

positive impact of the instruction on learning. Learners in general benefited from

the instruction to improve their basic interpreting skills.

2. Introductiona. Background

In 2008, the Defense Language Institute Foreign Language Center

(DLIFLC) initiated a test project of developing Translation and Interpretation

(T&I) training curriculum for its language students. The purpose of the project

was to tap into the possibility of launching T&I training programs to selected

military linguists to enable them to perform basic interpretation and translation

tasks between English and the foreign language of their study. The DLI was

hoping to have T&I curriculum developed with the help of outside sources and

use their existing language instructors, who were mostly not familiar with

translation or interpretation, to deliver T&I instruction internally.

Due to the lack of T&I expertise at the DLI’s end, the entire curriculum

development and pilot course delivery were outsourced to the faculty of the

Graduate School of Translation, Interpretation, and Language Education

(GSTILE) at the Monterey Institute of International Studies. The MIIS faculty in

four language programs – Korean, Spanish, Chinese, and Arabic – participated in

the 1-year project of developing T&I training courses and delivered them to the

5

DLI in May 2009. The courses were designed to be delivered in the instructor-led

traditional classroom environment.

b. Problem Description

More than a year has passed since the T&I training courses were delivered

to the DLI, but the DLI has not yet implemented the T&I training or taken any

action to follow up the project. According to email communications with the

project manager at the DLI, the institute does not have specific plan to implement

the training in the near future. Although it was not DLI’s official opinion, the

project manager stated two major reasons for non-action: 1) lack of budget, and 2)

lack of resources, especially adequate teaching staff that has enough expertise to

teach translation and interpretation. Initially, the DLI planned to utilize its own

instructors to deliver T&I training using the courses developed by the MIIS

faculty. However, most instructors, especially senior members, felt uncomfortable

about the idea of teaching translation and interpretation skills with which they

were unfamiliar. The courses developed by the MIIS faculty were designed for

instructor-led learning environment, and thus, they required instructors to

demonstrate significant knowledge and skills during instruction. While the

institute understands this limitation, due to budget constraints, it cannot hire new

instructors who are capable of teaching translation and interpretation.

The capstone project was envisioned as a potential solution to the above

mentioned problems that make implementing T&I training difficult at the DLI.

Consecutive interpreting learning modules developed as the capstone project are

self-paced learning modules which are intended to replace or supplement the

existing instructor-led interpretation training courses. The web-based online

modules which include various interactive multimedia components are designed

to assist learners step by step through the process of acquiring necessary skills for

interpreting. As compared to traditional classroom instruction, the online modules

6

provide more flexibility that allows students to learn wherever and whenever they

desire and to repeat as many times as they want. As a result, the modules

minimize instructor intervention in the learning process, which contributes

significantly to lowering burden on instructors. Also, the institute will be able to

bypass budgeting issue since providing instruction through online modules will

cost significantly less than conducting training through existing instructors or

hiring new teaching staff for the purpose.

c. Target Audience

The target audience is active members of the US Armed Forces military

linguists who have completed the basic course in their choice of foreign language

at the Defense Language Institute Foreign Language Center at the Presidio of

Monterey. Students are taught by highly educated instructors who are native

speakers of each language. Language classes are given five days a week, seven

hours a day. All learners have received high school education, some with a

college degree, and are fluent in English. Age varies among learners, but most are

between 25 and 35 years of age. The majority of learners are men and there is no

significant cultural diversity present among them.

Not all students at the DLIFLC are eligible for T&I training. In order for a

person to be able to interpret efficiently between two languages, he/she should

demonstrate a certain degree of mastery in both languages. Therefore, the project

is targeted specifically at qualified linguists who have achieved a minimum

foreign language proficiency level of 3 or higher (listening and reading) on the

Defense Language Proficiency Tests (DLPT) based on the guidelines of the

Interagency Language Roundtable (ILR). Although qualified target audience

demonstrates a certain level of proficiency in speaking, listening, reading and

writing, they have not had any systematic interpretation education and are not

familiar with the interpreting mechanism. A significant imbalance in language

7

proficiency between their A (native) and B (foreign) languages usually makes it

hard for them to perform interpretation from A to B language.

d. Literature Review

Unlike in other areas, the use of advanced technological tools has not been

actively pursued in the field of interpreter education, mainly because of the fact

that interpreting comprises auditory and verbal activities which require heavy

human intervention for guidance and assessment. As a result, most interpreter

education curriculums still rely on a traditional, instructor-centered approach

where expert instructors, as the source of expertise and authority, play the major

role of judging and assessing trainee interpreters’ performance (Hartley et al.

2003).

According to the same authors, however, the acquisition of interpreting

skills by students requires not only such professional guidance during classes, but

also extensive practice hours outside these classroom instructions. Students

cannot attain the required performance standards simply by sitting in classes. To

enhance their interpreting performance, students typically engage in intensive

group practice and seek feedback from their peers. Sandrelli and Hawkinsi (2006)

point out that while interpreter training curriculums unanimously emphasize the

importance of students’ autonomous practice as the key to improving interpreting

performance, students, especially beginners, do not always have access to

adequate study support or practice materials. Thus, a number of researchers

including Sandrelli (2005) have examined more extensive use of technological

tools and the development of dedicated computer software for the purpose of

supplementing traditional teaching methods and assisting trainees in making

unsupervised group- and self-practice sessions more productive and useful.

However, the development of such software tools have not shown much

progress. A couple of CAIT(Computer Assisted Interpreter Training) tools have

8

been designed so far, and one of them was a dedicated interpreter training module

called Black Box which was developed by Melissi Multimedia Ltd (U.K.) in 2002

for the University of Hull. Unfortunately, Black Box software has not been widely

adopted for its limitations. In software evaluation sessions organized by the

Schools for Interpreters and Translators of the Universities of Trieste and

Bologna, trainees who tested the software suggested ‘improved degree of

interactivity’ and ‘inclusion of more extensive resources’ for improvement

(Sandrelli, 2005).

Based on these research, it has become clear that computer assisted

interpreter training tools have great potential to contribute significantly to student

learning of interpreting skills. The remaining task is to enhance the degree of

interactivity between the user and the software as well as among users. Also, it is

important for such tools to include resources of sufficient quantity in order for

them to be useful to learners.

3. Solution Descriptiona. Goals

The capstone project serves three key purposes:

1) Provide basic knowledge about interpreting

The project introduces learners to interpreting. Even without much help

from instructors, learners will grasp the nature of interpreting as a mechanism

different than simple learning of languages, and will have opportunities to

practice interpreting between English and Korean using the modules. Learners

will acquire basic interpreting skills that will help them perform simple

interpreting tasks assigned to them in the field.

2) Assist DLIFLC instructors in teaching a course with contents

unfamiliar to them

9

The DLI plans to utilize its own language instructors to deliver the

interpretation training course to the students. Even though these instructors are

competent language teachers, the vast majority of them have not had formal T&I

education, and therefore, they do not have sufficient knowledge or understanding

about the mechanism of interpreting. Especially in a situation where instructors

have to deliver instruction as non-SMEs, the role of well designed learning

modules becomes even more critical to the success of instruction. Instead of

taking full responsibilities of teaching each learning content themselves,

instructors will rather function as facilitators. Relying on useful resources of self-

guided learning modules, they would feel strongly supported through the course

with specific techniques and guidance provided in detail by the modules.

3) Provide flexibility in learning and an alternative to a human

interpreting practice partner

The project is designed to provide flexibility in learning through the use of

computer-based multimedia platform that is not confined to the instructor-

delivered classroom instruction setting. Unlike the trainer-centered course which

as developed two years ago, the new web-based modules are designed to help

learners exercise enhanced control of their own learning. Learners can access the

modules at any place – either in or out of classroom- , how much they want, and

at times convenient to them. It creates a student-centered learning environment

where learners can pace their own learning depending on their differing needs and

capabilities.

b. Solution to Fill the Gap

The interpreting modules also provide a solution to a common limitation

experienced by all students who study interpretation; in interpreting, group

practice is very important to improving interpreting performance, and such

practices cannot be done efficiently without a study partner. In general, interpreter

10

trainees are strongly encouraged to form a small study group in which members

take turn to become a designated speaker to read a text or deliver a speech in the

source language for the other members to practice interpreting into the target

language. A significant number of hours spent on practicing are the key to the

success of a student as an interpreter. While practice partner(s) play such a critical

role in improving one’s interpreting skills, in reality, it is not always easy to find a

partner, and even if a student has one, finding a mutually convenient practice time

between the two is often limited. The interactive multimedia interpreting modules

will supplement the role of a human practice partner especially for learners who

do not have a practice partner or those who want to practice more on their own.

The modules are especially useful for less proficient or shy learners who

feel uncomfortable about performing interpretation in front of other people.

Interpreting almost always involves speaking in front of the audience and the

practice items will provide ample opportunities for them to feel comfortable about

speaking in public and build confidence. Until they build confidence, learners will

be able to exercise interpreting repeatedly on their own using these modules.

The capstone project focuses mainly on helping learners acquire basic

skills required for performing consecutive interpreting that involves auditory and

verbal activities. It comprises four modules that cover; 1) basic concepts of

interpreting, 2) memory building exercise, 3) paraphrasing exercise, and 4) note-

taking exercise. Three of the four modules (except for the basic interpreting

concept module) include demonstrations and multiple practice items in varying

topics, which introduce learners to key skills they need to master to be able to

perform interpreting. Learners are guided through the process of acquiring these

skills in a scaffold manner. Each module begins with a relatively easier task with

sufficient guidance and support, and progresses gradually toward more

challenging tasks. In the process, scaffolding is removed gradually and students

are asked to complete exercise items on their own.

11

In typical interpreter training curriculums, instructions begin with mono-

lingual exercises especially in the stage of teaching key skills described above.

Accordingly, the scope of the capstone project is limited to mono-lingual

exercises that ask learners to perform activities within the same language -

English. Learners listen to an audio in English and perform required verbal tasks

in English.

Each module includes a selection of carefully chosen audio or video

materials that adequately and effectively assist learners in achieving each learning

objective. This approach is useful especially for beginner interpreters, who

usually do not have much knowledge with regards to what type of materials are

really useful for them in enhancing their skills. The target audience of the

capstone project is beginner interpreter trainees who have not had interpreter

education before, and thus, this methodology is expected to prove useful.

c. Instructional Strategy

The main instructional strategy used for developing this project is Gagne’s

Nine Events of Learning. However, not all components of the nine events are

strictly included. Instead, some elements of Merrill’s First Principles of

Instruction are applied to complement the main instructional strategy.

Each module starts with a media component that grabs learners’ attention,

which is followed with clear explanation of the learning objectives as well as

what activities learners are expected to perform within each module. As the

instruction begins, learners are provided with learning guidance. Based on

Merrill’s demonstration principle, each module first provides learners with an

example solution to a problem. Then, learners are shown a series of problems and

asked to perform learning activities on their own. The problems progress into

increasing difficulty to scaffold the learning process. After each problem,

learners have a chance to perform self-assessment as they compare their own

12

answer to the best answer provided in the module. Instruction closes by

highlighting and reviewing important elements of the contents.

The main focus is on keeping the modules dynamic and interesting, while

avoiding the overuse of multimedia decorations. For example, various images are

presented on the screen while a source text is being played, so that learners can

easily associate and recall the contents of the text they hear. For voice recording,

multiple native speakers of English are used to read the texts to add variations and

liveliness to the modules.

The practice items are carefully arranged in a manner that transitions

progressively from very easy to more difficult tasks, so that every learner can

experience a sense of achievement at the beginning. The purpose of this approach

is to stimulate learners’ interest in interpreting and make them want to challenge

further.

Throughout the course, special attention is paid to encourage learners to

actively engage in verbal activities of interpreting. While learners hardly need to

speak or talk in many self-learning web modules, the Capstone interpreting

modules require learners’ active participation in verbal activities. In case the

modules are used within the classroom under the supervision of an instructor, the

instructor will work as a facilitator who will make sure that learners complete

activities verbally.

Some passive learners often choose not to attempt their own interpretation

when given a chance to practice on their own. Instead, they simply listen to the

texts and think about possible interpretation only to themselves without

verbalizing it. The key to the successful learning of interpretation depends heavily

on how active learners become in completing practice items verbally. To that end,

learners are reminded, within the modules, of the importance of verbal

engagement as well as the importance of using an audio recording device to

record their output and to listen to their own performance afterwards.

13

d. Media Components

The modules incorporate various interactive multimedia elements. For

example, in the memory exercise module, recording of a source text is played

back upon clicking of a button for learners to listen and perform required tasks.

For lower-level tasks at the beginning, relevant visual images or videos are

presented while the audio is played back to assist learners remember key

information in the source text they are listening to. Whenever needed, learners can

click buttons to go back to the previous page, move on to the next page, or return

to the main menu to choose another exercise. Below is the detailed list of all

media element used in the project:

Audio

- Background audio

- Voice recordings of the texts

- Audio recording/editing software

Video

- Useful video clips to assist instruction (Either retrieved from the web or

created by the designer using a camcorder)

Images

- Images retrieved from the web (with appropriate acknowledgement) used

across the modules, especially for visualization.

- Photos of the designer (or an animated character) used to create a virtual

instructor.

Human Instructor

- Minimum intervention. The role of an instructor is a facilitator throughout

the instruction.

Text

14

- Written texts prepared for translation exercises and transcripts.

Handwriting simulation

- Livescribe Smartpen and Livescribe Desktop applications to display

handwritten notes on the screen along with audio.

The delivery of the course is completely web-based. The interpreting

modules are created using Adobe Captivate 5, and the completed modules are

organized into the Capstone website, which was created by using Adobe

Dreamweaver 5 and is hosted by the CSUMB’s ITCDLAND server. Authorized

learners with the URL to the website will have easy access to the course and

assume learning anytime, anywhere and at their own pace.

4. Methods and Procedure a. Design and Development

The project development began by deciding which topics or skills to

cover. When the four topics –basics of interpreting, memory retention,

paraphrasing, and note-taking – were finalized, a significant amount of time was

spent on the course design. Each module was structured in a way that resembled

offline interpreting courses targeted at beginners. The universal flow of the course

across all modules was to promote learners’ understanding of the topic – what it is

about and why it is important – and then to present practice items to help learners

acquire the skill.

When the decision was made in regard to the topics and course structure,

the next step was to gather necessary texts for exercises. For memory exercise

module, for example, a total of five authentic texts were selected in varying

difficulty level. Text selection was a very important task that took into

consideration many factors such as text difficulty, length, subject, etc. More than

15

two months were spent on selecting texts for exercise items in all four mini

modules.

Of four modules, memory exercise module was the first to be developed.

Screen layout, colors, graphic design and background music which were selected

for this module were repeated in the subsequent modules for consistency. Upon

completion of memory exercise module, it was tested on six prospective learners

for usability and learning gains. Testing was based on observation of each

individual tester as he/she was using the module to learn the subject.

Based on the testing results, revisions were made to the prototype module.

After the prototype had been brought to completion with all the necessary

changes, corrections and revisions, the development of the subsequent three

modules began. Thanks to the trial and error on the prototype module, the

development of the following modules was completed more easily in a shorter

period of time. Because various design elements and course organization rules

were finalized on the prototype module, the designer did not have to waste time

on making frequent changes in the development process and reflecting them

across all modules repeatedly.

b. Challenges and Solutions

One of the biggest challenges for the project came from the nature of

interpreting; it involves auditory and verbal activities, and thus, it is hard to assess

learner performance or give feedback solely within the module without human

intervention. It is difficult to incorporate assessment into the modules, because

learners’ verbal interpreting performance cannot be evaluated on written

assessment. Throughout the modules, learners are asked to perform tasks verbally,

but they are unable to get instant feedback on how well or poorly they performed.

Continuing to complete practice items without exactly knowing whether or not

they are making progress in the right direction will likely to frustrate learners.

16

Therefore, the biggest obstacle to completing the modules was developing an

adequate method to check learners’ understanding of the contents.

As a solution to this problem, learners are asked before each practice item

to have a recording device ready and record their verbal performance. For

example, in memory exercise module, learners are reminded of using a recording

device to record their performance before they start practicing an item. Then, an

audio is played for learners to listen and recall from memory. After each practice

item, the written text is shown on the screen, so that learners can play their audio

and check for content accuracy as they compare the audio against the text

provided. The recorded audio can also be sent electronically to instructors who

will then provide valuable feedback and comments electronically. To further

enhance this self-assessment process, learners are also provided after each

exercise an example of best practice demonstrated by a professional interpreter in

a text format. Learners can compare their own answer to the example and make

more constructive and detailed assessment of their own performance.

In order for learning to be successful for students, it is necessary for the

learning modules to make sure that students record their performance. The chance

of using an audio recording device will increase if a recording function is

seamlessly incorporated into the modules, but within the current scope of the

project and limited resources and skills, such a solution is unattainable. Instead,

learners are constantly reminded for each practice item to record their own

performance, and they are provided with example answers to make self-

assessment.

5. ResourcesResources needed to complete the project are listed below. Most hardware

equipments and software applications including Adobe Captivate 5, the main tool

17

used to create modules, have been either purchased already or available freely

online. The only device that was needed for purchase was Livescribe Smartpen

and specially designed dot paper to be used along with the Smartpen. Instead of

purchasing it, this costly device ($200 +) was rented from the CSUMB’s

TechRent store at $5 per week.

Other financial costs include monthly subscription fee of $25 for

Lynda.com, the website that provides extensive tutorials and technical supports

for various software applications including Captivate and Dreamweaver.

Although there are many free software tutorials available online, Lynda’s tutorials

are of more superior quality than any other free tutorials available on the Internet,

and their tutorials provide in-depth knowledge about software in a systematic

manner. A total of three-month subscription was needed to learn these two pieces

of software to complete the project.

Additional miscellaneous costs of $60 were incurred from buying small

gift items for testers who participated in the time-consuming testing session. A

total of six testers participated and each was given a box of candies as a token of

appreciation for their time. In total, $145 was spent on the completion of the

project.

In addition to financial resources, the project required the use of

interpretation lab for video recording, as well as a classroom for conducting

usability testing. These facilities were reserved from the Monterey Institute of

International Studies where the designer worked as adjunct faculty. The

interpreting lab was used for four hours for video recording, and a classroom was

used for tests over the period of three weeks in May 2010.

Hardware

Computer

Camcorder

18

Tripod

Livescribe Smartpen

Dot paper for Smartpen

High performance headset

Introduction to Conference Interpretation (Reference book, by

Jeonghwa Choi)

Software

Adobe Captivate 5

Adobe Dreamweaver

Microsoft Word 2007

Microsoft Excel 2007

Microsoft Windows Moviemaker 7

Livescribe Desktop Application for handwriting recognition

File format converter software

Native speakers of English (for voice recording of English texts)

6. Timeline

Key Milestone Duration/Completed by

Gather bilingual texts and contents materials for prototype 60 days/ Feb 15, 2011

Prototype storyboarding 25 days/ Mar 10, 2011

Develop a prototype module 15 days / Mar 30, 2011

Recruit testers and arrange testing

Prepare pre-test/post-test materials.

Prepare observation/evaluation rubric

20 days / Apr 8, 2011

Perform usability testing on prototype 24 days / May 2, 2011

Make revisions 15 days / May 20, 2011

Develop subsequent modules 60 days / July 25, 2011

Develop capstone website 40 days / Aug 17, 2011

19

Beta-testing/expert reviews 20 days / Aug 20, 2011

Final revisions 40 days / Dec 7, 2011

7. Evaluation/Testing ResultsThe tryout was intended for testing and evaluating one of the four modules

planned for the capstone project. For this tryout, the module for building short-

term memory skill was used. Through observations, pre- and post-tests, and

questionnaire, the tryout measured the usability of the module, learning gains

resulting from the instruction, and users’ reaction to it. The tryout also helped the

developer understand areas needed improvement in terms of clarity, efficacy, and

enjoyability of the instruction.

The prototype used for evaluation was one of the four capstone modules,

which was designed to help learners improve memory retention. It briefly

explains the importance of short-term memory to interpreting, presents a

demonstration, and provides four practice items of varying difficulty levels for

learners to complete on their own. All the practice items included in the module

are mono-lingual exercises that ask learners to listen to an English audio and

render it in English.

For this tryout, however, learners did not necessarily need to be bilingual

speakers. The prototype only comprised English to English mono-lingual

exercises, and therefore, the usability of it could be successfully tested on test

subjects regardless of their foreign language proficiency as described above. At

the same time, test subjects did not need to be military linguists either. As a result,

the criteria used for test user selection in this tryout were; 1) native speakers of

English, 2) no prior experience of interpreting education, 3) basic familiarity with

computer usage, and 4) 18 years of age or older.

A total of six test users were recruited for the purpose of the tryout, all of

whom were native speakers of English. Two of them were military language

20

instructors (MLI) at the DLI, three were housewives who were Carmel

Highschool parents, and one was a student at the Monterey Institute of

International Studies. None of the six test subjects were familiar with the

interpreting mechanism.

The data collection included three parts; 1) pre- and post-tests to measure

learning, 2) observations, 3) survey questionnaire to measure reaction.

1) Pre- and post-tests: An audio of 211-word English passage

(appendix A) was prepared as the pre-test to measure learner’s entry

skill. Before starting the module, the audio was played to the

learners who were then asked to recall what they heard from

memory. Learners’ output was recorded for analysis. The last

practice exercise of the module was used as the post-test item

(appendix B) to measure learning gains. The length of the post-test

passage (209 words) was equivalent to the pre-test passage. Pre- and

post-test items comprised different passages but were carefully

designed to be of the same difficulty level and of the same length.

Use of the same passage for both tests was avoided because of the

concern that learners might be achieving higher scores in the post-

test due to memory effect. Post-test learner performance was also

recorded.

2) Observation: The main mode of evaluation used for this tryout was

one-on-one observation of how a potential user was actually using

the module. Test participants were contacted and arranged for

individual tryout which took place during a three-week period from

April 14 through May 2. Time spent on observation ranged from 35

to 50 minutes. Instructor intervention was kept at minimum

throughout the observation process. At the beginning, learners were

briefed on what the instruction was about, and the purpose of

21

testing. They were not given any further information than what

would be given to actual users who will be using this self-paced

module in real-life situations. For testing’s sake, learners were

recommended to complete the module in whole, but were told to do

whatever they felt necessary. When learners had a question in the

middle of the learning process, the developer first checked what

their understanding was before answering the question. The

developer used the observation checklist (appendix F) during

observation.

3) Questionnaire

Participants were asked to fill out a short online questionnaire

(appendix G) that was designed to measure efficacy and clarity of

the instruction, as well as how they liked the module. Online

questionnaire by Google Form was used to avoid bias caused by

lack of anonymity, e.g. responding to questions at the presence of

the developer.

In order to perform tryout in as much the same environment as the one in

which potential learners will normally use the module, participants were asked to

bring their own laptop. The module was provided to the participants in a USB

thumb drive. In case participants’ computers did not support sound recording, e.g.

no sound card, a separate audio recorder was prepared by the developer. Because

it involved audio playing and recording in most part, tryout could not be done in a

public place. One of MIIS (Monterey Institute of International Studies)

classrooms was used for tryouts.

Analysis of pre- and post-test results (appendix D) showed that there was a

statistically significant difference between the two sets of scores, indicating that

the modules are effective in teaching the basic interpretation skills to the target

audience. Learners’ pre- and post-test results were graded based on the grading

22

rubric (appendix C), and then paired two sample t-test was performed to see if the

instruction contributed significantly to improved memory retention skill. The

absolute value of t-stat was greater than the t critical value (|t|=3.663 > 2.015) and

it was concluded that the instructional module resulted in learning gains.

Although only a single passage was used for pre- and post-test respectively, test

reliability was not compromised significantly because each passage was divided

into 15 meaning units, each of which was graded as a separate test item (4 points

per each meaning unit). Even if passages used for pre- and post-tests were

selected very carefully to ensure they were at the same difficulty level, even a

slight gap in difficulty level between the two could lead to improved mean scores

from pre-test to post-test (67.8 to 76.3).

When it came to learners’ reaction, analysis of survey responses (appendix

G) demonstrated that learners considered the instruction as effective overall. On

average, test users evaluated the module’s contribution to learning at 3.5 on a 5

point scale. Learners rated very positively (average 4.2 on a 5 point scale) their

own level of confidence with the learned skill after instruction.

In terms of clarity of instruction, learners rated the module at 3.6 out of 5.

When asked whether or not they experienced any difficulty, all learners answered

they experienced some degree of frustration, where 67% managed to figure out

the solution on their own and the remaining 33% asked the developer for

clarification. This result was congruent with the result of observation during

which two participants asked questions for further explanation.

While learners considered instruction to be contributing to learning

overall, they did not enjoy the instruction (2.8 points on average on a 5 point

scale). The reasons for the instruction being ‘effective but not enjoyable’ could be

inferred from learners’ responses to open-ended questions that asked likes,

dislikes and suggestions for improvement. Out of total 9 responses in

dislike/improvement categories, 5 responses were about the lack of dynamism,

23

lack of video/music/photos, and lack of fun elements. Learners also mentioned a

need for clearer instruction for recording (3 responses), more practice items (2

responses), and a more visible menu (2 response). Some minor comments

included grammar/pronunciation errors.

Based on the above findings, the following recommendations were

provided:

Include more multimedia elements to avoid the module from being boring

or monotounous: It is suggested that the module actively incorporates various

multimedia elements including music, video and photos to keep the instruction

more lively and dynamic, and have learners more engaged. Currently, it has only

several static images that are not sufficient enough for the instruction to be called

‘multi-media interactive learning’. Instead of simply showing a photo for each

exercise, a related video or a slideshow of photos can be used.

Include more number of practice items in varying level of difficulty: At

present, the whole module has only 4 practice items – 2 for each level. It is

strongly recommended that the module provides wider options for learners of

varying skill levels. Instead of having only two levels – easier and more

challenging – it is advisable that practice items are organized into more number of

levels. Also, the level of difficulty has to be carefully adjusted to make sure lower

level items are easier than higher level ones.

Provide clearer instruction regarding audio recording: As it was

confirmed by both observation and survey responses, learners experienced

confusion – though in varying degree – as to how audio recording of their own

performance works in the module. Observation of test users indicated that 5 out of

6 were not using audio recording as it was originally intended by the developer.

They were able to use it correctly as they moved on, but initial frustration was

obvious. It is advisable that a clearer and more articulate instruction on voice

24

recording is provided in the module so that learners understand clearly why they

need to record their output and how they are going to use the recording.

Although the tryout process might have had faults and limitations, the

findings obtained through the process were very useful and beneficial for

improvement of the instructional module. For this tryout, only 6 testers were used

due to the significant amount of dedicated time required from each tester (an hour

and half). For future testing, however, more number of testers are desired to

further improve the reliability of the findings.

8. ConclusionInterpretation learning modules are developed to provide efficient self-

paced learning opportunities for prospective interpretation trainees – especially

military linguists at the Defense Language Institute (DLI). Language

interpretation curriculums require students to learn and acquire basic skills at the

beginning of curriculums. These skills are considered key foundation for

successful interpreting, and thus, students are required to obtain the skills through

repeated practicing. The interpreting learning modules are designed to enhance

students’ understanding of interpreting mechanics, and to provide step-by-step

instruction on what and how to practice to acquire each skill.

The project is meaningful in that it assists DLI instructors, who haven’t

had interpreting education or training, in their teaching of interpreting – the

subject they are not familiar with. Not only they serve as reliable teaching

resources, they provide flexibility in learning and an alternative method to a

human practice partner. While interpreting practice usually requires verbal

interaction with human partners who will provide useful feedback to each other,

the modules allow learners to practice repeatedly on their own without depending

entirely on human practice partners. Shy learners, in particular, are expected to

25

benefit greatly from these modules as they can build their skills using these

modules until they feel confident about performing verbal tasks in front of other

people.

Analysis of pre-& post-tests, observations and surveys support that not

only learners demonstrated significant learning gains, but they also self-reported

positive impact of the instruction on learning. Learners in general benefited from

the instruction to improve their memory retention skills.

The experience of designing and developing the Capstone project over the

period of 11 months was the process of learning some invaluable lessons about

instruction.

First, similarly as in instructor-led traditional classroom settings, web-

based instruction requires well-laid out plans at early stage of design process.

Hastily setting to work on development without thoroughly reviewed and

finalized plans will likely to result in inefficient use of time as it could be wasted

on having to make changes and reflecting them to already produced modules.

Building a fully tested and finalized prototype is a very time-consuming process,

and therefore, designers, who want to make progress with their project, may

easily become impatient at this step. However, it is worthwhile to take sufficient

time at this stage to have a solid prototype ready because this will significantly

save time later with subsequent modules.

Second, it is crucial for designers to look at their instructional modules

from learners’ point of view, not from designers’. In the intense process of

development, however, designers often think only about what the instruction

should look like based on their insight, not what actual learners may feel about it.

Learners, one of the most important players and elements in instructional design,

are often forgotten amongst the bells and whistles of designing process. Only after

the prototype is tested on prospective learners, designers may belatedly realize the

26

huge gap between what they originally envisioned and what learners are actually

experiencing.

Lastly, instruction requires elements that effectively keep learners

engaged. This is especially true to web-based learning circumstance where, in the

absence of an instructor or facilitator, learners become solely responsible for their

own learning. Where possible, instructional designers need to strive to incorporate

interactive learning elements in their instruction to encourage ‘action’ from

learners. This will keep learners focused throughout the instruction and motivate

them to continue learning.

Due to its limited scope, the Capstone project covers the very basic

aspects of interpreting with only limited number of exercise items. Therefore,

further development efforts will be focused on producing additional exercise

items in many difficulty levels to meet varying needs of learners at different

proficiency levels. With ample exercise items, learners will be given more

chances to practice and build skills needed for interpreting.

ferences

27

Hartley, A., Mason, I., Peng, G., & Perez, I. (2003). Peer and self-assessment in

conference interpreter training. Retrieved from:

http://www.lang.ltsn.ac.uk/prf.aspx#lang1

Sandrelli, A. (2005). Designing CAIT(Computer-Assisted Interpreter Training)

tools: Black Box. Paper presented at MuTra Conference on Challenges of

Multidimensional Translation. Saarbrücken, Germany.

Sandrelli, A., & Hawkins, J. (2006). From Black Box to the Virtual Interpreting

Environment (VIE): Another step in the development of Computer

Assisted Interpreter Training. Paper presented at the Conference on the

Future of Conference Interpreting: Training, Technology and Research.

London, UK.

Appendices

28

A. Pre-test passage

San Bruno Gas Explosion ( 211 words)

A huge PG&E gas line explosion and fire rocked San Bruno Thursday

night around 6 pm PST destroying at least 55 homes and damaging dozens

of others. The death toll stood at 4 Friday evening, with 52 people,

including 4 firefighters, treated for smoke inhalation and burn wounds.

The most critical were taken to the premier burn center in the Bay Area.

Officials did not release victims' names. However, Agustin Macedo of San

Francisco told the Associated Press his 44-year-old daughter and his 7-

year-old granddaughter were killed. The explosion happened near Skyline

Boulevard and San Bruno Avenue. A huge crater was left at the initial

explosion site and a huge pipe was seen nearby on a flattened street.

PG&E President said a 30-inch steel gas pipe ruptured about 3 feet

underground.

A woman who lives in the neighborhood said the blast quickly escalated

down her street, destroying one house at a time in quick succession. She

said she got out of her house with her two cars and her pets. Other

neighbors said they saw the street rip apart and ran for their lives from a

huge wall of flames. The explosion was followed by a hail of asphalt

falling from the sky as the pipe line ripped through the street. 

B. Post-test passage

29

Trapped Chilean Miners (209 words)

After more than two months trapped underground in a collapsed mine, 33

Chilean miners were pulled to safety. The men were buried alive in a

copper-and-gold mine 2,000 feet below ground in northern Chile. They

spent 69 days mostly in a dark, small space—about the size of a one-room

apartment—with temperatures around 90°F. So how did they survive the

difficult conditions?

The first couple of weeks were reportedly the most challenging for the

trapped miners. During this time, the miners had to survive on very limited

supplies. They had only enough food to last two days. Each miner allowed

himself two spoonfuls of tuna, half a cookie, and half a glass of milk every

two days. It was not until 17 days after the cave-in on August 5 that the

group made contact with rescuers on the outside.

On August 22, rescuers above ground discovered the men. They broke

through the ceiling of the miners' shelter with a long, narrow drill. The

miners attached a message to send back to the rescuers, letting them know

they were all alive. Soon after, the rescuers began sending the miners

supplies through narrow plastic tubes. These items included camping cots,

flashlights, food and water, clothing, and medicine.

C. Pre- and Post-Test Grading Rubric

30

Category Contents ScoreKey Ideas 15 meaning units 60

Details

Numbers

List of items/names/places

Other details of accident

10

10

10Order of Incidence Logical progression of story 10

Total 100

D. Pre- and Post-Test Scores

User 1 User 2 User 3 User 4 User 5 User 60

10

20

30

40

50

60

70

80

90

100

Pre-test scorePost-test score

E. T-test Paired Two Sample for Means

31

t-Test: Paired Two Sample for Means

  Pre-Test Score  Post-Test ScoreMean 67.83333333 76.33333333Variance 155.7666667 74.26666667Observations 6 6Pearson Correlation 0.919211953Hypothesized Mean Difference 0df 5t Stat -3.663475485P(T<=t) one-tail 0.007271016t Critical one-tail 2.015048372P(T<=t) two-tail 0.014542032t Critical two-tail 2.570581835

F. Observation Checklist

Checklist Yes No CommentsDoes the learner complete each and every item in the module? Does the learner follow the instruction correctly? Does the learner conduct recording for each practice item? Does the learner encounter any problem, but manage to solve it on his/her own without developer’s intervention?Does the learner encounter any problem, and need developer’s intervention to be able to continue? Does the learner navigate freely without much problem?

G. Questionnaire and Response Summary

32

H. Summary

1. Was the instruction clear enough to follow? Please rate it on a 5 point scale.

Very unclear

Very clear

1 -

Very unclear

0 0%

2 0 0%

3 2 33%

4 4 67%

5 -

Very clear 0 0%

2. How significantly doyou think the instruction contributed to memory building skill? Please rate it on a 5 point scale.

33

Very insignificant

Very significant

1 -

Very insignificant

0 0%

2 0 0%

3 4 67%

4 1 17%

5 -

Very significant 1 17%

3. Did you experience any frustration while learning with the module?

Never 0 0%

Occasionally, but I managed to figure out the solution

4 67%

Occasionally, and I had to ask for clarification 2 33%

Very often, so I couldn't move on properly. 0 0%

4. How would you describe the difficulty level of level-specific practice items?

34

Too easy overall 0 0%

Just right 3 50

%

Too challenging overall 1 17

%

Some were too easy, some were too challenging

2 33

%

5. Do you now feel more confident about recalling a passage from memory? Please rate your confidence on a 5 point scale.

Not confident at all

Very confident

1 -

Not confident at all

0 0%

2 0 0%

3 0 0%

4 5 83%

5 -

Very confident 1 17%

6. Did you enjoy learning with the module? Please rate on a 5 point scale.

35

Did not enjoy at all

Enjoyed it very much

1 -

Did not enjoy at all 0 0%

2 2 33%

3 3 50%

4 1 17%

5 -

Enjoyed it very much

0 0%

N. 7. What did you like about the instruction? Please write O. learning about a new skillIt guided me through learning step-by-step. I liked that it offered

two different levels of exercises from which I could choose. consistent screen layout. I got to know about interpreting. software was clear and simple. Easy to maneuverThe choice of passages was adequateI could move around in the program anywhere I wanted to go.

P. 8. What did you dislike about the instruction? Please write.Q. Having to record my voice (I hate listening to my own voice), but I know it helped a lot.

problems with English pronounciation (use a native speaker for a narrator???)I am a very visual and analytical learner. This verbal and aural exercise was hard without much visuals to use. I wasn't sure whether I was doing the right thing first time off with recording stuffs At first I didn't understand why I had to do the recording, but I figured that out towards the end. I think the program has not explained sufficiently about recording. I kinda moved on not knowing what I was doing.... At some p...

R. 9. Any suggestions for improvement?S. More fun elements. I wish I had more varieity of practice items. There were only 4. There

were some grammatical errors that needed correction. Clearer explanation about the instruction at the beginning. The program lacks colors/dynamism(?) as compared to many online programs thesedays. To keep people more engaged. I enjoyed the program. I am just thinking you could have more easier stuffs there so other people who are less trained can try various items. It was fine by me though. Shorten the length of training. Use of music, videos, etc... Use of more visible menu. Having the menu visibl...

36