Measuring Linguistic Gains in Immersion Settings: Empirical Studies from French and

Spanish as a Second Language

Dennis M. Wiseman

Council for International Education Exchange Annual Conference

Breaking Barriers: Cost, Curriculum, Culture Baltimore, Md

November 19-22, 2014

Wofford College

•  Four-year, private, liberals arts college •  Founded in 1854 •  Methodist affiliated •  Phi Beta Kappa •  # 4 in the Open Doors survey •  1600 students

National Survey of Student Engagement

National Survey of Student Engagement

National Survey of Student Engagement

National Survey of Student Engagement

Linguistic Gains: High Impact Practices

All French or Spanish students in these data have engaged in at least one of the following activities in study abroad. – Home stay – Language course work – Internship – Volunteer or service learning

Methodology •  OPIc •  Immediately upon return from study abroad,

set / proctored •  French students – Paris, Nantes, Rennes, Bordeaux, Aix-en-

Provence, Dijon, Arles •  Spanish students –  Spain, Argentina, Panama, Chile, Dominican

Republic, Peru, Ecuador, Bolivia, Honduras, Nicaragua

LAT. AMER.

SPAIN

0 20 40 60 80 100 120 140

LAT. AMER. SPAIN Series1 118 77

SPANISH_STUDY_ABROAD_2007_'13

0 20 40 60 80 100 120 140

LAT. AMER.

LAT. AMER DNT

SPAIN

SPAIN DNT

LAT. AMER. LAT. AMER DNT SPAIN SPAIN DNT Series1 118 16 77 29

ACTFL_OPI_DID_NOT_PARTICIPATE_#

0 5 10 15 20 25 30 35 40 45 50

AH

AM

AL

IH

IM

IL

DNT

AH AM AL IH IM IL DNT Series1 1 6 31 44 18 2 16

SPANISH_LAT._AMER._ACTFL_2007_2013_#

0 5 10 15 20 25 30 35

AH

AM

AL

IH

IM

IL

DNT

AH AM AL IH IM IL DNT Series1 1 7 25 15 29

SPANISH_SPAIN_ACTFL_2007_2013_#

0 5 10 15 20 25 30 35 40

AH

AM

AL

IH

IM

IL

DNT

AH AM AL IH IM IL DNT Series1 0.8 5.17 26.27 37.28 15.25 1.6 13.56

SPANISH_LAT._AMER._ACTFL_2007_2013_%

0 5 10 15 20 25 30 35 40

AM

AL

IH

IM

IL

DNT

AM AL IH IM IL DNT Series1 1.3 9.1 32.47 19.48 37.66

SPANISH_SPAIN_ACTFL_2007_2013_%

French_ACTFL_2007_2013

AH, 3

AM, 6

AL, 21

IH, 16

IM, 12

AH

AM

AL

IH

IM

IL

0 5 10 15 20 25

French_ACTFL_’09_’13

AH, 3

AM, 4

AL, 19

IH, 13

IM, 7

AH

AM

AL

IH

IM

IL

0 2 4 6 8 10 12 14 16 18 20

French_ACTFL_’11_’13

AH, 3

AM, 2

AL, 11

IH, 10

IM, 4

AH

AM

AL

IH

IM

IL

0 2 4 6 8 10 12

French_ACTFL_’12_’13

AH, 3

AM, 1

AL, 7

IH, 6

AH

AM

AL

IH

IM

IL

0 1 2 3 4 5 6 7 8

French_ACTFL_2013

AH, 2

AM, 1

AL, 6

IH, 1

AH

AM

AL

IH

IM

IL

0 1 2 3 4 5 6 7

French_ACTFL_Advanced_2007_2013

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

2007_2013

2009_2013

2011_2013

2012_2013

2013

Spanish_ACTFL_Advanced_2007_2013

0% 5% 10% 15% 20% 25% 30% 35%

2007_2013

2009_2013

2011_2013

2012_2013

2013

What is COPI telling us about Spanish

improvement in the study abroad context?

CIEE Conference

Baltimore, November 2014

Vera Cerqueiras, BA Spanish Coordinator

CONTENTS

• Describe the process of training, implementation and correction of the exam;

• Analyze the progress made by students between their entrance COPI and their exit COPI;

• Show some of the challenges and the subsequent effects (washback) resulting from this evaluation.

2

What is COPI (Computer based Oral Proficiency Interview)?

3

• The COPI exam is a computerized semi-adaptive exam aimed at determining the ability for using oral spanish language at a specific moment taking into account the ACTFL Guidelines.

• It is intended for high school

students, college students, and professionals whose native language is English or who have an advanced English level

• Standarized exam • Measuring linguistic production

holistically • Providing feedback to both the

test takers and their professors.

• Exam results can be used to show the progress of a student after a semester of study.

• It cannot be used to make

decisions to pass or fail Spanish courses

Characteristics

Evaluation Levels

4

1. Superior: The candidate gives opinions, hypothesizes, and uses abstract language in unfamiliar language situations 2. Advanced: The candidate narrates and describes situations with some level of complication 3. Intermediate: The candidate demonstrates creative use of the language in simple and everyday situations 4. Novice: The candidate uses fixed expressions, short phrases and lists of memorized words The three latter levels are subdivided into 3 sublevels: High: The candidate does not have the ability to reach and maintain the next level Medium: Represents the prototypical level Low: The candidate has difficulty in meeting the level requirements

Administration of the Exam

The exam has the following components:

• A CD for the administration of the COPI exam

• An application for the certification of the exam

• Lab with PC`s ,Computers, headset with microphone and Speaker, keyboards

• The candidates must select their Spanish level taking in account the summary description of each level given by the program

• The program selects from 7 to 11 tasks

• The candidates must complete the exam by recording their answers

• Teachers / Raters should be in the computer

lab to assist.

• The exam lasts between 45/60 minutes

5

Interview Evaluation and Grading and Final Report of the Evaluation

of the level reached by the candidate.

6

• The program saves the candidate’s activities.

• A double assessment is performed by two certified professors who correct all the activities completed by each candidate.

• The evaluator assigns a corresponding level for each answer he/she listens to, taking into account the level assigned by the student or the program

• The evaluator has the option to write comments about the activity, which the program then saves as part of the evaluation.

• The program establishes an overall average after the evaluator assigns a level for each of the activities performed by the candidates.

• A final report with the

certification of each candidate’s obtained level is generated by the program including the overall average and a description of the proficiency

Exam and Certification of COPI Evaluators • Evaluators are trained by using a kit provided by CAL

(Center of applied linguistics)

• They have to pass an exam according to ACTFL Guidelines

• This exam consists of 15 comprehensive COPI exams that should be rated

• Evaluators approve the exam if at least 60% (9/15) scores coincide with those indicated by the CAL experts.

7

8

9 >17%

10

11

Why COPI implementation represents a challenge?

12

• BECAUSE it implies working with isolated language samples that must be analyzed without consideration of neither its production context nor its acquisition process.

• BECAUSE the evaluator's work needs permanent training and peer feedback in order to adjust and perfect the administration, correction and considerations of the exam.

• The internalization of the practices incorporated by giving the COPI exam allow us to focus on the elaboration of evaluation instructions and on the importance that the "knowledge of the world" has towards the successful achievement of that instruction.

• The progress data between entry exams and exit exams tests the results obtained by the very same students in the evaluations of the Spanish courses they take as part of their immersion program studies. Until now they are consistent. In other words, those students who advance the most are the same students who achieve the highest grades in their Spanish courses.

Washback Effects

Effects of International

Service-Learning on the

Development of Spanish Oral

Proficiency

CIEE Conference: Breaking Barriers

November 19-22, 2014, Baltimore

Francisco Salgado-Robles, Ph.D.

The City University of New York

Overview of presentation

①  Goal ②  Community Service-Learning

a)  General notions b)  SL in higher education c)  SL in the Spanish classroom d)  SL at-home: Linguistic gains e)  SL abroad: Linguistic gains

③  Research Questions ④  Instrument ⑤  Participants ⑥  Blogging & Service-Learning ⑦  Results ⑧  Discussion ⑨  Conclusions ⑩  Challenges & Recommendations

GOAL

¡  Gain further insight into the effects of context (i.e., service-learning in an international setting) on the L2 acquisition process (i.e., the development of Spanish oral proficiency) through a case-control study analysis of participants’ experiences enrolled in a Language for Specific Purposes (LSP) course (i.e., Spanish for the working environment) during a summer semester abroad.

3

¡  SL? “Experiential education that engages students in activities that address human and community needs together with structured opportunities intentionally designed to promote student learning and development” (Jacoby, 1996)

¡  Students’ service experiences are tied to the academic content of an on-campus course or curriculum (Giles, Honnet, & Migliore, 1991)

¡  From the tenets of experiential education and constructivist theories that advocate for learning through first-hand discovery (Furco, 2001)

¡  “One of the most effective approaches for engaging students and boosting learning” (Learn and Serve America, 2011)

¡  Shape learners’ knowledge and cultural beliefs, professional aspirations, and community engagement.

SERVICE-LEARNING (SL): GENERAL NOTIONS

4

¡ Based on statistics from 2009-2010, ½ + of community colleges and approximately ¼ of all universities integrate SL in different academic curricula (Learn and Serve America, 2011)

¡ SL pedagogy has been adopted not only due to

the overwhelming amount of research in support of this type of educational experience, but also because it has been witnessed how life-changing these opportunities can be for their students (Jeandron & Robinson, 2010)

SL IN HIGHER EDUCATION

5

¡  The incorporation of SL in Spanish curricula follows similar trends observed in other disciplines in higher education

¡ Community engagement has been a “growing

presence in Spanish classes” and that there is presently “significant activity and interest within our ranks (Hellebrandt & Jorge, 2013)

¡ Rapidly expanding Hispanic population in the United

States (Rabin, 2009; Zlotkowski, 1999) ¡ Convincing evidence of the positive impact SL can

have on students taking Spanish as a second or heritage language (Petrov, 2013)

SL IN THE SPANISH CLASSROOM

6

¡ Offer students authentic communicative contexts in the target language

¡ Communicative competence improved from regular interaction with NSs after participating in SL (Barreneche, 2011; Navarro, 2012; Olazagasti-Segovia, 2004; Salgado-Robles, 2014)

¡  Facilitate linguistic improvements in the L2 (Hale, 1999; Mullaney, 1999; Over field, 1997)

¡  Spanish SL students scored higher on L2 reading, speaking, and writing tasks (Malkin, 2010)

¡  Students increased their use of Spanish outside the classroom and felt more confident speaking in Spanish (Pellettieri, 2011)

SL AT-HOME: L2 GAINS

7

¡  SL as an effective way enhance student’ language skil ls, knowledge of the host country, intercultural sensitivity, and global competence (Kiely, 2011)

¡  Students gained confidence in their Spanish skil ls,

developed intercultural skil ls (Kopischke Smith & Moreno-López, 2012)

¡  Facilitate linguistic improvements in the L2 (Cubillos,

2013) ¡  Students perceived a positive relationship between

their participation in SL projects and the development of their l inguistic and cultural proficiency (Brown & Purmensky, 2014)

8

SL ABROAD: L2 GAINS

RESEARCH QUESTIONS

①  What is the effect of a community-enhanced Spanish for Specific Purpose course on the oral proficiency at the Intermediate-level?

②  What aspects of oral proficiency are most and least influenced by the community engagement experience?

9

INSTRUMENT

10

PARTICIPANTS (CONT’D)

Experimental group

22 students (14 f / 8 m)

19-21 yrs old

3rd year Spanish

Spanish minors/majors

Spanish for the Professions

Control group

18 students (11 f / 7 m)*

19-23 yrs old

3rd year Spanish

Spanish minors/majors

Spanish for the Professions

* 1 heritage speaker removed 11

PARTICIPANTS

Experimental group Control group

12

13

BLOGGING & SPANISH FOR THE WORKING ENVIRONMENT

14

BLOGGING & SPANISH FOR THE WORKING ENVIRONMENT (CONT’D)

BLOGGING & SPANISH FOR THE WORKING ENVIRONMENT

15

ü  Pre & Post Score Comparison (Experimental Group, N=22)

RESULTS (CONT’D)

16

ü  Pre & Post Score Comparison (Control Group, N=18)

RESULTS (CONT’D)

17

RESULTS

*** High Statistical Significance

ü  Paired t-tests run to determine significance of trends over time

Pre Mean

Post Mean

p-value

Mastery sentence 43.10 50.00 0.0003*** Experimental Vocabulary 36.77 51.22 <0.0001***

Group Fluency 45.52 54.19 <0.0001***

(+SL) Pronunciation 46.18 57.30 0.0003***

Mastery sentence 43.94 46.04 0.0002***

Control Vocabulary 39.75 52.76 <0.0001***

Group Fluency 47.00 49.15 0.1320

(-SL) Pronunciation 44.60 48.63 0.0302

18

DISCUSSION

①  What is the effect of a community-enhanced Spanish for Specific Purpose course on the oral proficiency at the Intermediate-level? ü  A statistically significant impact on the proficiency level of

Intermediate-level learners. ü  Just one participant experienced L2 gains towards the

advanced-low proficiency level in this experiment.

②  What aspects of oral proficiency are most and least influenced by the community engagement experience? ü  SL had an observable and statistically significant impact on all

sub-skills of oral proficiency measured by the Versant Test.

19

CONCLUSIONS

①  The incorporation of the SL component into a Spanish for Specific Purposes course in an international setting had a significant effect on oral proficiency;

②  The learning conditions provided by the community

engagement facilitated oral skil l development across all sub-skil ls;

③  Overall, the SL pedagogy worked successfully for this LSP course and, thus, should be further promoted, since:

ü  Provide more opportunities for L2 use in meaningful ways; ü  Practice different L2 skills: speaking, reading, writing, listening, cultural understanding, etc.; ü Debrief and reflect on SL experience with others; ü  Introduction to the professional field in a hands-on way; ü Give more confidence to students when they are in their services.

20

q  Challenges: §  Time and effort: More so than the ‘normal’ course; §  Technological problems and conquering the learning curve; §  Lots of smaller tasks (visit report via Google Docs; blog entry

and comments; in-class led discussions); some students had a hard time keeping up with due dates.

q  Recommendations:

§ Cooperative efforts among researchers, students, instructors, administrators, and community supervisors are plainly needed;

§  The development and dissemination of SL in LSP pedagogic materials are warranted;

§ More large-scale studies and longitudinal assessment of L2 acquisition (both qualitative and quantitative) within the SL (and LSP) contexts would help to further understand a new strand of experiential learning (Kolb, 1984).

CHALLENGES & RECOMMENDATIONS

21

ACKNOWLEDGEMENTS

² Department of Hispanic Studies, U of Kentucky

² Service-Learning Council, U of Kentucky

² Education Abroad Office, U of Kentucky

² Spanish Studies Abroad, Seville, Spain

² Non-profit organizations

² 40 students

THANK YOU!

QUESTIONS? COMMENTS?