DataDrivenLanguageLearning:** LearnerProficiency ... · DataDrivenLanguageLearning:**...
52
DataDriven Language Learning: Learner Proficiency, Performance, and Percep9on Nina Vyatkina University of Kansas Humboldt-Universität zu Berlin December 3, 2014
Transcript of DataDrivenLanguageLearning:** LearnerProficiency ... · DataDrivenLanguageLearning:**...
Data-‐Driven Language Learning: Learner Proficiency, Performance, and
Percep9on
Nina Vyatkina University of Kansas
Humboldt-Universität zu Berlin December 3, 2014
Overview
• Results of two empirical studies that explored the effects of Data-‐Driven Learning (DDL) of German lexico-‐gramma?cal construc?ons (verb-‐preposi?on colloca?ons) by North American university students
2
Corpus research and L2 teaching and learning
3
applied corpus research
direct applica?ons: DDL
computer-‐based (hands-‐on)
paper-‐based (hands-‐off)
indirect applica?ons: grammars and textbooks
Theore?cal background: Usage-‐Based Grammar (UBG)
• language is learned induc?vely through repeated exposure to and prac?ce with specific language models
• Langacker 1987; Robinson & Ellis 2008; Bybee 2008 • inseparability of grammar and the lexicon
• Conrad, 2000; Chambers, 2005; Flowerdew, 2011
• primacy of target language input • Krashen 1982; Doughty & Long 2003; Schmidt 1990; Sharwood Smith 1993
4
DDL pedagogical applica?ons • Benefits:
– work with aZested usage examples (Jones, 1986; 1991) – induc?ve (discovery) learning (Bernardini, 2000; 2002; Ellis, 2002;
Richards et al., 1992; Thornbury, 2006) – developing analy?cal and problem-‐solving skills (Johns, 1991, 1997;
Qiao & Sussex, 1996; Kennedy & Miceli, 2002)
• Challenges: – Language learners are not expert corpus users
• “most widely accessible corpora were created as tools for linguis?c research and not with pedagogical goals in mind” (Brown, 2007)
– Corpus examples are ‘genuine’ but not ‘authen?c’ for language learners (Widdowson, 2000; 2003)
5
DDL research • DDL effec?ve for L2 learning (vocabulary, colloca?ons,
improving student wri?ng) – e.g. Horst, Cobb & Nicolae 2005; Lin 2008; Sun & Wang 2003; Yoon 2008
• Hands-‐on DDL superior or equivalent to tradi?onal instruc?on – e.g. Cobb 1997, 1999; Daskalowska 2013; Garner 2013; Kaur & Hegelheimer 2005; Liou et al. 2012
• Hands-‐off DDL superior or equivalent to tradi?onal instruc?on – Allan 2006; Boulton 2008, 2009, 2010; Koosha & Jafarpour 2006, Tian 2005
6
Does DDL work?
• Cobb & Boulton (in press): meta-‐analysis – up to 2012, further analysis currently ongoing
• Is DDL effec?ve? – Within groups effect size: 1.68 SD units
• SD d = .84, 95% CI = [1.36, 2.00]
• Is DDL efficient? – Within groups effect size: 1.04 SD units
• SD d = .73, 95% CI = [.83, 1.25]
• => high ES (Oswald & Plonsky 2010) 7
Research gaps • morpho-‐syntac?c features • non-‐English • correla?ons with proficiency and percep?ons • hands-‐on vs. hands-‐off • delayed effects • low L2 proficiency levels • applica?ons with ‘real’ teachers • new vs. previously learned language features
8
Target structure: verb-‐preposi?on colloca?ons
• some verbs subcategorize preposi?onal arguments • form-‐meaning mismatches lead to L2 errors
– to wait for = warten auf (not für) • e.g. Nesselhauf 2003
• German pronominal adverbs extremely frequent – davon, dabei -‐ thereof, thereby…
• either the verb or the preposi?on also assigns gramma?cal case => a lexico-‐gramma9cal construc9on
• warten + auf + accusa?ve NP • Difficult construc?ons for L2 German learners
• e.g. Baten 2011, Diehl et al. 2000, Vinagre and Muñoz 2011
9
Instruc?onal sewng
• large public US university • intact classes of German as a Foreign Language • age: 21 mean (range 18-‐35) • L1: American English • gender: approx. equal # female and male • no familiarity with corpora
10
Study 1
• Short-‐term and delayed effects of two DDL methods for teaching L2 colloca?ons: – computer-‐based (hands-‐on) – paper-‐based (hands-‐off)
• Correla?on of learners’ performance with their L2 proficiency and DDL recep?vity (percep?ons)
11
Par?al replica?on of Boulton (2012)
• DDL of English verb structures by high-‐intermediate college-‐level ESL students (L1 French)
• hands-‐on and hands-‐off DDL equally effec?ve • only hands-‐off results correlated with proficiency
• correla?on with recep?vity posi?ve but not significant
12
Par?cipants and instruc?onal sewng
• 9 German majors and minors, 1 High School student • Enrolled in 16-‐week-‐long ‘Advanced German’ course • L2 proficiency: B1 (CEFR) average • Course goals:
– development of L2 proficiency AND corpus literacy: the ability to use corpora for consulta?on and language analysis (Mukherjee, 2002)
– regular hands-‐on and hands-‐off DDL ac?vi?es in addi?on to tradi?onal extensive reading, discussion, and gramma?cal analysis
13
DDL instruc?on
• DWDS core corpus of the 20th cent. German (www.dwds.de); 100 M words
• Biweekly computer lab mee?ngs, in-‐class and homework corpus-‐based assignments
• Teaching method: – DDL “TRIPLE I” principle: Illustra?on-‐Interac?on-‐Induc?on (McCarthy
and Carter 1995) – Focus on forms encountered by learners during reading the course
novel: top-‐down, text-‐to-‐corpus approach (Charles 2007; Frankenberg-‐Garcia 2012; Tyne 2012)
– The en?re text of the novel is archived in DWDS
• Study: 20 verb-‐preposi?on colloca?ons from the novel 14
The course novel
15
Example worksheet
16
Research ques?ons 1. Did learner wriZen performance on the gap-‐filling task and the
sentence-‐wri?ng task improve following DDL instruc?on and were the gains retained a month later?
2. Which DDL method was more effec?ve: hands-‐on or hands-‐off? 3. Did performance outcomes correlate with learner proficiency? 4. Did learners prefer hands-‐on or hands-‐off DDL, did their
preferences change during the course, and did their preferences correspond with their performance and proficiency?
• Modifica?on of Boulton 2012
17
Data collec?on • All students: hands-‐on and hands-‐off DDL condi?on • Language background ques?onnaire • Performance: pretest – post-‐test -‐ delayed post-‐test
– 10 colloca?ons per condi?on – gap-‐filling and sentence-‐wri?ng test tasks – 10 possible pts per test / condi?on / task
• Proficiency: pre-‐course – post-‐course – standardized online diagnos?c test (www.ondaf.de)
• Recep?vity: pre-‐course – post-‐course – wriZen ques?onnaires
18
Aug. 28 (SW1) Nov. 6 (SW11) Nov. 11 (SW12) Dec. 4 (SW15) Dec. 9 (SW16)
Proficiency and receptivity pretest; language background questionnaire
Performance hands-on: pretest, treatment, and posttest
Performance hands-off: pretest, treatment, and posttest
Proficiency and receptivity posttest
Performance: delayed posttest
Gap-‐filling test task
19
Sentence-‐wri?ng test task
20
Performance outcomes: raw scores
21
0
1
2
3
4
5
6
7
8
9
10
pretest posZest delayed
On-‐gap
Off-‐gap
On-‐sentence
Off-‐sentence
Methods
• Raw test scores – abnormally distributed ⇒ mul?level Poisson regression (z-‐tests)
• Gain scores – normally distributed ⇒ mul?level linear regression (t-‐tests)
22
Performance outcomes by task • Pretest-‐PosZest:
– gap: z = 4.94*, p < .0001 (+1.97) – sentence: z = 3.52*, p = .0004 (+1.71)
• Pretest-‐Delayed PosZest: – gap: z = 2.05*, p = .04 (+1.35) – sentence: z = 0.91, p = .36 (+1.16 )
• PosZest-‐Delayed PosZest: – gap: z = −3.01*, p = .004 – sentence: z = −2.65*, p = .01
23
Performance outcomes: gains by condi?on
24
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Hands-‐off Hands-‐on
immediate gap
immediate sentence
delayed gap
delayed sentence
Interaction between conditions not significant: Immediate gains: t(34) = −0.80, p = .43 Delayed gains: t(34) = 0.47, p = .64
Proficiency outcomes
onDaF Test
N Mean Min Max SD N in CEFR bands
A2 B1 B2
Pre-course
9 73.78 (B1.1)
36 115 26.88 3 4 2
Post-course
10 80.70 (B1.2)
48 118 22.94 1 6 3
25
Correla?ons (r) between DDL performance and proficiency
measure N method pretest immediate posttest
delayed posttest
total
Gains 9 hands-on N/A 0.30 -0.16 N/A
hands-off N/A 0.70a 0.58 N/A
26
a: two-tailed t-test: p=.046 (t=2.37; df=8)
Recep?vity outcomes
Recep9vity to corpus use Pre-course Post-course
Mean (1-5)
SD Mean (1-5)
SD
1 The German corpus (DWDS) will be / has been easy to use for language learning purposes.
3.7 0.82 3.7 0.82 2 The German corpus (DWDS) will be / has been useful for
learning German. 3.6 0.96 3.9 1.10
3 Corpus work will be / has been interesting. 3.4 1.08 4.1 1.20 4 I liked doing corpus activities on computer. N/A N/A 3.5 1.43 5 I liked doing activities with corpus concordances on paper. N/A N/A 3.8 1.14
Overall mean: 3.57 0.13 3.8 0.22
27
Individual recep?vity
28
0
1
2
3
4
5
6
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10
Precourse
Postcourse
Recep?vity: correla?ons • Strong, significant:
– hands-‐on – hands-‐off recep?vity p=.03 (r=0.68, t(8)=3.0) – overall recep?vity – hands-‐on recep?vity p=.001 (r=0.87, t(8)=5.8) – overall recep?vity – hands-‐off recep?vity p=.003 (r=0.83, t(8)=4.27)
• Moderate but not significant – pretest – posZest recep?vity (r=0.48, t(8)=1.57) – recep?vity – proficiency (r=0.46, t(8)=1.46)
• Weak, not significant: – overall recep?vity – performance, both posZests – hands-‐on recep?vity – performance, both posZests – hands-‐off recep?vity – performance, immediate posZest
• (moderate, not significant for the hands-‐off condi?on on the delayed posZest)
29
Study 1 results: summary • Did learner wriZen performance on the gap-‐filling task and the
sentence-‐wri?ng task improve following DDL instruc?on? -‐> Yes • Were the gains retained a month later? -‐> Yes for gap-‐filling results • Which DDL method was more effec?ve: hands-‐on or hands-‐off? -‐> No difference • Did performance gains correlate with overall proficiency gains? -‐> Yes for hands-‐off, no for hands-‐on • Did preferences correspond with performance and proficiency? -‐> No
30
Study 2
• Comparison of short-‐term effects of a paper-‐based DDL method and a non-‐DDL method for teaching L2 colloca?ons to low-‐proficiency learners – lexical vs. syntac?c gains – new vs. previously learned colloca?ons
31
Par?cipants and instruc?onal sewng
• Intact classes of intermediate-‐low (3rd semester) and intermediate-‐mid (4th semester) German
• L2 proficiency: < or = A2 (CEFR) • Mul?-‐sec?on program, uniform syllabus • Students enrolled to fulfill a language requirement
32
Design • 4 groups: LD (n=15); LT (n=13); MD (n=13); MT (n=16)
– Instruc?onal level (“proficiency”): L (low) and M (mid) – Treatment condi?ons: D (DDL) and T (textbook-‐based)
• Instruc?on: – 10 verb-‐preposi?on colloca?ons from regular textbook – D: III (Illustra?on-‐Interac?on-‐Induc?on, McCarthy & Carter 1995) – T: PPP (Presenta?on-‐Prac?ce-‐Produc?on, DeKeyser 1998)
• Timeline: – day 1: pretest (10 min.), language background ques?onnaire – day 2: instruc?on (40 min.), posZest (10 min.)
• Only gap-‐filling produc?on data: – 10 possible pts for accurate lexical items (preposi?ons) – 5 possible pts for accurate syntac?c items (ar?cle inflec?on)
33
DDL worksheet I
34
DDL worksheet II
35
Test task I
36
Test task II
37
Research ques?ons 1. Did learner wriZen performance on the gap-‐filling task
improve following focused instruc?on? 2. Were the gains higher for the DDL or the non-‐DDL group? 3. Was there interac?on among the factors treatment, course
level, and linguis?c items (lexical or syntac?c)?
38
Lexical gains
39
0
1
2
3
4
5
6
7
pretest posZest
MT
MD
LT
LD
Syntac?c gains
40
0
0.5
1
1.5
2
2.5
3
pretest posZest
MT
MD
LT
LD
Results • Overall gains:
– lexical: z = 6.14*, p < .0001 (by 1.94) – syntac?c: z = 3.63*, p = .0002 (by 1.87)
• DL beZer than TL: – lexical: t(104) = -‐2.39*, p = .02 (by 1.66) – syntac?c: no difference
• DM vs. TM: – lexical: no difference – syntac?c: no difference
41
Study 2b: DL (n=16), TL (n=15) Lexical gains
42
0
0.5
1
1.5
2
2.5
3
pretest posZest
TL
DL
Study 2b: DL (n=16), TL (n=15) Syntac?c gains
43
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
pretest posZest
TL
DL
Results • Pretest:
– lexical: no difference between groups – syntac?c: no difference between groups
• Overall gains: – lexical: z = 4.13*, p < .0001 (by 3.16) – syntac?c: z = 3.66*, p = .0002 (by 4.63)
• DL beZer than TL: – lexical: t(42) = 3.81*, p = .0004 (by 1.73) – syntac?c: t(42) = 2.58*, p = .01 (by 1.17)
44
Study 2 results: summary • Did learner wriZen performance on the gap-‐filling task
improve following focused instruc?on? -‐> Yes • Were the gains higher for the DDL than the non-‐DDL group?
Was there interac?on among the factors treatment, course level, and linguis?c items (lexical or syntac?c)?
-‐> DDL was more effec?ve than non-‐DDL only for the lower-‐level group (for learning new colloca?ons) -‐> DDL advantage retained on a delayed posZest
45
Scores for the same 10 colloca?ons across 3 course levels
46
0
2
4
6
8
10
12
14
Pretest PosZest
DH
DM
TM
DL
TL
Study 1 revisited: Learner comments
47
Study 1 revisited: Learner comments (cont.)
48
…a coda
• all five students who con?nued in the next semester advanced German course (with no explicit focus on corpora) reported having used it as a reference resource independently throughout the course.
49
Conclusion • DDL effec?ve for teaching colloca?ons and well received by learners
= Boulton 2012; Cobb 1997, 1999; Daskalowska 2013; Liou et al. 2012
• can be applied: – to languages other than English – to linguis?c targets beyond vocabulary (morpho-‐syntax) – as regular course component and one-‐?me interven?ons – with regular teachers
• lexical gains higher than morpho-‐syntac?c gains • both hands-‐on and hands-‐off DDL is effec?ve and can be alternated to
teach structures resistant to tradi?onal teaching methods • hands-‐on method equally beneficial for faster and slower learners • more efficient than non-‐DDL for learning new structures
– vs. Nesselhauf 2004
• benefits of guided induc?on and consistent prac?ce -‐> learner autonomy
50
Future research • More replica?ons • Longitudinal study • Singling out DDL effects
– input richness and enhancement (Frankenberg-‐Garcia 2014) – guided and unguided induc?on (Smart 2014) – learner corpus in DDL (Cotos 2014)
• Hands-‐on for lower-‐level learners • More aZen?on to morpho-‐syntax • Transfer to wri?ng and other free produc?on • More integra?on into regular L2 syllabi
51
Acknowledgements
• Terrence D. Jorgensen Jr. for help with sta?s?cal analysis
• The department of Germanic Languages and Literatures, University of Kansas
52
