Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied...

53
Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging replication research in the field of AL and SLA,” Graeme Porte convener. The American Association for Applied Linguistics Annual Conference, Denver CO, March 23. Copyright © Lourdes Ortega, 2009

Transcript of Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied...

Page 1: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Please cite as:

Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging replication research in the field of AL and SLA,” Graeme Porte convener. The American Association for Applied Linguistics Annual Conference, Denver CO, March 23.

Copyright © Lourdes Ortega, 2009

Page 2: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

A Sociology of Replication and Replicability in Applied

Linguistics

Lourdes OrtegaUniversity of Hawai‘i at Mānoa

AAAL colloquium, Graeme Porte ConvenerDenver CO, March 23, 2009

Page 3: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Sociology of knowledge: “study of the relationship between human thought and the social context within which it arises” (Wikipedia)Replicability: conditions that improve a field’s ability to support replication activities (McCullough, 2007, inter alii)

Page 4: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

In art,as in

science...

Page 5: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“Original”

1656

Page 6: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“Replica/tion”

1950s

Page 7: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Fidelity? Value?

Page 8: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

FidelityVerification

ValueWorthy Knowledge,

Worthy Purpose

In science, as in art...

Page 9: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Difference…?Value…?

contested understandings…

Page 10: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“how different/similar”?

Page 11: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Rosetta Stone (c. 203 BC) in British Museum… these

Rosetta Stone bookends?

Page 12: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

uncertain understandings of

“replication”…

Page 13: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Polio & Gass (1997):

“many more studies than claim to be are actually some type of replication […] clearly, a fine line exists between replication and extending research”

(p. 501)

Page 14: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Neuliep & Crandall (1993):

Everyone was wrong: There are lots of replications out there. Journal of Social

Behavior and Personality, 8(6), 1-8.

As much as 78% of studies were (conceptual) replications that were not labeled by authors as such.

Page 15: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Norms for replication

Incentives for replicability

Sociological perspective:

Page 16: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Replication, the norm

Page 17: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Based on Rosenthal (1991)

Supports the theory (& investigator)

Damages the theory (& impugns investigator)

Internal validity,results easy to interpret, no “novelty”

Extends the theory (& supports investigator)

Limits the theory (& impugns the investigator very little)

External validity, greater “novelty,” any difference in results difficult to interpret

Result of replication

Same Different

Value

Exact

Conceptual,etc

TYPEofREPLICATION

“doing it again” “contributing to

a valued dialogue”

“doing it right”

Page 18: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Replication as dialogue

1998, TQ 2001, Chapter 2006, LL

Longitudinal component

Validation of instrumentvia L1 baseline & interviews

RQs & focus, instrument, analysesEFL sample from a

purposefully differentpopulation

Original:

Bardovi-Harlig & Dörnyei

Replication +1:

Niezgoda & Roever

Replication +1+ extension:

Schauer

Different:

Same:

Page 19: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Verification of findings?

ESL (n=173, Indiana, college):pragmatic > grammaticalEFL (n=370, Hungary, coll & hs):pragmatic < grammatical

Barvodi-Harlig &Dörnyei (1998)

ESL (n=48, Hawaii, college):pragmatic > grammaticalEFL (n=124, Czech Rep, college):pragmatic = grammatical(& better than ESL sample in both)

Niezgoda &Roever (2001)

“ESL” (n=16, Germans in UK, college):pragmatic > grammatical(and over 9 months close to L1 baseline)EFL (n=17, Germany, college):pragmatic < grammatical

Schauer (2006)

Pragmatic awareness of ESL vs. EFL learners

Page 20: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Contributions to knowledge?

CONTEXT: a powerfulinfluence on L2 pragmaticdevelopment

Barvodi-Harlig &Dörnyei (1998)

CONTEXT: powerful,but not deterministic[program evaluation to shed lighton optimal context+instruction design?]

Niezgoda &Roever (2001)

CONTEXT: Always worthwhile toapply longitudinal, developmentallens

[context+proficiency?]

Schauer (2006)

Pragmatic awareness & Context

Page 21: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“The thinking presented in a single study may turn the movement of the literature, but the results in a single study are important primarily as one contribution to a mosaic of study effects.”

(Wilkinson & APA Task Force on Statistical Inference, 1999, p. 602)

Knowledge construction

as multi-study effort

Page 22: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

The more polemic a research area is, the more important the choice becomes between verificationist or value-oriented and value-aware approaches to replicating

“doing it again”

“doing it right”

“contributing to a valued dialogue”

Page 23: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Yet… is the replication

norm only an unrealistic

ideal?

Page 24: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“Economists treat replication the way teenagers treat chastity – as an ideal to be professed but not to be practised”

Hamermesh (2007, p. 715)Sociologists…

Medical researchers…

Educational researchers…

Page 25: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Replication, the

“incentives”

Page 26: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Graduate programs

Journal editorsAuthors (replicated &

replicating)

What things could be done?

Page 27: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Things graduate training programs

could do

Page 28: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

cf. papers in this colloquium by

Abbuhl, Fitzpatrick

Make replication central to graduate training experiences

Page 29: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Things journal editors could do

Page 30: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

To encourage within-study exact and conceptual replication (or “systematic” replication, Hendrick, 1991; also Hamermesh, 2007, pp. 731-732)

“multi-trait multi-method validation (Campbell & Fiske, 1959) “sensitivity analyses” (Saltelli et al., 2000)

e.g., Long, Inagaki, &

Ortega (1998)

Foster within-study replication activities

Page 31: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Several journals across fields have featured special replication sections at various times

SSLA, Valdman (1993)LT, Porte (2008)

Devote dedicated space to replication articles

Page 32: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“… journal editors [could] commission leading senior empirical researchers to undertake a replication study of a paper of their choice, one that had previously been published in the journal. If editors of each of the three leading general journals commissioned two replication studies per year, with publication guaranteed subject to refereeing (NOT by the author of the original study) to assure some minimum quality level, more replications would be undertaken. Original authors would be expected to write a short reply to the final version of the replication study”

(Hamermesh, 2007, p 726)

possible in our field?

Commission replication by senior researchers

Page 33: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“Data distributed with an article should become accessible to the scientific community without having to obtain permission from the author for each use. Having to obtain permission from an author to read a published article, such as by agreeing ex ante not to criticize it in print, is so obviously unacceptable it no longer occurs. The same should be true for data. Science requires the transmission of information through public means, not private agreements.”

(Gary King, 2007, p. 177)

Make data-sharing mandatory for publication in a journal

Page 34: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“information about replication [should be] part of the price of admission to competitive journals, rather than as an act of individual honor or graciousness following publication”

Mandatory data- and code-sharing should be “akin to the expectation of a full citation for all references”

(Freese, 2007, p. 155 & 156)

possible in our field?

Page 35: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Things (replicated and

replicating) authors could do

Page 36: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

“…think at the onset of their work about data sharing and opportunities for replication.”

Schneider (2004, p. 1477)How

often/regularly do we do this in

our field?

Design studies with the forethought of replication

Page 37: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

include by design: within-study exact and conceptual replication elements (Hendrick, 1991)“multi-trait multi-method validation (Campbell & Fiske, 1959) “sensitivity analyses” (Saltelli et al., 2000)

Enhance the quality of our studies with replication elements

Page 38: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

In the pre-Internet era, complaints about journal limited space may have justified lower reporting standards and lower levels of methodological transparency (cf. Polio & Gass, 1997)

In our Internet-dominated era, full public availability of instruments/data/code is made easier through personal webpages, professional archiving, and public repositories

(Freese, 2007; King, 2007)

Make your instruments/data/code publicly available (don’t wait to be asked!)

Page 39: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Make your instruments/data/code publicly available (don’t wait to be asked!)

Skehan’s (1989) longitudinal follow-up study of Wells’ (1985) Bristol project

Page 40: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Don’t wait to be asked!

CHILDES (McWhinney, 2000)

FLLOC & SPLLOC (Myles, 2008; Mitchell et al.,

2008)

Page 41: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

IRIS: Marsden & Mackey’s UK-US collaborative grant proposal to create a Digital Database of Instruments for Research Into Second-Languages

Don’t wait to be asked!

Page 42: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

The freeloading problem: Why should I go to the effort to obtain grants and collect my own data if I am then required to share my data with others?

The ‘‘I-might-be-scooped’’ problem: Not only will there be freeloaders, but they might become famous at my expense by publishing key results before I am able to.

[…] Too much work: The extra work for authors […]

would be onerous.(Firebaugh, 2007, p. 207)

Overcome our own prejudices about data sharing:

Page 43: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

e.g., Contact the to-be-replicated researcher when planning a replication, for a collaborative (not oppositional) footing if possible.

e.g., Respond (positively or negatively but

promptly and with courtesy) to requests for instrument/code/data sharing.

e.g., Make collegial and measured discourse choices in published exchanges (replicating or replicated)

Be mindful of standards of collegiality

Page 44: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Conclusion: Where do

we go from here?

Page 45: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

The promised benefits of heightened replication activity are many…

…but will they be fulfilled?

Page 46: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

FidelityVerificati

on

ValueRecognizing knowledge &

purpose,contributing to

dialogue

Page 47: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Thank [email protected]

Page 48: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

References: Bardovi-Harlig, K., & Dörnyei, Z. (1998). Do Language Learners Recognize

Pragmatic Violations? Pragmatic Versus Grammatical Awareness in Instructed L2 Learning. TESOL Quarterly, 32, 233-262.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Child Language Data Exchange System (CHILDES): http://childes.psy.cmu.edu/

Firebaugh, G. (2007). Replication Data Sets and Favored-Hypothesis Bias: Comment on Jeremy Freese (2007) and Gary King (2007). Sociological Methods & Research, 36, 200-209.

Freese, J. (2007). Replication Standards for Quantitative Social Science. Sociological Methods & Research, 36, 153-172 .

French Learner Language Oral Corpora (FLLOC): http://www.flloc.soton.ac.uk/index.php

Hamermesh, D. S. (2007). Viewpoint: Replication in economics. Canadian Journal of Economics, 40, 715-733.

Hendrick, C. (1991). Replications, strict replications, and conceptual replications: Are they important? In J. W. Neuliep (Ed.), Replication research in the social sciences (pp. 41-49). Newbury Park, CA: Sage.

Page 49: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

References: King, G. (2007). An Introduction to the Dataverse Network as an

Infrastructure for Data Sharing. Sociological Methods & Research, 36, 173-199.

Language Teaching Review Panel. (2008). Replication studies in language learning and teaching: Questions and answers. Language Teaching, 41, 4-14.

Long, M. H., Inagaki, S., and Ortega, L. (1998) The role of implicit negative feedback in SLA: Models and recasts in Japanese and Spanish. Modern Language Journal, 82, 357–71.

MacWhinney, B. (2000). The CHILDES Project: Tools for Analyzing Talk. Volume 1: Transcription format and programs. Volume 2: The database. Mahwah, NJ: Lawrence Erlbaum Associates.

McCullough, B. D. (2007). Got replicability? The Journal of Money, Credit, and Banking archive. Eco Journal Watch, 4, 326-337.

Mitchell, R., Dominguez, L., Arche, M. J., Myles, F. and Marsden, E. 2008. “SPLLOC: A new database for Spanish second language acquisition research.” EuroSLA Yearbook 8, 287-304. Muñoz, C. (Ed.). (2006). Age and the rate of foreign language learning. Clevedon, UK: Multilingual Matters.

Page 50: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

References: Myles, F.(2008). Investigating learner language development with

electronic longitudinal corpora: Theoretical and methodological issues. In Ortega, L. & Byrnes, H. (Ed.), The longitudinal study of advanced L2 capacities (pp. 58-72). New York: Routledge.

Neuliep, J. W., & Crandall, R. (1993). Everyone was wrong: There are lots of replications out there. Journal of Social Behavior and Personality, 8(6), 1-8.

Niezgoda, K., & Roever, C. (2001). Pragmatic and grammatical awareness. In K. R. Rose & G. Kasper (Eds.), Pragmatics in language teaching (pp. 63–79). New York: Cambridge University Press.

Polio, C., & Gass, S. M. (1997). Replication and reporting: A commentary. Studies in Second Language Acquisition, 19, 499-508.

Rosenthal, R. (1991). Replication in behavioral research. In J. W. Neuliep (Ed.), Replication research in the social sciences (pp. 1-30). Newbury Park, CA: Sage.

Rosenthal, R. (1966). Experimenter effects in behavioral research. New York: Appleton-Century Crofts.

Saltelli, A, Chan, K., & Scott, E. M. (Eds.).(2000). Sensitivity Analysis. Malden, MA: Wiley.

Page 51: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

References: Schauer, G. A. (2006). Pragmatic Awareness in ESL and EFL Contexts:

Contrast and Development. Language Learning, 56, 269–318.

Schneider, B. (2004). Building a scientific community: The need for replication. Teachers College Record, 106, 1471–1483.

Skehan, P. (1989). The relationship between native and foreign language learning ability: Educational and linguistic factors. In H. W. Dechert (Ed.), Current trends in European second language acquisition research (pp. 83-106). Clevendon, UK: Multilingual Matters.

Spanish Learner Language Oral Corpus (SPLLOC): http://www.splloc.soton.ac.uk/index.html

Valdman, A. (1993). Replication study (editorial introduction). Studies in Second Language Acquisition, 15, 505.

Valdman, A. (1997). Editorial.Studies in Second Language Acquisition, 19, 67.

Wells, G. (1985) Language development in the pre-school years. Cambridge, UK: Cambridge University Press.

Page 52: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Image copyright credits:

Velazquez’s Meninas: http://www.richardsonandgray.com/Theme%20Tours/Art%20Tours/Spanish%20Treasure.htm

Picasso’s Meninas: http://www.flickr.com/photos/32357038@N08/3286365519/

Rosetta Stone bookends: http://www.egyptianmarketplace.com/index.php?act=viewProd&productId=1490

Page 53: Please cite as: Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging.

Please cite as:

Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging replication research in the field of AL and SLA,” Graeme Porte convener. The American Association for Applied Linguistics Annual Conference, Denver CO, March 23.

Copyright © Lourdes Ortega, 2009