Please cite as:

Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging replication research in the field of AL and SLA,” Graeme Porte convener. The American Association for Applied Linguistics Annual Conference, Denver CO, March 23.

Copyright © Lourdes Ortega, 2009

A Sociology of Replication and Replicability in Applied

Linguistics

Lourdes OrtegaUniversity of Hawai‘i at Mānoa

AAAL colloquium, Graeme Porte ConvenerDenver CO, March 23, 2009

Sociology of knowledge: “study of the relationship between human thought and the social context within which it arises” (Wikipedia)Replicability: conditions that improve a field’s ability to support replication activities (McCullough, 2007, inter alii)

In art,as in

science...

“Original”

1656

“Replica/tion”

1950s

Fidelity? Value?

FidelityVerification

ValueWorthy Knowledge,

Worthy Purpose

In science, as in art...

Difference…?Value…?

contested understandings…

“how different/similar”?

Rosetta Stone (c. 203 BC) in British Museum… these

Rosetta Stone bookends?

uncertain understandings of

“replication”…

Polio & Gass (1997):

“many more studies than claim to be are actually some type of replication […] clearly, a fine line exists between replication and extending research”

(p. 501)

Neuliep & Crandall (1993):

Everyone was wrong: There are lots of replications out there. Journal of Social

Behavior and Personality, 8(6), 1-8.

As much as 78% of studies were (conceptual) replications that were not labeled by authors as such.

Norms for replication

Incentives for replicability

Sociological perspective:

Replication, the norm

Based on Rosenthal (1991)

Supports the theory (& investigator)

Damages the theory (& impugns investigator)

Internal validity,results easy to interpret, no “novelty”

Extends the theory (& supports investigator)

Limits the theory (& impugns the investigator very little)

External validity, greater “novelty,” any difference in results difficult to interpret

Result of replication

Same Different

Value

Exact

Conceptual,etc

TYPEofREPLICATION

“doing it again” “contributing to

a valued dialogue”

“doing it right”

Replication as dialogue

1998, TQ 2001, Chapter 2006, LL

Longitudinal component

Validation of instrumentvia L1 baseline & interviews

RQs & focus, instrument, analysesEFL sample from a

purposefully differentpopulation

Original:

Bardovi-Harlig & Dörnyei

Replication +1:

Niezgoda & Roever

Replication +1+ extension:

Schauer

Different:

Same:

Verification of findings?

ESL (n=173, Indiana, college):pragmatic > grammaticalEFL (n=370, Hungary, coll & hs):pragmatic < grammatical

Barvodi-Harlig &Dörnyei (1998)

ESL (n=48, Hawaii, college):pragmatic > grammaticalEFL (n=124, Czech Rep, college):pragmatic = grammatical(& better than ESL sample in both)

Niezgoda &Roever (2001)

“ESL” (n=16, Germans in UK, college):pragmatic > grammatical(and over 9 months close to L1 baseline)EFL (n=17, Germany, college):pragmatic < grammatical

Schauer (2006)

Pragmatic awareness of ESL vs. EFL learners

Contributions to knowledge?

CONTEXT: a powerfulinfluence on L2 pragmaticdevelopment

Barvodi-Harlig &Dörnyei (1998)

CONTEXT: powerful,but not deterministic[program evaluation to shed lighton optimal context+instruction design?]

Niezgoda &Roever (2001)

CONTEXT: Always worthwhile toapply longitudinal, developmentallens

[context+proficiency?]

Schauer (2006)

Pragmatic awareness & Context

“The thinking presented in a single study may turn the movement of the literature, but the results in a single study are important primarily as one contribution to a mosaic of study effects.”

(Wilkinson & APA Task Force on Statistical Inference, 1999, p. 602)

Knowledge construction

as multi-study effort

The more polemic a research area is, the more important the choice becomes between verificationist or value-oriented and value-aware approaches to replicating

“doing it again”

“doing it right”

“contributing to a valued dialogue”

Yet… is the replication

norm only an unrealistic

ideal?

“Economists treat replication the way teenagers treat chastity – as an ideal to be professed but not to be practised”

Hamermesh (2007, p. 715)Sociologists…

Medical researchers…

Educational researchers…

Replication, the

“incentives”

Graduate programs

Journal editorsAuthors (replicated &

replicating)

What things could be done?

Things graduate training programs

could do

cf. papers in this colloquium by

Abbuhl, Fitzpatrick

Make replication central to graduate training experiences

Things journal editors could do

To encourage within-study exact and conceptual replication (or “systematic” replication, Hendrick, 1991; also Hamermesh, 2007, pp. 731-732)

“multi-trait multi-method validation (Campbell & Fiske, 1959) “sensitivity analyses” (Saltelli et al., 2000)

e.g., Long, Inagaki, &

Ortega (1998)

Foster within-study replication activities

Several journals across fields have featured special replication sections at various times

SSLA, Valdman (1993)LT, Porte (2008)

Devote dedicated space to replication articles

“… journal editors [could] commission leading senior empirical researchers to undertake a replication study of a paper of their choice, one that had previously been published in the journal. If editors of each of the three leading general journals commissioned two replication studies per year, with publication guaranteed subject to refereeing (NOT by the author of the original study) to assure some minimum quality level, more replications would be undertaken. Original authors would be expected to write a short reply to the final version of the replication study”

(Hamermesh, 2007, p 726)

possible in our field?

Commission replication by senior researchers

“Data distributed with an article should become accessible to the scientific community without having to obtain permission from the author for each use. Having to obtain permission from an author to read a published article, such as by agreeing ex ante not to criticize it in print, is so obviously unacceptable it no longer occurs. The same should be true for data. Science requires the transmission of information through public means, not private agreements.”

(Gary King, 2007, p. 177)

Make data-sharing mandatory for publication in a journal

“information about replication [should be] part of the price of admission to competitive journals, rather than as an act of individual honor or graciousness following publication”

Mandatory data- and code-sharing should be “akin to the expectation of a full citation for all references”

(Freese, 2007, p. 155 & 156)

possible in our field?

Things (replicated and

replicating) authors could do

“…think at the onset of their work about data sharing and opportunities for replication.”

Schneider (2004, p. 1477)How

often/regularly do we do this in

our field?

Design studies with the forethought of replication

include by design: within-study exact and conceptual replication elements (Hendrick, 1991)“multi-trait multi-method validation (Campbell & Fiske, 1959) “sensitivity analyses” (Saltelli et al., 2000)

Enhance the quality of our studies with replication elements

In the pre-Internet era, complaints about journal limited space may have justified lower reporting standards and lower levels of methodological transparency (cf. Polio & Gass, 1997)

In our Internet-dominated era, full public availability of instruments/data/code is made easier through personal webpages, professional archiving, and public repositories

(Freese, 2007; King, 2007)

Make your instruments/data/code publicly available (don’t wait to be asked!)

Make your instruments/data/code publicly available (don’t wait to be asked!)

Skehan’s (1989) longitudinal follow-up study of Wells’ (1985) Bristol project

Don’t wait to be asked!

CHILDES (McWhinney, 2000)

FLLOC & SPLLOC (Myles, 2008; Mitchell et al.,

2008)

IRIS: Marsden & Mackey’s UK-US collaborative grant proposal to create a Digital Database of Instruments for Research Into Second-Languages

Don’t wait to be asked!

The freeloading problem: Why should I go to the effort to obtain grants and collect my own data if I am then required to share my data with others?

The ‘‘I-might-be-scooped’’ problem: Not only will there be freeloaders, but they might become famous at my expense by publishing key results before I am able to.

[…] Too much work: The extra work for authors […]

would be onerous.(Firebaugh, 2007, p. 207)

Overcome our own prejudices about data sharing:

e.g., Contact the to-be-replicated researcher when planning a replication, for a collaborative (not oppositional) footing if possible.

e.g., Respond (positively or negatively but

promptly and with courtesy) to requests for instrument/code/data sharing.

e.g., Make collegial and measured discourse choices in published exchanges (replicating or replicated)

Be mindful of standards of collegiality

Conclusion: Where do

we go from here?

The promised benefits of heightened replication activity are many…

…but will they be fulfilled?

FidelityVerificati

on

ValueRecognizing knowledge &

purpose,contributing to

dialogue

Thank Youlortega@hawaii.edu

References: Bardovi-Harlig, K., & Dörnyei, Z. (1998). Do Language Learners Recognize

Pragmatic Violations? Pragmatic Versus Grammatical Awareness in Instructed L2 Learning. TESOL Quarterly, 32, 233-262.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Child Language Data Exchange System (CHILDES): http://childes.psy.cmu.edu/

Firebaugh, G. (2007). Replication Data Sets and Favored-Hypothesis Bias: Comment on Jeremy Freese (2007) and Gary King (2007). Sociological Methods & Research, 36, 200-209.

Freese, J. (2007). Replication Standards for Quantitative Social Science. Sociological Methods & Research, 36, 153-172 .

French Learner Language Oral Corpora (FLLOC): http://www.flloc.soton.ac.uk/index.php

Hamermesh, D. S. (2007). Viewpoint: Replication in economics. Canadian Journal of Economics, 40, 715-733.

Hendrick, C. (1991). Replications, strict replications, and conceptual replications: Are they important? In J. W. Neuliep (Ed.), Replication research in the social sciences (pp. 41-49). Newbury Park, CA: Sage.

References: King, G. (2007). An Introduction to the Dataverse Network as an

Infrastructure for Data Sharing. Sociological Methods & Research, 36, 173-199.

Language Teaching Review Panel. (2008). Replication studies in language learning and teaching: Questions and answers. Language Teaching, 41, 4-14.

Long, M. H., Inagaki, S., and Ortega, L. (1998) The role of implicit negative feedback in SLA: Models and recasts in Japanese and Spanish. Modern Language Journal, 82, 357–71.

MacWhinney, B. (2000). The CHILDES Project: Tools for Analyzing Talk. Volume 1: Transcription format and programs. Volume 2: The database. Mahwah, NJ: Lawrence Erlbaum Associates.

McCullough, B. D. (2007). Got replicability? The Journal of Money, Credit, and Banking archive. Eco Journal Watch, 4, 326-337.

Mitchell, R., Dominguez, L., Arche, M. J., Myles, F. and Marsden, E. 2008. “SPLLOC: A new database for Spanish second language acquisition research.” EuroSLA Yearbook 8, 287-304. Muñoz, C. (Ed.). (2006). Age and the rate of foreign language learning. Clevedon, UK: Multilingual Matters.

References: Myles, F.(2008). Investigating learner language development with

electronic longitudinal corpora: Theoretical and methodological issues. In Ortega, L. & Byrnes, H. (Ed.), The longitudinal study of advanced L2 capacities (pp. 58-72). New York: Routledge.

Neuliep, J. W., & Crandall, R. (1993). Everyone was wrong: There are lots of replications out there. Journal of Social Behavior and Personality, 8(6), 1-8.

Niezgoda, K., & Roever, C. (2001). Pragmatic and grammatical awareness. In K. R. Rose & G. Kasper (Eds.), Pragmatics in language teaching (pp. 63–79). New York: Cambridge University Press.

Polio, C., & Gass, S. M. (1997). Replication and reporting: A commentary. Studies in Second Language Acquisition, 19, 499-508.

Rosenthal, R. (1991). Replication in behavioral research. In J. W. Neuliep (Ed.), Replication research in the social sciences (pp. 1-30). Newbury Park, CA: Sage.

Rosenthal, R. (1966). Experimenter effects in behavioral research. New York: Appleton-Century Crofts.

Saltelli, A, Chan, K., & Scott, E. M. (Eds.).(2000). Sensitivity Analysis. Malden, MA: Wiley.

References: Schauer, G. A. (2006). Pragmatic Awareness in ESL and EFL Contexts:

Contrast and Development. Language Learning, 56, 269–318.

Schneider, B. (2004). Building a scientific community: The need for replication. Teachers College Record, 106, 1471–1483.

Skehan, P. (1989). The relationship between native and foreign language learning ability: Educational and linguistic factors. In H. W. Dechert (Ed.), Current trends in European second language acquisition research (pp. 83-106). Clevendon, UK: Multilingual Matters.

Spanish Learner Language Oral Corpus (SPLLOC): http://www.splloc.soton.ac.uk/index.html

Valdman, A. (1993). Replication study (editorial introduction). Studies in Second Language Acquisition, 15, 505.

Valdman, A. (1997). Editorial.Studies in Second Language Acquisition, 19, 67.

Wells, G. (1985) Language development in the pre-school years. Cambridge, UK: Cambridge University Press.

Image copyright credits:

Velazquez’s Meninas: http://www.richardsonandgray.com/Theme%20Tours/Art%20Tours/Spanish%20Treasure.htm

Picasso’s Meninas: http://www.flickr.com/photos/32357038@N08/3286365519/

Rosetta Stone bookends: http://www.egyptianmarketplace.com/index.php?act=viewProd&productId=1490

Please cite as:

Ortega, L. (2009). A sociology of replication and replicability in applied linguistics. Paper presented in the invited colloquium “Encouraging replication research in the field of AL and SLA,” Graeme Porte convener. The American Association for Applied Linguistics Annual Conference, Denver CO, March 23.

Copyright © Lourdes Ortega, 2009