The Counseling Psychologist

download The Counseling Psychologist

of 29

Transcript of The Counseling Psychologist

  • 7/28/2019 The Counseling Psychologist

    1/29

    http://tcp.sagepub.com

    The Counseling Psychologist

    DOI: 10.1177/00110000042722572005; 33; 299The Counseling Psychologist

    Carolyn A. Karr and Lisa M. LarsonFrom 1990 to 1999

    Use of Theory-Driven Research in Counseling: Investigating Three Counseling Psychology Journ

    http://tcp.sagepub.com/cgi/content/abstract/33/3/299The online version of this article can be found at:

    Published by:

    http://www.sagepublications.com

    On behalf of:

    Division of Counseling Psychology of the American Psychological Association

    can be found at:The Counseling PsychologistAdditional services and information for

    http://tcp.sagepub.com/cgi/alertsEmail Alerts:

    http://tcp.sagepub.com/subscriptionsSubscriptions:

    http://www.sagepub.com/journalsReprints.navReprints:

    http://www.sagepub.com/journalsPermissions.navPermissions:

    http://tcp.sagepub.com/cgi/content/refs/33/3/299Citations

    at KANSAS STATE UNIV LIBRARIES on May 14, 2009http://tcp.sagepub.comDownloaded from

    http://www.div17.org/http://tcp.sagepub.com/cgi/alertshttp://tcp.sagepub.com/cgi/alertshttp://tcp.sagepub.com/subscriptionshttp://tcp.sagepub.com/subscriptionshttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsPermissions.navhttp://tcp.sagepub.com/cgi/content/refs/33/3/299http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/cgi/content/refs/33/3/299http://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsReprints.navhttp://tcp.sagepub.com/subscriptionshttp://tcp.sagepub.com/cgi/alertshttp://www.div17.org/
  • 7/28/2019 The Counseling Psychologist

    2/29

    10.1177/0011000004272257THE COUNSELING PSYCHOLOGIST / May 2005Karr, Larson / USE OF THEORY-DRIVEN RESEARCHUse of Theory-Driven Research in Counseling:

    Investigating Three Counseling Psychology Journals

    From 1990 to 1999

    Carolyn A. KarrUniversity of MissouriKansas City

    CCS Assessment Center

    Lisa M. LarsonIowa State University

    Three major journals in counseling psychology were sampled from 1990 to 1999 to

    assess the percentage of quantitative, empirical articles that were theory driven. Only

    43%of thestudies utilized a theoryor model,and57% predicted therelationbetweenthevariables, withfew studies specifyingthe strength of the relation. Studies sampledin the

    Journal of Counseling Psychology (63%) andthe Journal of Vocational Behavior (65%)

    reported a significantly higher percentage of theory-driven research than the Journal of

    Counselingand Development(43%). A higher proportion ofJournal of Counseling Psy-

    chology studies compared with Journal of Counseling and Development studies

    anchored findingsto theory.Fewstudies replicatedpreviousstudies.This paucity implies

    that replication is underutilized by the field as a potentially fruitful way to substantiate

    empirical knowledge. A binary logistic regressionwas conductedto examine trends over

    time. No trends emerged. Suggestions for future implementation of theory and

    implications for counseling psychology as a field are offered.

    Counseling psychologists have alerted the field that too little theory-

    driven research is being generated (Harmon, 1982; Heppner, Kivlighan, &

    Wampold, 1999; Strong, 1984, 1991a, 1991b; Tracey, 1991; Tracey &

    Glidden-Tracey, 1999). An excellent example of theory-driven research has

    been thetestof John Hollands hexagonof sixinterests (e.g.,Fouad,Harmon,

    & Borgen, 1997). Recent studies in multicultural (e.g., Prosser, 2003) and

    vocational psychology(e.g., Blanchard& Lichtenberg, 2003) indirectlysug-

    gest a greater need for theory-driven research in these areas as well.

    A recent review of the process and outcome literature for test interpreta-

    tion found that only 9% of studies were based on theory (Hanson & Good-

    year, 2000). This observation is not limited to counseling psychology, how-

    ever. Meehl (1978, 1990, 1993) voiced similar criticisms about psychology

    in general, and social psychology has been concerned with the related areas

    of meaningful hypothesis construction (e.g., Wallach & Wallach, 1994) and

    299

    Theauthorswould like to thankDougBonett forhis suggestions andguidancewith thestatisticalanalyses and Jennifer Swaim for her aid in data compilation.

    THE COUNSELING PSYCHOLOGIST, Vol. 33 No. 3, May 2005 299-326

    DOI: 10.1177/0011000004272257

    2005 by the Society of Counseling Psychology

    at KANSAS STATE UNIV LIBRARIES on May 14, 2009http://tcp.sagepub.comDownloaded from

    http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/
  • 7/28/2019 The Counseling Psychologist

    3/29

    replication of studies (e.g., Neuliep & Crandall, 1991). Their concerns are

    noteworthy giventheusefulness of theory-driven research and thepaucity of

    empirical studies that test established theories.

    Theories andmodelsare crucial forseveral reasons. Theories aremeant to

    be organized explications of the dynamics that underlie a given psychologi-

    cal phenomenon (Strong, 1991a). They are fundamental tools to understand

    the construct being researched. Meehl (1990) articulated a more specific

    function of theories in empirical studies by advocating that theories should

    be used to help determine whether the substantive theory, the statistical

    hypotheses, and the empirical observations are related in terms of logical

    inference and theoretical causality.

    The definition of theory used for this study was a general principle for-

    mulated to explain a group of related phenomena (Chaplin, 1985, p. 467).For the purposes of this study, a model was construed as a description of the

    assumed structure of a set of observations (Everitt & Wykes, 1999, p. 119).

    Although similar, the former utilizes a general tenet to explain related inter-

    actions, while the latter describes the expected observable interactions in

    more detail. By definition, theories and models are similar in function and

    scope. Forster (2000) stated that the best way to distinguish theories and

    models is to discuss each in conjunction with predictive hypotheses. In his

    conceptualization, the threeare hierarchically arranged, with theories at the

    most general level, models applied to concrete systems in the middle, and

    predictive hypotheses at the lowest level, which result from fitting models to

    data (Forster, 2000, p. 233). He emphasized that the essential point of this

    tripartite distinction is that predictive accuracy is a property of predictive

    hypotheses at the very bottom of the hierarchy, and is traded-off against thetruth at thenext levelupthe levelof models (Forster, 2000, p. 233). In this

    way, both theories1 and models are tested by the utilization of tailored

    predictive hypotheses.

    Theory-driven research also provides a conceptual framework that mini-

    mizes the chances that researchers efforts will lead to the generation of a

    multitude of unconnected facts (Strong, 1991a). Strong (1991b) explained

    the furthering of scientific thought necessarily involves a circular pro-

    cess that begins with theory, carries this theory to the context of testing, is

    held to specific observations, and is interpreted through the context of dis-

    covery. Discovery then follows into further redefinition and redescription of

    theory. This grounded, circular process ensures that utility is blended with

    conceptualization.

    Moreover, use of theory easily guarantees that a meaningful question is

    asked because of the network of supplementary and predictive informa-

    tion that is part of the theory (Meehl, 1978). Kazdin (1992) noted that much

    theory-free research is considered meaningful by using the rationale that a

    300 THE COUNSELING PSYCHOLOGIST / May 2005

    at KANSAS STATE UNIV LIBRARIES on May 14, 2009http://tcp.sagepub.comDownloaded from

    http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/
  • 7/28/2019 The Counseling Psychologist

    4/29

    particular study has not yet been conducted. He emphasized, however, that

    literally thousands of variables could be researched together. Conglomera-

    tions of variables without links to theoretical networks suggesting the spe-

    cific andsignificant reasons why theyare chosen areessentially meaningless.

    The specificity of the hypotheses is also important in conducting theory-

    driven research. In particular, research questions should be asked in terms of

    whether the data will fall within a specific meaningful range, based on theo-

    ries that include components that are likewise relatively narrow and focused

    (Meehl, 1993). Instead, many hypotheses based on specific theories do not

    identify the strength of the relations among the theoretical constructs.

    Meehl (1978,1990, 1993) haschampioned thebenefitsof hypotheses that

    predict whether experimental observations will fall within a certain, rela-

    tively narrownumeric range andwhether theobservationswill be specifiedapriori. He explains the advantages of numeric hypothesizing over traditional

    significance testing; numeric hypothesizing allows researchers to ask, does

    the theory have sufficient verisimilitude to warrant our continuing to test it

    and amend it rather than, is the theory literally true? (Meehl, 1990). He

    defines verisimilitude as the relationship between the theory and the real

    world which the theory speaks about (Meehl, 1990, p. 113).

    Wampold, Davis, and Good (1990) have further specified the need for

    hypothesis validity, which is the extent to which research results reflect

    theoretically derivedpredictions aboutthe relations betweenor amongcon-

    structs (p. 360). In this manner, hypothesis validity ensures that the rela-

    tion between constructs is informed by theory. Therefore, by utilizing

    hypotheses that include precise numeric ranges (e.g., effect sizes, variability,

    and confidence intervals), we can better assess how expectations and result-ing observations are linked, while simultaneously evaluating thecurrent the-

    ory (Greenwald, Gonzalez, Harris, & Guthrie, 1996; Howard, Curtin, &

    Johnson, 1991; Levin & Robinson, 1999; Wampold et al., 1990).

    The magnitude and direction of the relation should be stated a priori

    because theory identifies more than the important constructs. Theory also

    should identify the posited strength of the relation among the constructs

    not the exact statistical value (e.g., .09) but rather small, medium, and large

    effects and positive and negative relations. This specification provides the

    template for hypotheses to be formulated. Examining the expected magni-

    tude of the relation a priori, in comparison with the actual magnitude of the

    relation, contributes important information in determining how variables

    are related. For example, based on social cognitive theory, if we expect self-

    efficacy andoutcome expectations to be moderately related (rvalues ranging

    from .3 to .5) and find instead that the relation across five studies is weaker

    (i.e., r= .15), then the theory must be adjusted.

    Karr, Larson / USE OF THEORY-DRIVEN RESEARCH 301

    at KANSAS STATE UNIV LIBRARIES on May 14, 2009http://tcp.sagepub.comDownloaded from

    http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/
  • 7/28/2019 The Counseling Psychologist

    5/29

    Identifying discrepancies can further hone theoretical relations, contrib-

    uting to advances in theory. By examining expectations of effect magnitude

    in theory and prior research, and then incorporating these expectancies into

    the studys hypotheses, potential extra variables or relations are more easily

    identified, which could lead to new discoveries. In practice, identifying the

    magnitude of relations a priori is much like the use of effect sizes (Cohen,

    1969) in meta-analyses, in that effect size provides a quantified measure of

    the relation for comparison purposes. Allen (1995) stated that effect size

    indicates how strongly the independent variable influences the dependent

    variable, or the strength of the relationship (p. 82). Effect sizes, used in this

    way, provide a measure of themagnitude of a givenrelationand aregenerally

    more informative than hypotheses and analyses that simply aim to reject the

    null hypothesis (Cohen, 1994).Effect sizes are also used in replication of research, because strict null

    hypothesis testing does not include the probability that a result can be repli-

    cated(Gigerenzer & Murray, 1987;Lykken,1968). Replication of researchis

    important in the assessment and reformation of theory, using effect sizes as

    necessary vehicles to assess relations and further knowledge (Cohen, 1994).

    Prior research can be useful in determining numeric range estimates for

    hypothesized relations, as wellas informingthe directionof these relations.

    Some researchers question the viability of theory-driven research and

    even suggest that theory can thwart research progress under certain condi-

    tions. Greenwald, Pratkanis, Leippe, and Baumgardner (1986) argued that

    over-reliance on theory can cause confirmation bias when researchers per-

    severe by revising procedures until obtaining a theory-predicted result.

    Researchers discovered this bias when they noticed poor preliminary datatendedto cloud theinterpretationof laterdata of betterquality(Bruner & Pot-

    ter, 1964; Wyatt & Campbell, 1951).

    Others have agreed that research is frequently developed by way of

    confirmation-biased practices (e.g., Popper, 1959). More specifically, chang-

    ing conditions and procedures until the desired result is obtained is consid-

    ered theoryconfirming (Greenwald et al., 1986) or prejudiceagainst thenull

    hypothesis (Greenwald, 1975), which strays from the ideal empirical meth-

    ods of hypothesis testing and resulting theory disconfirmation. Remedies are

    offered in the form of utilizing condition-seeking and design approaches,

    which are used, respectively, to discover conditions on which an existing

    finding depends and to specify conditions that can produce an unobtainable

    result. Greenwald et al. (1986; refer for lengthier discussion) stated that the

    need for theorytoguide empiricalprogress, along with thestimulus to theory

    that is provided by new findings, rescues thecondition-seeking methodfrom

    being just a means of cultivating empiricaltrivialities(p. 224). Thefinal step

    302 THE COUNSELING PSYCHOLOGIST / May 2005

    at KANSAS STATE UNIV LIBRARIES on May 14, 2009http://tcp.sagepub.comDownloaded from

    http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/
  • 7/28/2019 The Counseling Psychologist

    6/29

    is then to revise, expand, and/or rearticulate theories to incorporate the new

    conditions.

    A relatedobjectionto theory-drivenresearchis thecontention that theory-

    driven researchnecessitates abandoningstudies derived fromempiricaldata.

    In actuality, empirical data can inform future studies and inform/revise the-

    ory. The authors are not suggesting that all research should be based on the-

    ory. Rather, most studies should be theory-driven, with initially atheoretical

    data used to specify and refine existing theories. Or new theories can be con-

    structed to explain existing empirical and currently atheoretical data. By uti-

    lizing remedies for theory-driven research problems, we ensure the produc-

    tion of meaningful, theory-grounded data. The current study attempts to

    identify the current proportion of theoretical to atheoretical research. By

    identifying thetheoriescurrently utilizedin thesamplewewill also launchaneffort to determine the extent to which theories have been substantiated or

    further developed by new data.

    Still, despite these objections, theory-driven research may unify con-

    ducted research, especially among findings linked to similar, but different,

    fields within psychology (Staats, 1983). For example, many objectives of

    counseling psychology are similar to those of clinical psychology, social

    work, and school psychology.Yet findings and viewpoints have been largely

    linked within one field without integrating findings into a larger, unifying

    framework. Shared theoretical frameworks would serve to unify both psy-

    chological research and disciplines (Forsyth & Strong, 1992), benefiting not

    only the pursuit of science but also professional relationships and activities

    that can then contribute to a more common scientific ground. In short, the

    generation of theory-driven research is vital.The purpose of this study was to empirically examine during 10 years

    (1990 to 1999) the extent to which major counseling psychology journals

    havepublished theory-driven research. Specifically, the authors will address

    two questions. For the first question, the overall proportion of theory-driven

    research articles to quantitative research articles will be determined in a

    representative sample of counseling psychology journals. Although the

    profession has called for more theory-driven research, no studies have been

    conducted to examine the extent to which major journals in counseling psy-

    chology publish theory-driven research. Omer and Dar (1992) examined the

    research purposes posited in studies appearing in the Journal of Consulting

    and Clinical Psychologyacrossthe 1960s,1970s,and1980s. They found that

    the trend in clinical research has moved away from theoretical bases for

    research questions and has moved toward pragmatism.

    Two factors led to the selection of this particular time frame: (a) The aim

    of the study was to measure trends in utilization of theory for the 10 years

    Karr, Larson / USE OF THEORY-DRIVEN RESEARCH 303

    at KANSAS STATE UNIV LIBRARIES on May 14, 2009http://tcp.sagepub.comDownloaded from

    http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/
  • 7/28/2019 The Counseling Psychologist

    7/29

    before the new century (2000) to assess how the field should conduct

    research in the current century, and (b) an active discussion regarding the

    optimal degree of theory utilization in counseling psychology surfaced just

    before the1990s (e.g., Heppner & Claiborn,1989; Serlin, 1987) andagain in

    the early 1990s (Gelso, 1991; Patton & Jackson, 1991; Strong, 1991a,

    1991b), with similar concerns voiced elsewhere in psychology during this

    time (e.g., Gergen, 1991). Potential trends and/or changes in theory utiliza-

    tion in counseling psychology during the 1990s could then at least be par-

    tially attributed to discussions held around this time. The field would benefit

    if future research addressed these concerns for the first 10 years of the cur-

    rent century (2000 to2009) forthe purpose of comparison. Giventhe valueof

    theory-driven research, data must be collected to ascertain the extent to

    which researchers are embedding research questions in theory.For the second question, for those studies that appear to be theory driven,

    the authors will determine the extent to which the studies report a priori the

    directionand strengthof therelations of certain variables in thetheory. Meehl

    (1978) argued that if research is to inform the broader discipline of psychol-

    ogy, the parameters of the strength of the hypothesized relation must be

    specified a priori.

    METHOD

    Sample: The Studies

    The three most prominent and primarily quantitative journals in counsel-ing psychology were sampled: Journal of Counseling Psychology (JCP),

    Journal of Vocational Behavior(JVB),andJournal of Counseling and Devel-

    opment(JCD). The Counseling Psychologistwas excluded because its arti-

    cles are more conceptual. These three journals were selected because they

    have traditionally been associated with empirical research. For example,

    when studies have examined the prominence of counseling psychology pro-

    grams across the United States, the extent to which the faculty publish in

    these journals (including The Counseling Psychologist) is used to measure

    the productivity of the counseling psychology faculty.

    To obtain a representative sample with sufficient power, every seventh

    article was sampled. A power analysis was conducted to determine adequate

    sample size. The power analysis was conducted before data collection, and

    indicated sampling every seventh article would generate a sample size withsufficient statistical power. By utilizing Power and Precision for Windows

    (Biostat,2001), an estimated150 total studies analyzed viac2 with sevencri-

    teria acrossthree journals (effect size = .50;a= .05) yielded a powerestimate

    304 THE COUNSELING PSYCHOLOGIST / May 2005

    at KANSAS STATE UNIV LIBRARIES on May 14, 2009http://tcp.sagepub.comDownloaded from

    http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/
  • 7/28/2019 The Counseling Psychologist

    8/29

    of .99. Actual sample size of the studies2 consisted of the following:JCP, n =

    70;JVB, n = 63; andJCD, n = 60;N= 193. Because the total number of stud-

    iessampled wasmore than 150, power is considered to be of a sufficient level

    for this study.

    Procedures

    Many decision rules were employed during the selection and analysis of

    the sampled studies. The sampling began with the first volume in each jour-

    nalthatwaspublished in1990. If anarticle containedmore than onestudy, all

    the studies in that article were used, resulting in a total of 193 studies in JCP,

    JVB, and JCD. If the seventh article was not data-based, the next article

    would be sampled. Thus, reviews and conceptual articles were skipped ifthey were the seventh article. Given that qualitative articles are generally

    used to build, rather than disconfirm, theory, they also were not sampled.

    Thus, if the seventh article was qualitative, the next article was sampled.

    Qualitative studies were excluded for three primary reasons. First, this

    studywasa foray into measuringresearchtrends in this mannerin counseling

    psychology; qualitative studies were not included to keep the decision crite-

    riauniform for theentiresample. Second, qualitative studies arebased on the

    inductive, rather thandeductive, approach.Thisstudyexaminedthe quantita-

    tive research tradition in the field rather than the qualitative, which contains

    different assumptions. Third, research would benefit the assessment of the-

    ory utilization for qualitative studies but was beyond the scope of this study,

    partly because of the aforementioned factors.

    Development of the Content of the Questions

    The seven questions presented in Table 1 were derived from the literature

    calling for more theory-driven research. Questions 1 through 4 and Question

    7 were derived from Tracey andothers (Meehl, 1978; Tracey, 1991; Tracey &

    Glidden-Tracey, 1999), who argued that theory should be used at all stages,

    from conceptualization of questions and hypotheses to interpretation of

    results and the discussion of the findings within the field. Question 3 was

    based on Wallach and Wallach (1998), who argued that useful hypotheses

    shouldbespecific.For example, Wampoldet al.(1990) noted that thefollow-

    ingstatedpurpose of a study is an ambiguous research hypothesis (one iden-

    tified threat to hypothesis validity):

    Thepresentstudy attempted todetermine a) therelationof parentaladjustmentmeasuresof suchvariablesasdepression,marital satisfaction, parenting stress,and other negative life stressors to mothers and fathers perceptions of their

    Karr, Larson / USE OF THEORY-DRIVEN RESEARCH 305

    at KANSAS STATE UNIV LIBRARIES on May 14, 2009http://tcp.sagepub.comDownloaded from

    http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/http://tcp.sagepub.com/
  • 7/28/2019 The Counseling Psychologist

    9/29

    TABLE1:P

    ercentagesofSampledStudiesCodedYesor

    NoontheQuestionsand

    2byJournalandO

    verall

    JCP

    JVB

    JCD

    Overall

    JCP/

    JVB/

    JCP/

    n=70

    n=63

    n=60

    N=193

    JVB

    JCD

    JCD

    Question

    Yes

    No

    Yes

    No

    Yes

    No

    Yes

    No

    c2

    c2

    c2

    1.Isthestudy

    embeddedinatheory/model?

    44.3

    5

    5.7

    47.6

    52.4

    36.7

    63.3

    43.0

    57.0

    .15a

    1.51

    .77

    2.Ifso,istheintroductionorganizedaround

    thistheory/m

    odel?b

    90.3

    9.7

    80.0

    20.0

    81.8

    18.2

    84.3

    15.7

    1.29

    .03

    .81

    3.Ofthestudiesthatutilizeatheory/model,are

    thehypothesesdrawndirectlyfromt

    heory?

    87.1

    1

    2.9

    73.3

    26.7

    77.3

    22.7

    79.5

    20.5

    1.83

    .11

    .88

    4.Ofthestudiesthatutilizeamodel/theory,are

    thefindings

    anchoredtothatmodel/theory?

    86.7

    1

    3.3

    73.3

    26.7

    59.1

    40.9

    74.4

    25.6

    1.67

    1.17

    5.15*

    5.Istherelatio

    nbetweenthevariablespredicted?

    62.9

    3

    7.1

    65.1

    34.9

    43.3

    56.7

    57.5

    42.5

    .07

    5.86*

    4.96*

    6.Isthepredic

    teddegreeoftherelationbetween

    thevariablessignificant?c

    17.1

    8

    2.9

    7.9

    92.1

    6.7

    93.3

    10.9

    89.1

    2.52

    .07

    3.29

    7.Isthestudy

    adirectextensionofoneormore

    previousstu

    dies?

    17.1

    8

    2.9

    11.1

    88.9

    8.3

    91.7

    12.4

    87.6

    .99

    .27

    2.21

    NOTE:JCP=

    JournalofCounselingPsychology;JCD=JournalofCounselingandDevelopment;JVB=Jour

    nalofVocationalBehavior.

    a.Eachc

    2equationincludedonedegreeoffreedom.

    b.ForQuestions2through4,asubsetofthesamplewasused(JCP,n=31;JVB,n=30;JCD,n=22).

    c.Nostudiesi

    npilotdatapredictedthestrengthoftherelatio

    nsbetweenthevariables.

    *p