aj620103

8/9/2019 aj620103

1/7

Survey Course on Research Methods: Integrating StatisticalAnalysis and Study Design

JoLaine R. Draugalis, Jean T. Carter and Marion K. Slack

CollegeofPharmacy, TheUniversityofArizona,P.O.Box210207, TucsonAZ85721-0207

The purpose of the three-credit (45 hours) required course is to ensure PharmD student competence inselecting, interpreting, and evaluating clinical drug literature. The course is a prerequisite for a three-creditDrug Information/Drug Literature Evaluation course in the professional curriculum. Topics covered includeprinciples of research design, acquisition and interpretation of data, statistical analysis, and validity ofconclusions. The essential connection between research design and statistical analysis is emphasizedthroughout the course in lectures, examples, assignments, and exams. The purpose of this article is to sharethe methods used to convey statistical analysis and research design concepts as well as methods forassessing the students. The techniques and approaches described are also applicable to other courses inthe pharmacy curriculum. Examples also show how instructors used evaluation and revision to cultivateintegration of concepts, encourage more active learning and class participation, and enhance assessmentprocedures.

INTRODUCTION

In 1992, an article was published that provided a descriptionof a three credit (45 hours) required research evaluationcourse in an entry-level PharmD program(1). The articleconcluded with a list of course improvement efforts includ-ing content changes, additions of assignments, and examina-tion improvement. In that more institutions will be offeringa course of this nature by virtue of increased numbers ofentry-level PharmD programs, this update details our on-going improvement efforts as well as other refinements ofthe past six years. The materials and methods of this courseaddress a number of specific educational outcomes andgeneral ability-based outcomes (including thinking abili-

ties, communication abilities, responsible use of values andethical principles, and self-learning abilities and habits)deemed requirements of a competent pharmacy practitio-ner(2). As well, the course meets Standards 12 (teaching andlearning processes) and 13 (student evaluation) in the newACPE accreditation manual(3). Particularly relevant areGuidelines 12.2 and 12.3 which encourage teaching ap-proaches that require problem solving capabilities and pro-mote life-long learning to in turn, ensure professional com-petence. Guideline 13.1 specifically argues for an evaluationsystem that examines higher levels of learning, such asintegration, application, and synthesis rather than simplytesting for rote memorization of specific facts. Finally, aphilosophy of teaching as scholarship as espoused by Boyer

guides instructors in that the goal is to transform and extendknowledge, rather than simply transmit information(4).Although this paper provides examples for a specific con-tent area, the methods embraced and the techniques andapproaches used most certainly can be applied to manyother courses in the pharmacy curriculum.

COURSE DESCRIPTION

In order for students to use the skills gained in this coursethroughout their study plan, a curriculum revision resulted

in moving the course from the final didactic semester (priorto the final year of clerkship rotations) to first semester ofthe second professional year. The Drug Information/Litera-ture Evaluation (DI) course is now in the semester immedi-ately following the Methods course, rather than in thesemester preceding it. The DI course was also increased byone unit to three units. Some topics previously covered inthe Methods course (e.g., Meta-Analysis, Proposal WritingEvaluation Methods, and article critique) have been movedto the DI course, in order to dedicate more time to statisticatechniques/data analysis as an introductory biostatisticscourse is no longer a prerequisite. This decision was made because prerequisite course content and student retentionlevels were so varied. However, as the prepharmacy curriculum is evaluated over the next year, an entry level biostatis-tics course requirement will be revisited. The pharmaco-economic evaluation component was deleted several yearsago as this topic is now covered in much more detail in theProfessional Practice Management course.

The overall course purpose is to provide the studenwith the skills and principles of clinical research design and biostatistics needed for interpretation and evaluation of thmedical literature and assessment of research reports andproposals. The overview of statistics module begins with aquotation from H.G. Wells, Statistical thinking will oneday be as necessary for efficient citizenship as the ability toread and write. Our quest for conceptual understandingover mathematical plug-and-chug begins in earnest onday one and continues throughout the course. Teachingstudents to be healthy skeptics, arming them with tools to be proficient professional decision makers, and challenginthem to accept responsibility for their actions are necessaryEncouraging open-mindedness and discouraging the urgeto over-generalize based on an N of 1 are crucial. Theoverriding objective is to ensure competence in selectinginterpreting, and evaluating clinical drug literature, includ-ing principles of research design, acquisition and interpreta-tion of data, statistical analysis, and validity of conclusions

American Journal of Pharmaceutical Education Vol. 62, Spring 1998 17

8/9/2019 aj620103

2/7

Fig. 1. Course outline. Each topic is covered in one 75-minutesessionunlessotherwisenoted.

The specific objectives addressing this charge have not beenchanged (other than deleting those related to deleted/reas-signed material) since the original report was presented(1).The original report also details the specific learning objec-tives. The course is now offered in two 75-minute classperiods per week (see Figure 1).

COURSE METHODSThe first day of class we disseminate a 360-page coursenotebook. This notebook contains the syllabus, lecture notes,reprints, objectives, and references to be used during thesemester. Class materials, including in-class examples, arechosen to make concepts more meaningful for students be itclinical or practice relevance. Articles and reports from themedical literature as well as the lay press provide an endlesssource of new material. The course coordinator oversees allaspects of course preparation and planning and provides 80 percent of the lectures. The other instructor provides 20percent of the lectures. Occasionally, a guest lecturer mayprovide one or two lectures (e.g., epidemiology topics). Agraduate teaching assistant is available to prepare and gradeassignments.

The course is still primarily lecture-based, however wehave sought to provide increased opportunities for interac-tion as well as in-class activities for certain topics. In fact, onthe first day of class, each of the instructors provides about25 minutes of lecture on terminology and the scientificmethod and the remaining 25 minutes is spent having thestudents (either in groups or the class as a whole) generate a

scientific prediction, describe an example of a cause-and-effect relationship in health care, and identify the indepen-dent (including levels) and dependent variables. Some re-cent examples include: ill-fitting shoes cause blisters, sunexposure causes skin cancer, drinking Tucson water causes pregnancy, and completing lecture early results in clasdismissal. The class found out early-on that they would beunable to test the last hypothesis. From day one, we beginthe discussion of plausible alternative hypotheses and therole of research design in controlling these threats.

In Part 2, Statistical Techniques/Data Analysis, thecourse notes provide definitions and compare and contraseach of the statistical techniques. See Figure 2 for an in-classexercise for the chi square test of distribution. The formaincludes a very brief scenario, data display, null and alterna-tive hypothesis, calculation example, and finally an exercisewhere students use a table of critical values to ultimatelymake a statistical decision.

RESEARCH DESIGN SECTION (Part 3 of Course)

The goal of the research design section is for students tolearn basic research design concepts. Specifically, studentsshould be able to distinguish among different types ofresearch design including experimental, quasi-experimen-

tal, pretest-posttest, and case studies, and associated threatsto internal validity when they complete the section. Thesection is based on the presentation of research design andinternal validity by Campbell and Stanley(5). To learnconcepts, students must see examples of each type of re-search design and learn to identify the critical characteris-tics that differentiates one design from another(6). The basic instructional strategy is to present a definition andescribe the critical characteristics of each research designthen to analyze a research report using that design. Follow-ing the discussion of research designs, related issues such astypes of randomization, how to handle compliance andwithdrawals in a clinical study, and how to interpret Pvalues are presented in Part 4. Additional practice analyz-

ing research reports is provided in the following semester inthe drug information course.Every effort is made to connect research design issues

to practice. To introduce the concept of research design, anexample study problem is presented. The current example problem is a practitioner wanting to conduct a study tdetermine if using community health workers will increasethe number of people who complete prophylactic therapyfor tuberculosis. Students are presented with the questionHow will we know if this program works? and asked togenerate ideas about how they would answer the questionStudent responses are listed on the board and, as appropri-ate, their ideas are translated into research design languageFor example, a student may express concern over the type

of education that the patients in the control group wilreceive outside the study situation; this concern is trans-lated into a concern with internal validity. Or a studentmight suggest that a control group is needed which isfollowed by a brief discussion of how to assign patients togroups.

To further increase relevance to a practice situationexternal validity (generalizability) is presented as answer-ing the question Do the results of this study apply to patients that I see in my practice? Students are directedtoward the inclusion/exclusion criteria for the study and thedescription of the study sample in the results section. They

18 American Journal of Pharmaceutical Education Vol. 62, Spring 98

8/9/2019 aj620103

3/7

8/9/2019 aj620103

4/7

Fig. 3. Cognitivemapexample.

example is shown in Figure 3. The cognitive map enablesstudents to focus on critical characteristics and to differentiate relevant from irrelevant information. The students usethe cognitive map to analyze example research reportsStudents usually analyze six example reports. They areprovided with copies of the reports and the cognitive mapsand asked to read the report and fill in the map before theycome to class. At the beginning of class, they turn in one copyof the map (they keep a second copy for reference duringclass) and then we discuss the characteristics of each study

and the issues associated with it. The discussion follows aprescribed sequence in which the type of research is identified, the purpose of the study, the independent and dependent variables, the statistical results, the primary threat tointernal validity and generalizability to patients seen in practice. In general, the discussion is focused on researcdesign issues; strictly clinical issues may be mentioned buare not discussed in depth.

Experience from teaching the course has shown us thaconcept learning is not a linear, straight forward processTypically, students need to revisit basic concepts before theycan readily identify differences. For example, students canhave a great deal of difficulty learning to differentiaterandom selection from random assignment. The topic is

introduced early in the course and revisited at every oppor-tunity. In the 13th lecture (Topics in Sampling), the conceptsof random selection and random assignment are definedand illustrated. Then as we analyze example research reports, students are asked whether the researchers userandom selection or random assignment. We point out thedifferences between the two techniques and the reasonsrandom selection is not used with clinical studies. Whenstudents analyze example research reports, most studentscan correctly differentiate random selection from randomassignment. Last year, 52 of 53 students correctly identifiedrandom assignment as the technique used to assign subjectsto treatment groups in an experimental study. Another

Fig. 4.Judgmentstepsforassessingaresearchreport.


8/9/2019 aj620103

5/7

difficult objective is learning to differentiate independentfrom dependent variables; identification of variables is re-visited during most research design lectures. On the relevantexam questions, about 92 percent of the students answer thequestions correctly.

Experience also has shown us that students have a naivemisconception concerning the relative importance of inter-nal validity versus external validity. Students typically judgethe quality of a research study by perceived external validityand they tend to treat studies conducted in major research

and teaching hospitals as having little external validity. Tofacilitate learning to assess internal validity first and exter-nal validity only if the study is judged internally valid, asequential process is followed when discussing the report inclass. A supplemental handout (see Figure 4) also is usedthat clearly delineates the sequence of steps in assessing aresearch report: assessing the statistical conclusion, theninternal validity, and finally, external validity(7).

During an informal in-class evaluation of the researchdesign section, students were asked what they knew nowabout research design that they didnt know before. Theresponse of several students was everything, that they didnot know anything about research design before taking theclass. Several students commented that they liked the frame-

work (cognitive map) for analyzing the research reports andone student stated they wished other professors would teachthis way. They also said the homework was very helpful andthey seemed to see value in the class. One student remarkedthat the research design analysis was very basic, an observa-tion that accurately reflects the level of the class.

Human subject considerations is the first topic in Part 4(The Fundamentals of Clinical Trials). Real life examples ofunethical research and violation of informed consent areused to underscore the need for institutional review boardscritical analysis of research protocols. In addition, studentsare shown official written informed consent and projectapproval forms. Ensuing discussion invariably raises differ-ing opinions leading to teachable moments about the role ofinstitutional review boards.

The final part of the class contains two sessions on eachof the topics, survey research and epidemiologic methods.In survey research we concentrate on comparing and con-trasting the four sources of error (coverage, sampling,nonresponse, and measurement) as defined by Salant andDillman(8). Because surveys are so prevalent, it is a widelyheld belief that surveys are easy to conduct. Instructorsstress that survey research requires significant amounts oftime, energy, money, and expertise; particularly when rela-tionships are examined and concepts generated. Therefore,sampling, questionnaire construction, data analysis, andother technical aspects are covered. A journal reprint detail-ing the specifics of evaluating the results of mail surveyresearch is provided to students(9).

The epidemiology objectives were added to the coursefive years ago due to the increased numbers of literaturereports using these methods. The primary objectives are:describe and evaluate the measures of disease in a popula-tion and compare and contrast the study designs used inepidemiologic research with respect to their design andanalyses. Interpretation of relative risk ratios and odds-ratios and associated confidence intervals along with evalu-ation criteria for cohort and case-control studies round outthe materials.

Fig. 5.Examplesofassignmentitemsovertime.

ASSIGNMENTS

Assignments were only 10 percent of the semester gradewhen originally introduced to the course, now twenty-fivepercent of the course grade is based on assignments. Five 20- point homework assignments are interspersed throughouthe semester serving as study guides for examinations. Theassignments are intended to provoke discussion and weencourage students to work together in completing courseobjectives and homework.

Homework assignments have evolved in several waysItems are based on scenarios from research reported in theliterature. The format of items on the assignments waschanged to include a mixture of multiple choice questionssimple calculations, completion of tables, (e.g., source tablefor an analysis of variance or blocked random samplingscheme). The question portion of the multiple choice itemswas converted from one-sentence statements about a con-cept to simple scenarios that required the student to identifythe concept based on the description. For example, ratherthan ask the student which potential threat to internavalidity is most likely representing an experimental design


8/9/2019 aj620103

6/7

the student received a three or four sentence description ofa study with an experimental design (i.e., prospective, con-current comparison groups with random assignment) andwas asked which potential threat to internal validity wasmost likely with this study design. The identification re-quirement was used to further discourage rote memoriza-tion and provide experience in determining study designsbased on descriptions of research (see Figure 5).

The multiple choice items were further revised to in-clude several questions with each scenario. This approach

has several advantages. Well-written scenarios and sets ofresponses require a substantial amount of time and numer-ous iterations. Using the same set of responses for a concept,such as potential threats to internal validity for differentscenarios that may or may not have the same threats re-quires students to compare and contrast the research de-scribed. The assignment contains examples of experimental,quasi-experimental, and crossover designs to provide thestudent with convenient comparisons.

Another advantage of this assignment preparation strat-egy is the ability to change scenarios from year to yearwithout having to completely re-write response items. Stu-dents with access to homework assignments from previousyears should not have an advantage over students without

such access. The effect of access to the additional examplesmay help the student learn the concepts.The third advantage of this approach is the ability to use

reported research as the basis for the scenarios. On-linesearches provide quick and easy access to material forscenarios that are based on actual research. The reports arestill cleaned up somewhat to help the student focus on thestatistical and design concerns, but some red herrings areoccasionally left in to maintain some sense of reality. Theuse of reported studies begins to bring the students closer toworking with the literature. The scenarios may also includetables of data or results, a correlation matrix, or a sourcetable to further simulate the literature. The final assignmentasks a series of questions about a recently published fullresearch article.

The response items used in the multiple choice items arechosen carefully and the importance of knowing why eachresponse is or is not correct is emphasized throughout thesemester. The information in the course builds on itself, soconcepts covered early in the semester are woven into theassignments. For example, a scenario in the last assignmentmay ask the student to determine the level of measurementor identify the dependent variable.

After observing several episodes of a student copyingthe answers to the multiple choice format, assignments weredeveloped with approximately half multiple choice ques-tions and half simple calculation or other hands-on tasks.This was done to encourage students to spend more timewith the material and hopefully discourage last minutecopying.

The calculations are integrated into an overall schemafor conducting a statistical analysis. A template has beendeveloped that may be adapted to a variety of parametricand nonparametric procedures. A series of questions leadsthe student from the research question to the statisticalhypothesis to be tested, selection of an appropriate test,calculation of the statistic, selection of a critical value, a judgment of whether the null hypothesis is retained orrejected, and how this all relates to the original researchquestion. This approach allows the course instructor to

Fig. 6.Assignmentexampleoftestsforcategoricalfrequencydata

change numbers in the small data set or use an entirely newscenario each year without changing the questions thatprovide a model of an approach to statistical analysis. Figure6 shows the use of the template with the chi-square statisticThis commonly used statistic appears to be a source ofconfusion among practitioners and PharmD candidates whocall seeking advice. It is hoped that this increased experiencewith tests for categorical frequency data will improve under-standing in the future.

Assignments have been based on practitioner inquiriesas well as the medical and pharmacy literature. Anothersuccessful approach has been to base an assignment on a fularticle (particularly appealing if research was conducted athe home institution). It has been very gratifying hearinghall talk and seeing debates rage in the student loungeover these assignments. Students rated assignments veryfavorably on course evaluations.

EXAMINATIONS

We assess application and other higher levels of learning, byconstructing items using examples that require students toapply concept skills rather than simply recognize or recalinformation. The multiple choice format has been retainedfor several reasons: item analysis may be performed andtests may be machine graded, while retaining the ability oassessing higher levels of cognition (1,6). Examinations are


8/9/2019 aj620103

7/7

evaluated and revised based on item analysis; such analysisdifferentiates a poorly written item missed by all levels ofperformers from a well written item about a difficult conceptmissed by all but the top performers. There are also coreor very basic items that everyone is expected to get right.Each examination (three total) contains fifty multiple choiceitems and constitutes 25 percent of the course grade.

For the past five years, reliability coefficients havetypically ranged from 0.65 to 0.85 using the KR-20 estima-tion method. In that several assumptions (e.g., normal distri-

bution of both item and test scores, some amount of varia-tion in the item scores) are violated, this is a conservativereliability estimate(10). However, reliability estimates forclassroom examinations are typically much lower. A table ofspecifications is constructed for each exam in order to assesscontent validity(11).

FUTURE PLANS

We continue to evaluate and refine the course based oninstructor, teaching assistant, and student input. Summativeinstructor and course evaluations are conducted each timethe course is offered using standardized forms. Studentsreport that they use the course notes as a reference through-out the remainder of the program and in some cases, at their

residency or practice site. We continuously gather and interpret quantitative data on student, instructor, and course performance. As suggested by Miles and Huberman, wefind qualitative information serves as an additional richsource to validate and reinterpret the quantitative find-ings (12). In the future we hope to have the students manipu-late data sets using statistical software programs.

Am.J.Pharm.Educ., 62, 17-23(1998);received 9/16/97, accepted 1/15/98.

References(1) Draugalis, J.R. and Slack, M.K., A course model for teac hing re

search evaluation in colleges of pharmacy, Am. J. Pharm. Educ., 5648-52(1992).

(2) Educational Outcomes, Center for the Advancement of Pharmaceutical Education Advisory Panel, American Association of Colleges oPharmacy, Alexandria VA (1994).

(3) Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to th eDoctor ofPharmacy DegreeAdopted June 14,1997, American Council on Pharmaceutical Education, Chicago IL (1997).

(4) Boyer, E.L., Scholarship Reconsidered - Priorities of theProfessoriateTheCarnegieFoundationfor theAdvancementofTeaching, Princeton

NJ (1990).(5) Campbell, D.T. and Stanley, J.C., Experimental andQuasi-Experi

mentalDesignsforResearch. Houghton Mifflin Company, BostonMA (1963).

(6) Merrill, M.D. and Tennyson, R.D., Teaching Concepts: AnInstructionalDesignGuide. Educational TechnologyPublications , EnglewoodCliffs NJ (1977).

(7) Cook, T.D. and Campbell, D.T., Quasi-Experimentation-Design &AnalysisIssuesforFieldSettings, Houghton Mifflin Company, BostonMA (1979), pp. 37-39.

(8) Salant P. and Dillman D.A., How toConductYourOwnSurvey, JohnWiley and Sons, New York NY (1994).

(9) Harrison, D.L. and Draugalis, J.R., Evaluating the results of mailsurvey research,J.Am.Pharm.Assoc., 37, 662-666(1997).

(10) Speedie , S.M., Reliability: The accuracy of a tes t, Am. J. PharmEduc., 49,76-79(1985).

(11) Kimberlin, C.L. Character istics desired in tests: Validity, ibid., 4973-76(1985).

(12) Miles, M.B. and Hube rman, A .M., QualitativeDataAnalysis: AnExpanded Sourcebook, 2nd Ed., Sage Publications, Inc., ThousandOaks CA (1994) p.10.


aj620103

Documents

Transcript of aj620103