Slide 1

Chapter 14

Designing ExperimentsPart A1

This is Dr. Olson and the topic of todays lesson is Chapter 14, Designing experiments, part a.1

AgendaDescribe features of experimental designDefine and describe measures to eliminate bias and reduce the influence of sampling error2

In todays session, we are going to define and describe measures to eliminate bias and reduce the influence of sampling error.2

Key TermsConfoundingExperimental artifactExperimental design componentsGoal of experimental designControl groupRandomizationCompletely randomized designSingle blindDouble blind

BlindingReplicationBalanceBlockingRandomized block designExtreme treatmentsFactorFactorial designInteractionMatching Sample size

3

There are a number of key terms to learn as it relates to experimental design. Ive listed these terms that come from your text as a friendly reminder to ensure that you focus on learning and discriminating between them. If you work in someones lab or decide to work in the clinical or industrial research industry, these terms will be used frequently. Take the time to learn them.3

Correlational StudiesNaturalistic observationArchival researchSurveys and questionnairesOne of the most common of methods in social sciencesRandom sample of participants completes a survey, test, or questionnaire that relates to the variables of interestRandom sampling is vital to generalizability of results4Available at: http://psychology.about.com/od/researchmethods/a/correlational.htm

First, we are going to briefly talk about correlational research and then well focus more on experimental methods.While correlational research can suggest that there is a relationship between two variables, correlational research cannot prove that one variable causes a change in another variable. In other words, correlation does not mean causation.

So, some of the ways that correlational research plays out is in naturalistic observation such as in the field or the wild where you are observing humans or animals in an unobtrusive fashion. This is called naturalistic observation. In my field of work in helping companies to design medical devices, I will go into a healthcare context and conduct what is called contextual inquiry. Ill simply observe people doing their jobs with patients and caregivers that will lead to impressions about how a device might be designed to accomplish some function.

Archival research is performed by analyzing studies conducted by other researchers or by looking a historical patient records. For example, researchers recently analyzed the records of soldiers who served in the Civil War to learn more about PTSD.

Survey and questionnaires are one of the most common methods used in psychological research. In this method, a random sample of participants completes a survey, test, or questionnaire that relates to the variables of interest. Random sampling is a vital part of ensuring the generalizability of the survey results.

4

Observational ResearchAdvantages of naturalistic observationGives experimenter opportunity to view variable of interest in natural settingIdeas for further researchMay be only option if lab experiments are not possibleDisadvantages of naturalistic observationTime-consuming and expensiveDoes not allow for scientific control of variables5Involves observing and recording the variables of interest in the natural environment without interference or manipulation by the researcher

Naturalistic observation entails observing and recording the variables of interest in the natural environment without interference or manipulation by the experimenter.

If offers ideas for research and is oftentimes the only option if lab experimentation is not possible. It can be time-consuming and expensive and does not allow for scientific control of variables and the experimenter cant control extraneous variables. So, too, the subject of the research may be aware of the observer and may act differently.5

Limitations of Observational ResearchCant distinguish between two completely different reasons behind an association between an explanatory variable and a response variableExplanatory variable may have no effect at all on the response variableMay be associated only because other variables affect both X and Y at the same time6

Correlational research is one type of observational research looks for relationships between variables. There are three possible results of a correlational study: a positive correlation, a negative correlation, and no correlation. While correlational research can suggest that there is a relationship between two variables, they cannot prove that one variable causes a change in another variable. In other words, correlation does not mean causation.

For example, a correlational study might suggest that there is a relationship between academic success and self-esteem, but it cannot show if academic success and increases or decreases self-esteem. Other variables might play a role, including social factors.

6

Confounding VariableVariables other than the variable that you manipulate that can have an effect on the response variableCan increase the variance of the measurements of the response variableMasks or distorts the causal relationship between an explanatory variable and a response variable in a study

7

Variables other than the variable that you manipulate can have an effect on the response variable by increasing the variance of the response variable.Confounding variables can mask and distort your measures and make it very difficult to determine causality. They can even get in the way to the extent that they reverse any apparent effect that one has on the other. With random assignment, no confounding variables will be associated with the experimental treatment except by chance alone.

7

Overview of ExperimentsMust be at least two treatmentsExperimenter must assign treatment to the research participant or subjectRandom assignment minimizes influence of confounding variables, fostering isolation of the effects of the treatment variableDifferences between groups would be caused by the treatment8

Many experiments compare two sets of measurements of the same variable. In conducting experimental research, the experimenter must assign research participants to the experimental or control conditions.

By using random assignment, we minimize the influence of confounding variables and foster isolation of the effects of the treatment variable. That way. Can differentiate between groups.

Weve talked in this class about paired designs and independent designs. This is where experimental design comes to play.

Before you start an experiment, it is a good idea to put some thought into which type of experimental design you will use. When the dependent variable cannot be manipulated, you probably need an independent design.A good example of this point would be an experiment that compares males with females. You cannot test the same people under each condition (male and female) because you cannot change their gender, so unless you are pairing them in some other way (twins, for example) you must compare one group of males with a group of different females.

When you cannot repeat a measurement on the same experimental unit, you probably need an independent designWhen using a paired design, you often test (or measure) the same experimental units twice. If the first test is likely to have an effect on the second test or if the unit can be tested only once, then the second, repeated test is not possible.

If you need to test the same experimental unit under different conditions, you need a paired design.

There are some other considerations when choosing an experimental design. It is easier to control for confounding variables with a paired design. Anything other than the independent variable that might affect your measurements is referred to as a confounding variable. There is always a risk that an experiment might produce a difference that is due to something other than the independent variable. For example, if you wanted to compare the heights of males and females but you chose one group from a basketball team and the other from a group of jockeys, you might find a difference, but would it be due to gender? By measuring the same subjects twice, you reduce the risk of introducing confounding variables..

If you are testing people, you may find that giving them the same test twice under different conditions allows them to practice the task in the first condition, which naturally produces an improvement in the second condition. Similarly, subjects might grow bored of the task and show fatigue effects when asked to perform it under the second condition.8

Design ComponentsBiasSimultaneous control groupRandomizationBlindingSampling errorReplicationBalanceBlocking9

A good experiment is designed with two objectives-Reduce bias in estimating treatment effects-Reduce effects of sampling error

A good experiment is designed with two objectives in mind. I call this keeping the end in mind.One is to reduce the bias in estimating treatment effects and the other is to reduce the effects of sampling error.

9

Eliminate biasReduced by use of controls, randomization, and blindingReduce sampling errorIncrease precisionIncrease powerGoal of Experiments

The goal of experimental design is to eliminate bias and to reduce sampling error when estimating and testing the effects of one variable on another. This is achieved by the use of design controls, randomization, and blinding.

It results in a more precise measurement and increases the power of your statistical tests.

To reduce the effects of sampling error, the experiment should be designed in such a way that the study is carried out on multiple independent subjects. In the clinical trial world, we do that be enrolling subjects at clinical locations across North America and sometimes even across the globe.

We want to make sure that we achieve balance by enrolling equal numbers of men and women and ethnic groups.For instance in my work, I often will travel to sites on the east coast, west coast, central US, north and southern cities to ensure that my sample reflects the population.

We also then group subjects by the clinical location where they undergo study procedures. That is often recorded as some kind of numbering scheme.10

Design Features that Reduce BiasControlsRandom assignment to treatmentsBlinding11

A control group is a group of subjects who do not receive the treatment but who otherwise experience similar conditions as the subjects that do receive the experimental treatment. Treatment and control subjects should undergo study procedures simultaneously or in a random order to ensure that any temporal (time) changes in experimental conditions do not affect the outcome.

In clinical trials, a control may be a placebo or the standard of care. A placebo is an inert substance that is indistinguishable from the treatment.

Once treatments and controls are designed for a trial, then randomization of subjects to treatments needs to be considered. The randomization scheme is generally done by a statistician to ensure that each subject has an equal chance of being selected for either the experimental condition or the control.A completely randomized design is an experimental design in which treatments are assigned to all units or subjects by randomization.

The reason that we randomize is that doing so breaks the association between possible confounding variables and the explanatory variable, allowing the causal relationship to be assessed. It doesnt eliminate variation due to the confounding variables, only their correlation with the treatment. It ensures that variation from confounding variables is similar between treatment groups. If randomization is properly carried out, then any remaining influences of confounding variables occurs by chance alone, which statistical methods can account for.

Now, lets talk blinding. Blinding refers to concealing from subjects and sometimes even researchers which group the subject has been assigned. Blinding prevents the experimenter from behaving differently across treatment groups. A couple of years ago, I worked on a study which the experimental treatment was one color and the control was another. For reasons that I was never aware, the control could not be made to look like the experimental treatment. The surgeon could not see what color the patient received, but others in the room could and you know that people talk. We really had a difficult time ensuring that we kept the blind.11

Experimental ArtifactsBias in a measurement produced by unintended consequences of experimental proceduresConduct under conditions as natural as possibleObservational studies can provide important insight into what is the best setting for an experimentSimulated use environment is used in medical device testingClinical trials are conducted in hospitals/clinics12

An artifact is a bias in a measurement produced by unintended consequences of experimental procedures.

To control for artifacts, the study should be conducted under conditions as natural as possible.If the medical device is to be used in the home, test it in a simulated use environment.

For instance, studies involving the use of computers are often tested by human factors scientists. The test environment will be set up to look like a home if that is where it will be used. If it is a computer game for kids, the test environment might be set up to look like a kids bedroom or a den.

If it is an autoinjector, well set it up to look like a kitchen table or perhaps even a mall for something like an Epipen. In this case the scenario introduces the look and feel of a real situation.12

Elements of Design in Clinical TrialsSimultaneous control groupTreatment group and control groupRandomization of subjects to treatmentsBlindingNeither subject nor investigator know which subject is assigned to which treatment13Reduce Bias

A control group is a group of subjects who are undergo all the study procedures except exposure to the experimental condition. If there is no control, cause and effect can not be determined.

There are a myriad of reasons for this including that people change their behavior just because they know they are being observed. Another reason is that they may experience a placebo effect. In clinical medicine and psychology, people get well simply because of the white coat phenomenon.

Some of the methods of ensuring that we reduce this type of bias is to administer saline to a control group if the other group gets an injection.

In animal studies, if a surgical procedure is conducted, the animal will be anesthetized and a surgical incision made and then closed. The experimental group will undergo the surgical treatment option while the control simply undergoes all but the treatment. Sham surgeries do happen in medicine, but it is rare.

In clinical medicine, you may see an experimental condition, a standard of care condition, and a placebo condition.I did a search and found a study that examined the effects of orally inhaled DHE, oral moxifloacin, and placebo. The study drug was an inhaler and there was a placebo inhaler as well.

In another study of the effects of phenylephrine on nasal congestion in seasonal allergies, there was the standard of care, a comparator, and a placebo. Oftentimes, if placebo is used, there will be a rescue medicine.

Certainly, with placebo, the goal is to ensure that no one knows who gets placebo.13

Control GroupGroup which is identical to the experimental treatment in all respects aside from the treatment itself Placebo group as controlSome illnesses, e.g., pain and depression respond to fact of treatment, even with no pharmaceutically active ingredientsControl inert substanceControl Groups in Clinical Trials

Just a little more here about controls and placebo, remember, a control group is identical to the experimental group except for exposure to the experimental condition.

Placebo as a control is so important and is real.Some particular illnesses such as depression and even the pain phenomena are notorious for needing special statistical measures to parse out the placebo effect.

14

Reducing BiasRandom Assignment

Averages out the effects of the confounding variables

There are many ways to randomize and as I mentioned previously, a statistician generally creates the randomization scheme.

Your book points out a variety of methods. The one you see here involves 8 subjects randomly assigned a number between 0 and 99. Treatment A, noted in dark orange in red was assigned to the four subjects with the lowest random numbers, whereas treatment B was assigned to subjects with the higher numbers. In other words, each subject had an equal chance of being assigned to conditions at the time they received a random number.

There are other methods as well, such as assigning treatment A to controls at one location and experimental conditions to another location. It has the potential of introducing confounding variables and so systematic differences may not be dispersed across both sites.

Another practice is to use a random number generator. Challenges with this methodology are that some random number generators use the same seed number each time, so that needs to be managed as well. Other types of assignment of subjects to conditions are outside the bounds of this introductory text.15

Reducing BiasPreventing knowledge of the experimenter (or subject) of which treatment is given to whomUnblinded studies usually find much larger effects(even threefold) demonstrating bias that results from lack of blindingBlinding measuresSingle-blindDouble-blindBlinding

Blinding or masking is intended to limit the occurrence of conscious and unconscious bias in the conduct and interpretation of a trial arising from the influence which the knowledge of treatment can have on research participants.

The essential aim is to prevent identification of the treatments until all opportunities for bias has passed. This may be after the results of the study are analyzed and all study sites closed.

There are a variety of blinding methods. Two are noted here. One is the single blind where the research participant does not know to which condition sh/e is has been assigned and the other is the double blind whereby the research participant nor the experimenter know to which condition the participant has been assigned.

16

17No confounding variables will be associated with treatment except by chanceCompletely randomized designTreatments are assigned to experimental units by randomizationReduces bias due to confounding variables by making non-experimental variables equal between treatmentsRandomized block designReducing BiasRandomization

When randomizing, we aim to ensure that no confounding variables will be associated with treatment, except by chance. The randomized block design and completely randomized design use methods to control these variables.Weve talked about differences among experimental units and how they can make a significant contribution to error variance, thereby masking or obscuring the effects of a treatment.

Similarly, administering the levels of a treatment under different environmental conditions, say, at different times of the day, locations, and seasons of the year can contribute to error variance and masks treatment effects. Variation in the dependent variable attributable to such sources is called nuisance variation and we can hold the nuisance variable constant.

We can also assign the experimental units randomly to the treatment levels so that known and unsuspected sources or variation are distributed over the entire experiment and thus dont affect just one or a limited number of treatment levels. We can include the nuisance variable as one of the factors in the experiment.

The latter approach uses a blocking procedure to isolate variation attributable to the nuisance variable so that it does not appear in estimates of treatment and error effects. The blocks are formed so that at the beginning of the experiment, the experimental units in each block are more homogeneous with respect to the nuisance variable than are those in different blocks.17

18Experiments with More than One FactorA factorial design is the most common experimental design used to investigate more than one treatment variable, or factor at the same timeIn a factorial design, every combination of treatments from two or more treatment variables is investigatedA factor is a single treatment variable whose effects are of interest to the researcher

Ronald A. Fisher introduced factorial designs in 1926. The design represented a fundamental advance in the scientific method because it enabled researchers to simultaneously investigate two or more treatments in a single experiment. It is one of the most widely used experimental designs. It is used in the life sciences, biology, behavioral sciences, and even engineering.

All factorial designs have two or more levels and the levels of different treatments are combined to form treatment combinations.

The factorial design is from three building block designs. The completely randomized design, randomized block design, and the Latin square design. Latin square is outside the bounds of this class, so I wont go into it here.

The simplest factorial design from the standpoint of analysis and assignment of experimental units to treatment combinations is the completely randomized factorial design. This design consists of at least two treatments and two or more levels. All levels of each treatment are investigated in combination with all levels of every other treatment. For example, if there are If there are p levels of one treatment and q levels of a second treatment, the experiment contains p x q treatment combinations. Each experimental unit must be assigned to only one combination.

18

InteractionsThe main purpose of a factorial design is to evaluate possible interactions between variablesAn interaction means that the effect of one variable on the response depends on the state of a second variable19An interaction between two or more explanatory variables means that the effect of one variable depends upon the state of the other variable

When two or more variables are investigated, there is more to consider that the main effect of each of the factors. The effect of one independent variable on the dependent variable of interest may not be the same at all levels of the independent variable.

A factorial design is a requirement to find an interaction effect. That means that two are more independent variables interact with one another so that there are observations at every combination of levels of the independent variables.

Lets take the Lake Lady Bird example from Exam 2 and talk about factors in that design.

We were interested in measuring the stress levels of people who either walked around Lake Lady Bird or did yoga. Lets say now we are interested in measuring their stress on a interval scale from 0 to 100. Lets measure these folks after 30 minutes of high intensity or 60 minutes of moderate intensity effort.

So, what we have is type of exercise and level of intensity as our factors and a measurement of stress as the response variable. So, this experiment if carried out would have two factors with two levels each.

What we would do is compute means and the grand mean for the study. One way to answer this question is to describe the main effects and to visually inspect the data using bar graphs or line graphs. Wed plot our data and depict the main effects of the two independent variables and their interactions.

19

20Examples of Interactions

F. Main effect of A No;Main effect of B - YesInteraction - NoMain effect of A Yes;Main effect of B No; Interaction - YesMain effect of A No;Main effect of B Yes;Interaction - Yes Main effect of A Yes; Main effect of B Yes;Interaction - NoD. Main effect of A No;Main effect of B No;Interaction effect - YesE. Main effect of A YesMain effect of B Yes;Interaction - Yeshttp://courses.washington.edu/smartpsy/interactions.htm

Here are six plots that I found on the web. They are important visual tools to determine main effects and interactions. For each plot, the dependent variable is on the vertical axis and the factors are on the horizontal axis. There is an interaction effect when one bar crosses the other or when one bar goes up while the other bar goes down.For more information, please click on the link.

20

Questions

21

So this concludes our discussion of key features of experimental design. Weve defined and described measures to eliminate bias and reduce the influence of sampling error.

Part B will be posted and will cover sample size and power calculations.

If you have any questions, please send me an email note via Canvas Email.21