Methodology and IRB

By Dr. James Lani

Methodology and IRB/URRDr. James Lani

Take Away Message

Research design is a blueprint with several components:

• Research method: Quant/Qual/Mixed• Operational constructs: How

constructs are measured• Sampling strategy and procedure

What is Research

Research follows the scientific method. What’s the scientific method?

• Formulation of testable questions or hypotheses

• It’s organized knowledge: logical (theory) and evidence based (observable)

• Precise constructs• Can be disproven (falsifiable)• Parsimonious (simplest

explanation)

Methodology: The Cookbook Metaphor

Cooking Researching

Making a stew… Examining research questions…

Ingredients: 2 lbs beef, I clove garlic…

Preparation: Cube beef, mince garlic…

Cooking Instructions: Bake at 350 for one hour

Celebrate: You’ve made a replicable) stew!

Ingredients: 20 participants, 15 item questionnaire, semi-structured interview.

Preparation: Administer questionnaire before and after lecture; semi-structure interview with participants for 20 minutes using a tape recorder.

Data analysis plan: Conduct dependent sample t-test; transcribe interviews then thematizeparticipants responses.

Celebrate: You’ve conducted a (replicable) research study!

Where the recipe can be replicated

Methodology Essential Ingredients Restate research

question and hypotheses

Overview

Research Design

Participants

Materials/ Instruments

Data Collection

Procedures

Data Analysis Plan

Sample Size

Limitations

Methods Quantitative Qualitative Mixed

You can count it Not quantitative Both

Goal

Tend to answer “What questions” (What is relationship or differences) or “When questions” (when is theory supported…after intervention

Tend to answer “Why questions” (understand why people feel that way) or “How questions” (explore how they see things)

Both

Research methods/ Strategies of data

collection

Experimental (random assignment) and Non-experimental methods (no control group), Observations (time participant)…

Semi-structured interviews, Archive data, Observations (write down positive and negative feeling words)… Both

Operationalize variables

Define constructs/ instruments

Define constructs/ instruments Both

Sampling strategy Discuss sampling process Discuss sampling process

Both

Theoretical (or Conceptual) Framework

Theory is a systematic explanation of behavior of phenomena.

• Theory guides analyses• List existing theories and how your

research questions relate to those theories

• E.g., Theory of mind: the ability to understand that others have their own beliefs, desires, intentions. Empathy. Tested by Faux Pas Task (ability to recognize a faux pas). Research question: Does alcohol abuse impact empathy?

Population

The population is the group you want to generalize to.

• Describe characteristics of population• Why is population relevant to problem

(look at other peer reviewed study’s justification)

• Distinguish the population from the sample

Sampling Frame and Sample

Universe: theoretical

population to generalize to

Population: largest target population

from universe you have access to or

“Sampling Frame”

Original Sample

Final Sample

Attrition

Survey Item

Old (one’s own SE) Change in item New (perceptions of others’ SE)

1I feel that I have a

number of good qualities.“I have” to “she has” I feel that she has a

number of good qualities.

2I feel I do not have much

to be proud of.“I do not” to “she

does not”I feel she does not have much to be

proud of.

3On the whole, I am

satisfied with myself.“I am” to “she is” and “myself” to “herself”

On the whole, she is satisfied with herself.

…

If you need to amend instrument, use a change matrix; do not create your own instrument!

Materials: Informed Consent

• State purpose of project

• State procedure and how long it will take

• State voluntary nature of participation

• State risks (if any)

• Have them sign or state that by filing our survey they are agreeing to participate.

Constructs vs. Variables

Th

eo

reti

cal

Em

pir

ical

Construct A: Social Environment

Construct B: Personality

Variable A: Birth-Order

Variable B: Introversion/ extroversion

Constructs vs. Variables

Constructs are the invisible abstract things we’re measuring (e.g., personality), while variables are the way we’re assessing (measure/operationalize) those invisible things.

Constructs and variables need to be precise (is personality measured by introversion scale or by conscientiousness scale?)

e.g., Intelligence is a construct, while the number of words remembered is a way of assessing intelligence.

e.g., Personality is a construct, while the scores on an introversion/extroversion test is a way to assess an aspect of personality.

VariablesOperationalize Constructs, Make Distinction between

IV’s/DV’s, and Describe Level of MeasurementExample:

Does Empathy differ by group (alcohol abuse vs. no alcohol abuse)?

Empathy is my dependent variable and Group is my independent variable.

Empathy (my construct) is measured by scores on the Faux Pas task. Alcohol abuse (my construct) is measured by 5 or more drinks in one day.

Empathy is a ratio-level variable measured with scores ranging from 0-25, while my Group variable is a nominal-level (categorical-level) variable because participants re in one of two groups (alcohol abuse group or not in alcohol abuse group).

Reliability and Validity

Valid but not Reliable

Reliable but not Valid

Reliable and Valid

Validity• Internal Validity: IV causes a change in the DV (not time or

other covariates, etc.

• External validity: can be generalized to the population

• Construct validity: does the scale measure the theoretical construct

• Translational validity:

• Face validity (items are reasonable)

• Content validity (items match the domains of interest)

• Criterion related validity: measures behave as theory predicts

• Convergent validity: how close the variable aligns with the construct (use EFA)

• Concurrent validity: construct relates to established instruments

• Predictive validity: measure can predict an outcome (GPA→Income)

Reliability

Internal consistency: Cronbach alpha. Average inter-item correlation

Inter-rater: if interval, correlate; if dichotomous, kappa

Test-retest: administer same test at two times

Split half: divide instrument into 2 parts and calculated totals, then correlate totals.

Brief Review: Units of Analyses

Quantitative (e.g., Age)

Nominal-level (Latin for name). Gender (M/F), Grouped (Old = 65+, middle age = 36-64, young = 35 or younger). Assign any number of groups (old = 1, middle = 2, young = 3).

Ordinal-level is ranked (Latin for showing order). GPA (A-F), or age (group 1 = age 1–15, group 2 = age 16-25, group 3 = age 25-54, group 4 = age 65+)

Interval/Ratio-level (also named scale or continuous; Latin for [equal] space [between numbers]). What is your age today in years? ____ (a number from 1-105)

Types of Methodology Models

Theories explain phenomena,

Models represent Phenomena.

• SEM and Path models• Regression models (linear, logistic, ordinal)• ANOVA models (repeated-measures)• Time-series• Heirarchical Linear Models (HLM)• Correlational Models

Relationship Among Variables

Employee Tenure

Leadership Style

EmployeeSatisfaction

Customer Satisfaction

Mediator

Moderator

Data Collection Method --Overview

• Describe the procedures used to administer the materials to the participants

• Remember to be as detailed as necessary so someone can literally replicate your study

Data Collection

Qualitative • Procedure for accessing participants

• Selection of data collected

• Number and duration of interviews

• How and when data is collected

• How data is recorded (e.g., hand notes, audiotaped)

• Role of researcher-relationship to researcher

Quantitative• Procedure to administer measures

Pilot Test (only if you develop instrument)

• Detect potential issues in the instrument

• Allows you to get feedback and to finalize your survey/interview items

• Makes sure participants understand survey items

• Assess typical responses to survey items or interview questions (were participants comfortable, long-winded, defensive, etc.

• Can you access data?

Data Analysis Plan

• Quantitative: Describe the analysis plan used to test each hypothesis, the assumptions of the statistical analyses, and a justification of the appropriateness of the analysis for each research question.

• Qualitative: Describe how the data will be analyzed (or thematized)

• Phenomenological

• Case Study

• Grounded theory

Sample SizeQuantitative

Varies by type of statistical analysis

1. Research questions in statistical language2. Level of measurement of variables3. Determine statistical analysis4. Conduct power analysis

Qualitative

Varies by which qualitative approach is taken

• Phenomenological• Case Study • Grounded theory

Assumptions, Limitation, Delimitations

Assumptions: Discuss things out of your control about the population and design, then justify assumptions (e.g., participants will answer honestly.)

Limitations: Are out of your control and describe weaknesses in design, threats to validity (e.g., generalizability).

Delimitations: Are in your control and relate to choices you will make to narrow the scope of the study (e.g., variables, research questions).

Ethical Considerations

• Describe informed consent procedures

• State whether your study will be anonymous or confidential with respect to the participants

• Describe considerations for children or vulnerable participants

See yourself graduate in 2014!

1-1 Personalized Dissertation Consulting

877-437-8600

Info@StatisticsSolutions.com

Thank you for your participation

and attention!

Join us for our next webinar on Wednesday, December 17th at 8:30pm

ET.Results and Discussion Chapters,

Editing, and Defense

Join us for our 1-day Dissertation Workshop

Dallas/Fortworth, Texas December 6th 9am-5pm

$199/student $89 lodging/night