Review of "Survey Research Methods & Design in Psychology"

Survey Design II

Lecture 12
Survey Research & Design in Psychology
James Neill, 2011

Survey Research & Design in Psychology:Review & Summary

7126/6667 Survey Research & Design in PsychologySemester 1, 2011, University of Canberra, ACT, AustraliaJames T. NeillHome page: http://ucspace.canberra.edu.au/display/7126Lecture page: http://ucspace.canberra.edu.au/display/7126/Lecture+-+ReviewNotes page:http://en.wikiversity.org/wiki/Survey_research_methods_and_design_in_psychology

Image name: Recycle symbol Taiwan.svgImage source: http://commons.wikimedia.org/wiki/File:Recycle_symbol_Taiwan.svgImage author:Bryan Derksen http://commons.wikimedia.org/wiki/User:Bryan_DerksenLicense: Public domain

Description:Reviews this semester-long (150-hour), third year undergraduate psychology research unit which focused on survey research methods and survey design. This lecture emphasises the second-half of the unit's content on MLR, ANOVA, significance testing, power, and effect size, as well as providing advice about the lab report and final exam assessment items.

Overview

Review

Unit aims and outcomes

Research process

Survey design

MLR, ANOVA, ES & Power

What type of analysis? (decision tree)

AssessmentLab report

Final exam

Evaluation & feedback

Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain

Image source: http://commons.wikimedia.org/wiki/File:Autoroute_icone.svgLicense: CC-BY-A 2.5Author: http://commons.wikimedia.org/wiki/User:Doodledoo

Unit aims & outcomes

http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Overview

Aims & outcomes

Knowledge & skills
for conducting ethical,
well-designed, survey-based research in psychology.

How confident are you that could conduct a good quality survey-based research study?

For 4th year
Honours?

In the
work-place?

Aims & outcomes

Theory & practice
of survey-based research, incl.:

Research questions / hypotheses

Survey design

Sampling

Interpreting & communicating results

Aims & outcomes

Use of SPSS for:

Data entry

Correlations

Factor analysis

Qualitative analysis

Reliability

MLR

Advanced ANOVA

Research process

Research process

1. Establish
need for info/
research2. Problem
definition/
Hypotheses3. Researchdesign4. Sampling/Data collection5. Data
analysis6. Reporting

Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/

Funnel model

Image source: Unknwn

Survey design

Survey design

Operationalisation of fuzzy concepts

Question types (objective/subjective, open/closed)

Response formats (e.g., dichotomous, frequency, Likert, multiple response, idiographic)

Levels of measurement (LOM)

Reliability & validity

Sampling (probability/non-probability)

Modes of administration (self-report (f2f, mail, web), interview (f2f/phone))

Items should
measure different
aspects of thelatent (underlying)
constructLatent Construct

Measured
Construct


Latent Construct

Measured ConstructPoor items
will create
brown sludge
(noisy
(unreliable)
measure)


Describing data


Describing data

Data screening (out of range, invalid etc.)

Discrete data: Frequencies & %s

Continuous data:
4 moments of a normal distributionCentral tendency

Dispersion

Skewness

Kurtosis

Visualisation of data

Aids interpretation of descriptives and tests of differences or relationships.

Univariate: histogram, bar graph, error-bar graph

Bivariate: scatterplot, clustered bar graph

Multivariate: Venn diagrams, multiple line graph, 3-d scatterplot

Software for
data visualisation (graphing)

Statistical packages

e.g., SPSS Graphs or via Analyses

Spreadsheet packages

e.g., MS Excel

Word-processors

e.g., MS Word Insert Object Micrograph Graph Chart

Correlations


Correlations

Strength & direction of bivariate linear relations

Building block for FA & MLR

Non-parametric correlations
(e.g., Point bi-serial, Phi/Cramer's V)

Scatterplots watch out for: Outliers

Non-linearity

Limited range

Caution with causal interpretation

Factor analysis


Factor analysis

Purpose Data reduction

Developing reliable & valid measures of fuzzy construct

Assumptions Linear relations

Sample size
(min. 5 cases: 1 variable)

Factor analysis

ExtractionPC vs. PAF

Rotation methodVarimax vs. Oblimin

Number of factorsKaisers criterion

Scree plot

Theoretical structure

Interpretation of factor loadings

Factor analysis

Decide how many factors

Iteratively eliminate itemsCommunality > .5?

Primary loadings > .5?

Cross-loadings < .3?

Diff. btw primary & cross-loadings > .2?

Sufficient items per factor

Face validity

Correlations between factors

Compare model across groups

Compare models across groups

% variance explained

No. of factors

Item loadings

Reliabilities & composite scores

Internal reliability: For each factor, calculate Cronbach s : > .8 = very good

> .7 = good

> .6 = OK

Composite scoresUnit-weighting

Regression-weighting

Reversing a scale e.g.,IM = mean(item1,item2,item3)EM = mean (item4,item5,item6)M = IM + (8 EM)1 2 3 4 5 6 77 6 5 4 3 2 1




Purposes Pilot study, exploratory research

Theory testing; Data reduction (synthesise meaning)

Validity-testing

Methods Quantitative (Content analysis, Multiple response analysis, Graph (e.g., bar graph)

Qualitative (Thematic analysis)

Inferential statistical
decision making tree


Statistical decision tree

Establish a research question and/or hypothesisDifferences or relationships?

No. of IVs and DVs

Identify levels of measurement

Use a statistical decision tree to identify an appropriate analysis

SeeInferential statistics decision-making tree

Multiple linear regression



Linear regression formula

Y hat = ax + bY = ax + b + eProportion of variance in a DV explained by one or more IVsR, R2, Adjusted R2

F (significance of R)

Change in R2 (Hierarchical)


Assumptions: LOMContinuous DV

Dichotomous or continuous IVs

Normality (helps satisfy other assumptions)

Linearity

Homoscedasticity

Multicollinearity

Multivariate outliers


Methods Standard / Direct

Hierarchical

Forward

Backward

Stepwise

Overall hypothesis: (Null) That the IVs do not explain variance in the DV (i.e., that R is 0)

+ one hypothesis per predictor: (Null) That each IV is not a significant predictor of variance in the DV (i.e., that t for each predictor is less than the critical value)


Consider/interpret:Direction of each predictor

Size of each predictor ( - standardised)

When IVs are correlated, interpret zero-order vs. partial correlations

sr2 = % variance in DV explained by each IV

Can use Venn or path diagrams to depict relationships between variables


ANOVA


ANOVA

Extension of t-test

Like MLR, ANOVA involves:One continuous DV (although ANOVA can handle multiple DVs)

One or more IVs

Unlike MLR, in ANOVA:Interactions are automatically tested

IVs must be categorical

Significant results may indicate need for followup or post-hoc tests

Types of ANOVA

1-way ANOVA

1-way repeated measures ANOVA

2-way factorial ANOVA

Mixed design ANOVA
(Split-plot ANOVA)

ANCOVA

MANOVA

ANOVA

AssumptionsCell size > 20 (Ideal)

Normally distributed DVs (for each level of the IVs)

Homogeneity of variance (b/w subjects)

Sphericity (w/in subjects)

Follow-up tests:Post-hoc tests

Planned comparisons

Effect sizes: 2 , p2 and Cohens d

Power, effect size, significance testing, publication bias & academic integrity


Significance testing

Significance testing has dominated psychology, but is problematic, mainly because:Results are dichotomous (sig. or not), which doesn't help us to understand the size of effect.

Sig. test results are influenced by power esp. if particularly high or low.

Power & effect sizes

Power and effect sizes have been neglected. Therefore:Calculate the power of studies (prospectively &/or retrospectively)

Report ESs & CIs to complement inferential statistics

r, r2, R, R2

2, p2, d

Publication bias &
academic integrity

Publication bias (low power; favouritism of sig. findings; funnel plots)Academic integrity - Integrity is doing the right thing, especially when no one is watching.

Lab report


Lab report - Tips

Check the lab report guidelines and the marking criteria

Use example HD lab report and sample write-ups as guides

Demonstrate your knowledge via independent thinking/work

Tell a story

Lab report - Structure

Cover Sheet

Title Page

Abstract

Body

References

Appendices

Lab report - Introduction

Explain the rationale what is the problem to be solved?

No waffle cut to the chase only review literature or argument relevant to the RQs / hypotheses

State clear RQs and/or hypotheses One per test/analysis/effect

Lab report - Method

Well-organised
(like a recipe)

Present relevant details efficiently (avoid getting bogged down in extraneous detail)

Lab report - Method

Sections:

Participants

Materials or Instrumentation

Procedure

Replicable?
A nave reader must be able to replicate the study

Lab report Results

Data screening

Consider LOM assumptions

Caution in use of overall scores

132132Overall Score not validOverall Score validImage source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/

Lab report Results (EFA)

EFA:PC/PAF?

Varimax/Oblimin?

2-6 factors?

5-30 items removed?

50%-60% of variance?

Table of factor loadings and communalities

Correlations between factors

Internal consistency for each factor

Lab report Results (MLR)

Example MLR: Hierarchical:DV = Campus Satisfaction

Step 1IV1 = Gender (M / F)

Step 2IV1 = Planning TM (Continuous)

IV2 = Time wasting TM (Continuous)

R2, Adjusted R2, Change in R2

Table of correlations and regression coefficients

Lab report Results (ANOVA)

ANOVA: e.g., 2 x (3) Mixed ANOVABetween-subjects IV:
Enrolment Status (FT / PT)

Within-subjects DV:
Satisfaction (Educational / Social / Campus)

Table of cell and marginal descriptives (M, SD, Sk, Kurt) + Graph

Effect sizes (p2 , d)

Lab report Results (Qualitative)

Example Qualitative analysis: Least or most satisfactory aspects of UCResearch question?

Qualitative or quantitative approach?

Table of main themes (names, description, example quotes, frequency/%)

Bar graph?

Thick description?

Sub-samples?

Lab report - Discussion

Provide insight about the results

Draw conclusions about the RQs & hypotheses in light of the results.

Discuss key strengths & limitations of the study. (Balanced criticism)

Draw out implications and recommendations

Lab report - Discussion

Offer specific, practical recommendations e.g.,

Theory: What are the implications for the theory/rationale upon which the study was based?

Methods: How could the research design (e.g., instrumentation) be improved?

Practice: Implications for students and universities e.g., for improving satisfaction?

Lab report Appendices

Optional: Include appendices where relevant and referred to in the body text. Appendices may not be consulted by a reader, so if its important make sure key content is covered in the text.

Use for content which would break the flow, but which is relevant to understanding the study e.g., the EFA correlation matrix.

Lab report Appendices

APA style not necessary.

Use headings (e.g., Appendix A, B, C etc.) and possibly titles e.g.,

Appendix A:
Bivariate correlations amongst the university student satisfaction items

Lab report - Submission

Insert & complete the Cover Sheet

Submit ONE DOCUMENT containing the coversheet, lab report and appendices

NO EXTRA ATTACHMENTS

Upload via the Moodle drop-box

You can re-submit up until the due date (after that, resubmission will incur late penalties)

Lab report - Marking

Lab report marks should be returned before the exam.

Keep an eye on your student email and Moodle Announcements.

Final exam


Final exam

Computer-based, supervised

Style, structure and content will be similar to the quizzes

~120 multi-choice questions in 180 mins (plenty of time).

Practice exams are available

Final exam

Permitted materials

Non-programmable calculator

Non-annotated foreign language dictionary

Marks and feedback

Upon submitting, you will receive your exam mark and feedback.

Evaluation & feedback


Evaluation & feedback: USS

Unit Satisfaction SurveyAvailable now on OSIS but you may wish to wait until after assessment results

UC takes these results very seriously ~40% response rate

Possible issues & topics

Please contribute your honest feedback:Lectures?

Tutorials?

Textbooks?

Assessment?

Website(s)?

Software SPSS?

Workload?

You are also welcome to
tell me directly or
email me or
post feedback on the
Moodle discussion forum.

Open Office Impress

This presentation was made using Open Office Impress.

Free and open source software.

http://www.openoffice.org/product/impress.html

Click to edit the title text format

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level







Review of "Survey Research Methods & Design in Psychology"

Education

Transcript of Review of "Survey Research Methods & Design in Psychology"