Academic Assessment at UTB Steve Wilson Director of Academic Assessment.
Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic...
-
Upload
nikhil-vasey -
Category
Documents
-
view
215 -
download
0
Transcript of Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic...
![Page 1: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/1.jpg)
Analyzing Survey Data
Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop
May 14th & 15th
UNLV
![Page 2: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/2.jpg)
Prior to Analysis
What would you like to discover? Perceived competence Preferences, satisfaction Group differences
Demographics
What are your predictions?
![Page 3: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/3.jpg)
Prior to Analysis
Your goals drive the make-up of the survey and how it should be analyzed.
Exploration can be informative, but with an analysis plan.
![Page 4: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/4.jpg)
Prior to Analysis
Survey design & layout Stylistic considerations are important because they
increase response, validity, and reliability
![Page 5: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/5.jpg)
Survey Design
Good questions reduce error
By increasing the respondent’s willingness to answer
Increases reliability and validity. Less error = better data
![Page 6: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/6.jpg)
Reliability & Validity
Reliability – Is the survey measuring something consistently? Typically measured using Chronbach’s alpha
Validity – Is the survey measuring what it’s supposed to be measuring? Typically measured using factor analysis
![Page 7: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/7.jpg)
Construct Validity
Does your measure correlate with a theorized concept of interest? Correlate measure with values that are known
to be related to the construct.
![Page 8: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/8.jpg)
Pilot
Piloting the survey can inform: Question clarity Question format Variance in responses
![Page 9: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/9.jpg)
Survey Analysis
Using data from Paper Surveys SurveyMonkey SelectSurvey.Net
![Page 10: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/10.jpg)
Survey Analysis
Paper surveys
Put data in spreadsheet format using excel or SPSS
Columns represent variables Rows represent respondents
![Page 11: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/11.jpg)
Survey Analysis
Paper surveys
Create a data matrixVariable name || Numeric Values || Numeric labels
Summarize open-ended questions separately Response group || frequency
![Page 12: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/12.jpg)
Survey Analysis
SurveyMonkey Available under the analyze results tab
Frequencies & crosstabs Download all responses for further analysis
Select Download responses from menu Choose type of download – select all responses
collected Choose format – select condensed columns and
numeric cells.
![Page 13: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/13.jpg)
Survey Analysis
SelectSurvey.NET Available under Analyze Results Overview
Frequencies Download all responses for further analysis
Select Export Data from Analyze page Export Format – CSV (excel) Data Format – SPSS Format Condensed
![Page 14: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/14.jpg)
Data Cleaning
Process of detecting, diagnosing, and editing faulty data
Basic Issues: lack or excess of data outliers, including inconsistencies unexpected analysis results and other types of
inferences and abstractions
![Page 15: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/15.jpg)
Data Cleaning
Inspect the data Frequency distributions Summary statistics Graphical exploration of distributions
Scatter plots, box plots, histograms
![Page 16: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/16.jpg)
Data Cleansing
Out of range Delete values and determine how to recode if possible
Missing Values Refusals (question sensitivity) Don’t know responses (can’t remember) Not applicable Data processing errors Questionnaire programming errors Design factors Attrition
![Page 17: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/17.jpg)
![Page 18: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/18.jpg)
Missing Data
Missing completely at random (MCAR) Cases with complete data are indistinguishable from
cases with incomplete data. Missing at random (MAR)
Cases with incomplete data differ from cases with complete data, but pattern of missingness is predicted from variables other than the missing variable.
Nonignorable The pattern of data missingness is non-random and it
is related to the missing variable.
![Page 19: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/19.jpg)
Missing Data Listwise or casewise data deletion: If a record has missing data
for any one variable used in a particular analysis, omit that entire record from the analysis. Default in most packages, including SPSS & SAS
Pairwise data deletion: For bivariate correlations or covariances, compute statistics based upon the available pairwise data. Useful with small samples or when many values are missing
Substitution techniques: Substitute a value based on available cases to fill in missing data values on the remaining cases. Mean Substitution, Regression methods, Hot deck
imputation, Expectation Maximization (EM) approach, Raw maximum likelihood methods, Multiple imputation
![Page 20: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/20.jpg)
Descriptive Statistics
Frequency distribution
![Page 21: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/21.jpg)
Descriptive Statistics
Cross-tabs Excel Pivot tables
Excel menu Data PivotTable and PivotChart
PivotTable menu Field setting summarize by count show data as % of row or column
![Page 22: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/22.jpg)
Data Analysis
Measurement scale determines how the data should be analyzed: Nominal, ordinal, interval, ratio
Move from categorical information, to also knowing the order, to also knowing the exact distance between ratings, to also knowing that one measurement in twice as much as another.
![Page 23: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/23.jpg)
Data Analysis
Three instructors are evaluating preferences among three methods (lecture, discussion, activities) 1) Identify most, second, and least preferred. 2) Identify your favorite. 3) Rate each method on a 10-point scale,
where 1 indicates not at all preferred and 10 indicates strongly preferred.
![Page 24: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/24.jpg)
Data Analysis
Nominal & ordinal variables are discrete Can be qualitative or quantitative
Interval & ratio variables are continuous Grades Age
![Page 25: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/25.jpg)
Data Analysis
Charts Pie charts & bar charts
used for categorical data
Histograms used for continuous data
Line graphs typically show trends over time
![Page 26: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/26.jpg)
Data Analysis
Other descriptive statistics Mean
preferred, uses all of the data Median
ordinal data open-ended scale outliers
Mode nominal data
![Page 27: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/27.jpg)
Data Analysis
Other descriptive statistics Interquartile range
Variability accompanying the median Standard deviation
Variability accompanying the mean
![Page 28: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/28.jpg)
Correlations
Are the variables related?
Determine variables that relate most to your item of interest
Correlate Likert-scale questions with each other Correlate interval/ratio demographic information
(e.g., age) to Likert-scale questions
![Page 29: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/29.jpg)
Correlation
Which correlation coefficient to use? Pearson’s r
Used with interval and ratio data Spearman & Kendall’s tau-b
Used with ordinal data Spearman used for linear relationship Kendall’s tau-b for any increasing or decreasing
relationship
![Page 30: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/30.jpg)
Mean Differences
Are there meaningful differences between groups? class sections instructors on-line vs. off-line courses major vs. non-major
![Page 31: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/31.jpg)
Mean Differences
Which test to run? Interval and ratio data
t-test when comparing 2 groups Independent Dependent (paired-samples in spss)
ANOVA when comparing > 2 groups Independent (One Way ANOVA in spss) Dependent (general linear model-repeated measure
in spss)
![Page 32: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/32.jpg)
Presenting Results
Describe the purpose of the survey List the factors that motivated you to conduct
this research in the first place. Include the survey!
On assessment reports When the survey is new/still being fine tuned
How it was administered
![Page 33: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/33.jpg)
Presenting Results
Present the breakdown of results Tables and graphs should complement text
Conclusions Explain findings, especially facts that were
important or surprising Recommendations
Describe an action plan based on concise concluding statements
![Page 34: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/34.jpg)
Presenting Results
Share results in formal venues
Familiarize yourself with key findings so that you can mention results at every opportunity
![Page 35: Analyzing Survey Data Angelina Hill, Associate Director of Academic Assessment 2009 Academic Assessment Workshop May 14 th & 15 th UNLV.](https://reader034.fdocuments.net/reader034/viewer/2022051820/56649cb55503460f9497a30c/html5/thumbnails/35.jpg)
Moving Forward
Continuously improve the survey Delete, add, change questions Evaluate method of administration
Compare results across semesters to look for improvements
Compare with other assessment data for a broader picture