Surveys as Student Learning Surveys as Student Learning MeasuresMeasures

CLAS Unit Head MeetingCLAS Unit Head MeetingSeptember 27, 2006September 27, 2006

Maria Cimitile, CLAS Associate DeanMaria Cimitile, CLAS Associate DeanPriscilla Carreras, CLAS Graduate AssistantPriscilla Carreras, CLAS Graduate Assistant

Julie Guevara, Accreditation and Assessment OfficerJulie Guevara, Accreditation and Assessment Officer

What are surveys?What are surveys?

Systematic efforts to collect information Systematic efforts to collect information about people by asking them to respond about people by asking them to respond to specific questions.to specific questions. Helpful in gaining backgrounds, experiences, Helpful in gaining backgrounds, experiences,

plans for the future, opinions, and attitudes.plans for the future, opinions, and attitudes. Can use paper, phone, or internet format.Can use paper, phone, or internet format. Provide insight into student attitudes about Provide insight into student attitudes about

programs, assignments, courses, exams etc.programs, assignments, courses, exams etc.

Survey Planning and Survey Planning and ConceptualizationConceptualization

Should you use a survey?Should you use a survey? What do you want to learn?What do you want to learn? Who is the sample or audience?Who is the sample or audience? How will you distribute the survey?How will you distribute the survey? Sampling techniques – interviews vs. Sampling techniques – interviews vs.

individual survey.individual survey.

Survey Planning and Survey Planning and ConceptualizationConceptualization

Obtain representative sample.Obtain representative sample. Determine time and people power.Determine time and people power. Evaluate survey process when done.Evaluate survey process when done. Timeline – when it’ll be created, Timeline – when it’ll be created,

returned.returned. IRB Institutional Review Board.IRB Institutional Review Board. Divide items into two categories: “have Divide items into two categories: “have

to know” and “nice to know”, then omit to know” and “nice to know”, then omit “nice to know”.“nice to know”.

Designing the Survey QuestionsDesigning the Survey QuestionsClose-ended QuestionsClose-ended Questions

Closed ended questions are those questions, Closed ended questions are those questions, which can be answered finitely. Quick and require which can be answered finitely. Quick and require little time investment. Can also provide little time investment. Can also provide incomplete responses. By definition, these incomplete responses. By definition, these questions can be restrictive. questions can be restrictive.

Examples:Examples: Multiple choice Multiple choice RankingsRankings Rating ScaleRating Scale

Ecosystem Rating ScaleEcosystem Rating Scale Likert ScaleLikert Scale

Multiple Choice Multiple Choice Quickly answered and tallied. No room for Quickly answered and tallied. No room for

unexpected answers. unexpected answers.

Chemistry Example: Chemistry Example: - Goal: Graduate students who are well - Goal: Graduate students who are well equipped to enter the job market.equipped to enter the job market.- Objective: Technical emphasis majors will - Objective: Technical emphasis majors will enter the workforce and apply their skills enter the workforce and apply their skills and knowledge.and knowledge.- Measure: develop and send employer - Measure: develop and send employer surveys to companies that hired our surveys to companies that hired our graduates 6-12 months after they’ve been graduates 6-12 months after they’ve been hired.hired.

RankingsRankings Tedious.Tedious. Assume respondents feel differently about Assume respondents feel differently about

each response.each response. Do not provide size of feeling between Do not provide size of feeling between

ranked items.ranked items. AVOID THEM!!!AVOID THEM!!!

Example: on a scale where “10” means you Example: on a scale where “10” means you have a great interest in a subject and “1” have a great interest in a subject and “1” means you have none at all, how would you means you have none at all, how would you rank your interest in each of the following rank your interest in each of the following subjects?subjects?

Rating ScaleRating Scale

Another type of multiple choice Another type of multiple choice question.question.

Example: States conclusions based on Example: States conclusions based on data collection.data collection.

ExcellentExcellent GoodGood FairFair PoorPoor

Eco-system Rating ScaleEco-system Rating Scale Ask for two ratings. The second gives Ask for two ratings. The second gives

information on the environment in which information on the environment in which the first was madethe first was made

Observes difference between growth Observes difference between growth obtained and growth desired.obtained and growth desired.

Measures attainment of goals and Measures attainment of goals and objectives.objectives.

Example: Ability to conduct a psychosocial Example: Ability to conduct a psychosocial assessment assessment

Importance: Not Critical __1 __2 __3 __4 __ 5 VitalImportance: Not Critical __1 __2 __3 __4 __ 5 VitalPreparation: Poor __1 __2 __3 __4 __ 5 Preparation: Poor __1 __2 __3 __4 __ 5

Excellent Excellent

Likert ScaleLikert Scale(Agreement Scale)(Agreement Scale)

EfficientEfficient Permits comparisonPermits comparison Can be used in place of rankingCan be used in place of ranking

Example:Example:Employees exhibit requisite safety skills in the lab.Employees exhibit requisite safety skills in the lab.

Strongly Agree Agree Disagree Strongly Disagree Strongly Agree Agree Disagree Strongly Disagree NANA

Numeric Open-ended Numeric Open-ended QuestionsQuestions

Example:Example:

In the past two years (1/04 – 1/06) In the past two years (1/04 – 1/06) how many of our graduates have you how many of our graduates have you hired?hired?

Text Open-ended QuestionsText Open-ended Questions

Participants compose their own answers. Participants compose their own answers. Provide infinite response. Provide infinite response.

Develop trust, are perceived as less Develop trust, are perceived as less threatening, allow an unrestrained or threatening, allow an unrestrained or free response, and may be more useful free response, and may be more useful with articulate users.with articulate users.

Not popular in surveys because:Not popular in surveys because: Lengthen time required to complete Lengthen time required to complete

survey.survey. Require more mental energy.Require more mental energy.Examples: How can we achieve the Examples: How can we achieve the

objectives of this course?objectives of this course?

Writing Good Survey QuestionsWriting Good Survey QuestionsKeep It Short and Simple (KISS)Keep It Short and Simple (KISS)

Keep questions short.Keep questions short. Each item should ask only one Each item should ask only one

question.question. Use simple vocabulary – NO JARGON!Use simple vocabulary – NO JARGON! Make definitions, assumptions, and Make definitions, assumptions, and

qualifiers clear.qualifiers clear. Do not make significant memory Do not make significant memory

demands.demands.

Writing Good Survey QuestionsWriting Good Survey Questions Do not ask for broad generalizations of Do not ask for broad generalizations of

attitudes “Do you consider yourself a attitudes “Do you consider yourself a good student?”good student?”

Do not ask questions which are Do not ask questions which are uncomfortable to answer honestly.uncomfortable to answer honestly.

When possible, give the option of not When possible, give the option of not knowing; Not applicable.knowing; Not applicable.

Avoid asking leading questionsAvoid asking leading questions Ex. Why are GVSU students your Ex. Why are GVSU students your

best employees?best employees? Define terms, qualifiers, assumptions, Define terms, qualifiers, assumptions,

etc.etc.

Maximizing Participation RateMaximizing Participation Rate

Professional appearance of survey is Professional appearance of survey is important. important. People are more likely to complete and People are more likely to complete and

return a survey if it looks like someone return a survey if it looks like someone cares about it.cares about it.

Be considerate of your participants.Be considerate of your participants. Include self-addressed & stamped envelope.Include self-addressed & stamped envelope. Give adequate time for response.Give adequate time for response. Remember that they are doing you a favor.Remember that they are doing you a favor.

Telephone SurveyTelephone Survey

Survey should be no more than 10 Survey should be no more than 10 minutes.minutes.

Use simple format, with easy to follow Use simple format, with easy to follow directions.directions.

Avoid open-ended questions.Avoid open-ended questions. Pre-test survey by reading it to people Pre-test survey by reading it to people

without looking at them.without looking at them. Document attempts to reach Document attempts to reach

participants.participants.

Employer SurveysEmployer Surveys

Before completing employer survey, Before completing employer survey, inform alumni.inform alumni.

SkillsSkills Importance of skills in particular Importance of skills in particular

field/work place.field/work place. Skills employee attained at university or Skills employee attained at university or

in specific program.in specific program.

ReferencesReferences Creative Research Systems (2006). The Survey System. Creative Research Systems (2006). The Survey System.

http://surveysystem.com/sdesign.htmhttp://surveysystem.com/sdesign.htm Introduction to Developing SurveysIntroduction to Developing Surveys. Georgia Tech Office for . Georgia Tech Office for

Assessment. Assessment. http://www.assessment.gatech.edu/eReports/slide_shows/Shttp://www.assessment.gatech.edu/eReports/slide_shows/Short_Guide_to_Survey_Research.pdf#search=%22developinhort_Guide_to_Survey_Research.pdf#search=%22developing%20surveys%22g%20surveys%22

Palomba, C.A. and Banta, T.W. (1999). Assessment Palomba, C.A. and Banta, T.W. (1999). Assessment essentials. San Francisco, CA : Jossey-Bass.essentials. San Francisco, CA : Jossey-Bass.

Richardson, Dr. John V. (2002). Open versus closed ended Richardson, Dr. John V. (2002). Open versus closed ended questions in the reference environment. questions in the reference environment. http://polaris.gseis.ucla.edu/jrichardson/dis220/openclosed.http://polaris.gseis.ucla.edu/jrichardson/dis220/openclosed.htmhtm

Survey How ToSurvey How To (2006). University of Texas at Austin (2006). University of Texas at Austin Division of Instructional Innovation and Assessment. Division of Instructional Innovation and Assessment. http://www.utexas.edu/academic/diia/assessment/iar/how_thttp://www.utexas.edu/academic/diia/assessment/iar/how_to/methods/survey.phpo/methods/survey.php

Suskie, L. (2004). Assessing student learning: A common Suskie, L. (2004). Assessing student learning: A common sense guide. Bolton, MA : Anker Publishing.sense guide. Bolton, MA : Anker Publishing.

USE SPARINGLYUSE SPARINGLYWhen there are more than 6 When there are more than 6 possible answers.possible answers.When asking for occupation.When asking for occupation.When structured item may bias When structured item may bias responses by steering responses by steering participants in one direction.participants in one direction.When asking a high interest When asking a high interest question, such as “How would question, such as “How would you change/improve/etc.”you change/improve/etc.”

Student SurveysStudent Surveys

Surveys of graduating seniors.Surveys of graduating seniors. Longitudinal Surveys of students Longitudinal Surveys of students

attending university, enrolled in attending university, enrolled in specific program.specific program.

Alumni SurveysAlumni Surveys

Employment related surveys. Employment related surveys. Programmatic.Programmatic.

Departmental missionDepartmental mission Skill attainmentSkill attainment

Who to Survey?Who to Survey?

StudentsStudents AlumniAlumni EmployersEmployers