Experiments as Reforms Persuasion in the Nation’s Service ...
Experiments on Education as a Service
-
Upload
martin-perez -
Category
Documents
-
view
15 -
download
0
description
Transcript of Experiments on Education as a Service
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011
Experiments on Education as a Service
Hyoyoung KimJames R. Morrison
Department of Industrial and Systems EngineeringKAIST
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 2
Contents
• Motivation• Experiments• Statistical Analysis• Concluding Remarks
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 3
Literature Review
Past Research My Research
University Experience O[1,2,4] X
Class experience X O
Kano Classi-fication
Basic Need O[1,4] X
Performance Need O[1,2,4] X
Excitement Need X O
QFD Approach O[1,3] X
SERVQUAL Approach O[3,4] O
Statistical Analysis
Wilcoxon signed-rank test X O
Kruskal-Wallis test X O
Experiment on single course X O[1] Bagchi, U.; , "Delivering student satisfaction in higher education: A QFD approach," Service Systems and Service Management (ICSSSM), 2010 7th International Conference on , vol., no., pp.1-4, 28-30 June 2010, doi: 10.1109/ICSSSM.2010.5530090[2] Mathew Joseph, Beatriz Joseph, (1997) "Service quality in education: a student perspective", Quality Assurance in Education, Vol. 5 Iss: 1, pp.15 – 21[3] Sangeeta Sahney, D.K. Banwet, S. Karunes, (2004) "A SERVQUAL and QFD approach to total quality education: A student perspective", International Journal of Productivity and Per-formance Management, Vol. 53 Iss: 2, pp.143 – 166[4] Kay C. Tan and Sei W. Kek, (2004) “Service Quality in Higher Education Using an enhanced SERVQUAL approach,” Quality in Higher Education, Vol. 10, No. 1, pp. 17-24
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 4
Motivation
Course Experi-ence
Satisfaction Evaluation TeachingBetterBetter
Take Home Point: Good service improves the student experience and increases satisfaction But what is good service?
ServiceService
Service Service
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 5
Motivation
• Identify how to increase customer satisfaction in Education
• Experiment 1– GOAL: Determine Basic, Performance, and Excitement needs– Kano’s model survey– Result: Identified 4 BN, 11 PN, 20 EN
• Experiment 2– GOAL: Determine which services (among excitement needs) make sta-
tistically significant difference in student satisfaction– SERVQUAL survey– Result:
• Observed drift of service and service memory effect• Numerous services improve/influence student perceptions
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 6
Experiment 1 - Spring, 2010
• GOAL: Determine Basic, Performance, and Excitement needs
• KAIST IE 200 Introduction to Operations Research class• Class size: 186 students (from all science and engineering majors)• Conducted Kano’s model survey
– 76 Functional and Dysfunctional questions
– 18 Solution questions
– 15 Overall satisfaction questions
Functional
Dysfunctional
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 7
Experiment 1 - Spring, 2010
RESULTS:• Basic Need (Must-be):
– Lecture room density is NOT high (large room size with small number of students)
– Lecture notes are uploaded at least one day before class– Lecture room is NOT disturbed by outside noise– Class time is well kept
Must-be
Dissatisfac-tion
Done very well
Done very poorly
Satisfac-tion
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 8
Experiment 1 - Spring, 2010
RESULTS:• Performance Need (One-dimensional):
– Exams and/or homework are directly related to lectures– You feel you learned a lot at the end of the semester– Lecture room is close to dormitory– Professor is confident about lecture contents– Professor enjoys teaching– Professor prepares the lecture well– Professor delivers the lecture clearly– Lecture is easy to understand– Professor respects you– TAs are kind– Lecture notes are easily understandable– Lecture room is clean
Satisfac-tion
Dissatisfac-tion
Done very wellDone very poorly
One-dimen-sional
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 9
Experiment 1 - Spring, 2010
RESULTS:• Excitement Need:
– Furniture and classroom equipment are up-to-date– Professor gives candies when answering questions– Professor and TAs provide sufficient and convenient office hours– Professor is humorous– Professor smiles while giving lectures– Professor cares about students– Professor’s appearance is neat– Professor calls students by name– Professor is kind– In-class activities are fun– Course material can be applied in the real world– Lectures make you think
Satisfac-tion
Dissatisfac-tion
Done very well
Done very poorly
Attractive
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 10
Experiment 2 - Fall, 2010
• KAIST IE 332 Operations Research II class, 60 students (IE ma-jors)
• SERVQUAL survey to measure expectation, performance, and importance
• GOAL: Determine which excitement needs measurably improve student satisfaction
Topic Service Dura-tion
Chap. 2 Professor is humorous 2 weeks
Chap. 3 Candies or rewards are given when asking or answering questions 2 weeks
Chap. 4 Fun in-class activities 2 weeks
Chap. 5 Nominal service 2 weeks
Chap. 6 Full service 3 weeks
Chap. 7 Real world application + full service 3 weeks
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 11
Experiment 2 - Fall, 2010
• Handling missing data: Regression• Data reliability: Cronbach’s alpha (α)
– 0 ≤ α ≤ 1, ↑ α, ↑reliable– Recommended to have a reliability of 0.70 or higher to be used as
data– Data of six surveys: 0.89 ≤ α ≤ 0.94 ⇒ can trust data
• Correlation– Measure of how much each pair of scores relates
• Statistical testing– SAS 9.1– Wilcoxon signed-rank test (paired sample test) – Kruskal-Wallis test
0.00-0.25: little if any0.26-0.49: low
0.50-0.69: moderate0.70-0.89: high
0.90-1.00: very high
non-parametric statistical hypothesis test for the case of two related samples
a non-parametric method for testing equality of population medians among groups
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 12
Statistical Analysis <Summary>
ServiceHumor Candy In-class Nominal Full Application
Survey
Humor 4.410 4.188 4.100 4.109 4.351 4.231
Candy 3.897 4.173 3.367 3.047 4.263 3.654
In-class 2.962 3.116 4.250 3.478 4.439 4.1
Application 3.372 3.348 3.550 3.522 3.632 3.712
• Perception 1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 13
Statistical Analysis <Humor>
ServiceHumor Candy In-class Nominal Full Application
Survey
Humor 4.410 4.188 4.100 4.109 4.351 4.231
Accept (p-value: 0.420 ≥ 0.05) Reject (p-value: 0.017 ≤ 0.05)
Reject (p-value: 0.003 ≤ 0.05)
Accept (p-value: 0.854 ≥ 0.05) Reject (p-value: 0.013 ≤ 0.05) Accept (p-value: 0.939 ≥ 0.05)
• Input– Jokes and funny/silly examples– Short funny music
• Output
– Spillover effect – effect of humor lasted longer than other services.– H0: there is no difference or relationship between the variables of interest– If p-value ≥ 0.05 , H0 is accepted → no difference in measurements– Students still remembered humor and rated humor higher on the next sur-
vey
1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 14
Statistical Analysis <Humor>
ServiceHumor Candy In-class Nominal Full Application
SurveyWell pre-
pared 4.359 4.116 4.283 4.304 4.456 4.346
Overall Ser-vice 3.949 3.783 3.917 3.891 3.982 3.885
Enjoyment 3.782 3.551 3.717 3.522 3.596 3.673
Kindness 4.385 4.203 4.167 4.153 4.390 4.308
• Outcome
Correlation of 0.6721: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 15
• Input– Give candy to students when they ask or answer questions
• Output– Perceived candy
– Drift of needs (Excitement need to basic need): 2.26/5 → 3.11/5 for the importance of giving candies
Statistical Analysis <Candy>
ServiceHumor Candy In-class Nominal Full Application
Survey
Candy 3.897 4.173 3.367 3.047 4.263 3.654
1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 16
• Input– Skits– Games– Team activity– Active learning activities
• Output
– Correlation of 0.43324 between in-class activities and confidence
Statistical Analysis <In-class Activity>
ServiceHumor Candy In-class Nominal Full Application
Survey
In-class 2.962 3.116 4.250 3.478 4.439 4.1
Time, effort 1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 17
Statistical Analysis <Nominal Service>
ServiceHumor Candy In-class Nominal Full Application
Survey
Enjoyment 3.782 3.551 3.717 3.522 3.596 3.673
Kindness 4.385 4.203 4.167 4.153 4.390 4.308
• Outcome
Significant changeInsignificant change
1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 18
ServiceHumor Candy In-class Nominal Full Application
Survey
Humor 4.410 4.188 4.100 4.109 4.351 4.231
Candy 3.897 4.173 3.367 3.047 4.263 3.654
In-class 2.962 3.116 4.250 3.478 4.439 4.1
• Input– Candy– Humor– In-class activity– Enthusiasm– Neat appearance
• Output
Statistical Analysis <Full Service>
1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 19
Statistical Analysis < Full Service >
ServiceHumor Candy In-class Nominal Full Application
SurveyWell pre-
pared 4.359 4.116 4.283 4.304 4.456 4.346
Overall Ser-vice 3.949 3.783 3.917 3.891 3.982 3.885
• Outcome– Positive
– Negative
ServiceHumor Candy In-class Nominal Full Application
Survey
Lecture 3.936 3.449 3.517 3.522 3.298 3.442
Lecture note 3.769 3.507 3.683 3.609 3.491 3.654
Concentrate 3.910 3.783 3.717 3.717 3.632 3.654
Significant changeInsignificant change
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 20
Concluding Remarks
• Students perceived the offering of such services
• We observed needs drift of such services
• We also observed the service memory effect
• Needs that require further improvement: Lecture notes are easy to understand Sufficient and convenient office hours Feel you learned a lot Exams and/or homework are directly related to lectures
• Future Research: Service increases student learning outcome
©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011
Thank You.
Any Questions?