+Don’t walk:

Rasch to

join

the

questionnai

re trend!

Ritsumeik

an

University

Keita

Kikuchi

Kanagaw

a

University

J. W.

Lake

Fukuoka

Women’s

University

+

Get ready to be Rasched…

(This is supposed to be a joke. Laugh.)

Structure of

this

workshop

Ritsumeik

an

University

Keita

Kikuchi

Kanagaw

a

University

J. W.

Lake

Fukuoka

Women’s

University

+

Rasch concepts

Terminology and explanations

Structure of

this

workshop

Ritsumeik

an

University

+

Steps to create a

questionnaire

Constructs, concepts, items, piloting, evaluation,

revision…

Structure of

this

workshop J. W.

Lake

Fukuoka

Women’s

University

+

Demotivation: An example

The flaws of traditional FA and why Rasch can

help questionnaire creation

Structure of

this

workshop

Keita

Kikuchi

Kanagaw

a

University

Teacher

Behavior

GETA

e1

.69

MET

e2

.85

ONE

e3

.75

PRON

e4

.69

EXPL

e5

.77

Environment

AUD

e14

TOPC

e13

INTR

e12

.76.74.67

VID

e15

.70

GRAM

e6

Experience

of difficulties

VOCI

e7

VOC

e8

TEST

e9

SELF

e10

.72 .68 .66.68

Lack of

Interest

NOP

e22

NON

e21

NOG

e20

NOI

e19

NOF

e18

.82

.83.77.78.81

NOTU

e11

.74

demotivation

.74

.68

.69

d1

d2

d3

d4

.66

FRN

e16

.64

MST

e17

.61

.50

GFI=.911

NFI=.905

CFI=.920

RMSEA=.064

SRMR = .055

+

Matthew Apple

Ritsumeikan University

Department of Communication

International Communication Program

Rasch

concepts

+

Rasch terms

Rasch log-odds (logits)

Rasch measures (logit scores)

Infit/Outfit (means sq. and z-score)

Item difficulty / endorsability

Person/item reliability / separation

Construct validity and

unidimensionality (not strictly

speaking Rasch, but…)

Rasch Principal components

analysis (Rasch PCA)

Loadings

Contrasts and Residuals

A probabilistic model

+Logits (log-odds)

The probability of a person correctly

answering an item 50% of the time

+Fit

0.75 to 1.3 logits

0.60 to 1.4 logits

“Within 2 standard deviations of the mean”

“Only Outfit z-scores of 3.0”

+Separation

The number of groups distinguishable by the

measurement instrument

(Wilson, 2005;

Wright, 1996)

The ratio of error-free variance and

observed variance

(Fisher, 1992)

+Item map

Persons and items on the same

linear logit scale

+

The “Line”Items and persons on the same scale

Item-person map (or)

Wright map

Ben Wright

+

The “Line”Items and persons on the same scale

Item-person map (or)

Wright map

+

How Rasch works

for questionnaires

Likert-scale data

Likert-type category data

Questionnaires do not produce

true interval but ordinal data

The steps in the “scale” can be

conceived as thresholds (τ)

named after Thurston, originator

of factor analysis)Refer to Andrich (1977, 1978)

+

How Rasch works

for questionnairesRefer to Andrich (1977, 1978)

+

How Rasch works

for questionnairesRefer to Andrich (1977, 1978)

N

1

2

3

4

5

+

How Rasch works

for questionnairesRefer to Andrich (1977, 1978)

N

1

2

3

4

5

1 + 3 = 4

+

How Rasch works

for questionnairesRefer to Andrich (1977, 1978)

N

1

2

3

4

5

1 + 3 = 4?

SD + N = A?

+

J W Lake

Fukuoka Women’s University

Steps to

create a

questionnaire

Steps in scale development: Issues to

consider (Netemeyer, Bearden, Sharma,

2003)

Step 1: Construct definition and content domain

The importance of clear construct definition,

content domain, and the role of theory.

Construct dimensionality: unidimensional,

multidimensional, or a higher-order construct?

Determine the purpose of the scale:

measurement or correlational analysis or

model building

Step 2: Generating and judging measurement items

Theoretical assumptions about items (e.g., domain

sampling)

Generating potential items and determining the

response format

How many items as an initial pool

Dichotomous vs. polytomous response formats

Item wording issues

The focus on “content” validity in relation to

theoretical dimensionality

Item judging (expert and layperson) --- the focus on

“content” and “face” validity

Step 3: designing and conducting studies to develop

and refine the scale

Pilot testing as an item-trimming procedure

The use of several samples from relevant populations for scale development

Designing the studies to test psychometric properties

Initial item analyses via exploratory factor analyses (EFAs)

Initial item analyses and internal consistency estimates

Retaining items for the next studies

EFA may be useful for correlational analysis or model building

Step 4: Finalizing the scales

The importance of several samples from relevant populations

Designing the studies to test the various types of validity

Item analysis via EFA

The importance of EFA consistency from Step 3 to Step 4

Deriving an initial factor structure—dimensionality and theory

Item analyses and confirmatory factor analyses (CFAs)

Testing the theoretical factor structure and model

+

Keita Kikuchi

Kanagawa University

Demotivatio

n: An

example

Teacher

Behavior

GETA

e1

.69

MET

e2

.85

ONE

e3

.75

PRON

e4

.69

EXPL

e5

.77

Environment

AUD

e14

TOPC

e13

INTR

e12

.76.74.67

VID

e15

.70

GRAM

e6

Experience

of difficulties

VOCI

e7

VOC

e8

TEST

e9

SELF

e10

.72 .68 .66.68

Lack of

Interest

NOP

e22

NON

e21

NOG

e20

NOI

e19

NOF

e18

.82

.83.77.78.81

NOTU

e11

.74

demotivation

.74

.68

.69

d1

d2

d3

d4

.66

FRN

e16

.64

MST

e17

.61

.50

GFI=.911

NFI=.905

CFI=.920

RMSEA=.064

SRMR = .055

Example Study

focused on specific external /internal forces that Japanese high school students may experience which might cause their motivation to be reduced or diminished

administered the questionnaire asking high school students to report what diminished their motivation to study in their high school days, which contained 40 Likert-scale questions (4-points)

analyzed the quantitative data using a confirmatory factor analysis using Amos and Rasch PCA of the residuals using Winsteps.

If you’d like to read this process thoroughly, please locate Kikuchi (forthcoming).

Demotivation

Dörnyei (2001)

Definition of demotivation

“specific external forces that reduce or

diminish the motivational basis of a

behavioral intention or an ongoing action” (p.

143).

I expand this definition and explore

demotivators including both internal and

external forces.

Previous Studies (Dörnyei, 1998)

Based on interviews with 50 secondary school students, he identified following as demotivators, the reason to get demotivated.

1. Teachers’ personalities, commitments, competence, teaching methods.

2. Inadequate school facilities (very big group, not the right level or frequent change of teachers).

3. Reduced self-confidence due to their experience of failure or success.

4. Negative attitude toward the foreign language studied.

5. Compulsory nature of the foreign language study.

6. Interference of another foreign language that pupils are studying.

7. Negative attitude toward the community of the foreign language spoken.

Previous Studies (Kojima, 2004, p.42)

English

Learning

Demotivation

Languag

e Level

Learning

situation

Amoun

t of

study

.58

.86

.71

.46

.41

Gramm

ar

Readin

g

Self-

confidenc

e

Writing Learning

method

Memori

zing

Vocab.

Learner

Level

Listening

problem

Class

atmospher

e

Change

of teaching

style

Teaching

approach

Teacher

personality.89

.77

.89

.89

.85 .90 .92 .80 .83 .85

GFI = 0.906

AGFI = 0.890

RMSEA = 0.052

Previous Studies

Kikuchi (2009)

47 university students

open-ended questionnaires

reflection on high school days

Kikuchi and Sakai (2009)

112 university students

a 35-item questionnaire with a 5-point scale

Sakai and Kikuchi (2009)

676 high school students

a 35-item questionnaire with a 5-point scale

Common demotivation factors Sakai and Kikuchi(2009)

F1: Learning Contents and Materials,

F2: Teachers’ Competence and Teaching Styles

F3: Inadequate School Facilities [Classroom Environment]

F4: Lack of Intrinsic Motivation

F5: Test Scores [Experience of Inferiority]

Kikuchi and Sakai (2009)

F1: Course Books

F2: Inadequate School Facilities

F3: Test Scores

F4: Non-Communicative Methods

F5: Teachers’ Competence and Teaching Styles

Both studies used a principal axis factor analysis using the direct

oblimin rotation

Six Original Constructs Teachers: Teachers’ attitude, teaching competence,

language proficiency, personality, and teaching style

Characteristics of classes: Course contents and pace,

focus on difficult grammar or vocabulary, monotonous and

boring lessons, a focus on university entrance exams and

the memorization of the language

Experiences of failure: Disappointment due to test scores,

lack of acceptance by teachers and others, and feeling

unable to memorize vocabulary and idioms.

Class environment: Attitude of classmates, compulsory

nature of English study, friends’ attitudes, inactive classes,

inappropriate level of the lessons, and inadequate use of

school facilities such as not using audio-visual materials

Class materials: Not suitable or uninteresting materials (e.g.,

too many reference books and/or handouts)

Lack of interest: Sense of English used at schools is not

practical and not necessary. Little admiration toward English

Participants (N=1,266)

MethodMaterials

background questions

a 40-item questionnaire “We would like to study the situations of English study in high schools.

The following statements are possible demotivating factors for English learning. To what extent are these statements true for you? Answer based on your experience.”

Questions are revised from what I used in my previous studies (Kikuchi and Sakai, in-press; Sakai and Kikuchi, 2009).

Example of Items (1=Strongly disagree, 2=disagree, 3=agree, and 4=Strongly agree) Teachers made one-way explanations too often.

The number of students in classes was large.

A great number of textbooks and supplementary readers were assigned.

I lost my understanding of the purpose of studying English.

I could not do as well on tests as my friends.

Results of EFA

An Exploratory Factor Analysis

40 items

principal axis factor analysis with a promax rotation procedure

a four-factor solution

teachers behaviors

class environment

experiences of difficulties

Lack of interest

Only 22 items left to be included in Confirmatory Factor Analysis.

Table 2:Factor Analysis of DemotivationNo. Item descriptions F 1 F 2 F 3 F 4

Factor 1: Experience of difficulties(α = .87)

i16 There were too many vocabularies that I did not understand in reading. 0.81 0.03 -0.14 -0.11

i15 I had difficulty in memorizing words and phrases. 0.80 -0.13 -0.01 -0.08

i13 I got low scores on tests (such as mid-term and final examinations). 0.79 -0.17 0.06 -0.09

i8 I did not understand grammar even though I studied. 0.75 -0.02 -0.19 0.05

i39 I started not to understand the content of the class. 0.71 0.09 -0.09 0.08

i7 I could not do as well on tests as my friends. 0.62 -0.11 0.10 -0.09

i33 I got lost in how to self-study for English lessons. 0.60 -0.02 0.03 0.09

Factor 2: Teacher behavior(α = .84)

i18 I thought that the approach that teacher used was not good. -0.05 0.93 -0.14 0.00

i5 Teachers' explanations were not easy to understand. 0.03 0.86 -0.22 0.02

i17 Teachers made one-way explanations too often. 0.06 0.78 -0.07 -0.02

i6 Teachers' pronunciation of English was poor. -0.10 0.73 0.02 -0.07

i34 I could not get along with teachers. -0.10 0.68 0.12 0.05

i31 The pace of lessons was not appropriate. 0.04 0.63 0.06 -0.03

Factor 3: Class environment(α = .85)

i28 Audio materials (such as CDs and tapes) were not used. -0.08 -0.01 0.82 -0.06

i23 The Internet was not used. -0.12 -0.16 0.81 0.04

i27 Topics of the English passages used in lessons were old. 0.04 0.02 0.71 -0.11

i35 Visual materials (such as videos and DVDs) were not used. -0.03 0.05 0.69 0.04

i10 My friends did not like English. 0.02 -0.05 0.67 -0.07

i40 The number of students in classes was large. -0.03 -0.11 0.65 0.13

Factor 4: Lack of Interest(α = .90)

i3 I lost my understanding of the purpose of studying English. -0.13 0.00 0.01 0.91

i2 I lost my goal to be a speaker of English. -0.08 -0.04 0.03 0.85

i26 I think that I will not use English in my future. 0.03 0.01 -0.10 0.81

i11 I don’t have specific goals for studying English. 0.04 -0.04 0.01 0.79

i24 I lost my interest in English. 0.12 -0.02 0.01 0.71

Method

Analysis

Rasch PCA of the residuals/Confirmatory

factor analysis of these six factors

Rasch PCA factor analysis /Confirmatory

factor analysis of these four factors were

conducted once again…

Conventional factor analysis confirmed

only four factors!

+

Enter the RaschWith Matt

+

Category utility

Measures the distance between

thresholds among the Likert-type

categories (“steps” of the scale)

+

Rasch PCA output

Loading

Measure

Infit means squared

Outfit means squared

Principal components analysis

+

Item fit analysis

Measure

Standard error

Infit Outfit means squared & z

+

Keita Kikuchi

Kanagawa University

Demotivatio

n, Part

Deux: The

Rasched

Teacher

Behavior

GETA

e1

.69

MET

e2

.85

ONE

e3

.75

PRON

e4

.69

EXPL

e5

.77

Environment

AUD

e14

TOPC

e13

INTR

e12

.76.74.67

VID

e15

.70

GRAM

e6

Experience

of difficulties

VOCI

e7

VOC

e8

TEST

e9

SELF

e10

.72 .68 .66.68

Lack of

Interest

NOP

e22

NON

e21

NOG

e20

NOI

e19

NOF

e18

.82

.83.77.78.81

NOTU

e11

.74

demotivation

.74

.68

.69

d1

d2

d3

d4

.66

FRN

e16

.64

MST

e17

.61

.50

GFI=.911

NFI=.905

CFI=.920

RMSEA=.064

SRMR = .055

Results

Rating Scale Instrument Quality Criteria (based on Fisher, 2007)

Criterion Poor Fair Good Very Good Excellent

Item Model Fit Mean-Square < 0.33 - >3.0 0.34 - 2.9 0.5 - 2.0 0.71 - 1.4 0.77 - 1.3

Person and item measurement

reliability <.67 .67-.80 .81-.90 .91-.94 >.94

Variance in data explained by

measures <50% 50-60% 60-70% 70-80% >80%

Unexplained variance in 1st

contrast of PCA residuals >15% 10-15% 5-10% 3-5% <3%

Table 1: Variance in measure explained by each demotivator

construct.

Six Demotivator

Constructs

Variance

explained by

measure

Unexplained

variance by

measure

Unexplained

variance

explained by 1st

contrast

Teachers 57.0% 43.0% 12.3%

Characteristics of classes 45.4% 54.6% 9.7%

Experiences failure 57.1% 42.9% 12.7%

Class environment 40.4% 59.6% 13.3%

Class materials 55.1% 44.9% 10.8%

Lack of interest 62.1% 37.9% 12.8%

No. Item description Logit

Score

Infit

MNSQ

Outfit

MNSQ

Factor

loadings

1. teachers(k=6, Rp=0.99, Gp=11.02)

15 Teachers shout or got angry 0.97 1.42 1.38 0.68

5 Teacher asked us to use accurate grammar 0.54 1.26 1.25 0.68

14 Teachers explanation not easy -0.85 0.79 0.85 -0.63

11 Teachers bad pronunciation 0.16 0.98 0.91 -0.45

40 Teachers’ bad teach method -0.31 0.75 0.72 -0.44

13 Teachers one-way explanation -0.51 0.83 0.84 -0.03

2. Characteristics of Classes (k=9, Rp=0.98, Gp=6.53)

10 Inappropriate pace of lesson -0.10 1.00 1.04 0.58

41 Monotonous class -0.22 0.98 1.04 0.58

1 Rare chance of communication -0.14 1.19 1.24 0.45

2 Focused on translation 0.06 0.81 0.83 -0.57

3 Focused on grammar -0.35 0.96 1.01 -0.56

42 Amount to study for mid-term/final tests -0.32 1.19 1.29 -0.20

6 Required memorizing passages in textbooks 0.26 0.94 0.95 -0.15

43 Amount of handout distributed 0.11 0.94 0.95 -0.13

4 Focused on college entrance exam. Prep. 0.69 0.90 0.89 -0.12

Results –Rasch PCA- Loadings of

Contrasts

No. Item description Logit

Score

Infit

MNSQ

Outfit

MNSQ

Factor

loadings

3. Experiences of Failure (k=6, Rp=0.99, Gp=8.89)

27 Did not do well on tests compared w friends 0.64 1.04 1.06 0.76

8 low scores on school test 0.25 0.84 0.84 0.68

37 Did not understand grammar -0.15 1.07 1.07 -0.64

36 Did not understand class 0.23 1.01 1.01 -0.56

9 Get lost in self-study -0.26 1.05 1.07 -0.12

7 Could not memorize vocabulary and idiom -0.71 0.95 0.96 -0.05

4. Class Environment (k=6, Rp=0.98, Gp=7.40)

31 English being compulsory subject -0.73 1.08 1.20 0.82

26 Too many students in class 0.00 1.05 1.00 0.23

22 Video and DVDs not used -0.41 0.91 0.90 0.05

25 Audio not used 0.04 0.87 0.85 -0.59

29 Friends did not like English 0.67 0.97 1.03 -0.42

23 Internet not used 0.43 1.08 0.99 -0.31

No. Item description Logit

Score

Infit

MNSQ

Outfit

MNSQ

Factor

loadings

5. Class Materials (k=6, Rp=1.00, Gp=7.30)

16 Topics of Passages uninteresting 0.14 1.06 1.09 0.57

20 Topics of Passages old 1.42 0.96 0.95 0.56

35 Unclear answers to Questions 0.22 1.01 1.03 0.41

19 too much reading 0.44 0.97 1.00 0.19

44 Many difficult Vocabulary -1.21 0.98 1.00 -0.56

18 Sentences were difficult to read -0.97 0.98 1.03 -0.55

17 Passages too long -0.04 1.00 1.01 -0.47

6. Lack of Interest (k=5, Rp=0.98, Gp=6.87)

34 No goal for being a person who can use Eng 0.31 0.92 0.89 0.75

39 No need studying English 0.63 0.85 0.84 0.73

33 Lost interest in English -0.30 1.15 1.14 -0.56

32 Lost purpose of study English -0.59 1.06 1.06 -0.42

46 No use of English in the future -0.05 0.97 0.95 -0.31

Figure 1: CFA of 6 factor models of demotivators

Notes.

GFI=800

CFI=808

RMSEA=

.072

teacher

i36e1

.56

i34e2

.73i30e3

.60

i18e4.82

i17e5 .76

i6e6.68

i5e7

.74

lesson

i29e8

i25e9

i22e10

i21e11

i20e12

i19e13

i4e14

.61

.58

.60

.59

.55

.60

.47

i31e15

.59

i32e16

.46

environment

i10 e17

i23 e18

i28 e19

i35 e20

i37 e21

i40 e22

.63

.67

.71

.70

.65

.62

material

i1 e23

i9 e24

i12 e25

i14 e26

i16 e27

i27 e28

.53

.57

.50

.56

.59

.57

interest

i2 e29

i3 e30

i11 e31

i24 e32

i26 e33

.82

.83

.77

.79

.81

failure

i39

e35

i33

e36

i15

e37

i13

e38

i8

e39

i7

e40

.74.68.67.71.67.60

i38 e41

.68

.79

.37

.27

.59

.54

.70

.56

.93

.79

.54

.88

.52

.73

.63

.57

Figure 2.

CFA of

4 factor

models

Notes.

GFI=916

CFI=926

RMSEA=

.062

Teacher

Behavior

GETA

e1

.69

MET

e2

.85

ONEW

e3

.75

PRON

e4

.69

EXPL

e5

.77

Environment

AUD

e14

TOPC

e13

INTR

e12

.76.74.67

VID

e15

.70

GRAM

e6

Experience

of difficulties

VOCI

e7

VOC

e8

TEST

e9

SELFS

e10

.71 .67 .66.68

Lack of

Interest

NOP

e22

NON

e21

NOG

e20

NOI

e19

NOU

e18

.82

.83.77.79.81

NOTU

e11

.75.67

FRN

e16

.64

MST

e17

.61

.46

.51

.44

.53

.23

.35

Teacher

Behavior

GETA

e1

.69

MET

e2

.85

ONE

e3

.75

PRON

e4

.69

EXPL

e5

.77

Environment

AUD

e14

TOPC

e13

INTR

e12

.76.74.67

VID

e15

.70

GRAM

e6

Experience

of difficulties

VOCI

e7

VOC

e8

TEST

e9

SELF

e10

.72 .68 .66.68

Lack of

Interest

NOP

e22

NON

e21

NOG

e20

NOI

e19

NOF

e18

.82

.83.77.78.81

NOTU

e11

.74

demotivation

.74

.68

.69

d1

d2

d3

d4

.66

FRN

e16

.64

MST

e17

.61

.50

GFI=.911

NFI=.905

CFI=.920

RMSEA=.064

SRMR = .055

Figure 3.

My tentative

model of

demotivation

Results –Rasch PCA of four factor

models-

Four Demotivator

Constructs

Variance

explained by

measure

Unexplained

variance by

measure

Unexplained

variance explained

by 1st contrast

Teachers 58.5% 41.5% 10.7%

Experience of difficulties 58.0% 42.0% 12.2%

Class environment 39.9% 60.1% 13.3%

Lack of interest 54.8% 45.2% 15.1%

This is how the poor factor was

working…

An activity for this workshop

Let’s try to make item bank of questionnaire items together for your practice.

Topic is Demotivating factors in English education for communication in Japan.

With your partner, think of constructs first and write items for each construct. How many constructs? How many items for each of them? Remember “the Line”!

Please use the questionnaire items that you have in your handout about demotivating factors in English education in high school English classroom to generate your discussion.

References

Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.

Dörnyei, Z. (2001). Teaching and researching motivation. Harlow: Longman.

Fisher, W. P. (2007). Rating Scale Instrument Quality Criteria. Retrieved November 25, 2007, from http://www.rasch.org/rmt/rmt211m.htm

Kikuchi, K. (2009). Student demotivation in Japanese high school English classrooms: Exploring with qualitative research methods. Language Teaching Research, 13(4), pp.453-471.

Kikuchi, K. (forthcoming). What are possible demotivators in SLA? –An insight from English teaching contexts in Japan. Multilingual Matters

Kojima, S. (2004). English learning demotivation in Japanese EFL students: Research in demotivational patterns from the qualitative research results of three different types of high schools. Unpublished master thesis, Kwansei Gakuin University, Hyogo, Japan.

Linacre, J. M. (1997). Guidelines for rating scales. Retrieved November 25, 2007, from http://www.rasch.org/rn2.htm.

Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage Publications.

Sakai, H., & Kikuchi, K. (2009). Japanese learners' demotivation to study English: A survey study? JALT Journal, 31 (2), pp.183-204.

Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum.

Q & A and discussion

The Rasch model

(Rasch, 1960)

The Rating Scale Model

(Andrich, 1978)

+Don’t walk:

Rasch to

join

the

questionnai

re trend!

Ritsumeik

an

University

Keita

Kikuchi

Kanagaw

a

University

J. W.

Lake

Fukuoka

Women’s

University

Send future inquires to Keita:

keita@kanagawa-u.ac.jp