The Development and Validation © 2015 SAGE Publications …...Erkut and Marx (2005) measures...

Journal of Psychoeducational Assessment 1 –18

© 2015 SAGE PublicationsReprints and permissions:

sagepub.com/journalsPermissions.nav DOI: 10.1177/0734282915571160

jpa.sagepub.com

Article

The Development and Validation of a Measure of Student Attitudes Toward Science, Technology, Engineering, and Math (S-STEM)

Alana Unfried1, Malinda Faber1, Daniel S. Stanhope1, and Eric Wiebe1

AbstractUsing an iterative design along with multiple methodological approaches and a large representative sample, this study presents reliability, validity, and fairness evidence for two surveys measuring student attitudes toward science, technology, engineering, and math (S-STEM) and interest in STEM careers for (a) 4th- through 5th-grade students (Upper Elementary S-STEM) and (b) 6th- through 12th-grade students (Middle/High S-STEM). Findings from exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) suggested the use of a four-factor structure to measure student attitudes toward science, math, engineering/technology, and 21st century skills. Subject matter experts and literature reviews provided evidence of content validity. Reliability levels were high for both versions. Furthermore, both the Upper Elementary S-STEM and Middle/High S-STEM Surveys demonstrated evidence of configural, metric, and scalar invariance across grade levels, races/ethnicities, and genders. The findings support the validity of interpretations and inferences made from scores on the instruments’ items and subscales.

KeywordsK-12, STEM, measurement, scale development, measurement invariance, attitudes

Workers with science, technology, engineering, and math (STEM) skills and competencies are in demand in the United States (Carnevale, Smith, & Melton, 2011; Rothwell, 2013), and research-ers and economists predict that this demand will continue for years to come (National Academy of Engineering, 2008; U.S. Department of Commerce, 2012). The likelihood that kindergarten through 12th-grade (K-12) students will participate in the STEM workforce depends in part on their attitudes toward STEM subjects and their interest in STEM careers (Business-Higher Education Forum, 2010; President’s Committee of Advisors on Science and Technology [PCAST], 2010). Thus, K-12 school systems at the local, state, and national levels have been undertaking special efforts to improve student attitudes and to increase student interest in STEM fields. Despite this work, there remains a dearth of comprehensive, validated, STEM-related

1North Carolina State University, Raleigh, NC, USA

Corresponding Author:Eric Wiebe, Department of STEM Education, North Carolina State University, Box 7801-326 Poe Hall, Raleigh, NC 27695-7801, USA. Email: [email protected]

571160 JPAXXX10.1177/0734282915571160Journal of Psychoeducational AssessmentUnfried et al.research-article2015

by guest on February 17, 2015jpa.sagepub.comDownloaded from

mailto:[email protected]

http://jpa.sagepub.com/

2 Journal of Psychoeducational Assessment

measurement instruments for educational programs (Minner, Ericson, Wu, & Martinez, 2012). This article describes the development of two surveys designed to measure STEM attitudes and career interests in two age ranges within the full range of 4th- through 12th-grade students (the development of an instrument measuring attitudes of kindergarten through third-grade students was beyond the scope of this study).1

Background and Theoretical Rationale

Student attitudes toward an academic subject can be thought of as being composed of self-effi-cacy and expectancy-value beliefs, two important subcomponents of his or her achievement motivation (Eccles & Wigfield, 2002). Self-efficacy is the belief in one’s ability to complete tasks or influence events that have an impact on one’s life (Bandura, 1986), and research has shown that students are more likely to pursue postsecondary schooling in STEM fields if they have high self-efficacy in math (Wang, 2013) or science (Scott & Mallinckrodt, 2005). Expectancy-value theory complements self-efficacy theory when investigating models of career aspirations and career-pathway persistence. Expectancy-value theories posit that individuals regularly assess the likelihood of attaining specific goals and appraise the value gained or lost from such attainment (Eccles & Wigfield, 2002; Wigfield & Eccles, 2000). Having high expectancy-value beliefs has been found to be associated with a student’s persistence in taking both advanced science and math courses (Fan, 2011; Simpkins, Davis-Kean, & Eccles, 2006). In this study, we use the term attitude to indicate a composite of both self-efficacy and expectancy-value beliefs.

In addition to such attitudes toward STEM disciplines, researchers are studying more closely the degree to which student interest in STEM careers predicts future participation in the STEM workforce. Historically, the association between career interests and participation in STEM career pathways has been less rigorously studied for elementary, middle, and high school stu-dents than for postsecondary students (Chen, Gully, Whiteman, & Kilcullen, 2000). Researchers recently, however, have begun to examine in more depth the career interests of younger students. Maltese and Tai (2011) found that eighth-grade students who believed science would be useful in their future and who were interested in a science career were more likely to earn degrees in STEM. Sadler, Sonnert, Hazari, and Tai (2012) found that students’ career interests when enter-ing high school were the strongest predictors of their career interests when leaving high school. Prior to the aforementioned studies, Kidd and Naylor (1991) found that course enrollments and occupational interests of Australian high school students had the greatest impact on students’ intentions to enroll in postsecondary STEM courses.

Together, STEM attitudes and career interests are key components in a larger theory of career development called social cognitive career theory (SCCT; Lent, Sheu, Gloster, & Wilkins, 2010). SCCT integrates individual, environmental, and behavioral variables into a model describing individuals’ academic and career choices (Lent & Brown, 2006; Lent, Brown, & Hackett, 2000). At the individual level, SCCT focuses on self-efficacy, outcome expectancy (an earlier, broader concept from which expectancy-value theory was derived), and goals (Lent & Brown, 1996). Researchers have applied SCCT in studies of STEM career pathways at the postsecondary level (e.g., Byars-Winston, Estrada, Howard, Davis, & Zalapa, 2010; Lent, Lopez, Lopez, & Sheu, 2008), but few studies have applied the theory at the elementary, middle, or high school levels. Wang (2013), however, used data from the Education Longitudinal Study of 2002 to study the effects of 12th-grade student characteristics on intentions to major in STEM. Wang found that a college student’s intent to major in STEM was directly affected by his or her 12th-grade math achievement, exposure to math and science courses, and math self-efficacy beliefs.

In addition to attitudes toward STEM disciplines and STEM career interests, educational pol-icy makers have paid increasing attention to general learning and career skills necessary for citi-zenship and a workforce in the globalized and information-rich 21st century. These skills include



Unfried et al. 3

critical thinking, complex communication skills, problem-solving, and self-management skills (National Research Council [NRC], 2010; PTCS, Partnership for 21st Century Skills, 2004; PCAST, 2010). These common practices, often referred to as 21st century skills, are increasingly included as instructional goals along with STEM content. For example, close alignment with 21st century skills has been found in elementary, middle, and high school education process stan-dards, in state science standards, and in several widely used frameworks for science and engi-neering curricula such as the 5E lesson plan, Learning By Design, and the Engineering Is Elementary design process (NRC, 2010; Rynearson, Douglas, & Diefes-Dux, 2014). For these reasons, national policy makers are calling for increased attention to improving student attitudes toward STEM content, learning, and career skills together (e.g., NRC, 2011).

In summary, researchers and policy makers have identified attitudes toward STEM content and career interests as key variables in predicting student participation in STEM-related careers, with attitudes toward 21st century skills as an important new variable to understand. Expanding on this, it is less clear from the literature whether students at the elementary, middle, and high school levels form attitudes about the multiple STEM subjects as a collective “meta-discipline” or as distinctive, individual subject areas. School organizational structures, curriculum develop-ers, and educators have historically treated STEM fields as primarily distinct areas of study (e.g., science or math), but there is increasing interest in treating STEM as a more unified area of instructional practice (National Academy of Engineering and the National Research Council [NAE-NRC], 2014). It is unknown whether this recent policy and practice discussion has influ-enced student perceptions.

Extant STEM Instruments

Several existing surveys measure younger students’ attitudes toward a single STEM subject. For instance, the Test of Science-Related Attitudes (TOSRA; Fraser, 1978) consists of seven sub-scales (10 items each) measuring high school student attitudes toward science. Scale reliabilities range from .64 to .93, including adequate test–retest reliability (Fraser, 1981). TOSRA has been used often, but has been criticized occasionally for low levels of discriminant validity (Khalili, 1987). Another measure of student attitudes toward science is the Affective Elements of Science Learning Questionnaire (Williams, Kurtek, & Sampson, 2011). This instrument consists of four subscales (35 items total) and has been formally tested on two high school chemistry classes. To our knowledge, no validity evidence has been provided. Another example is the Attitudes Toward Mathematics Survey (Miller, Greene, Montalvo, Ravindran, & Nichols, 1996), an instrument that aims to measure student attitudes toward math. The instrument measures high school and college students’ behavioral and cognitive engagement in math and was validated with high school stu-dents (Fredricks et al., 2011). Finally, unlike the aforementioned surveys, a survey created by Erkut and Marx (2005) measures attitudes toward multiple STEM subjects: science, math, and engineering. Erkut and Marx found Cronbach’s alpha levels to be above .85 for the science and math attitudes constructs and above .80 for the engineering attitudes construct. Despite the sur-vey having minimal psychometric support overall, some gender-biased items, and a narrow tar-get population (female, middle school students), the instrument provided a set of items from which to build a more reliable, valid, and fair survey.

Method

To support research on affective dimensions of students’ career development and to provide elementary, middle, and high school STEM education programs with a tool for measuring key constructs, we engaged in a deductive scale development process to produce two theoretically and empirically sound surveys: (a) the Upper Elementary S-STEM Survey and (b) the Middle/




High S-STEM Survey. To develop STEM attitudes items, we explored the psychometric proper-ties of Erkut and Marx’s (2005) STEM attitudes survey and then refined the items and constructs based on theory and empirical results. We simultaneously generated and tested items that mea-sured student interest in STEM careers. Finally, we examined and refined a construct measuring student attitudes toward 21st century skills. This construct contains only self-efficacy measures as 21st century skills are general tasks—that is, a person engages these skills only in connection with a subject-area task, and therefore task values would be confounded.

Survey Development

We piloted the three attitudes toward science, math, and engineering subscales of the Middle/High S-STEM Survey using the items in Erkut and Marx’s (2005) survey. Of note, none of these constructs specifically measured student attitudes toward technology. We adapted the construct for attitudes toward 21st century skills from the Student Learning Conditions Survey (Friday Institute, 2010). A 5-point Likert-type response scale (strongly disagree to strongly agree) was used for all four subscales. We also developed a pool of 43 items measuring student interest in STEM careers based on the Bureau of Labor Statistics’ (2011) Occupational Outlook Handbook. Each item named an occupation and included a brief description. This section used a 4-point Likert-type response scale (not at all interested to very interested). In total, the pilot Middle/High S-STEM Survey consisted of 94 items.

The pilot survey was administered to 109 students in Grades 6 through 12 who were partici-pating in after-school or summer STEM elective programs in North Carolina. We conducted exploratory factor analysis (EFA) to assess construct validity and collected content validity evi-dence through subject matter experts (SMEs). The three SMEs rated each item as essential, use-ful but not essential, or not necessary for measuring the given construct, and from their responses, we computed item-level content validity ratios (CVRs; Lawshe, 1975). Several items were removed or revised based on the total results. These analyses also indicated that the engineering subscale had insufficient evidence of content validity, so engineering education experts at a large university in the Southeast contributed to revising the construct. Through this process, gender-biased items were removed. Items relevant to engineering and technology careers that both do and do not require a bachelor’s degree were added as well; this more explicitly integrated atti-tudes toward technology into the subscale. The subscale was renamed Engineering/Technology Attitudes. After these changes, 43 items remained in the attitudes toward science, math, engineer-ing/technology, and 21st century skills subscales. Based on analysis of the pilot data, the 43 items measuring STEM career interests were collapsed into 12 STEM career pathways. Each new career-interest item consisted of a definition of the career pathway and examples of related occu-pations. We did not expect the 12 pathways to form a single construct; therefore, we ran no addi-tional psychometric tests on the career-interest section.

We developed a version of the S-STEM Survey for fourth- and fifth-grade students by slightly rewording each pilot item to be at a fourth-grade reading level. Two SMEs reviewed these sub-scales for content validity and results were calculated using Lawshe’s CVR. Finding results simi-lar to the Middle/High S-STEM, we made parallel revisions to the pilot Upper Elementary S-STEM. We then conducted interviews with five fifth-grade students, acquiring their feedback on the clarity of items and making subsequent language edits.

To further assess the accuracy of the reading levels to which both S-STEM surveys were writ-ten, 7 middle and high school teachers and 10 upper elementary school teachers rated each item as either too easy (below grade level), just right (at grade level), or too hard (above grade level). The teachers uniformly indicated that both surveys were at an appropriate length and difficulty for their students.



Unfried et al. 5

Sample and Descriptive Statistics

After the initial survey development process was complete, data were collected in fall of 2011 and 2012 from independent groups of students in 213 public schools in rural areas of a southeast-ern state. These schools were each implementing STEM education programs. The Middle/High S-STEM Survey was administered online to 17,485 students, and the Upper Elementary S-STEM Survey was administered to 4,232 students. Cases with zero variation in responses across items, indicating student apathy, were discarded, as were observations from students in grade levels outside 4th through 12th grades. This process left responses from 21,156 students.

Normality of the data was assessed using skew and kurtosis. Per the guidelines given in Kline (2005), absolute skew greater than 3 and kurtosis greater than 10 are considered problematic. For our data, the largest absolute skew was 1.73, and the largest absolute kurtosis was 3.40; thus, the normality assumption required for some of our analyses was supported.

Of the remaining 21,156 surveys, 94.6% had no missing data. For surveys with missing data, descriptive analysis led to the conclusion that the data should be assumed to be missing at ran-dom (MAR; Enders, 2013). Following that conclusion, missing values were imputed using the expectation-maximization (EM) algorithm (Truxillo, 2005). The final sample sizes from 2011 were 8,659 for 6th- through 12th-grade students (middle/high) and 768 for 4th- and 5th-grade students (upper elementary). The 2012 data set contained 8,316 middle/high students and 3,413 upper elementary students. In total, the middle/high students self-reported as 50.2% male, 65.7% White/Caucasian, 10.0% Black/African American, 9.9% Hispanic/Latino, 6.2% multiracial, 4.1% American Indian, 2.0% Asian, 0.2% Pacific Islander, and 1.8% Other. Furthermore, 87.2% were in middle school (Grades 6-8) and 12.8% were in high school (Grades 9-12). This did not come as a surprise as many of the STEM programs were in middle schools. The upper elementary students self-reported as 50.7% male, 59.7% White/Caucasian, 14.8% Black/African American, 9.8% Hispanic/Latino, 4.5% multiracial, 5.1% American Indian, 0.5% Asian, 0.6% Pacific Islander, and 5.1% Other.

Analyses

EFA was conducted with the first year of data (2011) for each survey. This was done to examine the factor structure and to remove any problematic items. Confirmatory factor analysis (CFA) was then used with the second year of data (2012) to test three theoretical models: a baseline model, a single-order factor model, and a second-order factor model. The single- and second-order models were chosen to examine whether students attitudinally respond to STEM as a meta-discipline (second-order) or as separate academic areas (single-order). We then used Cronbach’s alpha to assess the internal consistency of each subscale. Finally, to examine whether items and subscales functioned similarly for different groups of students, we conducted tests of measure-ment invariance on the second year of data.

Results

Exploratory Factor Analysis

We conducted EFA with SAS/STAT® software, Version 9.3, using principal axis factoring and promax rotation to allow correlation among factors. The Kaiser criterion (Kaiser, 1960), scree plots, and interpretability criteria were used to determine the number of factors. Items with loadings higher than .30 were classified as significant (Hair, Black, Babin, Anderson, & Tatham 2006). Items with two or more loadings greater than .30 were considered cross-loading.




In the Middle/High S-STEM Survey, a four-factor solution was found, suggesting a structure consisting of attitudes toward science, math, engineering/technology, and 21st century skills. One math attitudes item did not load at the .30 level on any factor, and no items cross-loaded. Factor correlations ranged from .16 to .37 (see Table 1), and when such factor correlations are found, Gorsuch (1983) recommends extracting higher order factors. Therefore, using the same criteria as the initial EFA, we conducted such analyses and found that the four first-order factors loaded significantly on one second-order factor. This could be considered a broader “STEM atti-tudes” factor. The Schmid-Leiman transformation to the bifactor solution was used to view direct relationships between items and factors, as well as the amount of extracted variance explained by each level separated into disjoint groups (Wolff & Preising, 2005). The second-order STEM attitudes factor explained 26.1% of the extracted variance, and the four first-order factors explained 73.9% of the extracted variance.2

A parallel four-factor structure was found for the Upper Elementary S-STEM Survey, although three math items and one science item did not load significantly on any factor (one was the same item that did not load on the Middle/High S-STEM Survey). The science item, however, and one of the math items had loadings of .29 on their respective factors—close to the target cutoff of .30. No items cross-loaded, and factor correlations ranged from .16 to .44 (see Table 2). One second-order factor that explained 32.3% of extracted variance after the Schmid-Leiman transformation was found. The four first-order factors explained 67.7% of the extracted variance.2

Gorsuch (1983) notes that second-order factors are of interest when they explain 40% to 50% of the extracted variance. The second-order factors for both surveys only explained 26% to 32% of the variance; therefore, the four lower-order factors are of more interest than the broader STEM attitudes factor. The math and science items that had EFA loadings of .29 on the upper elementary survey, and clearly loaded significantly on the Middle/High S-STEM survey, were retained in the survey—we used our judgment to determine that the items added substantial research value, especially as the .30 value is an arbitrary cutoff. We also intended that future psychometric tests conducted by us and others would determine whether these items should con-tinue to be retained. The other two math items that exhibited loadings much lower than .30 on the Upper Elementary S-STEM Survey were dropped from both survey versions. Based on a rational review, we also dropped two engineering/technology attitudes items and two 21st century skills attitudes items from both surveys. This was done to reduce respondent burden. The first-order factor analysis results showed clear structure, and we made no additional changes to the survey. We concluded that the Middle/High and Upper Elementary S-STEM Surveys were parallel and measured the constructs in an equivalent manner.

Confirmatory Factor Analysis

CFA was performed on both surveys with SAS/STAT® software, Version 9.3. Each survey con-sisted of 37 items. Factor variances were fixed at 1 for identification, and maximum likelihood estimation was used. These surveys were new measurement tools; therefore, we tested three theory-based models similar to those tested in the EFA. First, we considered a baseline model in which one latent factor undergirded all 37 items. The second model we tested was based on the first-order EFA results and specified four correlated STEM factors: attitudes toward science, math, engineering/technology, and 21st century skills. For each of the three subject-area factors, we allowed correlated residuals among the self-efficacy items and among the expectancy-value items—for example, in the science attitudes factor, we allowed correlated residuals among the four items measuring self-efficacy and separately among the five items measuring expectancy-value. We did this because research by others, such as those cited in the literature review, sug-gests a common source of variance among the self-efficacy items and also among the expectancy-value items. Allowing correlated errors accounts for systematic error by modeling



Unfried et al. 7

Table 1. Middle/High S-STEM EFA Promax-Rotated Loadings, CFA Loadings, Factor Correlations, and Final Reliability.

EFA results CFA results

Item Math ScienceEngineering/technology

21st century

skills Math ScienceEngineering/technology

21st century

skills

(D) Math is important for my life .38 — (-) Math has been my worst

subject.81 .57

I would consider choosing a career that uses math

.46 .67

(-) Math is hard for me .79 .52 (D) I will need a good

understanding of math for my future work

.12 —

I am the type of student to do well in math

.83 .64

(-) I can handle most subjects well, but I cannot do a good job with math

.75 .49

I am sure I could do advanced work in math

.64 .65

I can get good grades in math .62 .52 I am good at math .86 .65 I am sure of myself when I do

science.58 .72

I would consider a career in science

.81 .68

I expect to use science when I get out of school

.81 .64

Knowing science will help me earn a living

.75 .58

I will need science for my future work

.83 .54

I know I can do well in science .53 .68 Science will be important to me in

my life’s work.83 .59

(-) I can handle most subjects well, but I cannot do a good job with science

.35 .34

I am sure I could do advanced work in science

.58 .73

I like to imagine creating new products

.61 .64

If I learn engineering, then I can improve things that people use every day

.62 .76

I am good at building and fixing things

.68 .62

(D) Understanding engineering concepts will help me earn a living

.73 —

I am interested in what makes machines work

.78 .70

Designing products or structures will be important for my future work

.77 .80

(continued)






21st century


21st century

skills

I am curious about how electronics work

.64 .63

(D) I would choose a career that involves building things

.83 —

I would like to use creativity and innovation in my future work

.65 .70

Knowing how to use math and science together will allow me to invent useful things

.51 .73

I believe I can be successful in a career in engineering

.80 .75

I am confident I can lead others to accomplish a goal

.70 .75

I am confident I can encourage others to do their best

.74 .78

(D) I am confident I can make moral decisions

.73 —

I am confident I can produce high quality work

.65 .71

(D) I am confident I can act responsibly

.76 —

I am confident I can respect the differences of my peers

.75 .75

I am confident I can help my peers .79 .80I am confident I can include

others’ perspectives when making decisions

.72 .76

I am confident I can make changes when things do not go as planned

.68 .73

I am confident I can set my own learning goals

.71 .74

I am confident I can manage my time wisely when working on my own

.65 .67

When I have many assignments, I can choose which ones need to be done first

.64 .67

I am confident I can work well with students from different backgrounds

.65 .68

Math factor correlation — — Science factor correlation .16 — .27 — Engineering/technology factor

correlation.19 .32 — .36 .37 —

21st century skills factor correlation

.32 .37 .25 — .39 .43 .29 —

Reliability .90 .89 .90 .92

Note. (D) indicates that the item was deleted after EFA; (-) indicates that the item was reverse-coded during analysis. EFA extraction method: principal axis factoring, rotation method: promax, CFA extraction method: maximum likelihood. Loadings are only displayed for items on their expected factors (no other loadings were significant); loadings ≥ .3 are boldfaced. S-STEM = student attitudes toward science, technology, engineering, and math; CFA = confirmatory factor analysis; EFA = exploratory factor analysis.

Table 1. (continued)



Unfried et al. 9

Table 2. Upper Elementary S-STEM EFA Promax-Rotated Loadings, CFA Loadings, Factor Correlations, and Final Reliability.



21st century


21st century

skills

(D) Math is an important life skill .22 — (-) Math has been my worst subject .70 .31 When I am older, I might choose a

job that uses math.29 .45

(-) Math is hard for me .77 .28 (D) When I am older, I will need to

understand math for my job.14 —

I am the type of student who does well in math

.78 .44

(-) I can understand most subjects easily, but math is difficult for me

.75 .24

In the future, I could do harder math problems

.52 .58

I can get good grades in math .55 .49 I am good at math .82 .47 I feel good about myself when I do

science.47 .60

I might choose a career in science .80 .57 After I finish high school, I will use

science often.68 .60

When I am older, knowing science will help me earn money

.62 .50

When I am older, I will need to understand science for my job

.74 .46

I know I can do well in science .46 .59 Science will be important to me in

my future career.75 .54

(-) I can understand most subjects easily, but science is hard for me to understand

.29 .31

In the future, I could do harder science work

.43 .65

I like to imagine making new products

.51 .62

If I learn engineering, then I can improve things that people use every day

.59 .67

I am good at building or fixing things

.52 .46

(D) Understanding engineering will help me earn money

.65 —

I am interested in what makes machines work

.71 .55

Designing products or structures will be important in my future jobs

.74 .63

I am curious about how electronics work

.49 .50

(D) I would choose a job that involves building things

.75 —

I want to be creative in my future jobs

.46 .59

(continued)




the sources of variance that are not accounted for by the overall latent attitudes factor, whereas the common factor model assumes that errors are nonsystematic and uncorrelated. Finally, the third model we tested was a second-order factor model in which four first-order factors load on a general STEM-attitudes second-order factor. In this case, the correlation between the first-order factors is accounted for by the presence of the second-order factor. Correlated errors among items were allowed in the same manner as the first-order model.



21st century


21st century

skills

Knowing how to use math and science together will help me to invent useful things

.46 .65

I believe I can be successful in engineering

.68 .70

I can lead others to reach a goal .50 .61I like to help others do their best .75 .73(D) I usually know how to do the

right thing.58

In school and at home, I can do things well

.53 .61

(D) I can and usually do act responsibly

.55 —

I respect all children my age even if they are different from me

.69 .63

I try to help other children my age .73 .71When I make decisions, I think

about what is good for other people

.69 .64

When things do not go how I want, I can change my actions for the better

.48 .53

I can make my own goals for learning

.52 .60

I can use time wisely when working on my own

.52 .55

When I have a lot of homework, I can choose what needs to be done first

.54 .54

I can work well with all students, even if they are different from me

.66 .63

Math factor correlation — — Science factor correlation .22 — .52 — Engineering/technology factor

correlation.16 .44 — .46 .59 —

21st century skills factor correlation

.43 .36 .28 — .55 .51 .43 —

Reliability .85 .83 .84 .87

Note. (D) indicates that the item was deleted after EFA; (-) indicates that the item was reverse-coded during analysis. EFA extraction method: principal axis factoring, rotation method: promax, CFA extraction method: maximum likelihood. Loadings are only displayed for items on their expected factors (no other loadings were significant); loadings ≥ .3 are boldfaced. S-STEM = student attitudes toward science, technology, engineering, and math; CFA = confirmatory factor analysis; EFA = exploratory factor analysis.

Table 2. (continued)



Unfried et al. 11

The fit of the three models was assessed using χ2 values, standardized root mean square resid-ual (SRMR), root mean square error of approximation (RMSEA), and comparative fit index (CFI). A non-significant χ2 typically indicates a good model fit, with the notable exception of large sam-ple size applications, as large sample sizes allow for the detection of minute differences (Brown & Moore, 2012). Alternative fit indices are typically used in these cases (see Table 3). A good fit is indicated by SRMR less than .08, RMSEA less than .06, and CFI greater than .95 (Hu & Bentler, 1999). Marsh, Hau, and Wen (2004) note, however, that these values should be considered “rules of thumb” rather than strict cutoffs, especially regarding more complex models. Furthermore, χ2 difference tests can be conducted to compare the fit of nested models (see Table 3).

The baseline model (Model 1) fit poorly overall (see Table 3). The alternative fit indices indi-cate that Models 2 and 3 are both adequate, with only the CFI slightly below the recommended cutoff value of .95. Model 2 performed slightly better than Models 1 and 3 based on alternative fit indices and χ2 difference tests. Model 2 also aligned with the EFA results indicating that the four first-order factors were of greater significance than the general STEM-attitudes second-order factor. For these reasons, we considered the first-order factor model (Model 2) the best model for these surveys. The correlations found among the latent factors can be seen in Tables 1 (middle/high) and 2 (upper elementary). Furthermore, Figure 1 displays a diagram of Model 2 for the Middle/High S-STEM Survey.

Reliability

Cronbach’s alpha was used to measure internal-consistency reliability for each of the four con-structs. These values were calculated using the entire sample of data for each survey. All con-structs demonstrated sufficient levels of reliability for both the Middle/High S-STEM Survey (.89-.92; see Table 1) and the Upper Elementary S-STEM Survey (.83-.87; see Table 2).

Measurement Invariance/Equivalence

When comparing two or more groups (e.g., genders, age groups, races/ethnicities) with the same measurement instrument, the assumption is made that the instrument measures the same con-structs in the same ways across groups. Satisfying this assumption, referred to as measurement invariance (or equivalence), is critical if the instrument is intended for such comparisons. Following established guidelines (Vandenberg & Lance, 2000), we conducted confirmatory

Table 3. CFA Goodness-of-Fit Indices.

χ2 dfχ2

p value Δχ2 ΔdfΔχ2

p value CFI RMSEA SRMR

Middle/High S-STEM Model 1 107,129.159 629 <.0001 — — — .394 .143 .159 Model 3 10,433.788 570 <.0001 96,695.371 59 <.0001 .944 .046 .052 Model 2 10,254.388 568 <.0001 179.400 2 <.0001 .945 .045 .048Upper Elementary S-STEM Model 1 24,348.194 629 <.0001 — — — .474 .105 .111 Model 3 3,665.944 570 <.0001 20,572.25 59 <.0001 .931 .040 .046 Model 2 3,628.432 568 <.0001 37.512 2 <.0001 .932 .040 .044

Note. Models appear in a nested order from most restrictive to least restrictive. Model 1 is the baseline model, Model 3 is the second-order model, and Model 2 is the first-order model. S-STEM = student attitudes toward science, technology, engineering, and math; CFA = confirmatory factor analysis; CFI = comparative fit index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual.




factor analytic tests of measurement invariance using Mplus®, Version 7.3, statistical software. We examined configural invariance (equality of factor structures), metric invariance (equality of factor loadings), and scalar invariance (equality of item intercepts). A lack of invariance signals either that groups conceive of the particular construct differently or that groups apply a different calibration to the scaling (e.g., agree for one group is equivalent to strongly agree for a different group).

We tested all models as total disaggregation models, in which we specified each item as a mani-fest indicator of its respective, latent construct. To examine configural invariance, we analyzed both groups simultaneously (Horn & McArdle, 1992; Level 1). We allowed item parameters (i.e., factor loadings, item intercepts, and item uniqueness), factor variances, and latent means to vary

Figure 1. First-order factor model (Model 2).Note. Ranges of correlated error terms: mathematics attitudes = .11 to .40, science attitudes = .04 to .44, engineering/technology attitudes = −.12 to .21.



Unfried et al. 13

freely across groups. In successive models, we constrained factor loadings to test for metric invari-ance (Level 2), and we also constrained factor loadings and item intercepts to test for scalar invari-ance (Level 3). For model comparison, we examined multiple goodness-of-fit indices. Furthermore, the χ statistic is highly sensitive in large samples, so we supplemented this test with changes in CFI, Tucker–Lewis index (TLI), and RMSEA (Meade, Johnson, & Braddy, 2008). Cheung and Rensvold (2002) recommend examining changes in CFI (ΔCFI) as the primary test for invariance, with ΔCFI less than .01 indicating invariance.

We developed instruments for different age ranges, so it was important to ensure that measure-ment properties functioned similarly for 4th- and 5th-grade students and for 6th- through 12th-grade students (see Table 4). For the Upper Elementary S-STEM Survey, we examined fourth- versus fifth-grade students. For the Middle/High S-STEM Survey, we ran two sets of models: (a) Grades 6 to 8 versus Grades 9 to 12 and (b) Grades 6 to 7 versus Grades 11 to 12. The former comparison made intuitive sense based on school levels in the sample (middle school vs. high school), and the latter comparison was theoretically most likely to show differential func-tioning because the grade levels were further apart. For both the Upper Elementary and Middle/High S-STEM Surveys, ΔCFI never exceeded .003. Thus, all subscales demonstrated full config-ural, metric, and scalar invariance.

We also examined invariance across races/ethnicities and across gender. In Table 5, we dis-play results from the comparison of measurement properties across the racial/ethnic groups most represented in these data: White/Caucasian, Black/African American, and Hispanic/Latino. For all comparisons in both surveys, ΔCFI never exceeded .006, which suggests that both surveys demonstrated full configural, metric, and scalar invariance. For all gender comparisons (see Table 6), ΔCFI exceeded .007 only once, which was a ΔCFI of .011 for the scalar invariance model in the Upper Elementary S-STEM Survey sample. This is trivially higher (.001) than the suggested cutoff of .01; therefore, we concluded that this provides relatively sound evidence of invariance. We similarly concluded, however, that the potential lack of invariance based on gen-der should be an area of focus in future validation work.

Discussion

Using an iterative design, multiple methodological approaches, and a large representative sam-ple, this study presented reliability, validity, and fairness evidence for two surveys measuring

Table 4. Tests of Measurement Invariance Across Grade Levels.

χ2 dfχ2 p value Δχ2 Δdf

Δχ2 p value CFI ΔCFI TLI RMSEA

Upper elementary: 4th (n = 1,442) vs. 5th (n = 1,971) grades Level 1 4,419.329 1,136 .000 — — — .928 — .915 .041 Level 2 4,471.965 1,169 .000 52.636 33 .016 .927 −.001 .917 .041 Level 3 4,515.569 1,202 .000 43.604 33 .103 .927 .000 .919 .040Middle/high: middle school (n = 7,040) vs. high school (n = 1,276) Level 1 11,131.639 1,136 .000 — — — .943 — .934 .046 Level 2 11,254.553 1,169 .000 122.914 33 .000 .943 .000 .935 .046 Level 3 11,543.372 1,202 .000 288.819 33 .000 .942 −.001 .935 .045Middle/high: 6th to 7th (n = 4,206) vs. 11th to 12th (n = 449) grades Level 1 6,581.690 1,136 .000 — — — .942 — .932 .045 Level 2 6,706.650 1,169 .000 124.960 33 .000 .941 −.001 .933 .045 Level 3 6,997.008 1,202 .000 290.358 33 .000 .938 −.003 .931 .046

Note. CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation




4th- and 5th-grade and 6th- through 12th-grade student attitudes toward STEM and interest in STEM careers. SMEs and literature reviews provided evidence of content validity. Findings from EFA and CFA suggested the use of a four-factor structure to measure student attitudes toward science, math, engineering/technology, and 21st century skills. Student interest in STEM careers was not examined for a latent factor structure due to the nature of the items. Reliability levels were high for both survey versions. Also, from tests of three theoretical models, findings

Table 6. Tests of Measurement Invariance for Gender.

χ2 dfχ2 p value Δχ2 Δdf

Δχ2 p value CFI ΔCFI TLI RMSEA

Upper elementary: male (n = 1,743) vs. female (n = 1,670) Level 1 4,336.102 1,136 .000 — — — .929 — .916 .041 Level 2 4,400.606 1,169 .000 64.504 33 .001 .928 −.001 .918 .040 Level 3 4,922.603 1,202 .000 521.997 33 .000 .917 −.011 .908 .043Middle/high: male (n = 4,202) vs. female (n = 4,114) Level 1 10,869.190 1,136 .000 — — — .943 — .933 .045 Level 2 10,953.628 1,169 .000 84.438 33 .000 .943 .000 .935 .045 Level 3 12,176.441 1,202 .000 1,222.81 33 .000 .936 −.007 .929 .047

Note. CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation.

Table 5. Tests of Measurement Invariance for Race/Ethnicity.

χ2 dfχ2

p value Δχ2 ΔdfΔχ2

p value CFI ΔCFI TLI RMSEA

Upper elementary: White/Caucasian (n = 1,964) vs. Black/African American (n = 559) Level 1 3,772.196 1,136 .000 — — — .923 — .910 .043 Level 2 3,851.884 1,169 .000 79.688 33 .000 .921 −.002 .911 .043 Level 3 4,003.073 1,202 .000 151.189 33 .000 .918 −.003 .909 .043Middle/high: White/Caucasian (n = 5,289) vs. Black/African American (n = 938) Level 1 8,722.832 1,136 .000 — — — .944 — .935 .046 Level 2 8,864.060 1,169 .000 141.228 33 .000 .943 −.001 .935 .046 Level 3 9,127.735 1,202 .000 263.675 33 .000 .942 −.001 .935 .046Upper elementary: White/Caucasian (n = 1,964) vs. Hispanic/Latino (n = 359) Level 1 3,534.494 1,136 .000 — — — .925 — .912 .043 Level 2 3,561.192 1,169 .000 26.698 33 .773 .925 .000 .914 .042 Level 3 3,687.822 1,202 .000 126.63 33 .000 .922 −.003 .913 .042Middle/high: White/Caucasian (n = 5,289) vs. Hispanic/Latino (n = 1,003) Level 1 8,853.495 1,136 .000 — — — .943 — .933 .046 Level 2 8,992.338 1,169 .000 138.843 33 .000 .942 −.001 .934 .046 Level 3 9,450.839 1,202 .000 458.501 33 .000 .939 −.003 .933 .047Upper elementary: Black/African American (n = 559) vs. Hispanic/Latino (n = 339) Level 1 2,283.868 1,136 .000 — — — .899 — .881 .047 Level 2 2,342.854 1,169 .000 58.986 33 .004 .896 −.003 .882 .047 Level 3 2,445.341 1,202 .000 102.487 33 .000 .890 −.006 .878 .047Middle/high: Black/African American (n = 938) vs. Hispanic/Latino (n = 1,003) Level 1 3,328.721 1,136 .000 — — — .942 — .932 .045 Level 2 3,403.154 1,169 .000 74.433 33 .000 .941 −.001 .932 .044 Level 3 3,592.562 1,202 .000 189.408 33 .000 .936 −.005 .929 .045

Note. CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation.



Unfried et al. 15

suggested that students had separate but correlated attitudinal responses to the academic areas of science, math, and engineering/technology. Furthermore, study results indicated that attitudes toward 21st century skills could be considered as a separate factor. Also, borderline loadings on two items suggested a need for continued refining and testing of individual items in the upper elementary survey.

Results also suggested that all subscales in both the Upper Elementary S-STEM and Middle/High S-STEM Surveys demonstrated evidence of configural, metric, and scalar invariance across grade levels, races/ethnicities, and genders. Borderline results for scalar invariance in the fourth- and fifth-grade sample indicated that males and females, particularly at younger ages, may intui-tively calibrate the STEM attitude scales slightly differently. Ongoing validation work should investigate this finding further and determine its causes and implications. Overall, however, evi-dence of measurement invariance herein suggested that both surveys could be used to collect measurements on different groups and that these measurements could be meaningfully inter-preted and compared—the surveys are effectively free from bias and results are fair and trustwor-thy. These findings support the validity of interpretations and inferences made from scores on the instruments’ items and subscales.

Some limitations were present in the study. Most notably, some minor levels of selection bias may have been present in the respondent populations. The pilot survey was administered to stu-dents in elective STEM programs; however, we confirmed results from the pilot study with data that only included around 12% of students who were involved in elective STEM programs. This is the first study examining this instrument, and additional studies are needed to furnish more evidence of construct validity. Additional research is also needed to establish test–retest reliabil-ity, convergent and discriminant validity, and criterion-related validity. Research also could be directed at continued tests of measurement invariance for different groups. Finally, further devel-opment of instruments measuring additional key components of social cognitive career theory as they relate to STEM careers, such as student goals, would complement these surveys well.

The S-STEM surveys are robust instruments that elementary, middle, and high school STEM education program leaders can use to understand students’ psychological states and the impact programs may have on student attitudes toward STEM disciplines and 21st century skills and interest in STEM careers. Researchers can use these surveys to collect data that are important for expanding understanding of student participation and persistence in STEM career pathways. Experts predict that the social and economic demands for STEM professionals in the United States will only increase in the coming decades, and these instruments may serve as useful tools in the collective endeavor to meet that growing need. That said, these instruments are still somewhat new, and researchers are encouraged to continue testing and refining the surveys’ content and applications to help answer questions with important, national, educational policy implications.

Authors’ Note

All research reported in this article was conducted with permission of the authors’ institutional review board, with all relevant ethical guidelines for human research followed. Any opinions, findings, and conclu-sions or recommendations expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publi-cation of this article: Portions of this material are based upon work supported by the National Science




Foundation under Grant DUE-1038154. Portions of the work were also supported by the Golden LEAF Foundation.

Notes

1. For access to the full surveys, please visit our website at http://miso.ncsu.edu/articles/s-stem-survey and fill out the Instrument Request Form.

2. Supplementary materials including additional tables of factor loadings can be obtained from the cor-responding author.

References

Bandura, A. (1986). Social foundations of thoughts and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.

Brown, T. A., & Moore, M. T. (2012). Confirmatory factor analysis. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 361-379). New York, NY: Guilford Press.

Bureau of Labor Statistics. (2011). Occupational outlook handbook (2010-11 ed.). Washington, DC: U.S. Department of Labor. Retrieved from http://www.bls.gov/ooh/

Business-Higher Education Forum. (2010). Increasing the number of STEM graduates: Insights from the U.S. STEM education modeling project. Washington, DC: Author.

Byars-Winston, A., Estrada, Y., Howard, C., Davis, D., & Zalapa, J. (2010). Influence of social cognitive and ethnic variables on academic goals of underrepresented students in science and engineering: A multiple-groups analysis. Journal of Counseling Psychology, 57, 205-218. doi:10.1037/a0018608

Carnevale, A. P., Smith, N., & Melton, M. (2011). STEM: Science, technology, engineering, mathematics. Washington, DC: Georgetown University Center on Education and the Workforce.

Chen, G., Gully, S. M., Whiteman, J. A, & Kilcullen, R. N. (2000). Examination of relationships among trait-like individual differences, state-like individual differences, and learning performance. The Journal of Applied Psychology, 85, 835-847.

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. doi:10.1207/S15328007SEM0902_5

Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109-132.

Enders, C. K. (2013). Dealing with missing data in developmental research. Child Development Perspectives, 7, 27-31. doi:10.1111/cdep.12008

Erkut, S., & Marx, F. (2005). 4 schools for WIE (Evaluation report). Wellesley, MA: Wellesley College, Center for Research on Women. Retrieved from http://www.coe.neu.edu/Groups/stemteams/evaluation.pdf

Fan, W. (2011). Social influences, school motivation and gender differences: An application of the expec-tancy-value theory. Educational Psychology, 31, 157-175. doi:10.1080/01443410.2010.53652

Fraser, B. J. (1978). Development of a test of science-related attitudes. Science Education, 62, 509-515.Fraser, B. J. (1981). TOSRA: Test of science-related attitudes handbook. Hawthorn, Victoria: Australian

Council for Educational Research.Fredricks, J., McColskey, W., Meli, J., Mordica, J., Montrosse, B., & Mooney, K. (2011). Measuring stu-

dent engagement in upper elementary through high school: A description of 21 instruments (Issues & Answers Report, REL 2011-No. 098). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved from http://ies.ed.gov/ncee/edlabs

Friday Institute. (2010). Governor Perdue’s North Carolina Student Learning Conditions Survey (SLCS): Survey Implementation Study. Raleigh, NC: Author.

Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate data analysis

(6th ed.). Upper Saddle River, NJ: Prentice Hall.Horn, J. L., & McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging

research. Experimental Aging Research, 18, 117-144.Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional

criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.


http://miso.ncsu.edu/articles/s-stem-survey

http://www.bls.gov/ooh/

http://www.coe.neu.edu/Groups/stemteams/evaluation.pdf

http://www.coe.neu.edu/Groups/stemteams/evaluation.pdf

http://ies.ed.gov/ncee/edlabs


Unfried et al. 17

Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141-151. doi:10.1177/001316446002000116

Khalili, K. Y. (1987). A cross-cultural validation of a test of science related attitudes. Journal of Research in Science Teaching, 24, 127-136.

Kidd, G., & Naylor, F. (1991). The predictive power of measured interests in tertiary course choice: The case of science. Australian Journal of Education, 35, 261-272. doi:10.1177/000494419103500304

Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed.). New York, NY: Guilford Press.

Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563-575. doi:10.1111/j.1744-6570.1975.tb01393.x

Lent, R. W., & Brown, S. D. (1996). Social cognitive approach to career development: An overview. Career Development Quarterly, 44, 310-321.

Lent, R. W., & Brown, S. D. (2006). Integrating person and situation perspectives on work satisfaction: A social-cognitive view. Journal of Vocational Behavior, 69, 236-247. doi:10.1016/j.jvb.2006.02.006

Lent, R. W., Brown, S. D., & Hackett, G. (2000). Contextual supports and barriers to career choice: A social cognitive analysis. Journal of Counseling Psychology, 47, 36-49. doi:10.1037//0022-0167.47.1.36

Lent, R. W., Lopez, A. M., Lopez, F. G., & Sheu, H. (2008). Social cognitive career theory and the predic-tion of interests and choice goals in the computing disciplines. Journal of Vocational Behavior, 73, 52-62. doi:10.1016/j.jvb.2008.01.002

Lent, R. W., Sheu, H., Gloster, C. S., & Wilkins, G. (2010). Longitudinal test of the social cognitive model of choice in engineering students at historically Black universities. Journal of Vocational Behavior, 76, 387-394. doi:10.1016/j.jvb.2009.09.002

Maltese, A. V., & Tai, R. H. (2011). Pipeline persistence: Examining the association of educational experi-ences with earned degrees in STEM among U.S. students. Science Education, 95, 877-907. doi:10.1002/sce.20441

Marsh, H. W., Hau, K.-T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) Findings. Structural Equation Modeling, 11, 320-341.

Meade, A. W., Johnson, E. C., & Braddy, P. W. (2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568-592. doi:10.1037/0021-9010.93.3.568

Miller, R. B., Greene, B. A., Montalvo, G. P., Ravindran, B., & Nichols, J. D. (1996). Engagement in aca-demic work: The role of learning goals, future consequences, pleasing others, and perceived ability. Contemporary Educational Psychology, 21(4), 388–422. doi:10.1006/ceps.1996.0028

Minner, D., Ericson, E., Wu, S., & Martinez, A. (2012). Compendium of STEM student instruments PART II: Measuring students’ content knowledge, reasoning skills, and psychological attributes. Washington, DC: Abt Associates.

National Academy of Engineering. (2008). Grand challenges for engineering. Washington, DC: National Academies Press.

National Academy of Engineering and the National Research Council. (2014). STEM integration in K-12 education: Status, prospects, and an agenda for research. Washington, DC: National Academies Press.

National Research Council. (2010). Exploring the intersection of science education and 21st century skills: A workshop summary. Washington, DC: National Academies Press.

National Research Council. (2011). A framework for K-12 science education: Practices, crosscutting con-cepts, and core ideas. Washington, DC: National Academies Press.

PTCS, Partnership for 21st Century Skills. (2004). Retrieved from http://www.p21.org/President’s Committee of Advisors on Science and Technology. (2010). Prepare and inspire: K-12 educa-

tion in Science, Technology, Engineering, and Math (STEM) for America’s future. Washington, DC: Executive Office of the President.

Rothwell, J. (2013). The hidden STEM economy: The metropolitan policy program at Brookings. Washington, DC: Brookings Institution.

Rynearson, A. M., Douglas, K. A., & Diefes-Dux, H. A. (2014, June). “Engineering teaches problem solving”: Teachers’ perceptions of student learning through engineering lessons. Presented at the American Society for Engineering Education Annual Conference and Exposition, Indianapolis, IN.


http://www.p21.org/



Sadler, P. M., Sonnert, G., Hazari, Z., & Tai, R. (2012). Stability and volatility of STEM career interest in high school: A gender study. Science Education, 96, 411-427. doi:10.1002/sce.21007

Scott, A., & Mallinckrodt, B. (2005). Parental emotional support, science self-efficacy, and choice of sci-ence major in undergraduate women. Career Development Quarterly, 53, 263-273.

Simpkins, S. D., Davis-Kean, P. E., & Eccles, J. S. (2006). Math and science motivation: A longitudi-nal examination of the links between choices and beliefs. Developmental Psychology, 42, 70-83. doi:10.1037/0012-1649.42.1.70

Truxillo, C. (2005). Maximum likelihood parameter estimation with incomplete data. In Proceedings of the Thirtieth Annual SAS® Users Group International Conference (paper 111-30). Cary, NC: SAS Institute Inc.

U.S. Department of Commerce. (2012). The competitiveness and innovative capacity of the United States. Washington, DC: Author.

Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69. doi:10.1177/109442810031002

Wang, X. (2013). Why students choose STEM majors: Motivation, high school learning, and post-secondary context of support. American Educational Research Journal, 50, 1081-1121. doi:10.3102/0002831213488622

Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68-68.

Williams, K., Kurtek, K., & Sampson, V. (2011). The affective elements of science learning: A question-naire to assess-and improve-student attitudes toward science. Science Teacher, 78(1), 40-45.

Wolff, H.-G., & Preising, K. (2005). Exploring item and higher order factor structure with the Schmid-Leiman solution: Syntax codes for SPSS and SAS. Behavior Research Methods, 37, 48-58.



The Development and Validation © 2015 SAGE Publications …...Erkut and Marx (2005) measures...

Documents

Transcript of The Development and Validation © 2015 SAGE Publications …...Erkut and Marx (2005) measures...