ACS Faculty GrantBlended Inclusion: Innovative Pedagogy in Clinical Psychology Research Methods and

Statistics for Diverse Students

Assessment Measures

PageKnowledge of Research Methods & Statistics

Self-Reported Competence: Understanding of Research Methods & Statistics 2Knowledge Assessment: Mediation & Moderation (Fall 2017 & Spring 2018) 3Knowledge Assessment: Mediation & Moderation (Fall 2018 & Spring 2019) 6Knowledge Assessment: Communicating Research Findings 12Knowledge Assessment: Scale Development & Evaluation 18

Statistics AnxietyStatistics Anxiety Scale 24

Growth-Oriented Mindset for Quantitative SkillsImplicit Theory of Mathematical Ability 25

Blended Learning Module Usability Assessment 26Demographics 28

Project Lead: Laura E. Sockol, Ph.D., Davidson CollegeParticipants: William D. Ellison, Ph.D., Trinity University

Laura E. Knouse, Ph.D., University of RichmondSundi L. Richard, M.Ed., Davidson CollegeLauren A. Stutts, Ph.D., Davidson College

Self-Reported Competence: Understanding of Research Methods & Statistics(Developed for the current study.)

The following questions were developed to assess students’ self-rated competence regarding their ability to understand, interpret, and apply foundational concepts in research methodology and statistics.

_____________________________________________________________________________________

How would you rate your ability to:

1Not At All Competent

2Minimally Competen

t

3Somewhat Competent

4Competen

t

5Very

Competent1. Translate research questions into research

designs.2. Consider and evaluate the strengths and

weaknesses of elements of research designs.3. Understand statistical tests and how to

interpret them.4. Interpret and understand results of a study

presented as a graph.5. Interpret and understand results of a study

presented as a table.6. Interpret and understand results of a study

presented as text.7. Conduct and report results of statistics and

analyses.

2

Knowledge Assessment: Mediation & Moderation (Fall 2017 & Spring 2018) (Developed for the current study; abstracts and tables/figures adapted from Da Fonseca et al., 2008 & Rudolph, 2010.)

The following questions were developed to assess students’ understanding of the concepts of mediation and moderation. The questions assess students’ factual understanding of the concepts and ability to recognize, apply and interpret results of research studies of these concepts.

_____________________________________________________________________________________

1. What is a mediator? [Text Box]

Scoring:2 = Response is accurate & comprehensive.

“A mediator is a variable that explains the relationship between two other variables.”

1 = Response is accurate, but may not be comprehensive. “A mediator is a mechanism.” “A mediator is a variable that is related to two other variables.”

0 = No response or inaccurate response. “A mediator is a variable that changes the relationship between two other variables.”

2. What is a moderator? [Text Box]

Scoring:2 = Response is accurate & comprehensive.

“A moderator is a variable that changes the relationship between two other variables.”

1 = Response is accurate, but may not be comprehensive. “A moderator is an interaction.” “A moderator changes something.”

0 = No response or inaccurate response. “A moderator is a variable that explains/accounts for the relationship between two other variables.”

3

The following questions are based on the study below.

The aim of this study was to determine whether depression can explain the negative relationship between academic performance and the belief that intelligence is a fixed trait, i.e., entity belief. A sample of 353 French volunteer adolescents (age 11–16) completed questionnaires assessing entity theory and depressive symptoms. Academic performance was assessed by math performance. Results of this study revealed that entity theory is a significant negative predictor of academic performance and a significant positive predictor of depression. Our findings indicate that individuals who consider their abilities to be non-malleable (high entity theory) are more likely to develop depressive symptoms which, in turn, decrease academic performance. These findings contribute to tailoring specific prevention and treatment programs for childhood and adolescent depression.

3. Is this study assessing mediation or moderation?a. Mediationb. Moderationc. Neitherd. Unsure

4. Identify the role of each variable in this study. [Drag-and-drop.]

Roles VariablesPredictor(s) Academic/Math Performance Mediator(s) Depressive Symptoms Moderator(s) Entity TheoryOutcome(s)

5. Based on the figure above, do the results of this study show significant mediation or moderation? Check all that apply.

a. Yes, significant mediation.b. Yes, significant moderation.c. No.d. Unsure.

4

The following questions are based on the study below.

This research investigated the role of children’s implicit theories of peer relationships in their psychological, emotional, and behavioral adjustment. Participants included 206 children who reported on their implicit theories of peer relationships (entity vs. incremental theory), depressive and aggressive symptoms, and exposure to peer victimization. Results confirmed that entity theorists were more likely than incremental theorists to demonstrate depressive and aggressive symptoms when victimized. These findings contribute to social–cognitive theories of motivation and personality, and have practical implications for children exposed to peer victimization and associated difficulties.

6. Is this study assessing mediation or moderation?a. Mediationb. Moderationc. Neitherd. Unsure

7. Identify the role of each variable in this study. [Drag-and-drop.]

Roles VariablesPredictor(s) Aggressive Symptoms Mediator(s) Depressive Symptoms Moderator(s) Exposure to Peer VictimizationOutcome(s) Implicit Theories of Peer Relationships

Table 2. Predicting Depressive Symptoms from Peer Victimization, Implicit Theories of Relationships, and Peer Victimization x Implicit Theories of Relationships Interactions

Depressive Symptoms

Predictors β t ΔR2

Step 1: Main Effects .33Peer victimization .49 8.19***Implicit theories .21 3.58***

Step 2: Interaction .05Peer victimization x implicit theories .24 4.18***

Note: Values in the table are statistics from step entry in hierarchical multiple regression analyses. *** p <.001.

8. Based on the figure above, do the results of this study show significant mediation or moderation? Check all that apply.

a. Yes, significant mediation.b. Yes, significant moderation.c. No.d. Unsure.

5

Knowledge Assessment: Mediation & Moderation (Fall 2018 & Spring 2019) (Developed for the current study.)

Pre-Course Questions

What is a mediator? ________________________________________________________________________________________________________________________________

What is a moderator?

________________________________________________________________________________________________________________________________

Burnette and colleagues (2017) tested an online intervention designed to teach rural adolescent girls that they didn't have a fixed amount of intelligence but that instead their intelligence could be changed through education. They found that the intervention increased students' motivation to learn, and it did so by changing their beliefs about their intelligence. What relationship is occurring between these variables?

Mediation Moderation Neither Both

O'Donnell and colleagues (2017) measured what type of a certain gene (called COMT) children had and the amount of anxiety their mothers had while pregnant. They found that maternal prenatal anxiety predicted children's ADHD symptoms, and it had a particularly strong effect if the child had a particular COMT genotype. What relationship is occurring between these variables?

Mediation Moderation Neither Both

In a study by Gunnell and colleagues (2017), college students completed questionnaires about self-compassion, psychological need satisfaction, and mood at the beginning of their first semester and again five months later. They found that psychological need satisfaction explained the relationship between self-compassion and mood.

In this example, SELF-COMPASSION is a(n):

Predictor Mediator Outcome None of the above

6

Berntson, Patel, and Stewart (2017) found that stressful life events were associated with increased odds of heart disease. This relationship was especially strong in people with a history of depression.

In this example, DEPRESSION HISTORY is a(n):

Predictor Moderator Outcome None of the above

Which is true about mediation and moderation?

They both explain why X predicts Y They both indicate for whom and under what circumstances X predicts Y They both have a third variable (M) They both include the relationship among four variables

What is the missing variable in the above figure? Mediator Moderator Either mediator or moderator Neither mediator nor moderator

7

Does the evidence in this table show significant moderation?  Yes No Not enough information to determine Unsure

Does this figure show evidence for mediation? Yes No Not enough information to determine Unsure

8

Post-Course Questions

What is a mediator? ________________________________________________________________________________________________________________________________

What is a moderator?________________________________________________________________________________________________________________________________

Liu, Kraines, Massing-Schaffer, and Alloy (2014) asked 66 adults with a history of depression to complete a questionnaire of rejection sensitivity  and then interviewed them four months later about their depression symptoms. They found that rejection sensitivity predicted depression four months later, and it did so because more rejection-sensitive people experienced more stressful events, which made them more depressed.

What relationship is occurring between these variables? Mediation Moderation Neither Both

Feng and colleagues (2009) assessed 225 9-year-old girls on a measure of emotion regulation, and they assessed the girls’ mothers on a measure of parental psychological control. A year later, they assessed the girls’ depression symptoms. They found that low levels of positive emotions at age 9 predicted depression at age 10, but only in the context of high maternal psychological control.

What relationship is occurring between these variables? Mediation Moderation Neither Both

Al-Nima, Rosenberg, Archer, and Garcia (2013) surveyed college students about their stress, self-esteem, and depression. They found that stress explained the relationship between self-esteem and depression.

In this example, STRESS is a(n): Predictor Mediator Outcome None of the above

9

Ceschi and colleagues (2016) studied burnout in a sample of 208 service-sector workers. They found that exhaustion predicted “counterproductive work behaviors,” but this relationship was weaker in workers high in grit (i.e, a personality trait involving perseverance in the face of frustration).

In this example, is GRIT a(n): Predictor Moderator Outcome None of the above

Which is true about both mediation and moderation? They both explain why X predicts Y They both indicate for whom and under what circumstances X predicts Y They both include the relationship among four variables They both add complexity to how X predicts Y

What is the missing variable in the above figure? Mediator Moderator Either mediator or moderator Neither mediator or moderator

10

Does the evidence in this table show moderation? Yes No Not enough information to determine Unsure

Does this figure show evidence for mediation? Yes No Not enough information to determine Unsure

11

Knowledge Assessment: Scale Development and Evaluation (Fall 2018 & Spring 2019) (Developed for the current study.)

Pre-Course Questions

What is statistical reliability?________________________________________________________________________________________________________________________________

What is statistical validity?________________________________________________________________________________________________________________________________

Which of the following examples is a SELF-REPORT SCALE measure? Satisfaction measure Physiological monitor Interview fMRI image

Below is some information on a new scale designed to measure aspects of happiness, cleverly named the Happiness Scale.

Using the information below, which scale represents the total score for the measure?

Happiness Scale Scoring Instructions:Overall Happiness - Add items 1 through 18.Joy Subscale - Add items 1 through 6.Contentment Subscale - Add items 7 through 12.Fulfillment Subscale - Add items 13 through 18.

Overall Happiness Joy Subscale Contentment Subscale Fulfillment Subscale

People have higher scores on one item in the Happiness Scale tend to have higher scores on the other items.

This is an example of which type of reliability?

Test-retest reliability Internal consistency reliability Alternate forms reliability Convergent validity

12

The items on the Happiness Scale ask about different emotions clearly related to happiness such as joy, contentment, excitement, and fulfillment.

This is an example of which type of validity? Face validity Internal consistency Discriminant validity Convergent validity

Which of the following self-report items is most likely to yield valid results? In other words, which is the best item?

How satisfied are you with your job and your hobbies? Most people want to find enjoyment in their work. Is that true for you? Approximately how many hours per week do you spend on hobbies? Do you not want to miss out on activities with friends?

Which of the following best describes the stage of self-report scale development that is missing from the figure?

Define construct Calculate internal consistency Test participants Purchase scale

You calculate Cronbach’s alpha for the items on your new self-report scale based on responses collected from 100 participants. You obtain a Cronbach’s alpha of .80. Does this value indicate (at least) adequate internal consistency reliability?

Yes No Not enough information to determine Unsure

13

Using the information in the correlation table above, is there evidence for the convergent validity of the Happiness Scale?

Yes No Not enough information to determine Unsure

14

Post-Course Questions

What is statistical reliability?________________________________________________________________________________________________________________________________

What is statistical validity?________________________________________________________________________________________________________________________________

Which of the following examples is a SELF-REPORT SCALE measure? Physiological monitor EEG Interview Satisfaction measure

Below is some information on a new scale designed to measure aspects of happiness, cleverly named the Happiness Scale.

Using the information below, which scale represents a subscale score for the measure?

Happiness Scale Scoring Instructions:

Overall Happiness - Add items 1 through 18.Joy Subscale - Add items 1 through 6.Contentment Subscale - Add items 7 through 12.Fulfillment Subscale - Add items 13 through 18.

Overall Happiness Joy Contentment Both B and C

People who have higher scores on the Happiness Scale today tend to also have higher scores on the scale the following week.

This is an example of which type of reliability? Test-retest reliability Internal consistency reliability Alternate forms reliability Convergent validity

15

Scores on the Happiness Scale are unrelated to scores on a measure of level of education.

This is an example of which type of validity? Face validity Internal consistency Discriminant validity Convergent validity

Which of the following self-report items is most likely to yield valid results? In other words, which is the best item?

Which of the following is not your least favorite dinosaur? How satisfied are you with your current level of knowledge about dinosaurs? How much time do you spend reading and watching movies and TV shows about dinosaurs? Do you believe that dinosaurs are an exceedingly interesting topic to study?

Which of the following best describes the stage of self-report scale development that is missing from the figure?

Calculate internal consistency Pilot testing Demand characteristics Literature review

You calculate Cronbach’s alpha for the items on your new self-report scale based on responses collected from 100 participants. You obtain a Cronbach’s alpha of .45. Does this value indicate (at least) adequate internal consistency reliability?

16

Yes No Not enough information to determine Unsure

Using the information in the correlation table, is there evidence for the convergent validity of the Happiness Scale?

Yes No Not enough information to determine Unsure

17

Knowledge Assessment: Communicating Research (Fall 2018 & Spring 2019) (Developed for the current study.)

Pre-Course Questions

Name the four elements of research (i.e., major sections) typically included in scientific manuscripts, posters and presentations.________________________________________________________________________________________________________________________________

Name four types of figures used to present information in scientific manuscripts, posters and presentations.________________________________________________________________________________________________________________________________

A panel presentation is what type of research communication? Scientific Manuscript Scientific Poster Scientific Oral Presentation Wikipedia Page

Which of the following would be the best medium to include the most detailed information of a scientific study?

Scientific Manuscript Scientific Poster Scientific Oral Presentation Wikipedia Page

The measures used in a study are typically found in what section of a research communication? Introduction Method Results Discussion

The limitations of a study are typically found in what section of research communication? Introduction Method Results Discussion

Tables and figures that illustrate findings in a study are typically found in what section of research communication?

18

Introduction Method Results Discussion

The main study aims and hypotheses are typically found in what section of research communication? Introduction Method Results Discussion

Which of the following statements is true about the table above? There are many more women than men in each group The highest represented racial category in both groups is Bi/Multi-racial The two groups are much different in terms of racial distribution The control group and mindset group have an equal number of participants

19

Which of the following statements is true about the figure above? The control group's growth mindset score increased over time The mindset group's growth mindset increased over time Both group's growth mindset stayed the same over time The control group and the mindset group had much different scores at the pre-time

20

Post-Course Questions

Name the four elements of research (i.e., major sections) typically included in scientific manuscripts, posters and presentations.________________________________________________________________________________________________________________________________

Name four types of figures used to present information in scientific manuscripts, posters and presentations.________________________________________________________________________________________________________________________________

A journal article is what type of research communication? Scientific Manuscript Scientific Poster Scientific Oral Presentation Wikipedia Page

Which of the following would be the best medium to share your research in a concise form where the audience can come by and learn about your research individually?

Scientific Manuscript Scientific Poster Scientific Oral Presentation Wikipedia Page

The explanations of a study’s findings are typically found in what section of research communication? Introduction Method Results Discussion

The literature review of a study is typically found in what section of research communication? Introduction Method Results Discussion

The findings from the specific aims of a study are typically found in what section of research communication?

Introduction Method Results Discussion

21

The procedure of recruiting participants in a study is typically found in what section of a research communication?

Introduction Method Results Discussion

Which of the following statements is true about the table above? There are many more men than women in each group The highest represented racial category in both groups is Asian American The two groups are highly similar in terms of racial distribution The control group and the mindset group have a different number of participants

22

Which of the following statements is true about the figure above? The control group's self-compassion score largely stayed the same over time The self-compassion group's self-compassion score decreased over time Both groups' self-compassion score stayed the same over time The control group and the self-compassion group had much different scores at pre-time

23

Statistics Anxiety: Statistics Anxiety Scale(Pretorius & Norman, 1992; adapted from Betz, 1978)

The Statistics Anxiety Scale (SAS) is a 10-item measure which assesses the prevalence and intensity of statistics anxiety. The SAS has been validated in college students; high statistics anxiety predicted lower mathematics achievement test scores, higher test anxiety, and higher trait anxiety. The authors report internal reliability of α = 0.92.

_____________________________________________________________________________________

1Strongly Disagree

2Disagree

3Undecided

4Agree

5Strongly

Agree1. It wouldn’t bother me at all to take more

statistics courses.2. I have usually been at ease during statistics

tests.3. I have usually been at ease in statistics

courses.4. I usually don’t worry about my ability to solve

statistics problems.5. I almost never get uptight while taking

statistics tests.6. I get really uptight during statistics tests.

7. I get a sinking feeling when I think of trying hard statistics problems.

8. My mind goes blank and I am unable to think clearly when working statistics.

9. Statistics makes me feel uncomfortable and nervous.

10. Statistics makes me uneasy and confused.

24

Growth-Oriented Mindset for Quantitative Skills: Implicit Theory of Mathematical Ability(Adapted from Dweck, Chiu & Hong, 1995 and Pybus, 2015.)

Implicit theories of intelligence reflect the degree to which an individual endorses an entity (fixed) versus incremental (growth) mindset. Previous research suggests that implicit theories of intelligence are domain-specific (Hughes, 2015).

This 5-item measure assesses an individual’s implicit theory of mathematical ability. Higher scores indicate a stronger entity (fixed) implicit theory of mathematical ability. The questions were adapted from existing measures of implicit theories of general intelligence (Dweck, Chiu & Hong, 1995) and computer science ability (Pybus, 2015; Pybus & Gillan, 2015). The authors report internal reliability ranging from α = 0.94 to 0.98 for three items assessing general intelligence (items 1-3) and 0.88 for items assessing domain-specific intelligence (items 2, 4 & 5).

_____________________________________________________________________________________

1Strongly Disagree 2 3 4 5

6Strongly

Agree1. You have a certain amount of

mathematical ability, and you really can’t do much to change it.

2. Your mathematical ability is something about you that you can’t change very much.

3. You can learn new things, but you can’t really change your basic mathematical ability.

4. Practice, hard work, effort, and persistence can change your mathematical aptitude.

5. Your mathematical skills are something that you can develop.

25

Blended Learning Module Usability Assessment (Developed for the current study.)

The following questions ask about your experience using the [MODULE NAME] Module.

How much of the module's content did you complete? None Some About Half Most All

Please rate the module on the following categories:

Poor Fair Good Excellent Not applicable

Engaging o o o o oEasy to use o o o o oRelevant to Course

o o o o oEasy to Access o o o o oAccessibility (e.g., closed captioning)

o o o o o

Organized o o o o o

Please rate the CONTENT of the module on the following categories:

Poor Fair Good Excellent

Informative o o o oClear o o o oComprehensive o o o o

26

Please rate the difficulty level of the module: Far too easy A little too easy Just right A little too difficult Far too difficult

What elements of the module did you find most helpful or valuable?________________________________________________________________________________________________________________________________

What elements of the module did you find least helpful or valuable?________________________________________________________________________________________________________________________________

Do you have any recommendations for how to make the module better?________________________________________________________________________________________________________________________________

27

Demographics

1. What is your class year? 2018 2019 2020 2021 Other (Please specify.)

2. What is your major/minor? (Check all that apply.) Undeclared Archaeology Art/Art History Business (e.g., Accounting, Business Administration, Business Analytics & Technology,

Finance, International Business, Management, Marketing) Classical Studies Cultural Studies (e.g., Africana Studies, American Studies, Arab Studies, Chinese

Studies, East Asian Studies, Hispanic Studies, Jewish Studies, Latin American/Latino/Iberian Studies, Luso-Brazilian Studies, Middle East Studies, South Asian Studies)

Cognitive Science Communication/Communication Studies, Journalism Computer Science Digital Studies Economics (Business Economics, Mathematical Economics) Education Engineering English Language & Literature (Creative Writing, Comparative Literature) Film/Media Studies Foreign Languages and Literature (e.g., Arabic, Chinese Language & Literature,

Classical Languages & Literature, French & Francophone Studies, German Studies, Greek, Italian, Japanese, Studies, Latin, Spanish, Russian Studies)

Gender & Sexuality Studies Healthcare Studies, Health & Human Values History International Relations/International Studies Leadership Studies Linguistics Mathematics Natural Sciences (e.g., Astrophysics, Biology, Biochemistry, Biology,

Chemistry/Applied Chemistry, Environmental Science/Studies, Geography, Geosciences, Genomics, Molecular Biology, Neuroscience, Physics/Experimental Physics)

Performing Arts (Art, Visual Media & Arts Practice; Dance; Music; Theater) Philosophy Political Science/Government Psychology Religious Studies Social Sciences (e.g., Anthropology, Sociology) Statistics/Data Science Other (Please specify.)

28

3. What is your race? (Check all that apply.) African American/Black American Indian/Alaska Native Asian/Asian American Native Hawaiian/Pacific Islander White/Caucasian Other (Please specify.)

4. Do you identify as Hispanic/Latinx? Yes No

5. What is your gender? (Check all that apply.) Man Woman Transgender Non-Binary Other (Please specify.)

6. Are you a first-generation student? (Note. Federal guidelines classify students as first-generation if neither parent has completed a four-year college degree.)

Yes No

7. What is your disability status? (Check all that apply.) Physical Disabilities

Please specify. Learning Disabilities/Cognitive Disabilities (e.g., ADHD, history of concussions)

Please specify. Significant Physical or Mental Health Conditions

Please specify. I do not have a history of disability or health conditions.

8. Have you taken [Insert Institution-Specific Research Methods/Statistics Foundational Course Name Here]?

Yes, completed. Yes, currently enrolled. No.

9. Please list all quantitative courses you have you taken in college (e.g., calculus, econometrics, finite mathematics, statistics).

29