Measuring social impact: Challenges and benefits of alternative methods of measuring social impact
Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring...
Transcript of Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring...
![Page 1: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/1.jpg)
![Page 3: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/3.jpg)
![Page 4: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/4.jpg)
What are Future Skills anyway?What do we agree on?
Includes both hard and soft skillsProblem solvingInterpersonal skills
Lifelong learningFrom childhood through to adult
Changeable/modifiableA role for educatorsA role for ECEC!
![Page 5: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/5.jpg)
What are Future Skills anyway?What do we not agree on?
A name21st century skillsTransversal skillsTransferable skills Competencies…
A set of skillsMany frameworksAll include some set of social and emotional skills
![Page 6: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/6.jpg)
![Page 7: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/7.jpg)
![Page 8: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/8.jpg)
Why do we care about social and emotional skills?
Important in their own rightA core part of 21C skills frameworks Ability to function in a social worldIncreasingly important in a globalised world
SE skills cause gains in other domainsNew Australian evidence shows ES 0.3 SD for SE skills (age 5) on NAPLAN scores (age 8)https://doi.org/10.3102/0013189X19848724
![Page 9: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/9.jpg)
Reading Level 14
Level 13
Level 12
Level 11
Level 10
Level 9
Level 8
Level 7
Level 6
Level 5
Level 4
Level 3
Level 2
Level 1
Mathematics
Level 11
Level 10
Level 9
Level 8
Level 7
Level 6
Level 5
Level 4
Level 3
Level 2
Level 1
ReadingLevel 14
Level 13
Level 12
Level 11
Level 10
Level 9
Level 8
Level 7
Level 6
Level 5
Level 4
Level 3
Level 2
Level 1
Mathematics
Level 11
Level 10
Level 9
Level 8
Level 7
Level 6
Level 5
Level 4
Level 3
Level 2
Level 1
Early development Extending general skills to younger
age groups (if desired)
General abilities become increasingly
integrated into descriptions of domain-specific
proficiency
Range of descriptions for 4.2.1 Ex
Pre-reading skills Examples
Aural comprehension Identify print
Letter recognition
Pre-math skills Examples Counting
Recognise numbers Matching
Comparing
General abilities Examples
Fluid reasoning Executive function
Comprehension knowledge
Pre-reading skillsExamples
Aural comprehensionIdentify print
Letter recognition
Pre-math skillsExamplesCounting
ecognise numbersMatching
Comparing
Re
General abilities
![Page 10: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/10.jpg)
So, what are social and emotional skills?Social Interaction
The capacity to relate to and establish relationships with others
Social ResponsibilityThe capacity to function within the school environment and interact with groups of others in socially prescribed ways
Transcending Social DifficultiesSocial self-regulation and social problem-solving capacities
Other frameworks exist, eg, Collaborative for Academic, Social, and Emotional Learning (CASEL):
![Page 11: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/11.jpg)
So, what are social and emotional skills (CASEL 5)?
Self-Awarenessrecognise personal strengths and limitations
have well-grounded confidence and optimism.Self-Management
regulate emotions, thoughts, and behaviours Both self regulation and goal setting
Social Awarenessperspective taking and empathyunderstand social and ethical norms
![Page 12: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/12.jpg)
So, what are social and emotional skills (CASEL 5)?
Relationship Skillsmaintain healthy relationships communicating clearly, listening actively, cooperating, resisting inappropriate social pressure, negotiating conflict constructively
Responsible Decision-Making SkillsMaking choices based on considerations of ethical standards, safety, social norms, consequences, and the well-being of self and others.
![Page 13: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/13.jpg)
![Page 14: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/14.jpg)
ECEC and SE skillsPlay is essential to stimulate and integrate a wide range of children’s intellectual, physical, social and creative abilities. (VEYLDF)
OUTCOME 1: IDENTITY …feeling included and secure in the social settings that are part of everyday life.
OUTCOME 2: CONNECTED WITH AND CONTRIBUTE TO THEIR WORLDbeing in groups and contributing to family and social life
OUTCOME 3: STRONG SENSE OF WELLBEINGdevelop a range of social skills and dispositions.
OUTCOME 4: CONFIDENT AND INVOLVED LEARNERShttp://bit.ly/outcome4contribute positively and effectively to other children’s learning.
OUTCOME 5: CHILDREN ARE EFFECTIVE COMMUNICATORSinnately social, creative and motivated to exchange ideas, thoughts, questions and feelings.
![Page 15: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/15.jpg)
![Page 16: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/16.jpg)
Classroom Assessment Scoring SystemCLASS is A measure of process quality
Teaching through interactionsApplicable in multiple contextsNot reliant on particular curriculum or content
Emotional Support
1. Positive Climate2. Negative Climate3. Teacher Sensitivity4. Regard for Child
Perspectives
Instructional Support
8. Concept Development
9. Quality of Feedback
10. Language Modeling
Classroom Organization
5. Behavior Management
6. Productivity7. Instructional
Learning Formats
![Page 17: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/17.jpg)
1 2 3 4 5 6 7Classroom Assessment Scoring System (3 domains)
Emotional Support Domain Classroom Organization DomainInstructional Support Domain Average CLASS Score - 3 Domains
![Page 18: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/18.jpg)
Describing children’s SE learning How can educators measure SE skills and ability?See http://bit.ly/outcome4Example measures
SDQ, SSIS, MELQO, Early ABLESsee also EEF database http://bit.ly/ref_ec_measures
![Page 19: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/19.jpg)
SDQThree similar forms
2-4 years (parent- and educator-rated)4-10 years (parent- and educator-rated)11-17 years (self-rated)
3-5 dimensionsInternalizing (emotional symptoms + peer relationship problems) 5 + 5 itemsExternalizing (conduct/behavioural problems + hyperactivity) 5 + 5 itemsProsocial behaviour 5 items
![Page 20: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/20.jpg)
SDQ
![Page 21: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/21.jpg)
SDQ
Considerate of other people's feelings □ □ □Restless, overactive, cannot stay still for long □ □ □Often complains of headaches, stomach-aches or sickness □ □ □Shares readily with other children, for example toys, treats, pencils □ □ □Often loses temper □ □ □Rather solitary, prefers to play alone □ □ □Generally well behaved, usually does what adults request □ □ □Many worries or often seems worried □ □ □H l f l if i h f li ill □ □ □
NotTrue
SomewhatTrue
CertainlyTrue
![Page 22: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/22.jpg)
SSISCurrent version: SSIS SEL Screening and Progress Monitoring ScalesSingle, multi-rater version
4-14 years (parent- and educator-rated)5 SE dimensions + 1 motivation + 2 academic competence
CASEL 5Self-Awareness (8 items)Self-Management (15 items)Social Awareness (7 items)Relationship Skills (13 items)Responsible Decision-Making Skills (8 items)
![Page 23: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/23.jpg)
SSIS SELSelf-Management
Uses appropriate language when upsetCompletes tasks without bothering others.Follows your directions. Says when there is a problem.Has temper tantrums.Follows classroom rules. Responds appropriately when pushed or hit. …
![Page 24: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/24.jpg)
SSIS SELSocial Awareness
Tries to comfort others. Feels bad when others are sad. Shows kindness to others when they are upset. Forgives others. Is nice to others when they are feeling bad. Shows concern for others. Stands up for others who are treated unfairly.
![Page 25: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/25.jpg)
MELQOSingle, direct observation version
3-6 years Open source tool developed to support SDG 4.22 SE dimensions + literacy, numeracy, executive function
Perspective-Taking/Empathy (3 items)Understanding Feelings (2 items)
![Page 26: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/26.jpg)
MELQO
![Page 27: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/27.jpg)
MELQO
![Page 28: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/28.jpg)
MELQO
don’
t kno
w/ n
o
Now I’m going to ask you another
![Page 29: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/29.jpg)
Early ABLESSingle form
2-5 years (educator-rated)Specifically for children with additional needsValidation data up to 18 years
Validated and scaled using contemporary measurement theory (e.g., IRT)1 SE dimension + speaking and listening, and reading
![Page 30: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/30.jpg)
Early ABLESExample item from earlier study
![Page 31: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/31.jpg)
Study 1UNICEF Bogor
![Page 32: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/32.jpg)
Study 1Good fit to 1PL partial credit modelReliability lower than in other published studies (>0.7)
Internalizing 0.638, Externalising 0.610, Prosocial 0.456 Reliability != reliability (PV reliability vs alpha)
Internalizing and Externalising highly correlated 0.895
Prosocial negatively correlated with Internalizing (-0.230) and Externalising (-0.388)Adequate psychometric properties
![Page 33: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/33.jpg)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------- IInternalizing Externalizing PProsocial ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 3 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 2 | | |10.2 | | | |10.1 | | | | | | | |18.2 | | | |17.2 | | | | | | | |9.2 17.1 | | | | | 1 | | |7.2 18.1 | | | |25.2 | | | | | | | | | | | |21.2 | | | |9.1 15.2 23.2 | | | | | | | |7.1 8.2 12.2 16.2 | | | |20.2 21.1 22.2 | 0 | | |2.2 3.2 19.2 25.1 | | | |1.2 12.1 15.1 16.1 24.2 | | | |5.2 14.2 | | | |2.1 8.1 13.2 20.1 23.1 | | |19.1 22.1 | | | |3.1 5.1 6.2 24.1 | | | |1.1 4.2 11.2 | | | |14.1 | -1 | | |4.1 13.1 | | | |6.1 | | | |11.1 | | | | | | | | | | | | | | | | | | | | | -2 | | | |======================================================================================================================================================================Each ' ' represents 5.5 cases
![Page 34: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/34.jpg)
Study 2Unnamed study
![Page 35: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/35.jpg)
Growth?
500
550
600
650
0.0 0.5 1.0 1.5 2.0 2.5
Time
Soc
ial a
nd E
mot
iona
l
500
550
600
650
0.0 0.5 1.0 1.5 2.0 2.5
Time
Mat
hs
![Page 36: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/36.jpg)
Growth?
350
360
370
380
390
400
410
420
430
440
450
460
470
480
490
500
510
520
530
540
24 36 48 60 72 84 96 108120132144156168180192204Age in Months
W s
core
WJ-III TestApplied Problems
Brief Intellectual Ability
Concept Formation
Rapid Picture Naming
Sound Awareness
Understanding Directions
Verbal Comprehension
Visual Auditory Learning
Visual Matching
![Page 37: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/37.jpg)
What is the value added?Cross sectional (with temporal lag) shows significant effects
Social skills influence academic outcomesQuality practice influence social skills
Growth models show less-clear effectsWithin-child growth is flat by age 5 on many measures
What is the issue?Social skills strongly correlated with other domains (e.g., selection may play a role in cross sectional designs)Measures are inadequate to measure growth over medium termMeasures are not scaled using contemporary measurement theory (e.g., using sum scores instead)
![Page 38: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/38.jpg)
Are we measuring the right things?SE skills appear through ECEC and School curriculums We have some measures but they don’t capture the higher-order abilities- A focus on managing stress, controlling impulses,
internalising behaviours like withdrawing.- But what about empathy, working in diverse
communities, negotiation, resolving conflict, goal setting and decision making?
- E.g., is self awareness in SSIS really capturing empathy and perspective taking?
- Can these things be measured through checklist about frequency of behaviours (never, seldom…)?
![Page 39: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/39.jpg)
So what?Nearly all children attend ECEC
3 year-old K roll outPublic investment in ECEC is high
For 0-2-year-olds, private investment is also highCosts of ECEC are increasing (HILDA: childcare costs up 145% in since 2002,)
Need an effective and fair systemGive all children the best start in lifeQuantify the value-add of the ECEC sector
Identify examples of best practice Showcase the expertise of educatorsFacilitate clinical practice
![Page 40: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/40.jpg)
![Page 41: Measuring social and emotionalmwmccain.ca/_media/uploads/presentations/measuring... · Measuring social and emotional skills to support children’s growth Dan Cloney Dan.Cloney@acer.org](https://reader036.fdocuments.net/reader036/viewer/2022080507/5f7a7e560e72e32af326aa9e/html5/thumbnails/41.jpg)
Key linksMapping VEYLDF to F-10 curriculum
http://bit.ly/veyldf_f10Framework for 21C skills
http://www.lsce-mena.orgRelationship between social skills and academic achievement
https://doi.org/10.3102%2F0013189X19848724Assessing children as confident and involved learners
http://bit.ly/outcome4ACER LPs
http://bit.ly/acer_lpeMELQO
http://ecdmeasure.orgEEF database
http://bit.ly/ref_ec_measures