How to prove your edtech product works
-
Upload
joseph-wilson -
Category
Education
-
view
98 -
download
3
Transcript of How to prove your edtech product works
![Page 1: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/1.jpg)
Efficacyhow to prove that your edtech product works
@joeatmarsImpact8 2015
![Page 2: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/2.jpg)
![Page 3: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/3.jpg)
Source: GSV Advisors
![Page 4: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/4.jpg)
![Page 5: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/5.jpg)
![Page 6: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/6.jpg)
Tradition: focus on inputs Efficacy: focus on outputs
![Page 7: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/7.jpg)
…focused on teaching …focused on learning
…focused on inputs …focused on outputs
…tradition-driven …data-driven
…teacher as content authority …teacher as content guide
…teacher responsible for cause > effect …teacher iterates towards effect
…teacher is the sole proprietor of a private practice
…teacher is a member of a Professional Learning Community
…”best practices” …context sensitive
Educational researchEducational tradition
![Page 8: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/8.jpg)
New methods of measuring impact (for teachers and researchers)
1. Action-based research2. Developmental evaluation3. Design-based research4. Control groups, A/B tests5. SAMR
![Page 9: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/9.jpg)
![Page 10: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/10.jpg)
In medical terms, “efficacy” refers to the ability of a product to provide a beneficial effect.
![Page 11: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/11.jpg)
![Page 12: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/12.jpg)
![Page 13: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/13.jpg)
Impact Frameworks
1. IRIS 2. Pearson3. Learning Canvas
![Page 14: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/14.jpg)
![Page 15: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/15.jpg)
![Page 16: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/16.jpg)
• The learning benefit of your product is notself-evident
• Social metrics are often proxies for learning outcomes
• Social metrics should be used to make hypotheses about learning outcomes
From social metrics to learning metrics
![Page 17: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/17.jpg)
Example: you have a product that teaches math online with a parent-teacher bridge component. If you can increase parent-teacher communication by 20 min/week you can hypothesize that student achievement will increase based on “Parental Involvement and Student Achievement: A Meta-Analysis” (Jeynes, 2005)
Your social metric + existing learning research = learning hypothesis
From social metrics to learning metrics
![Page 19: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/19.jpg)
![Page 20: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/20.jpg)
![Page 21: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/21.jpg)
Impact Frameworks
1. IRIS 2. Pearson3. Learning Canvas
![Page 22: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/22.jpg)
![Page 23: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/23.jpg)
![Page 24: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/24.jpg)
![Page 25: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/25.jpg)
![Page 26: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/26.jpg)
From Good Intentions to Real Impact
5 Levels of Efficacy Method
Level 1: Why might the product have an impact onlearning?
Hypothesis
Level 2: Data shows some change amongst users Anecdotal evidence; surveys
Level 3: Data shows the change is because of the product
Control Groups
Level 4: Independent evaluation of impact/explanation of learning impact
Independent researchers
Level 5: Ability to replicate/scale the learningimpact
Replication studies
![Page 27: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/27.jpg)
Impact Frameworks
1. IRIS 2. Pearson3. Learning Canvas
![Page 28: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/28.jpg)
LEARNING GAINS
LEARNING PROPOSITION
END USER
TECHNOLOGY
THEORY
METRICS
LEARNING OBSTACLES
Who are the students who use the product?
What learning gains do you expect to see?
What hi-tech/low-tech do the students use?
What learning theory predicts these learning gains?
What metrics will you collect to prove the learning gain?
14 2
3
8
6
5
7
9
SCHEDULEHow often do the students interact with the product?
PARTNERS
What partners do you need to execute?
- Age- Grade- Gender- Special needs- Socioeconomic- Culture- Hand-outs
- iPad- Lab equipment- Sports equipment
- Once/week- 30 min/day- On demand- Beginning of
each course- Home use
- Increase in marks- 21st C skills- Complex thinking- EQAO scores
- Scientific studies- Theory search- Existing products- Benchmarks
- Marks- Self-assessment- Engagement- Attendance
- Iterate on hypothesis, unanticipated metrics
- Teachers- Researchers- Lead students- Principals- Superintendents
(After implementation) What learning gains resulted from the product’s use?
- Final metrics, confirmation of hypothesis
(After implementation) Did students get stuck anywhere? Were there any unpredictable negative side effects to the product’s use?
![Page 29: How to prove your edtech product works](https://reader033.fdocuments.net/reader033/viewer/2022042817/55a5ea981a28abb0128b48bb/html5/thumbnails/29.jpg)
LEARNING GAINS
LEARNING PROPOSITION
END USER
TECHNOLOGY
THEORY
METRICS
LEARNING OBSTACLES
SCHEDULEPARTNERS