Testing for Educational Gaming and Educational Gaming for Testing
description
Transcript of Testing for Educational Gaming and Educational Gaming for Testing
Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)
Microsoft Research Connections (Judith Bishop)
Testing for Educational Gaming andEducational Gaming for Testing
Tao XieNorth Carolina State University
contact: [email protected]
Testing Educational Gaming
Dynamic Symbolic Execution (Pex)Parameterized Unit Testing
secret
Pex for Fun (Pex4Fun)
Training
Support
http://research.microsoft.com/pex/ http://pex4fun.com
Agenda
BackgroundPex, Dynamic Symbolic Execution,Parameterized Unit Testing
Writing Code in a Browser
Coding Duels
Educational Platform
Conclusion
PEX, DYNAMIC SYMBOLIC EXECUTION,PARAMETERIZED UNIT TESTING
Background
http://research.microsoft.com/pex/
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Input
null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Ta==null
Input
null
Execute&Monitor
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Ta==null
Constraints to solve
a!=null
Input
null
Execute&Monitor
Choose next path
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Ta==null
Constraints to solve
a!=null
Input
null{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0F
TFa==null
Constraints to solve
a!=null
Input
null{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0F
TFa==null
Constraints to solve
a!=null a!=null &&a.Length>0
Input
null{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
Constraints to solve
a!=null a!=null &&a.Length>0
Input
null{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
Constraints to solve
a!=null a!=null &&a.Length>0
Input
null{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
Constraints to solve
a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890
Input
null{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
Constraints to solve
a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890
Input
null{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
T
Constraints to solve
a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890
Input
null{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
T
Constraints to solve
a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890
Input
null{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890
Done: There is no path left.
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
for any list,for any item,
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
… adding 1 item increases Count by 1
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Parameterized Unit Test = Unit Test with Parameters
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Parameterized Unit Test = Unit Test with Parameters
Separation of concernsSpecification of behaviorData to achieve coverage
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Parameterized Unit Testing in PracticeGetting Popular
Parameterized Unit Tests (PUTs) commonly supported by various test frameworks
• .NET: Supported by .NET test frameworks• Java: Supported by JUnit 4.X
Generating test inputs for PUTs supported by tools• .NET: Supported by Microsoft Research Pex• Java: Supported by Agitar AgitarOne
(http://www.agitar.com/)
Limitations
16K characters of code (single file / editor window)*
Single-threaded only
No environment interactions*
No non-determinism*
Try to avoid floating-point computations
Max. 20s total exploration time*
* Limitation only applies to pex4fun.com, but not standalone Pex tool
Dynamic Symbolic ExecutionSummary
“Ask Pex” sends code to cloud
Code is compiled and analyzed in cloud
Dynamic Symbolic Execution automatically finds relevant interesting test inputs that achieve high code coverage
Results are shown in browser
WRITING CODEIN A BROWSER
Code Auto-Completion
Auto-Completion
Writing Code in BrowserSummary
Code is compiled in cloud
Code is executed in cloud
Auto-completion via cloud/Javascript in browser
CODING DUELSFun and Engaging Serious Game – Win by Writing Code
Coding Duels
1,168,577 clicked 'Ask Pex!'
Coding Duels
Pex computes “semantic diff” in cloudcode written in browser vs.secret reference implementationYou win when Pex finds no differences
secret
Behind the Scene of Pex for Fun
Secret Implementation
class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}
Player Implementation
class Player { public static int Puzzle(int x) { return x; }}
class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}
behaviorSecret Impl == Player Impl
18
Behind the Scene of Pex for Fun
Secret Implementation
class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}
Player Implementation
class Player { public static int Puzzle(int x) { return x; }}
class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}
behaviorSecret Impl == Player Impl
18
Behind the Scene of Pex for Fun
Secret Implementation
class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}
Player Implementation
class Player { public static int Puzzle(int x) { return x; }}
class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}
behaviorSecret Impl == Player Impl
18
Coding DuelsFun and Engaging
Iterative gameplayAdaptivePersonalizedNo cheatingClear winning criterion
Example User Feedback
“It really got me *excited*. The part that got me most is about spreading interest in teaching CS: I do think that it’s REALLY great for teaching | learning!”
“I used to love the first person shooters and the satisfaction of blowing away a whole team of Noobies playing Rainbow Six, but this is far more fun.”
“I’m afraid I’ll have to constrain myself to spend just an hour or so a day on this really exciting stuff, as I’m really stuffed with work.”
Released since 2010
X
Coding DuelsSummary
Coding Duel: write function that matches specification given by another implementation
Semantic equivalence checked in cloud
Student’s tasks:
Infer likely intended behavior
Express behavior as code
http://pex4fun.com/CreatingAndPublishingCodingDuels http://pex4fun.com/TipsAndTricksForGreatDuels
SOCIAL EXPERIENCEConnecting Students – Competitive yet Self-Paced Environment
Social Experience
Social ExperienceSummary
CommunityHigh score lists, leaderboardLive feed
http://pex4fun.com/Community.aspxhttp://pex4fun.com/Livefeed.aspx
PEX4FUN, AN EDUCATIONAL PLATFORMPages – Courses – Automated Assessments
Teaching and Learning
Coding Duels for Automatic Grading @Grad Software Engineering Course
http://pexforfun.com/gradsofteng
Coding Duels for Training Testingpublic static string Puzzle(int[] elems, int capacity, int elem) { if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1))) return "Assumption Violation!"; Stack s= new Stack(capacity); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); int origSize = s.GetNumOfElements(); //Please fill in below test scenario on the s stack
//The lines below include assertions to assert the program behavior PexAssert.IsTrue(s.GetNumOfElements() == origSize + 1); PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty()); PexAssert.IsTrue(s.IsMember(elem)); return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “ + s.IsMember(elem).ToString() + "; " + s.IsEmpty(); }
Set up a stack with some elements
Cache values used in assertions
Usage Scenarios of Pex4Fun• Massive Open Online Courses (MOOC): Challenges– Grading, addressed by Pex4Fun– Cheating [Open Challenge]
• Course assignments (students/professionals)– E.g., intro programming, software engineering
• Student/professional competitions– E.g., coding-duel competition at ICSE 2011
• Assessment of testing/programming/problem solving skills for job applicants– Not just final results of problem solving but also process!
Teaching and LearningSummary
How to become a teacher:Sign in, choose nicknameSend us email with nickname
Teachers canCreate coursesReuse existing or author existing pagesGet automated assessments of studentsSee students code
http://pex4fun.com/TeachingWithPexForFun
Outlook
Analyzing code submissionsHow to guide students?Detect plagiarismFurther ranking
Support for more languages
More ReadingNikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith BishopTeaching and Learning Programming and Software Engineering via Interactive GamingIn Proceedings of the 35th International Conference on Software Engineering (ICSE 2013), Software Engineering Education (SEE), San Francisco, CA, May 2013.http://people.engr.ncsu.edu/txie/publications/icse13see-pex4fun.pdf
ConclusionPexTest generation for .NETbased on dynamic symbolic executionEnables Parameterized Unit Testing
Pex4Fun on the web.NET: C#, Visual Basic, F#Execution, code analysis with PexCoding DuelsGreat for teaching, learning
http://pex4fun.comhttp://research.microsoft.com/pex
Testing Educational Gaming
secret
Training
Support
http://research.microsoft.com/pex/ http://pex4fun.com
Q & AThank you!
contact: [email protected]
Acknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603, a Microsoft Research SEIF Award, and a Microsoft Research Award.