Crowdsourced Conformance

Testing via Remote

Sandboxing

Joe Gibbs Politz

http://www.batchgeo.com

“Remote Sandboxing?”

1600 times?

grade.rktStudents' interpreters

Students' interpreters grade.rkt

#lang racket/base

(require 'interp.rkt')

(command-line“--test” test-dir(run-tests interp test-dir))

#lang racket/base

(require 'interp.rkt')

(command-line“--test” test-dir(run-tests interp test-dir) “--report”(sign (run-tests interp our-tests) “sooper-seekret”))

#lang racket/base

(define interp (dynamic-require “interp.rkt” 'interp))

(command-line“--test” test-dir(run-tests interp test-dir) “--report”(sign (run-tests interp our-tests) “sooper-seekret”))

000101010001010111000111010111

000101010001010111000111010111

000101010001010111000111010111

000101010001010111000111010111

Students' interpreters grade.bin

Remote Sandboxing

“Crowdsouced Conformance Testing”?

correct-interp

for-update-after-test

plus-coerces-strings

arity-before-evaluate

.

.

.

• tests/• loop-tests/

• for-test1.psl• for-test2.psl

• operator-tests/• subtract-test1.psl• subtract-test2.psl• add-test1.psl

• function-tests/• apply1.psl• apply2.psl• closure1.psl• closure2.psl

discard-if-effects

✓✓✓

ParselTongue Testing Assignment

> 10,000 tests from 250 people

That's as many tests as JavaScript'sconformance suite.

Crowdsourced Conformance Testing

Lessons

● Racket's dynamic code support is awesome, and Racket's build tools for distribution just work

● Value proposition for teaching PL online – get crowdsourced tools and tests