Building Whole Applications Using Only Programming-by-

Demonstration

Richard McDaniel

January 14, 1999

2

Purpose

Allow nonprogrammers to build applications with original, dynamic behavior Use programming-by-demonstration No written language at all - programs

created solely through demonstration Implement in system called Gamut

3

State of the Art

Full-fledged languages and libraries: Visual C++, Java

Simplified languages with syntax-supporting editor: Authorware, Agentsheets

Tools with macro languages: Director / LINGO, Excel / Visual Basic

4

Outline

Introduction Interaction Techniques Video Inferencing User Study Conclusion

5

Domain

“Board Games” Has background “board” “Pieces” above board show state Pieces follow rule-like behavior Autonomous behavior controlled by timers

Examples Tic-Tac-Toe, Pacman, Reader Rabbit

6

Example: Pacman

7

Innovations in Gamut

Interaction Techniques Allows developers to specify important

details Inferencing

Infer conditions from sources unaffected by the behavior

Infer chains of expressions

8

Inferencing - Conditions

Relationship can be formed between arbitrary objects

– Relating a toggle switch to decide which way to move

Run Away

?

9

GO

Expression Chain

Expressions form chain of relationships– In Monopoly, the position to move a piece is, “the square

which is the sum of the dice’s number of squares from the original position”

Red’s Turn

10

Interaction Techniques

Nudges Do Something, Stop That Hint highlighting

Guide Objects Temporal Ghosts Cards and Decks

11

Nudges

12

Negative Examples

Standard technique for demonstrating examples: Macro Recorder Requires special controls to make a

negative example Negative examples in Gamut provided

by Stop That No special set-up required

13

Guides, Ghosts, and Hints

14

Cards and Decks

15

Video

16

Stages of Inferencing

TranslationMatchingActions

CodeGeneration

17

Translation

Read events from undo history list Convert events to actions Eliminate redundant and

inconsequential events

18

Matching Actions

Matches actions in trace to original behavior

– Unordered– Typically not many (three or four)

Considerations– Degree to which actions match– Priority of actions in original behavior– Actions which create new objects

19

Recursive Difference

Propagates changes to descriptions Each description contains heuristics for

comparing values Builds an And/Or tree result

20

Code Generation

Resolves each value difference by creating a new description Search for previously defined description Use heuristics to generate description from

highlighted objects Assemble values into a table and use

Decision Tree learning

21

Decision Trees

Created at places where Gamut chooses between different behaviors

Generate attributes using heuristics and hint highlighting

Build table of examples that relate behaviors to attributes

Is Eaten

RunHome

Is Blue

ChasePacman

RunAway

Y N

Y N

Condition For Pacman Monster

22

User Study

Goal Determine if nonprogrammers can use

Gamut Perform an informal study

Participants Nonprogrammers, CMU community

Location HCI Institute’s usability lab

23

Tasks

Safari

G-bert

Pawn Race

24

Expertise with Techniques

techniques P1 P2 P3 P4

Do Something X X X

Stop That X X

Highlights Ghosts X

Guide Objects X X

Cards and Decks X X

Player MouseIcons

X

25

Problems

Reluctance to create guide objects Needed explicit instructions in tasks Once told, participants would create

appropriate objects Highlighting inappropriate objects

Some objects seem too obvious to highlight, like ghosts

26

Future Work

Apply to other domains Feedback for just inferred relationships Display for language Higher-level inferencing

27

Conclusion

Builds entire applications without a written programming language

Infers a broader range of behavior than prior systems

Is usable by nonprogrammers