Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling

Symmetric Multi-modal Intelligent Interactive Development

Environment (SMIIDE) Seedling

Update as of

September 21, 2009

BAE AIT: Greg Sullivan, Basil Krikeles, Mike Cook

MIT CSAIL: Randy Davis, Howie Shrobe, James Oleinik, Kyle Mill

September 21, 2009 SMIIDEling Overview 22Distribution authorized to U.S. Government Agencies only

Symmetric Multimodal Interactive Intelligent Development Environments

•Dramatic reduction in time and cost of constructing software designs and related engineering artifacts

•Production of higher quality artifacts through interactive design.

•Metrics to be refined during seedling.

Multi-modal User Interfaces can now integrate sketch, gesture,& natural language to capture design, rationale more naturally.

Mixed-Initiative User Interfaces enable symmetric collaboration between human and computer-based designers.

Adaptive, Domain-Specific Model-Based Software Engineering tools enable flexible collaboration at a human level of abstraction.

Symmetric multimodal interaction brings intelligence to model-based development

Multi-modal Mixed-initiative User Interface

• Supports sketching, gestures, natural language.

• System can query human for clarification, suggestion, etc.

• System can take initiative in design and implementation, freeing the human from detailed specification of lower-level details

Adaptive Domain Specific Model-Based Software Engineering tools

• Support evolution of domain “vocabulary”

• Support diagramming using symbology specific to a domain.

ASSUMPTIONS AND LIMITATIONS:

• Anticipated productivity gains can be demonstrated on well-defined domains.

• Current natural language technology is sufficient.

• Current Integrated Development Environments (IDEs) are developer-centric, detail-intensive, and error-prone

•Modality is essentially paper-based and human developer-centric

•IDEs support limited consistency checking and debugging

• Demonstrate program feasibility and develop program structure

•Prototype combination of symmetric multimodal HCI with model-based code generation technology

•Determine feasibility of program leading to mixed-initiative high-productivity environments

•Identify technology gaps and refine DARPA-hard program challenges.

•Demonstrate on a DoD-relevant domain

•Survey and position with respect to previous research

QU

AN

TIT

ATIV

E IM

PA

CT

EN

D-O

F-S

EED

LIN

G G

OA

L

STA

TU

S Q

UO

NEW

IN

SIG

HTS

Code

Com

pile

Test

System Complexity

Co

st

(Tim

e)

Savings

With MMIIDE

Current IDEs

Does this process need access to the

database?

Let’s create a data interface component

September 21, 2009 SMIIDEling Overview 3

Some seedling goals (from Jim’s slides at August meeting)

• Arrive at a focused set of technical challenges that form the basis of a new program

• The purpose of the seedling is to create a new program initiative brief

• A “Scientific American” statement of the technical goals of the program

• A clear connection to compelling operational need• A convincing case for how this can be achieved• The nature of the technical challenges, and which

focused challenges will form the basis for a program?• How far can we reach in terms of operational capability

in the scope of a program?• What is the potential operational payoff?• What options for demonstrating the concept and

measuring success?


Outline

1. The Heilmeyer Questions2. A “Scientific American” statement3. Compelling operational need4. Research Topics for proposed program

1. Quick review of topics from August meeting.2. “Lots of Little Languages”.3. Rationale Capture via Natural Interaction4. Related research (“Why now?”)

5. Sample Scenarios1. Geospatial reasoning in Deep Green2. Building a wiki for after-action reports


1. The Heilmeyers…1. What is the problem, why is it hard?

Software design, development, and maintenance. Low level, error-prone, complex, hard to verify. Slow to adapt to mission needs.

2. How is it solved today? Code-level “forward engineering”. That is, human-intensive high-to-low level specification with

computer “automaton” support for detail tracking (e.g. parsing) and high-to-low level translation (e.g. compilation).

3. What is the new technical idea; why can we succeed now? Natural interaction and DSML interpretation combine to enable mixed initiative interaction, with

useful initiative from computer, augmented by captured rationale. Drill down and machine learning enable both computer and human input at all levels of abstraction.

4. What is the impact if successful? Revolutionary increase in productivity for creating software-intensive systems. Enables fast

development and deployment of latest technology, rather than traditional slow DoD software acquisition process.

Dramatic increase in development / adaptation allows rapid response to mission needs.

5. How will the program be organized?

6. How will intermediate results be generated?

7. How will you measure progress?

8. What will it cost?


A “Scientific American” statement of the technical goals of the program

• Every day, countless conversations take place between software engineers in front of white boards. The engineers take turns grabbing markers, and talk while drawing. In response to questions from each other, the engineers elaborate or erase parts of their diagrams. The interactions are highly verbal, and the drawings include text, traditional software diagrams, and ad-hoc notations made up on the spot.

• Now, what if one of those engineers was a computer?

• The goal of the SMIIDE program is investigate whether the combination of several cutting edge research areas can enable computer-based agents to become active participants in the software design, development, and maintenance process.

• The computer should be able to ask relevant questions, suggest useful approaches and designs, and thereby substantially contribute to the implementation of the eventual system.

Research Areas: Sketch recognition, Natural Language, Rationale Capture, Domain Specific Modeling, Dialogue management.


Compelling Operational Need

• The problem: in a fast-changing world, it’s easy to miss the mission window

• Response: Rapidly tailor systems to mission needs– Can leverage rationale captured in natural modalities.

• Why were certain decisions made?• What were assumptions?

• Increased speed of development while maintaining correctness

• Decreased costsWhat current operational needs are addressed by a dramatic increase in speed and flexibility?


Research Topics identified during August meeting

• Grounding complex abstract diagrams in sufficient meaning to support the design task;

• Determining how complex specifications can be segmented in ways that are functionally meaningful for reasoning and dialog;

• Determining what kind of human-computer interactions are needed at any given moment and mediating that interaction effectively;

• Determining the HCI principles that lead to productivity enhancements not attainable with model-based software engineering tools alone (including the advanced versions produced in the current DMT-SWP program).

• Is there a body of kn about MBSE applicable across domains?– Dev of DSMLs suggest any common body of language, lexicon,…– How wide a variety of specification forms to consider? tabular, text,

graphical

• Graphical models vs. behavioral specifications? How do we get the benefits of graphical structural models for behavioral specs?


Core Research Topic: Related Families of Overlapping DSMLs

• Benefits of DSLs:–Small language decreases

errors.–More abstraction increases

productivity, improves analysis–DSMLs capture expertise of

domain experts• How to get coverage / generality?

–Lots of DSLs–Overlaps in domains gain

coverage by transitivity.–Evolution of DSLs

• Generators / Interpreters / Transforms

–Provide overlap by translation–Often high-to-low level (where

“level” is loosely defined).–Generator logic can be hand-

generated or partially learned.

Geospatial

Geometry

Course of Action

TenneygramsUML classdiagrams

Correspondence, Transformations

Make

Java

Scala

Correspondence, Transformations

pathlocationunit


Core Research Topic: Natural Rationale Capture

• Can natural interaction make rationale capture (nearly) painless?– Why were certain decisions made?– What were assumptions?

• Can captured rationale provide the basis for “parametric design” of software?


Rationale Capture - Rationale

• Every software project involves thousands of design decisions– Most are routine (i.e., obvious to an experienced

programmer); many crucial ones are not

• Writing/modifying/debugging/ software requires constantly re-absorbing those decisions.

• This would be orders of magnitude easier if all the program authors were always there to explain how they made those decisions and why.

• Rationale capture can achieve this effect.


Rationale Capture – Technical Challenges

• Why isn’t rationale captured now?– Cost is too high (writing documentation is a lot of

work)– Task is aversive (writing documentation is unpleasant

work)– Task is disruptive (writing documentation interrupts

design)– The value equation is out of whack:

• The person incurring the cost is (almost) never the one who gains the benefit.

• Cost is now, benefit is in the future.

• Consequences: documentation of small but crucial decisions is deferred.

Can we change the equation by drastically lowering the cost and making it more like talking to another designer?


Core Research Topics: Sketch Understanding, Symmetric Multimodal Interaction

• Sketch understanding– grounding complex abstract diagrams in sufficient

meaning to support the design task; – structure (e.g., data structure) is easy to draw, but a

lot of software is about process, behavior• can we find ways to draw behavior that feel as natural?

• Symmetric multimodal interaction requires an intelligent observer, e.g., knowing what questions to ask and when– How task- and domain-specific is that knowledge?– Is there a body of knowledge about MBSE applicable

across domains? Or is the problem AI-complete?


Central Hypotheses

1. Natural interaction + modestly intelligent observer = drastic increase in agility and lower cost

2. Domain specificity enables intelligence of computer-based observer.

3. Evolvable and overlapping languages enables generality


How far can we reach in terms of operational capability in the scope of a program?

• What to rule out?– “Programmer’s common sense”

• Domain expertise encoded in DSML “smarts”.• Mixed initiative: human in the loop.

• System must be “smart about something” but cannot be smart about everything.


What options for demonstrating the concept and measuring success?

• Prototype and storyboards for demonstrating concept.


Metrics

• Ultimate metric: order of magnitude faster with system – allows missions / capabilities that simply weren’t available before.

• # of lines of code automated versus hand written.

• Some measure of complexity of code generated. In other words, abstraction + automation results in less quantity and lower complexity over course of task.


Program Structure

• Individual pieces, with an integrator

• Or entire systems from each team?


Related Research

• Sketch understanding– MIT (Davis), UWash (Landay), UC Riverside

(Stahovich), Illinois (Forbus)

• Dialogue management– Grosz (Harvard), Sidner (BAE)

• DSMLs– Karsai et al. (Vanderbilt), Metacase (Kelly, Tolvanen),

Gray (Alabama), Sprinkle (Arizona). – See dsmforum.org

• Natural Language• Mixed initiative


Random Thoughts

• Community aspect?– related to Framework Adoption Seedling– learning over many interactions with same

and different users.

• Related to Bootstrapped Learning?– Teaching computer to program.– Might be relevant to helping SMIIDE system

learn.

Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling

Documents

Transcript of Symmetric Multi-modal Intelligent Interactive Development Environment (SMIIDE) Seedling