Using Large-Scale Argumentation to Enable Open Innovation

Mark Klein

Open Innovation: A Powerful Emerging CI Tool

Open innovation can be a good match for this:

A Promising Approach for Complex Challenges

Idea synergyThe long tail (marginalsMany hands

Complex societal problems inherently involve many stakeholders and areas of expertise, and require innovative out-of-the-box solutions

Many eyesWisdom of the crowdsVolunteer ethic

“No matter who you are, most of the smartest people work for someone else” (Joy’s Law)

Organization Contributions Contributors

Open for questions 70K 100K

Google 10^100 150K 100K

IBM Idea Jam 46K 150K

Dell Ideastorm 21K

Challenges with Open Innovation at Scale

“Upon this gifted age, in its dark hour,Rains from the sky a meteoric shower Of facts…they lie unquestioned, uncombined.Wisdom enough to leech us of our illIs daily spun; but there exists no loomTo weave it into fabric…”

Edna St Vincent Millay. Huntsman, What Quarry? (1939)

Challenges with Open Innovation at Scale• High harvesting costs:

• redundancy, due to ignorance or manipulation• signal-to-noise: only ~10% of ideas are worth consideration

• Rating dysfunctions• With large corpuses, users tend to rate only the highest-rated

ideas, creating feedback loops => rating lock

• Shallow ideas: lots of rudimentary ideas• solo ideas w/o collaborative refinement• tough to track and reward IP – “I didn’t see he had the same

idea”• Unsystematic

• No mechanism to ensure coverage of critical issues/ideas

• Shallow evaluations:• quick personal impressions, not customer criteria-based • little/no analysis and fact-checking

• Google project10tothe100 needed 3,000 volunteers & 9 months to filter/sort ideas

• Open for Questions was shut down due to overload

• IBM flew 100 executives to New York for a week to filter Web Jam ideas

• Process opaqueness• “It’s tough to steer when you can’t see where you’re going”

Planeta.com (5/1/08) had a 13-page discussion on carbon offsetting pros and cons

Argument Mapping Can Help!

No redundancy: can ensure each unique point appears just once

Better coverage: easier to find gaps that need to be filled Deeper ideas: crowd builds issue/idea trees instead of

standalones Credit tracking: non-redundancy makes idea precedence

clear

Argument Mapping Can Help!

Open Issues: Argument Mapping at Scale

• Structure• How do we keep large growing maps well-structured?

• Content• How do we foster good (creative, detailed) solution ideas?• How do we rapidly prune the top 10% of ideas?• How do we ensure good (well-founded, complete) evaluations?

• Process• How do authors know where they can best contribute, in large

maps?• How do managers know what process problems need

intervention?

• Outcome• How do customers know which content is ready to ‘harvest’?

Overview of Approach

High-Speed Idea Pruning: Bag of Stars

0 0.2 0.4 0.6 0.8 1 1.20

0.2

0.4

0.6

0.8

1

1.2

LikertBOSBOL

false positive rate

tru

e p

osit

ive

rate

A key trick: assign posts to users in a way that maximizes value e.g. • to posts whose value is uncertain• to calibrate ratings across multiple rater pools• ...

Unbundle – break your thoughts into points that each contain just one issue, idea, or argument.

Locate – search the argument map to see where your point(s) belongEnter – If it’s a new point, create a new post, else refine existing post.

The live-and-let-live rule: only edit a post to strengthen it

The honest broker rule: remain strictly content-neutral

Certify well-structured posts - a coaching role

Rate to highlight worthy posts

Only certified posts can be viewed by readers

Structure Moderators

Idea: reduce the energy-intensity of products and services

Pro: typically has positive ROI (addresses: low-cost)

Issue: what actions can help avoid global warming?

criterion: low-costcriterion: effectivecriterion: few negative side-effects

Extended IBIS to Support Design Problems

• Added criteria to IBIS so we can track evaluation completeness:• Augment issues with list of criteria successful solutions should

satisfy• Annotate arguments with the criteria they address

Recombinant Design•Map defines a multi-dimensional

design space – Issues = design dimensions – Complete solutions are packages

(combinations) of ideas

•Benefits– Enables systematic exploration of

design space– Fosters creativity by recombination

•Challenges– Creates very large search spaces due to

exponential combinatorics

Nonlinear Negotiation• How find pareto-optimal solution packages in very large search spaces with

self-interested stakeholders?• Standard (linear) negotiation protocols fare poorly in such contexts• New protocols, often counter-intuitive, are needed

U(A

2)

Contracts

U(A

1)

optimum

optimum

selected best

Metrics & Attention Mediation• A free open-source web service that deliberation platforms can call to

calculate metrics for their maps• A growing library of metrics, defined based on systematic analysis of

deliberation needs and challenges• A language for defining “alerts” that notify participants of metrics signal

that they need to know about

Identifying Metrics:process-goal-exception analysis

Implementing Metrics

A little semantics, in abundance, allows us to define metrics of unprecedented power compared to those for social media

Data mining (dimensionality reduction , hierarchical clustering ...)

Belief propagation (e.g. Bayesian)

Network topology (includes social network) analysis

Statistical analysis

Graph-based query languages (e.g. SPARQL)

Time series analysis

Maturity Metric: Network analysis

versus

Groupthink Metric: Gini coefficient

Groundedness Metric: belief propagation

Balkanization Metric: SVD & Clustering

Apply dimensionality reduction to idea and argument ratings => ”bias space”

• each point represents a user• each axis represents a ”bias" (i.e. set of post with correlated ratings)

Balkanization = distinct clusters in bias space

Attention Mediation: A Keystone Capability

• Guide ideation towards gaps (maturity)• Guide ideation towards pareto front (negotiation)• Guide evaluation to where it is most needed (efficiency)• Guide managers to areas that need intervention• Guide customers to the good stuff

Harnessing the Collective Intelligence of the Research Community

http://deliberatorium.mit.edu/brainstorm-signup