The Participatory Web A medium for human-computer collaborative design of watershed

management solutions

Meghna Babbar-Sebens, Ph.D., Oregon State University

3rd CUAHSI Conference on HydroInformatics, July 17th 2015, Tuscaloosa, AL

1

Outline

1. Acknowledgements2. Introduction Need for participatory design in watershed planning Human-centered design on the Participatory Web

3. Research Goals4. Methodology5. Results Lessons in experiments with students and stakeholders

1. User learning and engagement2. User models for detecting revealed preferences3. Interactive search algorithms

6. Conclusions

2

1. ACKNOWLEDGEMENTS

Collaborators and Students Snehasis Mukhopadhyay, E Jane Luzar, Kristen Macuga, Edna Loehman,

Adriana D. Piemonti, Vidya B. Singh, Jon Eynon, Jill Hoffman, and the various student participants

Funding Agencies (NSF Award IDs ESE:1014693/ 1332385; NOAA/CPO/SARP Award ID NA14OAR4310253; ISDA Award ID A337-9-PSC-002)

Indiana Universitys FutureSystems (old name FutureGrid) (NSF Grant # 0910812)

Stakeholder participants and partners

3

Restoration of the hydrologic cycle in degraded watersheds need a network of

distributed conservation and storage practices Stakeholder engagement is

even more critical now!

acceptable choices will depend on the local site and stakeholder conditions and preferences

Image source: http://cdn.phys.org/

2. INTRODUCTION

References: Hey et al. 2004; Mitsch & Day Jr., 2006; Heisel, 2009

Filter Strips

No-till

Grassed Waterways

Wetlands

4

Participatory Methods for Engaging Stakeholders

Integrated Watershed & Water Resources Management (Schramm, G. 1980; Viessman, W. 1996; Hooper, 2007)

Shared Vision Modeling and Planning (Hamlet, A. et al. 1996; Palmer et al., 1995)

subjectivity and preferences

5

Participatory Design of Alternatives

Crowd-sourcing the design process to the community Bottom-up and democratic

problem-solving process Community building and

empowerment Emergence of a new human-

based computing paradigm, where human-computer interactions play an important role in design algorithms

References: von Ahn, 2009; Fraternali et al., 2012

6

Opportunities on the Participatory Web 2.0 ( and the Web 3.0 in the future)

The Internet provides opportunities for coordination of efforts of billions of humans Combined power of humans and computers can now be

harnessed to solve complex problems!

Consideration of issues that arise when human are included in the loop (Ipeirotis and Paritosh, 2011) Human factors Quality of human contribution Market, ethical, and legal aspects

7

The Participatory Optimization Concept

Optimization Algorithm Operations

Numerical Evaluation

Optimal Solutions

Initial Feed of Designs

Non-interactive Optimization Algorithm Approach

New Designs

(Interactive Optimization/ Human-guided search/ Human-Centered Optimization)8

User Web Interface

Feedback(Information

Gathering and

Decision Making)

Feedback(Likert Scale User Rating and UsabailityData)

Designs/Alternatives

OptimizationAlgorithm (IGAMII) Operations

Numerical Evaluation

Optimal Solutions

Initial Feed of Designs

SubjectiveEvaluation New Designs

Designs/Alternatives

Babbar-Sebens et al. (2015)

Online User Preference Models

The Participatory Optimization Concept(Interactive Optimization/ Human-guided search/ Human-Centered Search)

Interactive Optimization Algorithm Approach

9

3. RESEARCH GOALS

Represent stakeholders qualitative/unquantified knowledge (wisdom) within numerical optimization process through interaction

Participatory optimization algorithms and tools1. How do users engage with and learn from such

participatory optimization tools?2. Can we model users nonstationary and noisy feedback

provided through the interfaces to detect underlying revealed preferences?

3. How does the human-centered search process affect the design of watershed plans?

10

4. METHODOLOGYWRESTORE (http://wrestore.iupui.edu)11

WRESTOREs Client-Server Architecture

Arrows indicate data flow and workflow

HTTP

(Thin clients)

Web Interface

12

13

Participatory Optimization Introspection Loop to support Users Cognitive Learning

Participatory optimization/Human-

guided search sessions (Human or Simulated

User)

Introspection sessions (Human User)

Im sessionsHSn_m sessions

Where,n = number of iterations in optimization algorithmm = number of introspection sessions

n

m loops

Participatory optimization algorithm: IGAMII (Babbar-Sebens et al., 2015)

13

Restoring upland storage in Eagle Creek Watershed (HUC 11), Central Indiana

Conservation practices modeled using Soil and Water Assessment Tool (Neitsch et al., 2005), for a 2005-2008 simulation period.

Impact of practices estimated by comparing economic costs, peakflows, sediments, and nitrateswith the baseline scenario that does not contain any new practice.

Study site14

20 participants: Students and Stakeholders Decision variables: cover crops (0/1) and

filter strips (0-5m) in 108 sub-basins Quantitative objectives (Numerical

Evaluation): Economic Costs, Peak flow reduction, Nitrate reduction, and Sediment reduction at watershed scale

Qualitative objective (Subjective Evaluation): User Rating of alternatives based on users subjective assessment of suitability of practices in their local sub-basins

5. RESULTS

2 Loops

6

15

Q1. How do users engage with and learn from such participatory optimization tools?

a) Overall response times

Stakeholders exhibited more variability from DeJongs power learning curve in HSn_m sessions Small sample size, statistical detection of outliers, stakeholders might have a different

learning model structure

Im sessions HSn_m sessionsWhere,n = 1 to 6m = 1 to 3

- By tracking usability metrics (Tullis and Albert (2013))16

Q1. How do users engage with and learn from such participatory optimization tools?

b) Information Gathering versus Decision Making Stakeholders spent 16% more time on Information Gathering than students Stakeholders clicked 19% more in information gathering areas than students

I1 HS_1 I2 HS_2 I3

Students

Stakeholders52%

19%

29%45%

20%

35%

51%

12%

37% 46%

25%

29% 38%

5%

57%

35%

23%

42% 33%

25%

42%27%

13%60%

25%

32%

43% 32%

22%

46%

Mean percentage of response times

I1 HS_1 I2 HS_2 I3

Students

Stakeholders

39%

39%

22% 30%

51%

19%29%

30%

41%25%

56%

19%32%

40%

28%

55%30%

15%

46%

39%

15%

53%23%

24%48%

37%

15%

48%

7%

45%

Mean percentage of clicking events

Students

Stakeholders

Students

Stakeholders

17

Q1. How do users engage with and learn from such participatory optimization tools?

c) Response time vs. number of clicks per session per participant and based on trends in averages of self-reported confidence levels

y = 18.359x0.9401R = 0.6727

y = 10.023x0.8133R = 0.5581

y = 7.7771xR = 0.6645

0

50

100

150

200

250

300

0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00

Num

ber o

f Clic

ks

Time (min)

Information Gathering

y = 28.126x0.5678R = 0.6363

y = 29.703x0.6113R = 0.6488

y = -3.4978x2 + 26.858xR = 0.5815

0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00Time (min)

Decision Making

Positive trend Negative trend No trend

When users spend more time and make more mouse clicks in information gathering areas, their confidence levels are likely to increase over time

18

Q2. Can we model users nonstationary and noisy feedbacks provided through the interfaces to detect underlying revealed preferences?

Simulated Users (Avatars) To enable extensive search where individual users qualitative criteria is represented via user models

Which Machine Learning Algorithm to use?

User modeling in Environmental Problems is still in infancy! Lessons can be learned from HCI, Intelligent Interfaces, Adaptive Interfaces,

Cognitive Engineering, Intelligent Information Retrieval, Intelligent Tutoring, Expert Systems, etc.

Challenges:

High Number of input parameters

Varying size of input parameters

Limited Feedback data

Skewed training data

Online User Preference Models

19

Q2. Can we model users nonstationary and noisy feedbacks provided through the interfaces to detect underlying revealed preferences?

X axis: SDMs at end of introspection sessions1: I1 data = Epoch 12: HS_1 & I2 data = Epoch 23: HS_2 & I3 data = Epoch 34: Epoch 1 & 2 data5: Epoch 1, 2, & 3 data

Erro

r in

pre

dict

ed u

ser

ratin

g

Stakeholder SDM

Deep Learning tended to perform much better than any other machine learning algorithm we used

m loops20

Q3. How does the user-guided search affect the design of watershed plans?

10

15

20

25

PF

R (

cms)

Participant1

-2.6 -2.4 -2.2 -2 -1.8 -1.6x 107

4.5

5

5.5

x 106

Cost ($/Watershed)

NR

(kg

/Wat

ersh

ed)

PARTICIPANT 1

10

15

20

25P

FR (c

ms)

Participant2

-2.6 -2.4 -2.2 -2 -1.8 -1.6 -1.4x 107

4.5

5

5.5

x 106

Cost ($/Watershed)

NR

(kg/

Wat

ersh

ed)

PARTICIPANT 2

In Objective Space, differences and similarities can be detected in the acceptable or unacceptable alternatives found by the users

21

Q3. How does the user-guided search affect the design of watershed plans?

In Decision Space, Cover crops: Across all participants, 45% of sub-basins have low variability in

probability of cover crops (i.e. standard deviation< 0.13) Filter widths: Across all participants, 69% of the sub-basins have low variability in

modes (i.e. < 1.2 m)

Cover crops across participants

Higher average

probability across

participants

Filter Strip width across participants

Standard deviation

I like it Alternatives

Higher mode

Interface usability is important! Participants can have the same view for different

reasons. Cues are interpreted differently when they occur

Participants can have different views based on the same information different interpretations

User modeling needs to be adaptive to a humans own learning process

Putting stakeholders-in-the-loop enables identification of design features in alternatives that are satisficing from the humans perspective

6. CONCLUSIONS

Paradigm shift: Optimization based on human-computer collaboration via interaction. The participatory web provides an enabling framework. Knowledge and preferences of community included. Supports learning process of decision makers Need for collaborations across multiple disciplines

Foundation laid for further investigation in interactive and participatory design methodologies How will cognitive machines solve complex participatory design

problems for watershed groups in the future?

6. CONCLUSIONS24

Questions?

mathematical formulas alone do not produce consistently relevant results. Human intelligence is still a very important part of the process.

(Jimmy Wales, Founder of Wikipedia. Source - Businessweek.com, 12/2006)

25

The Participatory Web A medium for human-computer collaborative design of watershed management solutionsOutline1. ACKNOWLEDGEMENTSSlide Number 4Participatory Methods for Engaging StakeholdersParticipatory Design of AlternativesOpportunities on the Participatory Web 2.0 ( and the Web 3.0 in the future)The Participatory Optimization ConceptThe Participatory Optimization Concept3. RESEARCH GOALSSlide Number 11WRESTOREs Client-Server ArchitectureSlide Number 13Study siteSlide Number 15Q1. How do users engage with and learn from such participatory optimization tools?Q1. How do users engage with and learn from such participatory optimization tools?Q1. How do users engage with and learn from such participatory optimization tools?Q2. Can we model users nonstationary and noisy feedbacks provided through the interfaces to detect underlying revealed preferences?Q2. Can we model users nonstationary and noisy feedbacks provided through the interfaces to detect underlying revealed preferences?Q3. How does the user-guided search affect the design of watershed plans?Q3. How does the user-guided search affect the design of watershed plans?6. CONCLUSIONS6. CONCLUSIONSSlide Number 25