Download - De Liddo & Buckingham Shum ipp2014

Transcript
Page 1: De Liddo & Buckingham Shum ipp2014

Anna De Liddo

Knowledge Media Institute, The Open University, UK

Simon Buckingham Shum University of Technology Sydney, Australia

Page 2: De Liddo & Buckingham Shum ipp2014

Wh

Page 3: De Liddo & Buckingham Shum ipp2014

Model of Collective Intelligence (CI): from sensing the environment, to interpreting it, to generating good

options, to taking decisions and coordinating action...

Collec&ve(Ac&on(

Collec&ve(Decision(

Collec&ve(Idea&on(

Collec&ve(Sensemaking(

Collec&ve(Sensing((

(

Page 4: De Liddo & Buckingham Shum ipp2014

Model of Collective Intelligence (CI): from sensing the environment, to interpreting it, to generating good

options, to taking decisions and coordinating action...

Collec&ve(Ac&on(

Collec&ve(Decision(

Collec&ve(Idea&on(

Collec&ve(Sensemaking(

Collec&ve(Sensing((

(

Page 5: De Liddo & Buckingham Shum ipp2014

When tackling complex and contested problems: v there may not be one worldview, or clear option

v evidence can be ambiguous or of dubious reliability requiring the construction of plausible, possibly competing narratives;

v growth in intelligence results from learning, which is socially constructed through different forms of discourse, such as dialogue and debate.

Contested Collective Intelligence (De Liddo 2012)

Page 6: De Liddo & Buckingham Shum ipp2014

In the design space of CI systems u where there is insufficient data to confidently compute an

answer, u when there is ambiguity about the trustworthiness of

environmental signals,

u  and uncertainty about the impact of actions,

u  And there is not one view that fits all …then a more powerful scaffolding for thinking and discourse is required, in order to support the emergence of CI around complex socio political dilemmas.

Contested Collective Intelligence (De Liddo 2012)

Page 7: De Liddo & Buckingham Shum ipp2014

(Social,  Visual  and  Argumentation-­‐based  CI)

Collective  Intelligence  Online  Deliberation  

Human  Dynamics  of  Engagements  

Analytics, & Visualization

Crowdsourcing ideas, arguments

and facts

Structured Discourse and Argumentation

Democratic entitlements

New class of Online Deliberation tools

Citizen Voice

Social Innovation

Computational Services & Dialogic

Agents

Page 8: De Liddo & Buckingham Shum ipp2014
Page 9: De Liddo & Buckingham Shum ipp2014

¡  Poor Debate: No tools to identify were ideas contrast, where people disagree and why... popularity vs critical thinking

Page 10: De Liddo & Buckingham Shum ipp2014

Flat listing of posts and no insight into the logical structure of ideas and arguments the,  

such  as  the  coherence  or  evidential  basis  of  an  argument.    

Page 11: De Liddo & Buckingham Shum ipp2014

These tools are increasingly used to support online debate and facilitate citizens’ engagement in policy and decision-making. These are fundamentally chronological views which offer: ¡  No support for idea refinement and improvement

 

 

 

LINK  to  PETITION:      http://www.change.org/en-­‐GB/petitions/stand-­‐against-­‐russia-­‐s-­‐brutal-­‐crackdown-­‐on-­‐gay-­‐rights-­‐urge-­‐winter-­‐olympics-­‐2014-­‐sponsors-­‐to-­‐condemn-­‐anti-­‐gay-­‐laws    

Page 12: De Liddo & Buckingham Shum ipp2014

 

 

 

LINK  to  QUORA:      http://www.quora.com/Physics/Do-­‐wormholes-­‐always-­‐have-­‐black-­‐holes-­‐at-­‐the-­‐beginning#answers  

Page 13: De Liddo & Buckingham Shum ipp2014

¡  Poor Debate: No tools to identify were ideas contrast, where people disagree and why

¡  Poor idea evaluation: No mechanisms to identify, contribute and discuss the evidence for an idea

¡  Poor Summarization and Visualization ¡  Shallow contributions and Cognitive clutters ¡  Platform Island & Balkanization

This hampers both:

¡  quality of users’ participation and ¡  The quality of proposed ideas ¡  effective assessment of the state of the debate.

Page 14: De Liddo & Buckingham Shum ipp2014

That make the structure and status of a dialogue or debate visible Coming from research on Argumentation and CSAV, these tools make visually explicit users’ lines of reasoning and (dis)agreements.

¡  Deliberatorium ¡  Debategraph ¡  Cohere ¡  CoPe_it! ¡  Problem&Proposals ¡  YourView ¡  The Evidence Hub

Page 15: De Liddo & Buckingham Shum ipp2014
Page 16: De Liddo & Buckingham Shum ipp2014
Page 17: De Liddo & Buckingham Shum ipp2014

!

Page 18: De Liddo & Buckingham Shum ipp2014
Page 19: De Liddo & Buckingham Shum ipp2014

!

Page 20: De Liddo & Buckingham Shum ipp2014
Page 21: De Liddo & Buckingham Shum ipp2014

•  referred to as “deliberative aggregators” (van Gelder 2012b) •  produce a collective viewpoint or judgment on complex societal

issues by crowdsourcing discourse •  inherent purpose to support large communities to tackle

complex issues of public concern

Page 22: De Liddo & Buckingham Shum ipp2014
Page 23: De Liddo & Buckingham Shum ipp2014
Page 24: De Liddo & Buckingham Shum ipp2014
Page 25: De Liddo & Buckingham Shum ipp2014
Page 26: De Liddo & Buckingham Shum ipp2014

Collaborative Knowledge Production

Collaborative Web Annotation and Knowledge mapping

Social Network Analysis and Visualization

Structured Online Discussion and Argumentation

Advanced Analytics for: Attention mediation & Deliberation diagnostic

Page 27: De Liddo & Buckingham Shum ipp2014

http://litemap.open.ac.uk

Page 28: De Liddo & Buckingham Shum ipp2014

Flat listing of posts and no insight into the logical structure of ideas and arguments ,  such  as  

the  coherence  or  evidential  basis  of  an  argument.    

Page 29: De Liddo & Buckingham Shum ipp2014

Add rhetorical markers to distinguish post type and role in the conversation  

Page 30: De Liddo & Buckingham Shum ipp2014

agrees  with  

agrees  with  

disagrees  with  

Explore and Mark-up connections between people and ideas, and give them a semantic  

Page 31: De Liddo & Buckingham Shum ipp2014
Page 32: De Liddo & Buckingham Shum ipp2014

http://litemap.open.ac.uk demo...

Page 33: De Liddo & Buckingham Shum ipp2014

¡  used in a real user community to map the main issues, ideas and arguments raised in a conversation hosted on the Utopia platform.

¡  Utopia is a large German online community on sustainable consumption and lifestyles, which uses a content management system (built on Synphony) with a space for online discussion to host online debates within the virtual community.

¡  Three community managers created summary argument maps to visualise different topics of sustainable living based on discussions and articles on the Utopia platform

¡  The resulting argument maps were then embedded in the Utopia website to display them to the wider online community.

Page 34: De Liddo & Buckingham Shum ipp2014

¡  The study lasted 6 weeks

¡  3 argument maps were created with LiteMap and then embedded into the Utopia website.

¡  Three mappers collaborated to the creation of each argument map.

¡  The mappers did not receive any training neither with the tool nor with the argumentation mapping process, and they were not familiar with the IBIS argumentation model.

Page 35: De Liddo & Buckingham Shum ipp2014

¡  A newsletter was used to announce each new map to online community.

¡  Over the period of the testing, an advert was placed in the sidebar of the Utopia homepage which pointed to the testing Website to maximize visibility and promote partecipation to the online discussion.

¡  The Webpage in which the maps were embedded contained an explanation on how to use the map, information on the user study, and a link to a survey for user to provide feedback on the usability and usefulness of the argument map to improve the sensemaking and summarization of the online debate.

Page 36: De Liddo & Buckingham Shum ipp2014

¡  Over the testing period, over 800 people navigated to the argument maps.

¡  Most traffic was generated after the newsletters were sent, after each one, around 130 visitors came to see the argument maps on the same day.

¡  57 users survey. Twice as many women as men participated in the survey.  

¡  Whereas the quantitative analysis of the surveys results is still under development, in the following we report initial insights from the qualitative analysis of three interviews with the argument mappers.

Page 37: De Liddo & Buckingham Shum ipp2014

¡  Understanding the argument mapping tasks, specifically the challenges when collaboratively building a map in the same virtual space (example questions: How did you build an understanding of the task at hand?, how would you summarize the main process you went through?)

¡  Understanding of the use of the tool (LiteMap) to carry on the summarization and arguments structuring task (example questions: How did you use the tool? How did you coordinate your activities? How did you reach agreement on the node labelling, connections and map layout?)

¡  Reflecting on how the tool could be improved by implementing new features which solve some of the encountered limitation.

Page 38: De Liddo & Buckingham Shum ipp2014

¡  Users reckon that a bottom-up approach to create an argument map (from harvesting the arguments and then moving to ideas and finally defining the issues) was not an effective process.

“For the first topic we came from the lower end of all these clips and we basically had 30 clips in front of us and we had to cluster them. I.e. we had argument on the quality of organic food, arguments on the different work pays and condition in organic food shops, then arguments on strategies to attract people to buy organic etc... …many arguments each representing a different entry point to the conversation and all interlinked so it was hard for us to cluster them, make it into a logic and then fit them into the argument structure. If we just had the top down question to answer, then of course we could have just stick to adding the argument here or there and this would have been easy. But we had to find a good formulation to capture all the ideas that we wanted to put in there.”

Page 39: De Liddo & Buckingham Shum ipp2014

“It was quite hard for us to agree what each clip could mean, it is because the discussion in the Utopia website is completely unstructured…you have a topic, and idea and all the argument packed as list of comments and there is no hierarchy between them. We had to create as mapper this hierarchy and then see how the higher elements related with the lowers and vice-versa. That was not very easy in practice, especially because is a very subjective task to do. Each of us would have done it in a different way and also the comment from some of the viewer was that they would have done it differently than us. That made it hard for any topic to map, and even harder for the ill-structured topics that were less defined.”

Page 40: De Liddo & Buckingham Shum ipp2014

¡  the nature of the problem to map affected the complexity of the argument mapping process in a way that a collective mapping procedure that may work for one issue can fail for a different one.

¡  Background knowledge and nature of the topic are notably key component to affect the effectiveness of collaborative mapping, as well as training with the tool and knowledge of the argumentation model.

Page 41: De Liddo & Buckingham Shum ipp2014

¡  If the issues are open ended and ill-defined collaborative mapping can be trickier and even the definition of issues as such would require a long negotiation effort between mappers.

¡  For these cases a bottom up approach to summarization seem to be even more counterproductive because it forces users to label and connect clips before the group has clarified the higher level structure of the map (the meta issues to map).

Page 42: De Liddo & Buckingham Shum ipp2014

¡  advancement of personal knowledge of the topic,

¡  Acknowledge the power of structuring ideas for personal reflection,

¡  and future use and topic exploration.

“It was absolutely useful, especially for topics as sustainable food in which we had no common knowledge on the many aspects of the topic and in which there are no well established metrics and definitions to assess it, it is very good to see how you can cluster the issues around this topic. Also the aspects of organic food are now laid down and I can read them and then dig only on the arguments of the issues and ideas I am interested in.” “I think it is very useful to have the idea visualised and structured and see how they come together.”

Page 43: De Liddo & Buckingham Shum ipp2014

“The think I am not sure is how much this mapping task fits to the Utopia’s online community because the discussion on Utopia are quite shallow, just one two opinions and no depth even to the level of a newspaper article. So doing argument maps based on people contributions is not only very subjective but also not very useful. I would find it more interesting if the arguments mapped were based on a more scientific level. In terms of the output form our point of view (the community managers) if we had more validated information behind the map the map would have more value as well.”

¡  online debates, even debates about complex socio technical issues, consist of often very shallow conversations.

“An important added value that can be played by the community manager is the mapping of more validated information to support participants’ learning and reflection on new aspects of a given debate. Argument maps seems to be better suited to bring new evidence into the conversation rather than for mapping and summarising the existing online debate.”

Page 44: De Liddo & Buckingham Shum ipp2014

¡  Improving negotiation mechanisms

¡  Enabling different deliberation processes for different issue types (open-ended vs more focused problems)

¡  Building argument map to bring evidence to the debate and improve deliberation quality rather than summarization

Page 45: De Liddo & Buckingham Shum ipp2014

Thanks! [email protected]

Twitter: Anna_De_Liddo http://kmi.open.ac.uk/people/member/anna-de-liddo

Catalyst: http://catalyst-fp7.eu/ EDV – Election Debate Visualizations: http://edv-project.net/

Follow us on Twitter @CATALYST_FP7

Watch the Demos on Youtube CATALYST FP7

Page 46: De Liddo & Buckingham Shum ipp2014

¡  De Liddo, A., & Buckingham Shum, S. (2014). New Ways of Deliberating Online: An Empirical Comparison of Network and Threaded Interfaces for Online Discussion. In E. Tambouris, A. Macintosh, & F. Bannister, Lecture Notes in Computer Science (Vol. 8654, pp. 90–101). Springer.

¡  De Liddo, A. (2014). “Enhancing Discussion Forum with Combined Argument and Social Netwrok Analytics. In Okada, A., Buckingham Shum, S. and Sherborne, T., (Eds) Knowledge Cartography. Springer. Second Edition. In press.

¡  De Liddo, A. and Buckingham Shum, S. (2013) The Evidence Hub: Harnessing the Collective Intelligence of Communities to Build Evidence-Based Knowledge, Workshop: Large Scale Ideation and Deliberation at 6th International Conference on Communities and Technologies, Munich, Germany

¡  De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012) Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study, Computer Supported Cooperative Work (CSCW) Journal : Volume 21, Issue 4 (2012), Page 417-448

¡  Buckingham Shum, Simon (2008). Cohere: Towards Web 2.0 Argumentation. In: Proc. COMMA'08: 2nd International Conference on Computational Models of Argument, 28-30 May 2008, Toulouse, France. Available at:http://oro.open.ac.uk/10421/

¡  De Liddo, Anna and Buckingham Shum, Simon (2010). Cohere: A prototype for contested collective intelligence. In: ACM Computer Supported Cooperative Work (CSCW 2010) - Workshop: Collective Intelligence In Organizations - Toward a Research Agenda, February 6-10, 2010, Savannah, Georgia, USA. Available at: http://oro.open.ac.uk/19554/

¡  Buckingham Shum, Simon and De Liddo, Anna (2010). Collective intelligence for OER sustainability. In: OpenED2010: Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Available at: http://oro.open.ac.uk/23352/

¡  Buckingham Shum, Simon (2007). Hypermedia Discourse: Contesting networks of ideas and arguments. In: Priss, U.; Polovina, S. and Hill, R. eds. Conceptual Structures: Knowledge Architectures for Smart Applications. Berlin: Springer, pp. 29–44.