8/13/2019 All Lectures about articles
1/383
Research Methods in Computer Science
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 1 / 1
http://find/8/13/2019 All Lectures about articles
2/383
Introduction and Overview Research
Research Methods in Computer ScienceLecture 1: Introduction and Overview
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 2 / 21
http://-/?-http://-/?-http://find/http://-/?-http://-/?-8/13/2019 All Lectures about articles
3/383
Introduction and Overview Research
Today ...
1 Introduction and OverviewAimsLearning outcomesDeliveryAssessment
2 What is Research ?
Ullrich Hustadt Research Methods in Computer Science 3 / 21
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
4/383
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
Aims
1 To provide a deep and systematic understanding of the nature andconduct of Computer Science research
2
To equip students with the ability to undertake independent research3 To enhance existing transferable key skills
4 To develop high-order transferable key skills
5 To remind students of the Legal, Social, Ethical and Professional(LSEP) issues applicable to the computer industry
Ullrich Hustadt Research Methods in Computer Science 6 / 21
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
5/383
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
Learning Outcomes (1)
1 Have anunderstandingof how establishedtechniques of researchandenquiry are used to extend, create and interpret knowledge inComputer Science
2 Have a conceptualunderstanding sufficient to:
(i) evaluatecriticallycurrent researchand advanced scholarship inComputer Science, and(ii) proposepossiblealternative directionsfor further work
3 Beable to deal with complex issuesat the forefront of the academicdiscipline of Computer Science in a manner,
based on sound judgements, that is both systematic and creative; andbeable to communicate conclusions clearlyto both specialists andnon-specialists
Ullrich Hustadt Research Methods in Computer Science 8 / 21
I d i d O i R h Ai L i D li A
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://goback/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
6/383
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
Learning Outcomes (2)
4 Demonstrate self-direction and originalityin tackling and solvingproblems within the domain of Computer Science, andbeable to act autonomously in planning and implementing solutionsin
a professional manner5 Beable to define and plan a programme of independent research
6 Participate within the professional, legal and ethical framework withinwhich they would be expected to operate as professionals within the IT
industry
Ullrich Hustadt Research Methods in Computer Science 10 / 21
I t d ti d O i R h Ai L i t D li A t
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
7/383
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
Learning Outcomes (3)
7 Make use of the qualities andtransferable skillsnecessary foremployment requiring:
(i) the exercise of initiativeandpersonal responsibility,(ii) decision making in complex and unpredictable situations, and
(iii) theindependent learning abilityrequired for continuing professionaldevelopment
8 Have theskills setto be able to continue toadvance their knowledgeand understanding, and todevelop new skills to a high level, with
respect to continuing professional development as a self-directedlife-long learner across the discipline of Computer Science
Ullrich Hustadt Research Methods in Computer Science 11 / 21
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
8/383
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
Learning Outcomes (4)
In short, you should learn
1 tounderstand researchandresearch methodsin Computer Science;
2 tobe able to plan, andconduct your own research, taking into accountethical, legal, and professional limitations; and
3 tobe able to communicate its results
Ullrich Hustadt Research Methods in Computer Science 12 / 21
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
9/383
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
Delivery of the module (1)
1 Lectures
Monday, 11.00 Ashton Building Seminar Room (Ground Floor)Monday, 12.00 Ashton Building Seminar Room (Ground Floor)Friday, 10.00 Ashton Building Seminar Room (Ground Floor)
2 Seminars
During lectures By members of staffTuesday, 16.00 Departmental research seminar
Ashton Building Seminar Room (Ground Floor)
3 Tutorials/Labs
To be arranged
Ullrich Hustadt Research Methods in Computer Science 13 / 21
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
10/383
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
Delivery of the module (2)
1 Office hours
Typically, Monday 15.00 to 17.00;
make an arrangement by e-mail ([email protected]) first
2 Website
http://www.csc.liv.ac.uk/~ullrich/COMP516/
Ullrich Hustadt Research Methods in Computer Science 14 / 21
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
http://-/?-http://-/?-http://www.csc.liv.ac.uk/~ullrich/COMP516/http://www.csc.liv.ac.uk/~ullrich/COMP516/http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
11/383
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
Recommended texts
Christian W. Dawson: Projects in Computing and Information Systems(A Students Guide). Addison Wesley, 2005.Harold Cohen Library, Class No 518.561.D27
Earlier edition:Christian W. Dawson: The essence of computing projects (A studentsguide). Prentice Hall, 2000.Harold Cohen Library, Class No 518.561.D27
Justin Zobel: Writing for Computer Science. Springer, 2004.Harold Cohen Library, Class No 378.962.Z81
Ullrich Hustadt Research Methods in Computer Science 15 / 21
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
12/383
g y
Assessment
2,000 word essay on a topic chosen by the examinerto be handed out on Friday, 28 September 2007to be submitted on Friday, 26 October 2007, 15.30accounts for15% of the module mark
5,000 word essay on an agreed subjectwork can be started as soon as the subject is agreed(Friday, 26 October 2007, at the latest)
to be submitted before the Xmas break(most likely Thursday, 13 December 2007, 15.30)
accounts for85% of the module mark
Pass mark, as usual for MSc modules, is 50%
Ullrich Hustadt Research Methods in Computer Science 16 / 21
Introduction and Overview Research Aims Learning outcomes Delivery Assessment
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
13/383
g y
Teaching and learning strategy
Lectures, seminars, tutorials/labs only make up a small part of thedelivery of the module
In total you are expected to commit 150 hours to the module, that is,
12.5 hoursper week over 12 weeks(more hours per week than for any other module)
Of those the timetabled activities only make up 4 hours per week
In addition you should spend3.5 hours per weekon reflection,consideration of lecture material and background readingplus5 hours per weekon theassessment tasks
Ullrich Hustadt Research Methods in Computer Science 17 / 21
Introduction and Overview Research
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
14/383
What is research?
Research (Dictionary)
Noun
1 Scholarly or scientific investigation or inquiry.
2 Close, careful study.
Verb
1 To study (something) thoroughly so as to present in a detailed,accurate manner.
(Example: researching the effects of acid rain.)
Note the difference between the definition of the noun and of the verb.
Ullrich Hustadt Research Methods in Computer Science 18 / 21
Introduction and Overview Research
http://find/http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
15/383
What is research?
Study (Dictionary)
Noun
1 The pursuit of knowledge, as by reading, observation, or research.
2
Attentive scrutiny.Verb
1 To apply ones mind purposefully to the acquisition of knowledge orunderstanding of (a subject).
2 To inquire into; investigate.3 To examine closely; scrutinise.
Ullrich Hustadt Research Methods in Computer Science 19 / 21
Introduction and Overview Research
http://find/http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
16/383
What is research?
Research (http://en.wikipedia.org/wiki/Research)
anactive, diligent, and systematic process of inquiryin order todiscover, interpret or revisefacts, events, behaviours, or theories, or to
make practical applicationswith the help of such facts, laws, or theories.a collection of information about a particular subject.
derives from the Middle French and the literal meaning isto investigate thoroughly.
Homework: Read the Wikipedia article!
Ullrich Hustadt Research Methods in Computer Science 20 / 21
Research Knowledge Knowledge Originality
http://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://find/http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
17/383
Research Methods in Computer ScienceLecture 2: Research (continued)
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 22 / 39
Research Knowledge Knowledge Originality
http://find/http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
18/383
Previously . . .
1 Introduction and OverviewAimsLearning outcomes
DeliveryAssessment
2
What is Research ?
Ullrich Hustadt Research Methods in Computer Science 23 / 39
Research Knowledge Knowledge Originality
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://goforward/http://find/http://goback/http://-/?-http://-/?-8/13/2019 All Lectures about articles
19/383
Today ...
3 What is Research ?
More Definitions of Research
4 KnowledgeA HierarchyData
InformationKnowledge
5 KnowledgeTheories
6 OriginalityDefinitionThe importance of repeating the work of others
Ullrich Hustadt Research Methods in Computer Science 24 / 39
Research Knowledge Knowledge Originality Research 2
http://-/?-http://-/?-http://find/http://-/?-http://-/?-8/13/2019 All Lectures about articles
20/383
What is research?
Research (http://en.wikipedia.org/wiki/Research)
anactive, diligent, and systematic process of inquiryin order todiscover, interpret or revisefacts, events, behaviours, or theories, or to
make practical applicationswith the help of such facts, laws, or theories.a collection of information about a particular subject.
derives from the Middle French and the literal meaning isto investigate thoroughly.
Homework: Read the Wikipedia article!
Ullrich Hustadt Research Methods in Computer Science 25 / 39
Research Knowledge Knowledge Originality Research 2
http://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://find/8/13/2019 All Lectures about articles
21/383
What is research?
Research (Higher Education Funding Council for England)Original investigationundertaken in order togain knowledge andunderstanding, including
work of direct relevance to the needs of commerce and industry and tothe public and voluntary sectors
scholarship (research infrastructure)
the invention and generation of ideas, images, performances andartifacts including design, where these lead to new or substantiallyimproved insights;
the use of existing knowledge in experimental development to producenew or substantially improved materials, devices, products andprocesses, including design and construction.
Ullrich Hustadt Research Methods in Computer Science 26 / 39
http://find/8/13/2019 All Lectures about articles
22/383
Research Knowledge Knowledge Originality Hierarchy Data Information Knowledge
8/13/2019 All Lectures about articles
23/383
Knowledge: Data and Information
Datum/Data
statements accepted at face value (a given) and presented as numbers,characters, images, or sounds.
a large class of practically important statements are measurementsorobservationsof variables, objects, or events.
in a computing context, in a form which can be assessed,stored,processed, andtransmittedby a computer.
Ullrich Hustadt Research Methods in Computer Science 28 / 39
Research Knowledge Knowledge Originality Hierarchy Data Information Knowledge
K l d D d I f i
http://find/8/13/2019 All Lectures about articles
24/383
Knowledge: Data and Information
Information
Dataon its own has no meaning, only wheninterpretedby some kind ofdata processing systemdoes it take on meaning and becomesinformation
Example:Thehuman genome projecthas determined the sequence of the 3 billionchemical base pairs that make up human DNA
identifying base pairs producesdata informationwould tell us what they do!
Ullrich Hustadt Research Methods in Computer Science 29 / 39
Research Knowledge Knowledge Originality Hierarchy Data Information Knowledge
K l d Al i d fi i i (1)
http://find/8/13/2019 All Lectures about articles
25/383
Knowledge: Alternative definitions (1)
Knowledge (Dawson 2005)
higher level understanding of things
represents our understanding of the why instead of the mere what
interpretation of information in the form of rules, patterns, decisions,
models, ideas, etc.
Innatural sciences, understanding why is too ambitious most of time;understanding how is usually what we aim for
In other areas, understanding how is trivial, understanding why ischallenging
Ullrich Hustadt Research Methods in Computer Science 30 / 39
Research Knowledge Knowledge Originality Hierarchy Data Information Knowledge
K l d Alt ti d fi iti (2)
http://goforward/http://find/http://goback/8/13/2019 All Lectures about articles
26/383
Knowledge: Alternative definitions (2)
Knowledge (Davenport et al. 1998)
a fluid mix offramed experience,contextual information, values andexpert insightthat provides aframework for evaluating andincorporating new experiences and information.
information combined with experience, context, interpretation, and
reflection
high-value form of information that is ready to apply to decisions andactions
Second point similar to last point in the previous definition
Last point seems to imply that knowledge has to be useful(is astrophysics useful?)
Ullrich Hustadt Research Methods in Computer Science 31 / 39
Research Knowledge Knowledge Originality Hierarchy Data Information Knowledge
K l d Alt ti d fi iti (3)
http://find/8/13/2019 All Lectures about articles
27/383
Knowledge: Alternative definitions (3)
Knowledge (http://en.wikipedia.org/wiki/Knowledge)
theawarenessandunderstanding of facts, truths or information gainedin the form of experience or learning (a posteriori), or through deductivereasoning (a priori)
an appreciation of the possession ofinterconnected detailswhich, inisolation, are of lesser value
both knowledge and information consist of true statements, butknowledge is information that has apurpose or use(information plus
intentionality)
Ullrich Hustadt Research Methods in Computer Science 32 / 39
Research Knowledge Knowledge Originality Theories
K l d d th i s D fi iti
http://en.wikipedia.org/wiki/Knowledgehttp://en.wikipedia.org/wiki/Knowledgehttp://find/8/13/2019 All Lectures about articles
28/383
Knowledge and theories: Definition
Scientific knowledge is often organised intotheories.
Theory (http://en.wikipedia.org/wiki/Theories)
alogically self-consistent modelor framework describing the behaviourof a certain natural or social phenomenon, thus either originating fromobservable facts or supported by them
formulated, developed, and evaluated according to thescientific method
Ullrich Hustadt Research Methods in Computer Science 33 / 39
Research Knowledge Knowledge Originality Theories
Knowledge and theories: Criteria
http://en.wikipedia.org/wiki/Theorieshttp://en.wikipedia.org/wiki/Theorieshttp://find/8/13/2019 All Lectures about articles
29/383
Knowledge and theories: Criteria
Theory (http://en.wikipedia.org/wiki/Theories)
A body of (descriptions of) knowledge is usually only called a theoryonceit has afirm empirical basis, that is, it
1 isconsistent with pre-existingtheory to the extent that the pre-existing
theory was experimentally verified, though it will often showpre-existing theory to be wrong in an exact sense,
2 issupported by many strands of evidence rather than a singlefoundation, ensuring that it probably is a good approximation if nottotally correct,
Ullrich Hustadt Research Methods in Computer Science 34 / 39
Research Knowledge Knowledge Originality Theories
Knowledge and theories: Criteria
http://en.wikipedia.org/wiki/Theorieshttp://en.wikipedia.org/wiki/Theorieshttp://find/8/13/2019 All Lectures about articles
30/383
Knowledge and theories: Criteria
Theory (http://en.wikipedia.org/wiki/Theories)
A body of (descriptions of) knowledge is usually only called a theoryonceit has afirm empirical basis, that is, it
3 makes (testable) predictionsthat might someday be used to disprove
the theory, and4 hassurvived many critical real world teststhat could have proven it
false,5 is a/thebest known explanation, in the sense of Occams Razor, of the
infinite variety of alternative explanations for the same data.
Ullrich Hustadt Research Methods in Computer Science 35 / 39
Research Knowledge Knowledge Originality Theories
Knowledge and theories: Facts versus theories
http://en.wikipedia.org/wiki/Theorieshttp://en.wikipedia.org/wiki/Theorieshttp://find/8/13/2019 All Lectures about articles
31/383
Knowledge and theories: Facts versus theories
This (e.g. evolution) is only a theorynot afact
Fact
1. atruth(statement confirming toreality)or
2.datasupported by ascientific experiment
Status of a truth is by and large unachievable
Atheoryis formulated, developed, and evaluated according to thescientific method
Given enoughexperimental supportatheorycan be(a scientific)fact
Ullrich Hustadt Research Methods in Computer Science 36 / 39
Research Knowledge Knowledge Originality Definition Repeating work
Originality (1)
http://find/http://-/?-http://-/?-8/13/2019 All Lectures about articles
32/383
Originality (1)
Research(HEFCE):Original investigationundertaken in order togain
knowledge and understandingOriginality
Doing something that has not been done before
Dawson (2005):There is no point in repeating the work of others and dis-covering or producing what is already known
Only true for what is truly known (i.e. very little)
Theories make predictions, which need to be testedThe people performing those tests are neitherinfalliblenortrustworthyTests need to be repeated and resultsreplicated
Ullrich Hustadt Research Methods in Computer Science 37 / 39
Research Knowledge Knowledge Originality Definition Repeating work
(In)Fallibility
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
33/383
(In)Fallibility
Cold fusion
(http://en.wikipedia.org/wiki/Cold_fusion)Cold fusion: Nuclear fusion reaction that occurs well below thetemperature required for thermonuclear reactions, that is, near ambienttemperature instead of millions of degrees Celsius
First reported to have been achieved by Pons (University of Utah) andFleischmann (University of Southampton) in 1989
Scientists tried to replicate their results shortly after initialannouncement
Teams at Texas A&M University and the Georgia Institute ofTechnology first confirmed the results, but then withdraw those claimsdue to lack of evidence
Vast majority of experiments failed
Ullrich Hustadt Research Methods in Computer Science 39 / 39
Investigation Knowledge Originality Gain Research
http://en.wikipedia.org/wiki/Cold_fusionhttp://en.wikipedia.org/wiki/Cold_fusionhttp://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
34/383
Research Methods in Computer ScienceLecture 3: Research (continued)
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 49 / 66
Investigation Knowledge Originality Gain Research
Previously
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
35/383
Previously . . .
3 What is Research ?
More Definitions of Research
4 KnowledgeA HierarchyData
InformationKnowledge
5 KnowledgeTheories
6 OriginalityDefinitionThe importance of repeating the work of others
Ullrich Hustadt Research Methods in Computer Science 50 / 66
Investigation Knowledge Originality Gain Research
What is research?
http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
36/383
What is research ?
Research (Dictionary)
1 Scholarly or scientific investigation or inquiry.2 Close, careful study.
Research (http://en.wikipedia.org/wiki/Research)
anactive, diligent, and systematic process of inquiryin order todiscover, interpret or revisefacts, events, behaviours, or theories, or tomake practical applicationswith the help of such facts, laws, or theories.
a collection of information about a particular subject.
Research (Higher Education Funding Council for England)
Original investigationundertaken in order togain knowledge andunderstanding
Ullrich Hustadt Research Methods in Computer Science 51 / 66
Investigation Knowledge Originality Gain Research
Today ...
http://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
37/383
Today ...
7 Investigation
8 Knowledge
9 OriginalityAreas of originality
10 Gain
11 What is Research?Summary
Ullrich Hustadt Research Methods in Computer Science 52 / 66
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
38/383
Investigation Knowledge Originality Gain Research
Knowledge: A hierarchy
8/13/2019 All Lectures about articles
39/383
g y
Datum/Data
statements accepted at face value (a given) and presented as numbers,characters, images, or sounds.a large class of practically important statements are measurementsorobservationsof variables, objects, or events.
InformationData interpretedby some kind ofdata processing systemwhich gives itmeaning
Knowledge (Dawson 2005)
higher level understanding of thingsrepresents our understanding of the why instead of the mere whatinterpretation of information in the form of rules, patterns, decisions,models, ideas, etc.
Ullrich Hustadt Research Methods in Computer Science 54 / 66
Investigation Knowledge Originality Gain Research Areas of originality
Research and Originality (1)
http://find/http://-/?-8/13/2019 All Lectures about articles
40/383
g y ( )
Research(HEFCE):Original investigationundertaken in order togain
knowledge and understandingOriginality
Doing something that has not been done before
Dawson (2005):There is no point in repeating the work of others and dis-covering or producing what is already known
Only true for what is truly known (i.e. very little)
Theories make predictions, which need to be testedThe people performing those tests are neitherinfalliblenortrustworthyTests need to be repeated and resultsreplicated
Ullrich Hustadt Research Methods in Computer Science 55 / 66
Investigation Knowledge Originality Gain Research Areas of originality
Research and Originality (2)
http://find/http://-/?-8/13/2019 All Lectures about articles
41/383
g y ( )
Areas of originality (Cryer 1996)Exploring the unknownInvestigate a field that no one has investigated before
Exploring the unanticipated
Obtaining unexpected results and investigating new directions in analready existing field
The use of dataInterpret data in new ways
Tools, techniques, procedures, and methodsApply new tools/techniques to alternative problemsTry procedures/methods in new contexts
Ullrich Hustadt Research Methods in Computer Science 56 / 66
Investigation Knowledge Originality Gain Research
Gain
http://find/http://-/?-8/13/2019 All Lectures about articles
42/383
Research(HEFCE):Original investigationundertaken in order togainknowledge and understanding
Contribution
Research is supposed to add to the worlds body of knowledge andunderstanding (in contrast to adding to the researchers knowledge andunderstanding)
Ullrich Hustadt Research Methods in Computer Science 57 / 66
Investigation Knowledge Originality Gain Research Summary
What is research?
http://find/http://-/?-8/13/2019 All Lectures about articles
43/383
In summary, what are thethree key aspects of research?
(10 minutes group discussion)
Ullrich Hustadt Research Methods in Computer Science 58 / 66
Investigation Knowledge Originality Gain Research Summary
What is research?
http://find/http://-/?-8/13/2019 All Lectures about articles
44/383
Research (http://en.wikipedia.org/wiki/Research)
An active, diligent, and systematic process of inquiry in order to discover,interpret or revise facts, events, behaviours, or theories, or to makepractical applications with the help of such facts, laws, or theories.
Research (Higher Education Funding Council for England)Original investigation undertaken in order to gain knowledge andunderstanding
Sharp et al. (2002)
Seeking through methodical process to add to ones own body ofknowledge and to that of others, by the discovery of non-trivial facts andinsights
Ullrich Hustadt Research Methods in Computer Science 59 / 66
Investigation Knowledge Originality Gain Research Summary
What is research?
http://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://find/http://-/?-8/13/2019 All Lectures about articles
45/383
Research (http://en.wikipedia.org/wiki/Research)
Anactive, diligent, and systematic process of inquiryin order todiscover,interpret or revisefacts, events, behaviours, or theories, or to makepractical applicationswith the help of such facts, laws, or theories.
Research (Higher Education Funding Council for England)Original investigationundertaken in order togain knowledge andunderstanding
Sharp et al. (2002)
Seeking through methodical process toaddto ones own body ofknowledge and to that of others, by the discovery ofnon-trivial facts andinsights
Ullrich Hustadt Research Methods in Computer Science 60 / 66
http://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://find/http://-/?-8/13/2019 All Lectures about articles
46/383
Process models
8/13/2019 All Lectures about articles
47/383
Research Methods in Computer ScienceLecture 4: Research process models
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 66 / 85
Process models
Previously . . .
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
48/383
7 Investigation
8 Knowledge
9 OriginalityAreas of originality
10 Gain
11 What is Research?Summary
Ullrich Hustadt Research Methods in Computer Science 67 / 85
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
49/383
Process models Sequential Generalised Circulatory Evolutionary
Research process models
8/13/2019 All Lectures about articles
50/383
All definitions agree thatresearchinvolves asystematicor methodicalprocess
Dawson (2005), following Baxter (2001), identifies
four common views of theresearch process:Sequential
Generalised
Circulatory
Evolutionary
Ullrich Hustadt Research Methods in Computer Science 69 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Sequential (1)
http://find/http://-/?-8/13/2019 All Lectures about articles
51/383
Research process asSeries of activities
Performed one after another (sequentially)
In a fixed, linear series of stages
Example:Research process model of Greenfield (1996):
1 Review the field2 Build a theory3 Test the theory4 Reflect and integrate
Ullrich Hustadt Research Methods in Computer Science 70 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Sequential (2)
http://find/http://-/?-8/13/2019 All Lectures about articles
52/383
Example:Sharp et al (2002):
1 Identify the broad area of study2 Select a research topic3 Decide on an approach4 Plan how you will perform the research
5 Gather data and information6 Analyse and interpret these data7 Present the result and findings
Ullrich Hustadt Research Methods in Computer Science 71 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Sequential (3)
http://find/http://-/?-8/13/2019 All Lectures about articles
53/383
Greenfield (1996):
1
Review the field2 Build a theory3 Test the theory4 Reflect and integrate
Sharp et al (2002):
1
Identify the broad area of study2 Select a research topic3 Decide on an approach4 Plan how you will perform the research5 Gather data and information
6 Analyse and interpret these data7 Present the result and findings
What do you think about this research process model?What is wrong with it?
(7 minutes group discussion)
Ullrich Hustadt Research Methods in Computer Science 72 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Sequential (4)
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
54/383
Greenfield (1996):
1
Review the field2 Build a theory3 Test the theory4 Reflect and integrate
Sharp et al (2002):
1
Identify the broad area of study2 Select a research topic3 Decide on an approach4 Plan how you will perform the research5 Gather data and information
6 Analyse and interpret these data7 Present the result and findings
Problems with the sequential (and generalised) process model:
1 Stages not subject specific
2 No repetition or cycles
3 Starting point and order fixed
Ullrich Hustadt Research Methods in Computer Science 76 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Generalised (1)
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
55/383
Thegeneralised research process model recognises that the stages of theresearch process depend on thesubjectandnatureof the researchundertaken
Example:
Data gatheringanddata analysisplay no role for research inpuremathematicsand large parts ofcomputer scienceInstead researchers makeconjectureswhich theyprove mathematically
Thegeneralised research process modelprovidesalternative routesdepending on thesubjectandnatureof the research undertaken
But eachrouteis stillsequential
Ullrich Hustadt Research Methods in Computer Science 77 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Generalised (2)
http://find/http://-/?-8/13/2019 All Lectures about articles
56/383
Example:
(1) Identify the broad area of study(2) Select a research topic
In natural sciences:(3) Decide on an approach(4) Plan the research(5) Gather data and information(6) Analyse and interpret these data
In mathematics:(3) Make a conjecture(4) Prove the conjecture
(7) Present the result and findings
Problems with the generalised process model:1 No repetition or cycles
2 Starting point and order fixed
Ullrich Hustadt Research Methods in Computer Science 78 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Circulatory
http://find/http://-/?-8/13/2019 All Lectures about articles
57/383
Thecirculatory research process modelrecognises that any research is
part of acontinuous cycleofdiscoveryandinvestigationthat never endsIt allows the research process to bejoinedat any point
One can alsorevisit(go back to)earlier stages
ConceptualFramework(theory, literature)
Research Question
Empirical ObservationData Collection
Data Analysis
Ullrich Hustadt Research Methods in Computer Science 79 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Evolutionary (1)
http://find/http://-/?-8/13/2019 All Lectures about articles
58/383
Theevolutionary research process model recognises that research(methods) itselfevolveandchange over time
That is, over time our concept of
What research questions are admissible
What extend and methods of data collection are possible, necessary, ethical,or reliable
What methods are data analysis are available
What constitutes sufficient evidence for a hypothesis
What we mean by a systematic approach to research changes
Ullrich Hustadt Research Methods in Computer Science 80 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Evolutionary (2)
http://find/http://-/?-8/13/2019 All Lectures about articles
59/383
Theevolutionary research process model recognises that research(methods) itselfevolveandchange over time
As an example, we can consider research inmathematics, in particular,its use ofcomputers
With respect tomathematical proofswe can make the following
distinctions:(1) Proofs created solely by humans
typically sketchy, omitting steps that are considered obvious
(2) Computer-aided mathematical proofs Structure and deductive steps still provided by humans, but
certain computations are delegated to a computer
(3) Fully formal, computer generated and validated proofs
Every step of a proof is conducted and validated by a computer,possibly under guidance by humans
Ullrich Hustadt Research Methods in Computer Science 81 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Evolutionary (3)
http://find/http://-/?-8/13/2019 All Lectures about articles
60/383
Theevolutionary research process model recognises that research(methods) itselfevolveandchange over time
Computer-aided mathematical proofs (1)
Four colour theoremAny planar map can be coloured with at most four colours in a
way that no two regions with the same colour share a border.
Conjectured in 1852 by Guthrie. Proved in 1976 by Appel and Haken.Proof involves a case analysis of about 10,000 cases for which the helpof a computer was used
Proof seems generally accepted, but not by all Mathematician
Ullrich Hustadt Research Methods in Computer Science 82 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Evolutionary (4)
http://find/http://-/?-8/13/2019 All Lectures about articles
61/383
Theevolutionary research process model recognises that research(methods) itselfevolveandchange over time
Computer-aided mathematical proofs (2)
Sphere packing theorem
Close packing is the densest possible sphere packing.
Conjectured in 1611 by Kepler. Hayes published a proof plan in (1997).Execution of the plan involved solving about 100,000 linear optimisationproblems using a computer. The computer files for the related programsand data requires more than 3GB of space
At one point it was suggested that the proof will be published with adisclaimer, saying that it is impossible for a human to check itscorrectness
Ullrich Hustadt Research Methods in Computer Science 83 / 85
Process models Sequential Generalised Circulatory Evolutionary
Research process models: Conclusion
http://find/http://-/?-8/13/2019 All Lectures about articles
62/383
Among the four common views of theresearch processSequential
Generalised
Circulatory
Evolutionary
theevolutionary research process model best describes the realresearch process
While theevolutionary research process modelallows for the rules of thegame to change over time, this does not imply there arent any rules
For a young researcher it is best to follow the current establishedresearch process
Ullrich Hustadt Research Methods in Computer Science 84 / 85
http://find/http://-/?-8/13/2019 All Lectures about articles
63/383
Scientific method Intellectual discovery Problem solving
Previously . . .
8/13/2019 All Lectures about articles
64/383
10 Research process modelsSequential
GeneralisedCirculatoryEvolutionary
Ullrich Hustadt Research Methods in Computer Science 79 / 94
Scientific method Intellectual discovery Problem solving
Topics
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/8/13/2019 All Lectures about articles
65/383
11 Scientific methodElements
12 Intellectual discovery
DeductionAbductionInductionProcess model
13 Problem solving
Ullrich Hustadt Research Methods in Computer Science 80 / 94
Scientific method Intellectual discovery Problem solving Elements
Scientific method
http://find/8/13/2019 All Lectures about articles
66/383
Scientists useobservationsandreasoningtodevelop technologiesandpropose explanationsfor natural phenomena in the form ofhypotheses
Predictionsfrom thesehypothesesare tested by experimentand furthertechnologies developed
Anyhypothesiswhich is cogent enough to make predictions can then betested reproducibly in this way
Once it has been established that ahypothesisissound, it becomes atheory.
Sometimesscientific developmenttakes place differently with atheory
first being developed gaining support on the basis of its logic andprinciples
Ullrich Hustadt Research Methods in Computer Science 81 / 94
Scientific method Intellectual discovery Problem solving Elements
Elements of a scientific method
http://find/8/13/2019 All Lectures about articles
67/383
The essential elements of a scientific method are iterations,recursions,interleavingsandorderingsof the following:
Characterisations(Quantifications, observations and measurements)
Hypotheses(theoretical, hypothetical explanations of observations and
measurements)Predictions(reasoning including logicaldeductionfrom hypotheses and theories)
Experiments
(tests of all of the above)Bothcharacterisationsandexperimentsinvolve data collection
Ullrich Hustadt Research Methods in Computer Science 82 / 94
Scientific method Intellectual discovery Problem solving Deduction Abduction Induction Process model
Intellectual discovery
K i h h l f i ifi h d d ll
http://find/8/13/2019 All Lectures about articles
68/383
Knowing what theelementsof ascientific methodare does not tell ushow to come up with the rightinstancesof these elements
What predictions does a theory make?What is the right hypothesis in a particular situation?
What is the right experiment to conduct?
These are commonly derived by a process involvingDeductive reasoning
Abductive reasoning
Inductive reasoning
Classification by Charles Sanders Peirce (1839-1914)See http://plato.stanford.edu/entries/peirce/for additionaldetails
Ullrich Hustadt Research Methods in Computer Science 84 / 94
Scientific method Intellectual discovery Problem solving Deduction Abduction Induction Process model
Intellectual discovery: Deduction (1)
D d d f k l d f h ld
http://plato.stanford.edu/entries/peirce/http://plato.stanford.edu/entries/peirce/http://find/8/13/2019 All Lectures about articles
69/383
Deductive reasoningproceeds from our knowledge of the world(theories) and predicts likely observations
Example:
Assume we know that A implies B. A has been observed. Then we should also obverse B.
Useful forexperiment generationfor theories
Example:
Newtons theory of gravity versus Einsteins theory of relativity
Largely make the same predictions
Both predict that the suns gravity should bend rays of light
However, Einsteins theory predicts a greater deflection
Correctness of Einsteins prediction confirmed by observation in 1919
Ullrich Hustadt Research Methods in Computer Science 85 / 94
Scientific method Intellectual discovery Problem solving Deduction Abduction Induction Process model
Intellectual discovery: Deduction (2)
D d i i i f id l d k l d
http://find/8/13/2019 All Lectures about articles
70/383
Deductive reasoningis often saidnotto lead to new knowledge(Note: This implies pure mathematicians largely waste
their time)
Seriously underestimates the computational effort involvedindeductive reasoning
Most theories areundecidable
(There is no algorithm that even given infinite time coulddetermine whether a statements follows from a theory ornot)
Thus, establishing that a statement follows from a theory
extendsour knowledge
Ullrich Hustadt Research Methods in Computer Science 86 / 94
Scientific method Intellectual discovery Problem solving Deduction Abduction Induction Process model
Intellectual discovery: Abduction
Abd ti i d f b ti t
http://find/8/13/2019 All Lectures about articles
71/383
Abductive reasoningproceeds from observations to causes
Example: The phenomenon X is observed. Among hypotheses A, B, C, and D,
only A and B are capable of explaining X. Hence, there is a reason to assume that A or B holds.
Requires atheorylinking A, B, C, D to X
Useful forhypothesis generation
Hypothesesmust then be confirmed / eliminated through furtherobservation
It is not easy from the outside to decide whether someone usesdeductionorabduction The two are often confused
Ullrich Hustadt Research Methods in Computer Science 87 / 94
Scientific method Intellectual discovery Problem solving Deduction Abduction Induction Process model
Intellectual discovery: Induction (1)
I d ti i d f t f b ti t l
http://find/8/13/2019 All Lectures about articles
72/383
Inductive reasoningproceeds from a set of observations to a generalconclusion
Example:
Tycho Brahe, a 16th century astronomer, collected data onthe movement of the Mars.
Johannes Kepler analysed that data which was consistent
with Mars moving in an elliptic orbit around the sun.
Inductive conclusion:Mars, and all other planets, move in elliptic orbits around theSun, with the Sun at one of the focal points of the ellipse.
Primary tool fortheory formation
Ullrich Hustadt Research Methods in Computer Science 88 / 94
Scientific method Intellectual discovery Problem solving Deduction Abduction Induction Process model
Intellectual discovery: Induction (2)
An incomplete set of observations can easily lead to incorrect inductive
http://find/8/13/2019 All Lectures about articles
73/383
An incomplete set of observations can easily lead to incorrect inductiveconclusions
Example:
All swans Ive ever seen are white Inductive conclusion: All swans are white
Ullrich Hustadt Research Methods in Computer Science 89 / 94
Scientific method Intellectual discovery Problem solving Deduction Abduction Induction Process model
Scientific method: A model
http://find/http://goback/8/13/2019 All Lectures about articles
74/383
Observations
induction
Theory Hypothesis
deduction
Predictions
test
New Observations
Confirm predictions?
no
yes
Theory
Observations
abduction
Fact Hypothesis
test
Ullrich Hustadt Research Methods in Computer Science 90 / 94
Scientific method Intellectual discovery Problem solving
Intellectual discovery: Problems
Deductive reasoning tells us that from A and A implies B we can
http://find/8/13/2019 All Lectures about articles
75/383
Deductive reasoningtells us that from A and A implies B we canconclude B
However, it cannot tell us whether A or A implies B holds, norwhether B is what we want to show
Abductive reasoningtells us that from B and A implies B we mayconclude A
However, it cannot tell us whether B or A implies B hold, nor how toestablish that A is the case
Inductive reasoningtells us that from A(o1), . . . , A(on) andB(o1), . . . , B(on) we may conclude x.A(x)B(x).However, it cannot tell us what the properties A( ) and B( ) are (nor
how large the number n needs to be)
To overcome these problems we need additional techniques.
Ullrich Hustadt Research Methods in Computer Science 91 / 94
Scientific method Intellectual discovery Problem solving
Problem solving
Analogy: Look for similarity between one problem and another one
http://find/http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
76/383
Analogy: Look for similarity between one problem and another onealready solved
Partition: Break the problem into smaller sub problems which are easierto solve
Random/Motivated Guesses: Guess a solution to the problem thenprove it correct
Generalise: Take the essential features of the specific problem and posea more general problem
Particularise: Look for a special case with a narrower set of restrictionthan the more general case
Subtract: Drop some of the complicating features of the originalproblem
Add: A difficult problem may be resolved by adding an auxiliary problem
Ullrich Hustadt Research Methods in Computer Science 92 / 94
Research classification Research methods
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
77/383
Research Methods in Computer ScienceLecture 6: Research methods
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 102 / 117
Research classification Research methods
Previously . . .
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
78/383
13 Scientific methodElements
14 Intellectual discovery
DeductionAbductionInductionProcess model
15 Problem solving
Ullrich Hustadt Research Methods in Computer Science 103 / 117
Research classification Research methods
Topics
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
79/383
16 Classifying research
17 Research methodsOverviewExperimentsQuestionnaires
Ullrich Hustadt Research Methods in Computer Science 104 / 117
Research classification Research methods
Classifying research (1)
Research can be classified from three different perspectives:
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
80/383
Research can be classified fromthree different perspectives:
1 Field
Position of the research within a hierarchy of topics
Example:Artificial IntelligenceAutomated Reasoning
First-Order Reasoning Decidability
2 ApproachResearch methods that are employed as part of the research process
Examples:Case study, Experiment, Survey, Proof
3 NaturePure theoretical developmentReview of pure theory and evaluation of its applicabilityApplied research
Ullrich Hustadt Research Methods in Computer Science 105 / 117
Research classification Research methods
Classifying research (2)
Pure theory:
http://find/http://-/?-8/13/2019 All Lectures about articles
81/383
yDeveloping theories and working on their consequences, with regard to
experimentation or applicationDescriptive studies:Reviewing and evaluating existing theories, including describing thestate of the art, comparing predictions with experimental data
Exploratory studies:Investigating an entirely new area of research, exploring a situation ora problemSee http://www2.uiah.fi/projects/metodi/177.htm
Explanatory studies:
Explaining or clarifying some phenomena or identifying the relationshipbetween things
Ullrich Hustadt Research Methods in Computer Science 106 / 117 Research classification Research methods
Classifying research (2)
Causal studies:
http://www2.uiah.fi/projects/metodi/177.htmhttp://www2.uiah.fi/projects/metodi/177.htmhttp://find/http://-/?-8/13/2019 All Lectures about articles
82/383
Assessing the causal relationship between things
Normative studies:Producing a theory of design (or of other development) likerecommendations, rules, standards, algorithms, advices or other tools forimproving the object of study
Problem-solving studies:Resolving a problem with a novel solution and/or improving somethingin one way or another
Development and Application studies:Developing or constructing something novel
Ullrich Hustadt Research Methods in Computer Science 107 / 117 Research classification Research methods Overview Experiments Questionnaires
Quantitative and qualitative research methods
Q i i h h d
http://find/http://-/?-8/13/2019 All Lectures about articles
83/383
Quantitative research methods
Methods associated withmeasurements(on numeric scales)Stemming from natural sciences
Used totest hypothesesor create aset of observationsfor inductivereasoning
Accuracy and repeatability of vital importance
Qualitative research methods
Methods involving case studies and surveys
Stemming from social sciences
Concerned with increasing understanding of an are, rather than anexplanation
Repeatability usually a problem
Ullrich Hustadt Research Methods in Computer Science 108 / 117 Research classification Research methods Overview Experiments Questionnaires
Research methods (1)
Action research
http://find/http://-/?-http://-/?-8/13/2019 All Lectures about articles
84/383
Action research:
Pursues action (or change) and understanding at the same timeContinuously alternates between action and critical reflection, while refiningmethods, data and interpretation in the light of the understanding developedin the earlier cycles
Example: Reflective teaching
Case study:
In-depth exploration of a single situation
Usually generates a large amount of (subjective) data
Should not merely report the data obtained or behaviour observed butattempt to generalise from the specific details of the situation observed
Example: Case study of open source software development
Ullrich Hustadt Research Methods in Computer Science 109 / 117 Research classification Research methods Overview Experiments Questionnaires
Research methods (2)
http://find/http://-/?-http://-/?-8/13/2019 All Lectures about articles
85/383
Survey:
Usually undertaken using questionnaires or interviewsQuestionnaire and interview design important!(See Dawson 2005 for details)Determination of sample size and sample elements important!(See specialist literature for details)
Example: Survey on the popularity or use of programming languages
Experiment:
Investigation of causal relationships using test controlled by the researcher
Usually performed in development, evaluation and problem solving projects
Example: Evaluation of processor performance
Ullrich Hustadt Research Methods in Computer Science 110 / 117 Research classification Research methods Overview Experiments Questionnaires
Key elements of an experiment
http://find/http://-/?-http://-/?-8/13/2019 All Lectures about articles
86/383
A precisehypothesisthat the experiment will confirm or refuteA completely specifiedexperimental system, which will be modified insome systematic way to elicit the effects predicted by the hypothesis
Quantitativemeasurementof the results of modifying the experimentalsystem
Use ofcontrolsto ensure that the experiment really tests the hypothesis
Analysisof the measured data to determine whether they are consistentwith the hypothesis
Reportof procedures and results so that others can replicate theexperiment
Ullrich Hustadt Research Methods in Computer Science 111 / 117 Research classification Research methods Overview Experiments Questionnaires
Key issues for questionnaires
http://find/http://-/?-http://-/?-8/13/2019 All Lectures about articles
87/383
Consider the following questions
What are the key issues for conducting a survey by questionnaire?
Regarding the questionnaire itself, what types of questions do you know
and what is each of them used for?
(7 minutes group discussion)
Ullrich Hustadt Research Methods in Computer Science 112 / 117
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
88/383
8/13/2019 All Lectures about articles
89/383
Research Methods in Computer ScienceLecture 7: Who is Who in Computer Science Research
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 118 / 143
Prizes and Awards
Scientific achievementis often recognised by prizesandawards
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
90/383
g y p
Conferencesoften give abest paper award, sometimes also abeststudent paper award
Example:IJCAR Best Paper PrizeFor the best paper, as judged by the program committee
Professional organisationsalso giveawardbased on varying criteria
Example:British Computer Society Roger Needham AwardMade annually for a distinguished research contribution in computer
science by a UK based researcher within ten years of their PhD.
Arguably, the most prestigious award in Computer Science is theA. M. Turing Award
Ullrich Hustadt Research Methods in Computer Science 119 / 143
Alan M. Turing (1912-1954)
Considered to be the father of modern computer science
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
91/383
Considered to be the father of modern computer science
In a 1936 paper introducedTuring machines, as athought experiment about the limits of mechanicalcomputationGives rise to the concept ofTuring completenessandTuring reducability
In 1939/40, Turing designed an electromechanical machine whichhelped to break the german Enigma codeHis main contribution was ancryptanalytic machinewhich usedlogic-based techniques
In the 1950 paper Computing machinery and intelligence Turingintroduced an experiment, now called theTuring test, to define astandard for a machine to be called sentient
Ullrich Hustadt Research Methods in Computer Science 120 / 143
Turing Award
TheA. M. Turing Awardis given annually by the Association forC ti M hi t
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
92/383
Computing Machinery to
an individual selected for contributions of a technical nature made tothe computing community. The contributions should be of lasting andmajor technical importance to the computer field.
Ullrich Hustadt Research Methods in Computer Science 121 / 143
Turing Award Winners
What contribution have the following people made?Who among them has received the Turing Award?
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
93/383
g g
Frances E. AllenLeonard M. AdlemanPaul BaranTimothy J. Berners-LeeVinton G. CerfEdgar F. CoddStephen A. CookLawrence J. EllisonDouglas Engelbart
William H. Gates IIIJames A. GoslingIrene Greif
Alan KayDonald E. KnuthRobin MilnerTheodor H. NelsonLawrence PageAlan J. PerlisAmir PnueliDennis M. RitchieRonald R. Rivest
Adi ShamirRichard M. StallmanKen Thompson
(12 minutes group discussion)
Ullrich Hustadt Research Methods in Computer Science 122 / 143
Turing Award Winners
What contribution have the following people made?Who among them has received the Turing Award?
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
94/383
Who among them has received the Turing Award?
Frances E. AllenLeonard M. AdlemanPaul BaranTimothy J. Berners-Lee
Vinton G. CerfEdgar F. CoddStephen A. CookLawrence J. EllisonDouglas Engelbart
William H. Gates IIIJames A. GoslingIrene Greif
Alan KayDonald E. KnuthRobin MilnerTheodor H. Nelson
Lawrence PageAlan J. PerlisAmir PnueliDennis M. RitchieRonald R. Rivest
Adi ShamirRichard M. StallmanKen Thompson
Ullrich Hustadt Research Methods in Computer Science 123 / 143
Turing Award Winners
What contribution have the following people made?Who among them has received the Turing Award?
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
95/383
Who among them has received the Turing Award?
Frances E. Allen Leonard M. Adleman Paul Baran Timothy J. Berners-Lee
Vinton G. Cerf Edgar F. Codd Stephen A. Cook Lawrence J. Ellison Douglas Engelbart
William H. Gates III James A. Gosling Irene Greif
Alan Kay Donald E. Knuth Robin Milner Theodor H. Nelson
Lawrence Page Alan J. Perlis Amir Pnueli Dennis M. Ritchie Ronald R. Rivest
Adi Shamir Richard M. Stallman Ken Thompson
Ullrich Hustadt Research Methods in Computer Science 124 / 143
The ones who havent made it (yet)
Paul BaranO f h h i f k i h d k l i h
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
96/383
One of the three inventors of packet-switched networks, along with
Donald Davies and Leonard Kleinrock in the early 1960s
Timothy J. Berners-LeeTogether with Robert Cailliau invented the World Wide Web in 1989
Lawrence J. Ellison
Co-founder and CEO of the database softwar company OracleWilliam H. Gates III
Co-founder, together with Paul Allen, of Microsoft; held thepositions of CEO, chief software architect, and chairman
James A. GoslingInvented theJava programming languagein 1994; devised theoriginal design of Java and implemented its original compiler andvirtual machine
Ullrich Hustadt Research Methods in Computer Science 125 / 143
The ones who havent made it (yet)
Irene GreifT h i h P l C h i d d h f C
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
97/383
Together with Paul Cashman introduced the concept of Computer
Supported Collaborative Work (CSCW) in 1984
Theodor H. NelsonCoined the term hypertext in 1963 and published it in 1965; theidea itself goes back to 1945, Vannevar Bushs Memex device
Lawrence PageCo-founder, together with Sergey Brin, of Google; developed thePageRank algorithm in 1998 on which Google is based
Richard M. StallmanSoftware freedom activist, hacker, software developer; lauched the
GNU Project in 1983
Ull i h H st dt R s h M th ds i C t S i 126 / 143
Frances E. Allen
Received the Turing award in 2006
F i i ib i h h d
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
98/383
For pioneering contributions to the theory and
practice of optimizing compiler techniques thatlaid the foundation for modern optimizingcompilers and automatic parallel execution.
First woman to receive the award
Her 1966 paper on Program Optimization and a 1971 paper with JohnCocke provide the conceptual basis for the systematic analysis andtransformation of computer programs
Work forms the basis for modern machine- and language-independent
program optimizersLead an IBM project which developed the concept ofprogramdependence graph, the primary structuring method used by mostparallelizing compilerstoday
Ull i h H t dt R h M th d i C t S i 127 / 143
Vinton G. Cerf, Robert E. Kahn
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
99/383
Received the Turing award in 2004For pioneering work on internetworking, including the design andimplementation of the Internets basic communications protocols,TCP/IP, and for inspired leadership in networking.
Led the design and implementation of the Transmission ControlProtocol and Internet Protocol (TCP/IP)
Basis for current internetworking
Ull i h H t dt R h M th d i C t S i 128 / 143
Alan Kay
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
100/383
Received the Turing award in 2003
For pioneering many of the ideas at the root ofcontemporary object-oriented programming languages,leading the team that developedSmalltalk, and forfundamental contributions to personal computing.
Development started in 1969, publicly available since 1980
First complete dynamicobject-oriented programming
Influenced the design ofC++ andJava
Included a completevisual programming environmentEnvisaged to be part of a user-centered approach to computing
Ull i h H t dt R h M th d i C t S i 129 / 143
Leonard M. Adleman, Ronald R. Rivest, Adi Shamir
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
101/383
Received the Turing award in 2002
For their ingenious contribution for making public-keycryptography useful in practice.
Created the worlds most widely used public-key cryptography system,
RSA, in 1977Clifford Cocks described an equivalent system in an internal GHCQdocument in 1973, but it was never deployed and kept secret until 1997
Ullrich Hustadt Research Methods in Computer Science 130 / 143
Douglas Engelbart
Received the Turing award in 1997
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
102/383
For an inspiring vision of the future of interactivecomputing and the invention of key technologies tohelp realize this vision.
Vision of a computer and communications based working environment
Invention of key tools and systems that helped start thepersonalcomputerrevolution:
Computer mouseMultiple on screen windowsLinked hypermediaShared screen teleconferencing and computer aided meetingsOnline publishing
Presented in 1968 as part of the mother of all demos
Ullrich Hustadt Research Methods in Computer Science 131 / 143
Amir Pnueli
Received the Turing award in 1996
F i l k i d i l l i i
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
103/383
For seminal work introducing temporal logic intocomputing science and for outstanding contributionsto program and system verification.
Major breakthrough in theverification and certification of concurrent
and reactive systems
Landmark 1977 paper The Temporal Logic of Programsin Proc. 18th IEEE Symp. Found. of Comp. Sci., 1977, pp. 4657.
Focus on ongoing behaviour of programs (rather than input/output
behaviour)Allows to easily specify qualitative progress properties of concurrent programs
Careful logic design enables automated verification of concurrent programs
Ullrich Hustadt Research Methods in Computer Science 132 / 143
Robin Milner
Received the Turing award in 1991
F h di i d l hi
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
104/383
For three distinct and complete achievements:1 LCF, the mechanization of Scotts Logic of
Computable Functions, probably the first theoreticallybased yet practical tool for machine assisted proofconstruction;
2 ML, the first language to include polymorphic type
inference together with a type-safe exception-handlingmechanism;
3 CCS, a general theory of concurrency.
In addition, he formulated and strongly advanced fullabstraction, the study of the relationship betweenoperational and denotational semantics.
Ullrich Hustadt Research Methods in Computer Science 133 / 143
Dennis M. Ritchie, Ken Thompson
Received the Turing award in 1983
F th i d l t f i ti t
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
105/383
For their development ofgeneric operating systemstheoryand specifically for the implementation of theUnixoperating system.
Development of UNIX began in 1969
Seminal paper published in 1973 on The UNIX Time-Sharing Systemat the Fourth ACM Symposium on Operating Systems Principles
UNIX was the first commercially important portable operating system
Usable (almost without change) across a wide range of hardware from
smartphones to supercomputers
Ullrich Hustadt Research Methods in Computer Science 134 / 143
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
106/383
Edgar F. Codd
Received the Turing award in 1981
For his fundamental and continuing contributions to
8/13/2019 All Lectures about articles
107/383
For his fundamental and continuing contributions tothe theory and practice of database managementsystems.
Developed therelational approachtodatabase management
Seminal paper published in 1970 on A Relational Model of Data forLarge Shared Data Banks
Provided the impetus for widespread research into numerous relatedareas, including database languages, query subsystems, database
semantics, locking and recovery, and inferential subsystemsOther contributions: Boyce-Codd Normal Form
Online analytical processing (OLAP)
Ullrich Hustadt Research Methods in Computer Science 136 / 143
Donald E. Knuth
Received the Turing award in 1974
For his major contributions to the analysis of
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
108/383
For his major contributions to the analysis ofalgorithms and the design of programming languages.
Author of the multi-volume book series The Art of ComputerProgramming
First volume published in 1968, seven volumes planned, currently working onfourth volume
One of the most highly respected references in the computer science field
Created the field of rigorousanalysis of algorithms
Creator of theTEX typesetting systemand of the Metafont font designsystem
Ullrich Hustadt Research Methods in Computer Science 137 / 143
Alan J. Perlis
First recipient of the Turing award in 1966
For his influence in the area of advanced
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
109/383
For his influence in the area of advancedprogramming techniques and compilerconstruction.
One of the developers of the ALGOL programminglanguage
Ullrich Hustadt Research Methods in Computer Science 138 / 143
How to become a Turing award winner
To increase your chances to become a Turing award winner it might beadvantageous to work in one of the following fields:
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
110/383
programming language design and implementation(Backus, Floyd, Hoare, Milner, Naur, Iverson (APL), Dijkstra, Naur,Perlis (Algol), Fortran (Backus), Pascal, Modula (Wirth), Dahl,Nygaard (Simula), Kay (Smalltalk))program compilation(Cocke, Perlis), program optimisation(Allen)
program verification(Floyd, Pnueli)analysis and theory ofalgorithms includingcomplexity theory(Blum, Cook, Hopcroft, Hartmanis, Knuth, Karp, Rabin, Scott, Stearns,Tarjan, Yao)theory and practice ofdatabases(Bachman, Codd, Gray)theory and practice ofoperating systems(Brooks, Corbato, Ritchie,Thompson, Lampson)artificial intelligence(Feigenbaum, Minsky, Newell, Reddy, Simon)
Ullrich Hustadt Research Methods in Computer Science 139 / 143
Practical 1 Reading research papers
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
111/383
Research Methods in Computer ScienceLecture 8: Reading research paper
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 143 / 159
Practical 1 Reading research papers
Topics
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
112/383
18 Practical 1
19 Reading research papers
Ullrich Hustadt Research Methods in Computer Science 144 / 159
Practical 1 Reading research papers
Todays questions
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
113/383
1 Each of you has compiled a list of five to ten concepts that you did notunderstand. Discuss those, see whether someone else in your groupcan give you an explanation, then, as a group, compile a short list ofconcepts that remain unclear
2 For each of the papers list at least three claims that they put forwardand note what evidence they provide to support those claims
(15 minutes group discussion)
Ullrich Hustadt Research Methods in Computer Science 145 / 159
Practical 1 Reading research papers
Practical 1: Claims and Evidence
Paper 1: A SAT-based decision procedure for ALC
http://find/http://-/?-8/13/2019 All Lectures about articles
114/383
Paper 1: A SAT baseddecision procedure for ALC
1 Ksatoutperforms KRISby several orders of magnitude empirical evidence on randomly generated samples
2 SAT-based decision procedures are intrinsically bound to be moreefficient than tableau-based decision procedures in contrast to SAT-based procedures, tableau-based proce-
dures consider the same truth assignment more than once
3 There is partial evidence of an easy-hard-easy pattern empirical evidence; easy-hard-easy pattern evident for
Ksat(but not KRIS)
Ullrich Hustadt Research Methods in Computer Science 146 / 159
Practical 1 Reading research papers
Practical 1: Claims and Evidence
Paper 2: On evaluating decision procedures for modal logic
http://find/http://-/?-8/13/2019 All Lectures about articles
115/383
1 Ksatdoes not qualitatively outperform KRIS withpre-processing empirical evidence on the same randomly generated sam-
ples, but using KRIS with pre-processing
2 Non-eager application of simplifications causes inferiorperformance ofKRIS (even with pre-processing) empirical evidence and analysis of behaviour ofKRIS
3 Easy-hard-easy pattern is an artificial phenomenon ofKsat empirical evidence; translation approach has no easy-hard-
easy pattern although it solves the hardest samples fasterthan Ksat
Ullrich Hustadt Research Methods in Computer Science 147 / 159
Practical 1 Reading research papers
Practical 1: Claims and Evidence
http://find/http://-/?-8/13/2019 All Lectures about articles
116/383
Paper 3: More evaluation of decision procedures for modal logics1 All the claims in Paper 1 are correct empirical evidence on a new, less flawed set of randomly
generated samples; repetition of the argument about truth
assignments2 KsatC also outperforms the translation approach empirical evidence
Ullrich Hustadt Research Methods in Computer Science 148 / 159
Practical 1 Reading research papers
Practical 1: Conclusion
Tableau methodsconstruct refutations bycase distinctionand theapplication ofdecomposition rules
http://find/http://-/?-8/13/2019 All Lectures about articles
117/383
SAT-based procedurescan be seen astableau methodsusinga specific kind ofcase distinctionanda specificstrategyfor the application ofdecomposition rules
Question is nottableau-basedvsSAT-basedprocedures, but
what kind ofcase distinctionis best (if any) andwhatstrategyfor the application ofdecomposition rulesisbest (if any)
These questions are still open
sets of modal logic/description logic expressions are infinitelack of real-world samples
Ullrich Hustadt Research Methods in Computer Science 149 / 159
Practical 1 Reading research papers
Practical 1: Learning points
Regardingresearch papers:
http://find/http://-/?-8/13/2019 All Lectures about articles
118/383
Notions need precise definitions
Claims need to be formulated unambiguously
Evidence needs to be constructed carefully
Regarding theresearch process:
Peer review does not prevent mistakes
Research can progress without resolving contradictions
Research is also a social process
Ullrich Hustadt Research Methods in Computer Science 150 / 159
Practical 1 Reading research papers
Reading research papers
http://find/http://-/?-8/13/2019 All Lectures about articles
119/383
Researchaims to add theworlds body of knowledge Requires a researcher to be aware of what the
worlds body of knowledge(in the area s/he works in)
Frontiersof theworlds body of knowledge arenotdocumented intext
books, but injournal articles
reliability conference papersworkshop papers timelinesstechnical reports
Ullrich Hustadt Research Methods in Computer Science 151 / 159
Practical 1 Reading research papers
Get organised
Maintain a database of all the books and papers you read
Data stored should at least include title, author, place of publication,
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
120/383
and storage locationPreferably you should also keep a record of the answers to some or all ofthe following questions:
1 What is the main topic of the article?2 What was/were the main issue(s) the author said they want to discuss?
3 Why did the author claim it was important?4 How does the work build on others work, in the authors opinion?5 What simplifying assumptions does the author claim to bemaking?
Ullrich Hustadt Research Methods in Computer Science 152 / 159
Practical 1 Reading research papers
Get organised
Maintain a database of all the books and papers you read
Data stored should at least include title, author, place of publication,
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
121/383
and storage locationPreferably you should also keep a record of the answers to some or all ofthe following questions:
6 What did the author do?7 How did the author claim they were going to evaluate their work and
compare it to others?8 What did the author say were the limitations of their research?9 What did the author say were the important directions for future research?
Ullrich Hustadt Research Methods in Computer Science 153 / 159
Practical 1 Reading research papers
Evaluating research papers
Whenever you read a research paper, you should try toevaluateat the
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
122/383
same time.Try toanswer the following questions:
1 Is the topic of the paper sufficiently interesting (for you personally or ingeneral)?
2 Did the author miss important earlier work?
3 Are the evaluation methods adequate?4 Are the theorems and proofs correct?5 Are arguments convincing?6 Does the author mention directions for future research that interest you?
Given the answers to these questions for a number of research papers,
you should be able to construct a research proposalby considering howyou could improve the work presented in them
Ullrich Hustadt Research Methods in Computer Science 154 / 159
Structure of research papers Hints
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
123/383
Research Methods in Computer ScienceLecture 9: Structure of research papers
Ullrich Hustadt
Department of Computer ScienceUniversity of Liverpool
Ullrich Hustadt Research Methods in Computer Science 160 / 183
Structure of research papers Hints
Previously . . .
http://find/http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
124/383
18 Practical 1
19 Reading research papers
Ullrich Hustadt Research Methods in Computer Science 161 / 183
Structure of research papers Hints
Topics
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
125/383
20 Structure of research papers
21 Hints
Ullrich Hustadt Research Methods in Computer Science 162 / 183
Structure of research papers Hints
Todays questions
Taking the research papers you have been given and others that you may
h i h (h f ll ) i l
http://find/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
126/383
have come across in the past as a (hopefully) representative sample,consider the following questions:
1 What elements constitute the structure of the papers? Are theelements and their order identical for all the papers? If not, whichelements do the papers have in common and which elements only
appear in only some of the papers?2 What characterises each of the elements of the papers? That is,
looking at each element of each of the papers, what do they havecommon?
(15 minutes group discussion)
Ullrich Hustadt Research Methods in Computer Science 163 / 183
Structure of research papers Hints
Structure of a research paper
1 Title
2 Li t f th ( d th i t t d t il )
http://find/http://goback/http://-/?-http://-/?-http://-/?-8/13/2019 All Lectures about articles
127/383
2 List of authors (and their contact details)3 Abstract
4 Introduction
5 Related Work (either part of or following introduction or before
summary).6 Outline of the rest of the paper
7 Body of the paper
8 Summary and Future Work (often repeats the main result)
9 Acknowledgements10 List of references
Ullrich Hustadt Research Methods in Computer Science 164 / 183
Structure of research papers Hints
Title
As short as possible, but without abbreviations or acronyms(unless they are commonly understood)
A ifi d l ibl
http://find/http://-/?-8/13/2019 All Lectures about articles
128/383
As specific as necessary and as general as possible(e.g.The Complexity of Theorem-Proving Procedures
introduced the notion of NP-Completenessstarting point ofcomplexity theory)
Include key phrases which are likely to be used in a search on the topicof the paper(e.g.modal logic, calculus, decision procedure)
Avoid phrases which are too common(e.g. novel)
Use phrases that describe distinctive features of the work(e.g.Real-world Reasoning with OWL)
Ullrich Hustadt Research Methods in Computer Science 165 / 183
Structure of research papers Hints
Authors (1)
Anauthorof a paper is an individual who1 made a significantintellectual contributionto the work described in the
paper
http://find/http://-/?-8/13/2019 All Lectures about articles
129/383
paper(in contrast, for example, to amonetary contribution);2 made a contribution todrafting, reviewing and/or revisingthe paper for
itsintellectual contribution(in contrast, for example, tospell checkingortypesetting); and
3 approved the final version of the paper including references
Some organisations / publishers have strict rules regardingauthorship
Order ofauthorsmay depend onsubject area: pure theory oftenalphabetical
applied research often based oncontribution
research assessment(e.g. bibliographic measures associating order with contribution)
cultural context
Ullrich Hustadt Research Methods in Computer Science 166 / 183
Structure of research papers Hints
Authors (2)
In Computer Science,academic degreesandmembership of professionalorganisationsare typically not indicated
List of authors is typically followed by contact information consisting of
http://find/http://-/?-8/13/2019 All Lectures about articles
130/383
List of authors is typically followed bycontact informationconsisting ofaffiliationande-mail address(not postal address)
Some journals allow authors to provide longer descriptions of themselvesincluding photographs
Ullrich Hustadt Research Methods in Computer Science 167 / 183
Structure of research papers Hints
Abstract
Typically not more than 100150 words
Should aim tomotivatepeople to read the paper
Highlight the problem and the principal results
http://find/http://-/?-8/13/2019 All Lectures about articles
131/383
Highlight theproblemand theprincipal resultsThe abstract will be included inliterature databases Make surekey phraseswhich might be used in searches are included
(same principle as for titles)
Keepreferencesto a minimumKeepequationsand othermathematical expressionsto a minimum
Ullrich Hustadt Research Methods in Computer Science 168 / 183
Structure of research papers Hints
Introduction
State the generalarea of research(unless this is obvious from the context in which the paper appears)
Introduce the problem /
http://find/http://-/?-8/13/2019 All Lectures about articles
132/383
Introduce theproblemstate why the problem is important and/or interesting
Outline theapproachtaken to solve the problem
Outline thesolutionorprincipal resultsstate why the results are important and/or interesting
Do not repeat theabstract
Avoid platitudes and cliches
Ullrich Hustadt Research Methods in Computer Science 169 / 183
Structure of research papers Hints
Related work
Related workis previous work by the same or other authors whichaddresses the same or closely relatedproblems/topics
Section on related work gives cr
Top Related