DD3001 - Research: Theory, Method, Practice...DD3001 - Research: Theory, Method, Practice Motivation...

DD3001 - Research: Theory, Method, Practice

Motivation

Hope DD3001 helps you avoid this situation

9-Oct-10 2

The Significance of Research

Am assuming you already avoid this

Obsolete Scientific Method

• Hypothesis

• Sequence of experiments.

• Change 1 parameter perexperiment.

• Prove/Disprove Hypothesis.

• Document for others to repro-duce results.

Computer Scientific Method

• Hunch

• 1 experiment &

change all parameters

• Discard if it doesn’t supporthunch

• Why waste time?

We know this.

Source: “How to have a bad career in research/academia” by David Patterson, Feb 2002.

Remember Science/Research is Hard

Why is science hard?

• Intrinsic - Our knowledge is imperfect. Makes it difficult tocorrectly update this knowledge.

• Personal - Scientists have biases and flaws in their approach.

• Community - The way we organize ourselves can limitprogress.

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Amherst

http://kdl.cs.umass.edu/courses/rmcs/schedule.html

Reason for Difficulty: Intrinsic

All possible theories

True theories


Amherst



True theories

Theories we think are true


Amherst


Much of what we consider today is probably wrong

“Our successors will be amazed by theamount of scientific rubbish discussedtoday – that is, if they have the patienceto trawl through the electronic archives ofobsolete journals.”

— Sydney Brenner, (1927–)

Sydney Brenner is a South African biologist and a 2002 Nobel prize in

Physiology or Medicine laureate.


True theories


Theories wethink we havetested well


Amherst


Truth perhaps beyond our current imagination

“..my own suspicion is that the Universeis not only queerer than we suppose, butqueerer than we can suppose”

— J.B.S. Haldane, (1892-1964)

J.B.S. Haldane was a British-born geneticist and evolutionary biologist.


True theories


Theories wethink we havetested well

Theories considered


Amherst


Facts are simple in retrospect

Somebody once observed to the eminent philosopher Wittgensteinhow stupid medieval Europeans living before the time ofCopernicus must have been that they could have looked at the skyand thought that the sun was circling the earth. Surely a modicumof astronomical good sense would have told them that the reversewas true. Wittgenstein is said to have replied:

“I agree. But I wonder what it wouldhave looked like if the sun had beencircling the earth.”

Source: James Burke. The Day the Universe Changed. TV Series 1985

Why we need help doing this?

• Isn’t science just “common sense”?

- Yes, but it is advanced common sense.

• Both lay observers and experts tend to...

- See patterns in random data.

- Generalize from incomplete samples.

- See confirmation for favored ideas in ambiguous data.

• We tend to believe...

- What we already think we know.

- What we hear second-hand.

- What our community believesSource: T. Gilovich. How We Know What Isn’t So. 1991.

Reason for Difficulty: Personal

Personal failings: the human condition

• Slop - Presenting and doing research so it is not possible toknow what was done or observed

• Confused or unclear procedures

• Confused or unclear data-collection

• Imprecise theorizing

• Unexpressed assumptions

• Informal derivation of predictions

• Sloth - Doing too little!

• Important data not obtained or recorded

• Partial analysis of the data

Source: Donald D. Jensen. Unpublished lecture notes, University of Nebraska-Lincoln. 1995.

Personal Failings: Self-deception

• Precipitance - Jumping to a conclusion

? Premature decision on an issue

? Accepting as established something that deserves further

investigation

• Propaganda - Biased presentation of a theory or data

? Special pleading

? Salesmanship rather than science.

• Prejudice - Biased evaluation of theory and data

? Expect more of other theories than of one’s own

? “Tilt the playing field” in favour of one’s own theory

• Preservation - Clinging to a theory despite clear evidencethat it is false.


Not a new idea (Francis Bacon 1600’s)

• Idols of the Tribe – We see evidence for our favouredhypotheses in random data.

• Idols of the Cave – Our personal likes and dislikes affect ourprofessional judgments.

• Idols of the Marketplace – Common language is too limitedfor accurate scientific communication.

• Idols of the Theatre – Poor methods and philosophicalsystems can doom scientific investigations.

Personal failings: Willful deception

• Finagle - “Adjusting” data so that it fits a favoured theory.

? Minor fraud.

• Filch - Stealing ideas or data without giving appropriate credit

? Plagiarism or other unauthorized use of the work of others.

• Fraud - Falsifying data and the reports of investigationsperformed.


Reason for Difficulty: Community

Community failings

• Science is a human community

• Like any other community, it has...

7 Fads and fashions

7 Rigid structures and traditions that may interfere with progress

7 Disagreements over goals and direction that consume resources

7 Outsize personalities who may detrimentally dominate the field

• Science progresses in spite of these problems.


Amherst


Community benefits

• Science is a human community

• Like any other community, it can...

X Share data, results and theories

X Quickly explore fads and fashions

X Give independent evaluation of one’s own theories and work

X Provide competition and deadlines

• Science progresses because of these benefits.

Haven’t scientists overcome these challenges?

• X Yes

• With practice

• When using tools developed over the past few hundredyears.

• 7 No when unaided and having little practice.

• Most undergraduate education trains you how to learnwhat is already known, not how to learn what is currentlyunknown or misunderstood.


Amherst


DD3001 - Research: Theory, Method, Practice

Course Overview

Answer to your questions

1. What is this course about?

We will review how research gets done and the important

components of your research career. More details...

2. Do I have to attend all the lectures?

No. But you do have to do all the assigned readings and homeworks.

Homeworks for non-attended lectures will be longer as you will have

missed the chance to participate in discussions during the lecture.

3. How can I access the reading material?

I have password protected the website containing the reading

material. I have put the password on the board.

Goals of DD3001

• Inform you about the research process −→ help yourburgeoning research career.

• Raise your awareness that progress in research is verynon-linear. And requires hard work and perseverance!

• Acquaint you with basic strategies for your research career.? Identify important and solvable research topics, questions and

hypotheses.

? Select between alternative research directions.

? Importance of communication via reading and writing papers.

• Make you consider some of the responsibilities of being amember of the research community.

• Teach you some strategies for clear and effective writing.

Outline of the lecture topics

1. Choosing a research topic.

? Strategy and Tactics for Discovery

2. You and your code; Reproducible Research.

3. Communication with other scientists:Reading & Reviewing papers.

4. Communication with other scientists:Paper writing.

5. Ethics.

6. Last two lecture will probably be devoted to paper editing!

In most of the lectures will devote some time to writing tips.

Papers and texts

• Papers & Essays

? General advice to young researchers

? Historical perspective on discoveries

? Stimulating papers for discussion

• Selected chapters from these books

? Advice to a Young Scientist by Peter Medawar.

? The Incomplete Guide to the Art of Discovery by Jack E.Oliver.

? Style - Lessons in Clarity and Grace by Joseph M.Williams.

Course structure

• Standard lectures interspersed with

? discussion of the pre-assigned readings,

? discussion of issues raised during the lectures,

? jotting down instant responses to questions.

What you will do in this course!

• Read some interesting articles and essays about science

• Think about their content.

• Reflect on their relevance to your thoughts about scientificresearch.

• Complete homeworks based on these thoughts and reflections.

• Write 2 essays to investigate an issue in more depth.

Details of assessment

Course is pass/fail.

To pass you must

• Read the Medawar book and write a ∼ 500 word response.

• Complete and submit the homeworks to an acceptable level.

• Complete and submit two essay assignment of sufficientmerit. See course website for details.

Details of assessment

Course is pass/fail.

To pass you must

• Read the Medawar book and write a ∼ 500 word response.

• Complete and submit the homeworks to an acceptable level.

• Complete and submit two essay assignment of sufficientmerit. See course website for details.

Remember you have to pass DD3001 to get your PhD.

Reading response homeworks

• At least 3 paragraphs

• Upload response to bilda.

• Content of response

? One paragraph summarizing main points in assigned reading.

? Make two or more key points that reveal the relevancy of thepaper to you or doing research.

Devote at least one short paragraph to each point.

• Focus does not have to be the main point of the paper. Butthe ones that are most relevant to you.

This course is a work in progress

• This course is evolving. So please be patient !

• However, hopefully together we can have an interestingjourney exploring the issues raised.

• Don’t expect the final word on how to do research.

• Expect ideas, partial and potentially conflicting answers

• Please engage in discussions. Question what I say. By correctprobing we should closer to the truth!

Selecting Research Projects & MakingDiscoveries

DD3001

October 8, 2014

Choose your research problem carefully

What problem to work on?

• Most important decision you are going to make as a researcher

• Don’t choose a problem that is

? trivial =⇒ trivial results

? too hard =⇒ years of frustration & little ac-complishment

? in a crowded field =⇒ hard to get noticed or be first

Are you working on a problem because...

• It was the first one that came to mind when you started.

• Suggestion made by a professor.

• It is related to what colleagues are working on.

Are you working on a problem because...

• It was the first one that came to mind when you started.

• Suggestion made by a professor.

• It is related to what colleagues are working on.

A good basis for making the choice on how you’ll spend yourtime for the next n years?

Careful deliberation is necessary

• Select a problem

? whose solution will be important to the flow of science

? that is capable of being solved by you given your skills,

experience and talents.

• As a young scientist

? this does not mean you have to limit yourself to problems of

modest significance.

? Remember

Significance not necessarily correlated with difficulty.

Tips for selecting a basic research project

1. Spend time and think very hard about the matter.

2. Do not settle for what happens to be at hand.

3. Choose the most fundamental significant and mostfundamental problem that may yield to your capabilities andendeavour.

Tips for selecting a basic research project

1. Spend time and think very hard about the matter.

2. Do not settle for what happens to be at hand.

3. Choose the most fundamental significant and mostfundamental problem that may yield to your capabilities andendeavour.

Do you agree? How did you make your choice?

!"#$%&'()*(+"&,-&(./0012(345&$(6*(7&8393$(.:;,&(<$5("4(5,&(!"-#=-&>2

6$"=3=?-?5@("4(!#%%&AA

63@"B("4!#%%&AA

CDE&%5&8(F3-#&

!"#$%&'()*+,-)'.++(&$)/&,01(23(

!"#$%#&'(')$*+,#

Source: C. Loehle (1990) after P. Medawar (“The Art of the Soluble”)

!"#$%&'()*(+"&,-&(./0012(345&$(6*(7&8393$(.:;,&(<$5("4(5,&(!"-#=-&>2

6$"=3=?-?5@("4(!#%%&AA

63@"B("4!#%%&AA

CDE&%5&8(F3-#&

!"#$%&'()*+,-)'.++(&$)/&,01(23(

!"#$%#&'(')$*+,#

How would you draw this curve?

Is your PhD research in the Medawar zone?Source: C. Loehle (1990) after P. Medawar (“The Art of the Soluble”)

Remember

• Competent research results can be ignored.

Why? 7 Worked on a trivial problem.

• Modest work can produce a paper of great consequence.

Why? X Brilliance is in the choice of problem and theapproach of the solution.

“A successful person isn’t necessarily better than her lesssuccessful peers at solving problems; herpattern-recognition facilities have just learned whatproblems are worth solving.”

– Ray Kurzweil

Interesting CS research problems

• Explore or explain the behavior of algorithms, systems,protocols, and other computational artifacts

• Typical form of research questions:

? “Why...”

? “How...”

? “Under what circumstances...”

? “What are the necessary and sufficient conditions for...”

• Answers

? 7 not yes/no

? Xparagraphs, mathematical or statistical models, simulations,

etc.


Amherst


Factors to consider

• Importance

? How important is the research topic within the larger research

and application community?

• State of Knowledge

? What do we know already?

? What is the position of the research with respect to “the

frontier”?

• Unique Competence

? Are you uniquely qualified to address this research?

? What is your “secret weapon”?

• Interest

? How much does this research problem interest you personally?

? Do you have a passion for this problem?Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Amherst


Assessing importance

• Audience

Who will care about the answer?

• Impact

Will different answers change...

? ...what research gets done next?

? ...what is done by practitioners?

• Longevity

How long will the answer be relevant and important?


Amherst


“It is fundamentally the confusion between effectivenessand efficiency that stands between doing the right thingsand doing things right. There is surely nothing quite souseless as doing with great efficiency what should not bedone at all.”

– Peter Drucker

How many CS talks begin

• Graph of rapid growth in...

? Processing power

? Heat generation

? Relative size or cost of caches

? Size of the Web

? Installed base of specific devices

? ....

• Why?


Amherst


“The Frontier”

• Important work typically takes place at the frontier of a field.

• You want your research findings to be relevant when they arepublished, not just now.

• The frontier in CS has two components:

? Changing face of computer science knowledge - a problem

common to all fields of science

? Changing environment of computer technology and

applications - a particularly challenging element of computer

science


Amherst


“I skate to where the puck is going to be, not where it hasbeen.”

– Wayne Gretzky

Methods for identifying frontiers

• New observations or findings that need explanation.

• New opportunities provided by instruments, methods of study,or theoretical frameworks

• Converging lines of research that combine to provide newopportunities.

• Emerging needs from outside the field (e.g., Web search,bioinformatics, ecological monitoring, autonomous vehicles)

• New questions or conjectures by researchers with good trackrecords.


Amherst


Is the frontier special in CS?

• Research groups often focus on very specific algorithms orsystems.Frequently one system per research group.

• But...

? ....Underlying concepts and fundamentals are the goal rather

than incidental algorithmic features.

? ...Exploratory work often converges.

? ...Core algorithms and principles emerge.

? ...Leaders in the field often agree on what work is central and

“at the frontier”.

• The frontier is always difficult to identify early on.


Amherst


“Good research is done with a shovel, not with tweezers...You should find an area where you can get a lot out of itfast.”

– Roger Needham

Unique competence

• What is it that makes you better able to address this researchquestion than others?

• What is your “secret weapon”?

• Poor answers

? “I am smarter than other people”

It is rarely true.

? “I will work harder, longer, or faster”

There is always someone who can do those things better than

you, and that “weapon” can cause unacceptable losses (harm

to family, friends, and personal happiness).


Amherst


“Always have a secret weapon – the biggest computer, aproblem imported from another field that others haven’theard of yet, a fact you stumbled on by being curiousabout everything, a friend who is smarter than you are.Anything, in short, that will give you an unfair advantagein getting there first.”

– Herbert Simon

Write down what your or your group’s secret weapon?

Personal interest

• Focus your work on areas of personal interest.

• You will work harder, smarter, and more creatively.

• Others will sense your strong interest and want to work withyou, support you work, and credit you with innovations.

• Personal interest can be fostered and destroyed.Track what affects your interest and use that knowledge

• One widely shared factor: Speed of progress.


Amherst


Review & Discussion of Tips in

The Incomplete Guide to the Art of Discovery,

Jack E. Oliver

Before we start

• Book assumes that researchers want to make discoveries.

• What are your motivations for completing a PhD?

? want to learn how to do research?

? want to make discoveries?

? want to understand something?

? want to tackle a certain problem?

? a fun job ?

? .....

• Whatever they are, if you remain in research then you’ll havea more successful and satisfying career if you are makingdiscoveries.

Strategies for Discovery

1. Don’t Follow the Crowd

2. Rebel, but Wisely

3. Strive to Enhance Serendipity

4. Avoid Science Eddies

5. Study the Earth, and the Science of Geology

6. Seek the Non-questions

7. See Your Era in Long-Term Perspective

8. Go with Intuition

9. Avoid Sidetracking to Trivia

10. Be Competitive, Be a Winner, Be First

11. Argue by Analogy

12. Vision, Hypotheses, and Objective Testing

13. The Strategy of Exploration for Understanding

Strategies for Discovery

Pick and write down your top 5

1. Don’t Follow the Crowd

2. Rebel, but Wisely

3. Strive to Enhance Serendipity

4. Avoid Science Eddies

5. Study the Earth, and the Science of Geology

6. Seek the Non-questions

7. See Your Era in Long-Term Perspective

8. Go with Intuition

9. Avoid Sidetracking to Trivia

10. Be Competitive, Be a Winner, Be First

11. Argue by Analogy

12. Vision, Hypotheses, and Objective Testing

13. The Strategy of Exploration for Understanding

Questions

• What strategies are missing from this list?

• Which ones would you remove?

• Why?

• Do they apply to you as a PhD student? Why?

Questions

• Have you seen these strategies in action by senior scientists inyour research field?

• Which ones?

Questions

• Have you got a long-term perspective of your field?

• What were people doing a decade or 2 decades ago in yourfield?

• What was the stimulus for a big discovery in your field?

Tactics for Discovery

1. Adapt and Adopt Instruments and Technique

2. Skim the Cream

3. Minimize Jargon

4. Speak (Listen) to the Earth, and It Shall Teach Thee

5. Go for the Spatial Pattern

6. Skim the Rest of the Volume, Any Volume

7. Do It Yourself

8. The Knack of the Fresh Perspective

9. Choose Your Problem Very Carefully

10. The Curve of Discovery

11. Overcoming the “Terminal” Paper

Tactics for Discovery

Which of these tactics do you or your group follow?

1. Adapt and Adopt Instruments and Technique

2. Skim the Cream

3. Minimize Jargon

4. Speak (Listen) to the Earth, and It Shall Teach Thee

5. Go for the Spatial Pattern

6. Skim the Rest of the Volume, Any Volume

7. Do It Yourself

8. The Knack of the Fresh Perspective

9. Choose Your Problem Very Carefully

10. The Curve of Discovery

11. Overcoming the “Terminal” Paper

Questions

• What tactics do you or your research group follow that arenot on the list?

• Write down examples where “Instruments and Technique havebeen adapted and adopted” by those in your field.

• What are the roots of the techniques you use in your researchfield?

Question

Tactics for Discovery 93

covery. If it seems to be the latter, and you are discovery bound,look at alternate specialties.

Availability of significant observations can be used as a mea-sure of position on the discovery curve. An abundance of unex-plained observations or opportunities to make important newobservations are indications that the discovery curve has not yetflattened. Judging whether a given set of observations is signifi-cant or important is not always easy. Sometimes, however, thedecision is obvious simply because of the large scale or broadscope of the feature under study or because of its role in a largercontext.

tQo

IME

Overcoming the 'Terminal Paper"As every scientist knows, science does not advance with thesteady and orderly progress that is sometimes implied in mediaaccounts of science for the nonscientist. Science is beset by fads,abrupt changes in direction, growth and decay of interest in aparticular subject, mundane activity, and sparkling, exciting,creative activity. The course, in fact, often seems helter-skelter,

Where do you think your research topic is on this curve?

Questions

• How do you keep a fresh perspective on your research?

• Do you think it is beneficial to read broadly?

• “Do It Yourself ”. In your opinion, what does this mean w.r.tyour field.

DD3001 - Research: Theory, Method, Practice...DD3001 - Research: Theory, Method, Practice Motivation...

Documents

Transcript of DD3001 - Research: Theory, Method, Practice...DD3001 - Research: Theory, Method, Practice Motivation...