11

Bert GunterBert GunterGenentechGenentech

April 2005April 2005

The Human Side of The Human Side of Statistical Consulting*Statistical Consulting*

*With Apologies to Doug Zahn and James Boen, who authored a book with this title.

22

DisclaimerDisclaimer

These are my personal views and These are my personal views and do not necessarily reflect the do not necessarily reflect the opinions or policies of my opinions or policies of my department, my colleagues, department, my colleagues, Genentech or your local sponsorsGenentech or your local sponsors

33

OutlineOutline

Psychology and SociologyPsychology and Sociology

StatisticsStatistics

PracticePractice

44

-ology-ology

There are There are nono statistical problems statistical problems• Only engineering, scientific, medical, Only engineering, scientific, medical,

finance, ... problems that require finance, ... problems that require statistical design and data analysisstatistical design and data analysis

• Someone else always owns the data Someone else always owns the data or is responsible for doing the or is responsible for doing the experiment or studyexperiment or study

• If you don’t like science, you shouldn’t If you don’t like science, you shouldn’t be doing statisticsbe doing statistics

55

ConsequencesConsequences

Learn more about the subject Learn more about the subject mattermatter• It is It is your your responsibility to understand responsibility to understand

what the important subject matter what the important subject matter issues are and apply appropriate issues are and apply appropriate “statistical” approaches to them “statistical” approaches to them (which could and often does mean a (which could and often does mean a good graph or two, as Bill Forrest also good graph or two, as Bill Forrest also emphasizes).emphasizes).

66

Success depends on:Success depends on:• Collaboration and communicationCollaboration and communication

To define the essential issuesTo define the essential issues

To determine what data might shed light on themTo determine what data might shed light on them

To determine useful analyses of the dataTo determine useful analyses of the data

To communicate the results of those analysesTo communicate the results of those analyses

• ImaginationImaginationBeware of the obvious solutionBeware of the obvious solution

But also beware of reinventing wheelsBut also beware of reinventing wheels

These are at least as important as mere These are at least as important as mere technical knowledgetechnical knowledge

77

The most important type of The most important type of statistical error is not I or II, but III:statistical error is not I or II, but III:• Right answer – Right answer – wrong questionwrong question

John Tukey: “An approximate answer to the right John Tukey: “An approximate answer to the right question is worth a great deal more than a precise question is worth a great deal more than a precise answer to the wrong question”answer to the wrong question”

George Box: “All models are wrong; but some are George Box: “All models are wrong; but some are useful.”useful.”

88

HR HR

Everything is personalEverything is personal• Subject matter experts/investigators Subject matter experts/investigators

rarely fully understand the statistical rarely fully understand the statistical issuesissues

• Hence, their acceptance of Hence, their acceptance of youryour methods for dealing with methods for dealing with theirtheir problem and data is based on problem and data is based on trusttrust

99

You should ...You should ...

CareCare• What is the context?What is the context?• Can we do it better than the way it is Can we do it better than the way it is

“usually” done?“usually” done?

TeachTeach• At right levelAt right level• Mostly informallyMostly informally

SellSell• the value of statistical methodsthe value of statistical methods

Share your enthusiasm. Statistics is not Share your enthusiasm. Statistics is not a spectator sport – get involved!a spectator sport – get involved!

1010

... (suggested by David Giltinan)... (suggested by David Giltinan)

Be wise – compromise!Be wise – compromise!

But if you must dig in, choose your But if you must dig in, choose your battles wiselybattles wisely

Find good people to work with – Find good people to work with – and nourish the working and nourish the working relationshiprelationship

1111

-ics-ics

AllAll problems are statistical problems are statistical• Experimental design and data Experimental design and data

analysis are part of the warp and weft analysis are part of the warp and weft of scienceof science

All experiments are designed; the All experiments are designed; the only question is whether well or only question is whether well or poorly.poorly.

1212

Consequences ... Consequences ...

You are a professional, so ...You are a professional, so ...• Be proactive in your collaborationBe proactive in your collaboration• Help the investigator Help the investigator askask the right question the right question• Emphasize importance of good designEmphasize importance of good design• Never accept data at face valueNever accept data at face value

How were they obtained?How were they obtained?

What systematic sources of variability might What systematic sources of variability might mislead?mislead?

• Never assumeNever assumerandomizationrandomization

replicationreplication– at what levels of the variability hierarchyat what levels of the variability hierarchy

relevance of past experiencerelevance of past experience

1313

In sum, In sum,

Do not undervalue what you can Do not undervalue what you can contributecontribute

Statisticians are trained to understand Statisticians are trained to understand and anticipate how variability, which is and anticipate how variability, which is an inherent part of all natural an inherent part of all natural phenomena, can affect observations of phenomena, can affect observations of reality. This gives us powerful insight that reality. This gives us powerful insight that many scientists do not possess. We need many scientists do not possess. We need to use that insight to “help catalyze the to use that insight to “help catalyze the scientific learning process.” (George Box)scientific learning process.” (George Box)

1414

But ... But ...

Do not overvalue it eitherDo not overvalue it either• Prior knowledge and experience Prior knowledge and experience

matter even if they cannot be easily matter even if they cannot be easily captured and quantifiedcaptured and quantified

• All relevant information does not All relevant information does not reside in the data at handreside in the data at hand

Frequently, very little does.Frequently, very little does.

1515

PracticePractice

Collaborators, colleagues, co-Collaborators, colleagues, co-workersworkers• Not “clients”Not “clients”• Attitude makes a differenceAttitude makes a difference

1616

Turf:Turf:

Whenever possible, meet on theirs, Whenever possible, meet on theirs, not yoursnot yours• CBWA: “Collaborating by Walking CBWA: “Collaborating by Walking

Around”Around”• Touch the equipment, meet the Touch the equipment, meet the

subjects, observe the critters, try out subjects, observe the critters, try out the product, ...the product, ...

1717

Rules to Practice ByRules to Practice By

1818

Rule 1:Rule 1:

NEVERNEVER give advice over the give advice over the phone.phone.• It will always be badIt will always be bad

1919

Rule 2Rule 2

NEVERNEVER answer when asked how answer when asked how many.many.• It is almost always the wrong questionIt is almost always the wrong question• The right question has to do with The right question has to do with

defining the goals of the experimentdefining the goals of the experimentTypically, you will have to help the Typically, you will have to help the investigator figure this out.investigator figure this out.

2020

Rule 3Rule 3

Always interrogate the Always interrogate the measurementmeasurement• Systematic sources of measurement Systematic sources of measurement

variability aboundvariability abound• Ivestigators rarely know how to Ivestigators rarely know how to

quantify themquantify themBut they often can tell you what could be But they often can tell you what could be therethere

2121

Rule 4 (suggested by Lisa Bernstein)Rule 4 (suggested by Lisa Bernstein)

Get the raw dataGet the raw data• Many instruments, vendor-supplied/user-Many instruments, vendor-supplied/user-

created software “preprocess” the data created software “preprocess” the data in ad hoc, crazy ways devised by folks in ad hoc, crazy ways devised by folks with no statistical trainingwith no statistical training

Producing irretrievable junk that no Producing irretrievable junk that no subsequent analysis can redeemsubsequent analysis can redeem

• Often difficult to get “raw data” and Often difficult to get “raw data” and difficult to deal with when you have difficult to deal with when you have themthem

e.g. imagese.g. images

2222

Rule 5Rule 5

Thermodynamics: Disorder rules Thermodynamics: Disorder rules unless you work hard to defeat it. unless you work hard to defeat it. So...So...• Take nothing for grantedTake nothing for granted• Provide explicit step by step Provide explicit step by step

instructions, data format instructions, data format specifications, ...specifications, ...

• Remember MurphyRemember Murphy

2323

Some useful things I’ve learnedSome useful things I’ve learned

Most science is about hypothesis Most science is about hypothesis generation, not hypothesis testinggeneration, not hypothesis testing

All scientists have strong priorsAll scientists have strong priors

All variation is causedAll variation is caused

Beware of the data that aren’t there Beware of the data that aren’t there (suggested by Bill Forrest)(suggested by Bill Forrest)

All replicates are not created equalAll replicates are not created equal

You can never know too much You can never know too much statisticsstatistics• But most of what you learn is sillyBut most of what you learn is silly

2424

A few more ...A few more ...

Listen more, talk lessListen more, talk lessAlways provide an “executive summary” Always provide an “executive summary” of your results in a graph or two and a of your results in a graph or two and a paragraph or so of textparagraph or so of textA little paranoia can be good thingA little paranoia can be good thingIt’s better to lead the parade than It’s better to lead the parade than sweep up after the elephants (good sweep up after the elephants (good design is more important than fancy design is more important than fancy analysis)analysis)If you don’t think it’s ethical, don’t do itIf you don’t think it’s ethical, don’t do it• Your job is to speak for the data – integrity is Your job is to speak for the data – integrity is

everything!everything!

2525

And most important ...And most important ...

HAVE FUN !HAVE FUN !• (else why do it?)(else why do it?)