A Critical Look at “Best Practices”

Jon BachQE/PM Directorjobach@ebay.com

“Best Practices and Context-Driven: Building a Bridge”

From 2003, someone wrote…

Definition of “best”

“Of the most excellent, effective, or desirable type or quality.”

Mine: “Optimal available value”

Best Practices that aren’t

For a keynote, it’s best to have audience seating?

via @mariakedemo

Best Practice: Don’t be an asshole?

But wait…

“Someone has to be an asshole.

Polite people aren't going to change the industry

-- they made it the way it is.”

James Bach, 8/3/13

How can you argue with these?

Install the software before testing it– …but not best when testing install…– …but not best if you need a “dirty” config

Use a keyboard to type characters into a field– …but not best if a stylus is needed…– …but not best if you’re testing buffered inputs

When you find a bug, write a report– … but not best if the programmer is your brother

who is sitting right next to you

Construx

“Best practices are proven software engineering techniques, methods, lifecycles, etc. CxOne is based on software industry best practices provide insight and guidance into optimal selection, efficient deployment, and effective use of industry best practices.”

Construx (from their website)

“Best practices are proven software engineering techniques, methods, lifecycles, etc. CxOne is based on software industry best practices provide insight and guidance into optimal selection, efficient deployment, and effective use of industry best practices.”

…such as…?

“It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.”

“Gathering and agreeing on requirements is fundamental to a successful project.”

“Choosing the appropriate architecture for your application is key.”

“A best practice for constructing code includes the daily build and smoke test.”

“It is important to review other people's work.”

“It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded.” 

© Construx (from their website)

…best, but why…?

“It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.”

“Gathering and agreeing on requirements is fundamental to a successful project.”

“Choosing the appropriate architecture for your application is key.”

“A best practice for constructing code includes the daily build and smoke test.”

“It is important to review other people's work.”

“It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded.” 

© Construx (from their website)

…best how, and according to whom…?

It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.

Gathering and agreeing on requirements is fundamental to a successful project.

Choosing the appropriate architecture for your application is key.

A best practice for constructing code includes the daily build and smoke test.

It is important to review other people's work.

It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded. 

IBM – this link is broken, nice practice there

http://www.standishgroup.com/sample_research/chaos_1994_1.php

http://www.construx.com/Practicing_Earl/Context_Matters/

In fact, context matters with any "best practice". The word "best" doesn't make it immune to the reality of the context. While it may be best somewhere, it could be a "worst practice" for your project. How do you know the difference? You can't until you fully comprehend the context and that usually means trying the practice out.

A Construx employee writes critically about the term “Best Practices”, which goes against its own website (see previous slide).

Website bug … (10/15/13)

(link still broken as of 5/23/14)

Best Practice: Have Trained and Certified Test Teams

“Certification can establish the minimum and expected skills needed for test positions.”

“Introduction of ISTQB certification program is raising the standard for tester skills uniformly around the world.”

http://www.rbcs-us.com/images/documents/testing%20best%20practices.pdf

From Rex Black (RBCS Consulting) 2006

My opinion is that you do not need to be certified before you can be considered “skilled”. Certification is not a “practice”, nor does it build skill.

From STP: an article about not using “best practices” under the section titled “Best Practices”

http://www.softwaretestpro.com/tag/best-practices

Context-driven testing

http://context-driven-testing.com/

The value of any practice depends on its context.There are good practices in context, but there are no best practices.People, working together, are the most important part of any project’s context.Projects unfold over time in ways that are often not predictable.The product is a solution. If the problem isn’t solved, the product doesn’t work.Good software testing is a challenging intellectual process.Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

“It depends”

pendeo pendere pependi [to hang; to hang upon, depend on; to hang loose, hover; to be suspended, discontinued; be in suspense, uncertain, undecided].

http://catholic.archives.nd.edu

Me and Rob Sabourin

I said to Rob: “How about a keynote that’s context-driven? Maybe the content can come from my workshop about exploring best practices the day before I present the keynote?”

“I love this idea.”“This will be really cool.”“This has never been done before (that I know of).”

So… What practices will you explore? What are the pros and cons of each? What are some interesting contextual considerations?

“As a testing consultant, I don’t need the low-hanging fruit, I want the fruit that’s on top of the tree, that’s really hard to get!”“It would be cool to see the favorite 10 from the group.”“It would be cool to have an artifact -- a mind map, maybe?”

When hosting a workshop, it’s important that

you face the audience and talk to them.

Mission

Mission

Help me build the closing keynote:

“A Critical Look at Best Practices”

Tactics

Part IImagine that you are a council of elders. What so-called “best” software testing practices come to mind? Vote on a top 10 list.

Part IIPros and cons of each

Part IIIWhat should testers consider when choosing and evaluating practices?

Story

Who was here? What practices did we explore? How did we explore them? Is there a “top 10”? What are the nuances testers should consider

before using a practice? What are some real world examples and

experiences? What conclusions did we make? What artifact do we have to share? Advice to our colleagues on the other sessions.

Who

16 people (8 male, 8 female) 8 countries Testers, Test Managers, Consultants 2 years’ experience, 38 years’ experience Games, Insurance, Regulated, Government,

Banking, Anti-Virus, Big Data Dell, Nortel, Spotify, World of Tanks, Unity Neil Thompson…

From Neil’s paper…

“Best Practices and Context-Driven: Building a Bridge” - 2003

Would not call them “always-good practices” now, but “considerations for any context”

Brainstorm of “good” practices

Full list (64)

Learn how to tell a good testing story Use a test management system Carefully choose testing tools Vary your test techniques Bug advocacy: learn how to sell bugs Work smarter, not harder Make it clear to PM how much it costs to test

or deliver proper software Have Dev test it first before it goes to Test

Full list (64)

Don’t spend time doing things a machine can do better

Make sure test coverage is approved by a stakeholder

Plan the test environment and data needs early

Document enough detail to save effort Work as a team an communicate Design code to be maintainable Use exploratory testing

Full list (64)

Test early Be able to explain the value you add Work with Developers to know who tested

what Work with Dev to build in testability and

logging Weak or unclear requirements will cause

problems Prioritize Always add time to estimates

Full list (64)

Be aware of your assumptions Certify the testers Do research -- self education is important Become part of the community Create clear exit / entry / stop criteria Give yourself more time You can always find bugs in your software Be aware (observation vs. reference) Treat bugs as something positive

Full list (64)

Be aware that metrics can be dangerous Be aware of pitfalls when communicating

metrics Talk the same language (“test”, “integration”) Don’t let Dev verify bugs Use Session-Based Test Management Test software Check fixed bugs against new versions Ask questions

Full list (64)

Send testers to Per Scholas Never trust developers Talk to each other Don’t plan too much; execute as well Get feedback often Write clear descriptions of bugs Use daily standups Automate Use prototypes

Full list (64)

Communicate with end users Consult a variety of stakeholders Get good at using your product Be aware of your emotional responses Don’t be afraid of failing Gain and apply experience Look out for the unexpected Assume specs are incomplete Expect that you will never have time enough

Full list (64)

Focus and defocus Think critically Use checklists Perform smoke tests first Use a bug-tracking system Discuss issues with Developers first before

reporting Try to be effective and efficient

Vote on your 3 favorites

PICTURE

Top 10

Ask questions Talk to each other Think critically Test early Learn how to tell a good testing story Don't waste time doing things a machine can do

better Work as a team Be able to explain the value you add Don't be afraid of failing Prioritize

Now let’s see how these so-called “good” practices can go wrong…

“Ask questions” -- pitfalls

Asking too many questions could be annoying Could take too much time from stakeholders Could offend political sensibilities The answer might provoke hidden agendas Not understanding importance of who you ask the question of Forgetting that you asked the same question of the same

person The person being asked does not want to reveal they do not

know the answer Did you ask enough questions? Someone could give the wrong answer, but you don't know

“Talk to each other” pitfalls

Meetings for the sake of meetings Conflicts can come up from talking to each other Risk of too much information / confirmation bias Risk of interrupting people who do not want ot be interrupted Cultural differences could provoke conflict Speak too much -- garrulous Annoying volume to other colleagues Not easy with distributed teams, tech and time zones, language Unintentionally spread confidential info Can forget or have a lack of formal agreement what we talked

about which means wasted time Book: “Quiet -- The Power of Introverts in a

World That Can’t Stop Talking

“Think critically” pitfalls

Don't think -- court case testimony Acceptance testing Testers can go too deep for minor issues Don't take your critical thinking home Dangerous for your health -- deep psychology

problems, nothing is good enough Can be seen as over-perfectionism Can be seen as criticism Can be seen as culturally insubordinate

“Test early” pitfalls

Testers report invalid bugs because product isn't ready

If you don't have full specifications yet Project isn't yet approved by official

stakeholders There are dependencies for certain types of

testing (integration) -- preconditions Code base is too unstable, and code base

rewritten in two weeks time anyway no matter what you find

Learn to tell a good testing story -- pitfalls

Since learning takes practice, you could lose credibility as you learn

The story isn’t necessary

Don't waste time doing things a machine can do better -- pitfalls

It make take you some time to figure out that it couldn't be automated

Takes too much time to automate cases when you can just check it by hand -- machine can do it better, but it's not worth

Maybe its faster to do it manually for now BVT manual at first – easy, quick, painless to

run UNTIL the build gets more mature and iterates every hour

“Work as a team” pitfalls

Difficult to work as a team when the team dynamics might change

Meetings might be forced on people who already know how to do it individually

Einstein worked alone Sometimes you need to remove yourself

from the team in order to focus

“Work as a team” pitfalls 2

Autism / Aspergberger's will get very stressed

Dependency on each person of the team, varies on confidence, role, responsibility

Maybe that you don't get along socially with the others on the team; less effective when negative energy

If more than one "leader" on the team cause conflict

Introverts

“Be able to explain the value you add” pitfalls

Don't have to do that if it's known already and you know

There are intrinsic things that may not need to be explained

You may never know the value that you really add anyway

You don't have to justify the job because the existance of the job itself has value

“Don’t be afraid of failing” pitfalls

Some healthy fear is reasonable Don't be cocky Fear drives people to be better ("Expert

Game") and do research Testing books on the shelf

“Prioritize” pitfalls

Wasted energy because everything is equal and needs to be tested anyway

Pressure to prioritize may cause you to incorrectly prioritize

If the test approach is galumphing, prioritization is wasted effort

If you're in modeling mode, there's nothing you need to do to prioritize

There's an opportunity cost to any prioritize

Considerations for practices

Social interactions Knowing what story you need to tell

(depending on the domain of the project) Time How do you be not afraid of failure when your

software is Mission critical? Size of project (2 or 200) Budget Locale

Considerations

Lifecycle of project (Agile, waterfall, v-model) Your own technical skill Domain Legal (regulatory) Cultural Gender

In closing

Best Practice lists hide ignorance and incompetence; but make money because consultants get paid for quick wins,

BUT….

sometimes we just want to be told what to do Brainstorming practices that provide value is

easy; scrutinizing those practices can be difficult, exhausting, and even disheartening

You have a tester mindset; scrutinize any so-called “best practice” you want to get rid of.

I’m done. Let’s rumble…