xUnit Test Patterns

(Some)xUnit Test Patterns

(in practice)by Adam Czepil

Who needs these?

What kind of problems xUnit test patterns may help to solve? Who will benefit the most?

Goals of Test Automation

Code Smells

Principles of Test Automation

Behavior Smells

Projects Smells

Test Patterns

Is it a fish?

Or is it a mammal?

Revenons a nos moutons!

What is Defect Localization and why developers in good projects (almost) don’t use debugger?

Debugging is an art

The problem

The problem

We run the test (possibly manual test)

The problem

We run the test (possibly manual test)The test fails

The problem

We run the test (possibly manual test)The test failsWe have to figure out why it fails?

The problem

We run the test (possibly manual test)The test failsWe have to figure out why it fails?

The problem is that this can be very time consuming...

Fortunatelly we have extra-skilled developers ...

... who can find the problem very quickly ;-)

What we expect from very good tests is that

What we expect from very good tests is thatif the tests fail they provide information

What we expect from very good tests is thatif the tests fail they provide information detailed enoughso we can find the source of the failure quickly.

Defect Localization

ability to very quickly find the actual cause of test failure

Defect Localization is

Defect Localization is

one of the major goals of test automation!

Defect Localization is

Defect Localization is

one of the major advantages of unit tests over customer (functional) tests!

What to do to have good Defect Localization?

Frequent Debugging

When we deal with Frequent Debugging?

When we deal with Frequent Debugging?

When manual debugging is required to determine the cause of most test failures.

When we deal with Frequent Debugging?

When manual debugging is required to determine the cause of most test failures.

This is also true when instead of debugging we use extensive logging during testing.

Frequent Debugging...

Frequent Debugging...

reduces productivity!

Frequent Debugging...

reduces productivity! andmakes development schedules much less predictable!

Frequent Debugging...

reduces productivity! andmakes development schedules much less predictable!

because a single manual debugging session could extend the time required to develop the software by half a day or more.

What are the causes of Frequent Debugging?

What are the causes of Frequent Debugging?

Frequent Debugging is caused by

What are the causes of Frequent Debugging?

Frequent Debugging is caused by missing tests or

What are the causes of Frequent Debugging?

Frequent Debugging is caused by missing tests or lack of Defect Localization in our suite of automated tests.

We should be able to quickly determine the root cause of test failure based on

We should be able to quickly determine the root cause of test failure based on Assertion Message

We should be able to quickly determine the root cause of test failure based on Assertion Message or through the pattern of test failures

We should be able to quickly determine the root cause of test failure based on Assertion Message or through the pattern of test failures andknowledge about changes being made.

What are the main causes for no Defect Localization?

What are the main causes for no Defect Localization?

• not enough detailed unit tests compared to functional tests,

What are the main causes for no Defect Localization?

• not enough detailed unit tests compared to functional tests,• missing component tests (for a cluster of classes) that would

point out an integration error between classes.

What are the main causes for no Defect Localization?

• not enough detailed unit tests compared to functional tests,• missing component tests (for a cluster of classes) that would

point out an integration error between classes, • Infrequently Run Tests, because it causes that it is difficult

to point out the change that caused the tests to fail.

Very good solution for Frequent Debugging problem is...

Very good solution for Frequent Debugging problem is...

TDD – Test Driven Development

Very good solution for Frequent Debugging problem is...

TDD – Test Driven Development (or at least Test First)

What I want you to remember?

What I want you to remember?

Defect Localization is about finding quickly bug causing the tests to fail.

What I want you to remember?

Defect Localization is about finding quickly bug causing the tests to fail.

Unit tests are usually much better in Defect Localization than customer tests.

What I want you to remember?

Defect Localization is about finding quickly bug causing the tests to fail.

Unit tests are usually much better in Defect Localization than customer tests.

Frequent Debugging usually means that project is either missing tests or has poor Defect Localization.

What I want you to remember?

Defect Localization is about finding quickly bug causing the tests to fail.

Unit tests are usually much better in Defect Localization than customer tests.

Frequent Debugging usually means that project is either missing tests or has poor Defect Localization.

Use TDD as often as possible

WARNING! Too much debugging may broke your eyes! ;-)

Questions?

How NOT to automate manual tests?What are some differences between

manual and automated test-case? How to create automated test

comparing to manual test?

Manual vs. Automated Tests

Manual vs. Automated Tests

Let’s compare some aspects...

Manual vs. Automated Tests

Let’s compare some aspects...-time

Manual vs. Automated Tests

Let’s compare some aspects...-time,-fragility.

Time

Executing manual tests requires A LOT OF time from HUMAN

Time

Executing automated tests requires almost NO time from HUMAN

Since HUMAN time is precious and COMPUTER time is cheap we should care much about the first anddon’t have to worry too much about the latter.

Conclusions

Opposite to manual tests we can create unlimited number of regression test cases.

Fragility

When the test fails/stops in some non-final point HUMAN usually finds a way (work around) to continue its execution.

Fragility

When the test fails in some non-final point test execution STOPS.

HUMAN, although much slower than COMPUTER, is smarter test executor, especially when the test fails before all test conditions have been checked.

Conclusions

Opposite to manual tests we have to care about assertions order andfind a way of handling not executed assertions.

Verify One Condition per Test

Verify One Condition per Test principle is not the same asOne Assertion Per Test principle

Let’s see an example...

example-refactored.zipExample.zip

example example refactored

When (not) to use setup (@Before) and teardown (@After) in xUnit?

Verify One Condition per Test helps to give us good Defect Localization to pinpoint the source of the failure.

Another reason we Verify One Condition per Test is to Minimize Test Overlap

Another reason we Verify One Condition per Test is to Minimize Test Overlap, so that we have fewer tests to modify if we later modify the behavior of the SUT.

So? How NOT to automate manual tests?

So? How NOT to automate manual tests?

Don’t write long, complex test scenarios

So? How NOT to automate manual tests?

Don’t write long, complex test scenarios,instead Verify One Condition per Test

So? How NOT to automate manual tests?

Don’t write long, complex test scenarios,instead Verify One Condition per Test,but not necesserily one assertion per test.

Questions?

Hey! Test code is code too!

Is test code different than “regular” code? How to organize test cases into classes/files/directories? What about test utils/helpers? Where to keep them?

Questions?

There is a lot of „patterns” described in the book. Here are „maps” of them...

The Maps

Why it’s worthwile to study xUnit patterns?

Questions?

Thank you!Go there: http://xunitpatterns.com/Really! You don’t have to read everything there...