Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission...
Transcript of Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission...
![Page 1: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/1.jpg)
Winning the Battle against Automated Testing
Elena LaskavaiaMarch 2016
![Page 2: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/2.jpg)
Quality
People Process Tools
Foundationof Quality
![Page 3: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/3.jpg)
Development vs Testing
Developers don’t test
Testers don’t develop
Testers don’t have to be skilled
Separate Developers and Testers
Make the Test team responsible for quality
![Page 4: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/4.jpg)
One TeamQuality is a team responsibility
![Page 5: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/5.jpg)
The Process
When quality is bad let's add more steps to the process
![Page 6: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/6.jpg)
Story about the broken Trunk
Thousands of developers
Continuous stability is a must
“Trunk is broken” too often
Huge show stopper for R&D
People did root-cause analysis
Came up with Improved Process
![Page 7: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/7.jpg)
“Improved” Pre-Commit
Process
Repeat for all hardware variants
Manually execute sanity test cases
Re-build and re-deploy whole system
Clean compile All
Pull/Update All Source
![Page 8: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/8.jpg)
Trunk is still broken. Why?
Process was not followed
Process is too complex
Process is too boring
Process is too time consuming
Environment / Hardware limitations
Developers don’t know about the process
Developers are lazy
![Page 9: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/9.jpg)
AutomatedPre-Commit Testing
![Page 10: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/10.jpg)
Pre-Commit Tests with Source Management System
Fix Push Checks
master
● Peer reviews
● Robots checks
![Page 11: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/11.jpg)
AutomationHack
Oh well we don’t have any more budget and time, let go back to manual testing
It does not work at all now!
Oops our tester quit, who knows how to run it?
Need a person to run it for every build
Spend 6 month developing testing framework
Randomly pick a tool
Let's slap on some automation!
![Page 12: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/12.jpg)
Continuous TestingContinuous Quality
![Page 13: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/13.jpg)
Cost of Automation
• Cost of Tools
• User Training
• Integration and Customization
• Writing Test Cases
• Executing Test Cases
• Maintaining Test Cases
![Page 14: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/14.jpg)
Jump Start
Make one team responsible
Setup continuous integration
Add pre-commit hooks
Establish simple self-verifying process
Add one automated test
![Page 15: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/15.jpg)
Key Principlesof successful automated testing
![Page 16: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/16.jpg)
Gate Keepertest system must guard the gate
![Page 17: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/17.jpg)
100% Success100% of tests must pass. zero tolerance
![Page 18: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/18.jpg)
NO random failures
• Remove such tests from automation
• Use repeaters to keep intermittent tests
• Be prepared for the noise
• Modify AUT to remove source of randomness for tests
![Page 19: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/19.jpg)
Fully Automated
• No monkeys pushing buttons to start the testing
• No monkeys watching automated UI testing
• Hooks on code-submission (pre-commit, fast)
• Hooks on build promotion (overnight)
![Page 20: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/20.jpg)
Fast and Furioius• Feedback for pre-commit <=10 min
• Overnight is absolute maximum
• More tests degrade the system response time
• Not all tests are born equal!
• Use tagging and filtering
• Distribute or run in parallel
• No sleeps
![Page 21: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/21.jpg)
Timeouts
• Make sure tests are not hanging!
• Use timeouts
• Use external monitors to kill hanging runs
• Do not overestimate timeouts
![Page 22: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/22.jpg)
Test Scripts are Programs
• Automated test cases are programs
• Treat them as source code
• They must be in text form
• They must go to same version control system
• Subject to code inspection, coding standards, build checks, etc
![Page 23: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/23.jpg)
Unit Tests • Mandatory with commit
• Use servers to run
Part of the process
• Use a mocking framework
• Use UI bot
• Use test generators
• Inline data sources
Easy to write
![Page 24: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/24.jpg)
Unit
• Unit tests cannot cover all
• Test actual installed AUT
• Run the program as user would
• Use same language for unit and integration testing
Integration
![Page 25: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/25.jpg)
Pick and Choose
• Difficult to set-up cases
• Rare but important scenarios
• Check lists
• Module is actively developed
• Long maintenance expected
Candidates
you should not automate everything
![Page 26: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/26.jpg)
Self-Verification: test the test system?
Automatically Check
Code submission is properly formatted (has bug id, etc)
Code submission has unit tests
Total number of tests is increased
Performance is not declined
Code coverage is not declined
![Page 27: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/27.jpg)
Failed Battles
![Page 28: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/28.jpg)
Tools we used or evaluated and failed
• after 3 month of writing tests realized that it won’t work on Linux
WinRunner
• was pretty good until it was bought and it stopped launching with new eclipse
WindowTester
• 4 years ago: database, no text for tests, no integration
Jubula
• slow, not debuggable, blocks on support. python.
Squish
• domain specific language
RCPTT
![Page 29: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/29.jpg)
Working Solution
![Page 30: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/30.jpg)
Continuous Integration
Tools
• Unit testing
JUnit
• Source Control and Code Review
Git/Gerrit
• Static Analysis
FindBugs
• Build System
• maven-surefire-plugin (for unit tests)
• maven-failsafe-plugin (for integration tests)
• findbugs-plugin for static analysis
Maven/Tycho
• Continuous Integration Server
• Gerrit Trigger plugin - pre-commit builds and voting
• FindBugs plugin - reports and status
• JUnit plugin - reports and status
Jenkins
• gui testing
SWTBot
• junit mocking
Mockito
• Code Coverage
Ecl Emma
• Lots of custom libraries, frameworks and bots
Custom
![Page 31: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/31.jpg)
Tips and Tricks
![Page 32: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/32.jpg)
Auto-BotsChecks that can be added to every test
App crashed during a test
Test timeout exceeded
App generated unexpected log
Temp files were not cleaned up
Resource or memory leaks
Runtime error detection
![Page 33: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/33.jpg)
AutoBots: Junit Rulespublic class SomeTest {
// tests that we don’t leave tmp file behind (this is custom rule not
// base junit)
@Rule TmpDirectoryCheck tmpRule = new TmpDirectoryCheck();
@Test
void testSomething(){
}
}
// base class with timeouts
public abstract class TestBase {
public @Rule Timeout globalTimeout = Timeout.seconds(1); // 1 second
}
![Page 34: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/34.jpg)
Jenkins Split VerifiersRegression
testing Linux
Regression testing
Windows
Static Analysis
+1 verify
To speed up verification
for pre-commit hooks
set up multiple jobs
which trigger on the
same event (i.e. patch
submitted)
![Page 35: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/35.jpg)
Inline Data Sources: Comments in Java
// template<typename T2>
// struct B : public ns::A<T2> {};
// void test() {
// B<int>::a;
// }
public void testInstanceInheritance_258745() {
getBindingFromFirstIdentifier("a", ICPPField.class);
}
![Page 36: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/36.jpg)
Code Coverage• Run tests with code coverage
• Not during pre-commit check
• Unless it has validation hooks
• Good tool for unit test design (IDE)
• Never ask for 100% coverageCode Coverage ->
Select Tests
Based on changed code
exclude tests that do not cover
the changes
![Page 37: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/37.jpg)
Static Analysis• Can be run independently
• Has to be a gatekeeper
• Spent time to tune it (remove all noisy checkers)
• Push to desktop (if running as you type - instantaneous feedback!)
• Use alternative UX on desktop (i.e. code formatter)
Jenkins Plugin:
Code Reviewer
Post defects from static
analysis as reviewer
comments on the patch
![Page 38: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/38.jpg)
Tagging and Filtering: Junit Categories// tag class with categories in test class
@Category({PrecommitRegression.class, FastTests.class})public class SomeClassTest {@Testpublic void someTest() {
}}
// in maven pom.xml
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<groups>com.example.PrecommitRegression</groups></configuration>
</plugin>
</plugins>
</build>
![Page 39: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/39.jpg)
Runtime Filtering: Junit Assume
// skip test entirely if not running in osgi
@Beforepublic void setUp() {
Assume.assumeTrue( Activator.isOsgiRunning() );}
![Page 40: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/40.jpg)
Intermittent Test: Junit Rule
public class SomeClassTest {public @Rule InterimittentRule irule = new InterimittentRule();
// repeat this up to 3 times if it failing@Intermittent(repetition = 3)@Testpublic void someTest() {}
}
You can create a runner or define a rule which repeats a test if it fails.
Junit itself does not define either, you have to add it yourself (2 classes, 62 lines) of code).
![Page 41: Winning the Battle against Automated Testing · automated UI testing •Hooks on code-submission (pre-commit, fast) •Hooks on build promotion ... frameworks and bots Custom. Tips](https://reader036.fdocuments.net/reader036/viewer/2022062414/5ede978bad6a402d6669eb2e/html5/thumbnails/41.jpg)
The End
One Team
Simple Process
Right Tools