John Stinson Test Automation In Finance

16
• Challenges in Finance • How to Automate 1. Automation Tools 2. Getting the Basics Right 3. One Size Does Not Fit All 4. Build the Team 5. Coverage – Science or Craft? 6. Get Inside the Code! 7. Pick your Arguments 8. Incentivise Good Behaviour Test Automation John Stinson

Transcript of John Stinson Test Automation In Finance

Page 1: John Stinson Test Automation In Finance

• Challenges in Finance• How to Automate

1. Automation Tools2. Getting the Basics Right3. One Size Does Not Fit All4. Build the Team5. Coverage – Science or Craft?6. Get Inside the Code!7. Pick your Arguments8. Incentivise Good Behaviour

Test Automation

John Stinson

Page 2: John Stinson Test Automation In Finance

Challenges in Finance• Finance IS technology• Complexity is higher than ever before• Mergers have lead to accumulation of systems• Increased regulations have increased pressure• Teams are frequently spread globally• Volumes have increased 20 times since 1995• Latencies have decreased to milliseconds• Proliferation of custom software. Approval to use open-source takes

time.• Operations separated from developers via entitlements• No sharing of information via blogs or social media• Different personalities, some driving automation, some not• Automation may pay off in 2-3 years. Bonus cycle is annual

Page 3: John Stinson Test Automation In Finance

1) Automation Approaches• Server

– Unit Tests eg JUnit– Fluent Tests eg AssertJ, FEST – Message Record/Replay, eg Rational Integration Tester– Custom Frameworks– BDD: Cucumber Ruby or JVM– Prod Log File Replay

• GUI– Unit Tests – BDD: Specflow – Selenium (Web)– QTP, UFT, TestComplete– Bitmap eg Sikuli

CheaperTo

Maintain

- How easy to Maintain?

Page 4: John Stinson Test Automation In Finance

2) Get the Basics Right

• Stub the Interfaces!• Lock the Data Down• Sort Out the Environments!

Page 5: John Stinson Test Automation In Finance

3) One Size Does Not Fit All

Page 6: John Stinson Test Automation In Finance

Different Approaches for Different Sizes

Statement,Method,Class,

Component

System

Unit/Fluent/BDDMessage Replay,

Custom Framework,BDD

GUI

Page 7: John Stinson Test Automation In Finance

Google Testing SizesSimon Stewart

Page 8: John Stinson Test Automation In Finance

Different Approaches for Different Lines of Business

Page 9: John Stinson Test Automation In Finance

Different Approaches for Different Lifecycle Stages

Cutting Edge Proceed Cautiously

Stable Move Forward

Legacy Handle with Care Focus on Changes or Wrap with Cheap Regression

Page 10: John Stinson Test Automation In Finance

Dev/BA

BA/QA

BA/QA

Dev/Envt/Auto

Dev/Envt/Auto

Dev/Envt/Auto

Dev/Envt/Auto

Dev/BA

Dev/BA

Dev/BA

Dev in Test

Classic BDD

Developer in Test

Dev/BA

Dev/Envt/Auto

Dev/Envt/Auto

Dev/Envt/Auto

Dev/Envt/Auto

Dev/Envt/Auto

BA

Dev/Envt/Auto

Developer Quality Mentality

Gherkins

Developers automate

Dev in Test

Dedicated automators help

keep paceAutomation focused

Dev lead drives team behaviour

4) Team Models that Work

Devswrite

frame-work Dev in

Test

BA/QA

QAQA checks coverage

Page 11: John Stinson Test Automation In Finance

5) Coverage

Science or Art?

Page 12: John Stinson Test Automation In Finance

Science of Coverage

2*12*2*2*2*4*10 = 7680

Page 13: John Stinson Test Automation In Finance

Understand RiskPerform Exploratory TestingSummarise Coverage to Stakeholders

Art of Coverage

Page 14: John Stinson Test Automation In Finance

6) Get Inside the Code!

Diff Code to understand risk of late changesView Unit Test Code Coverage in IntellijFind interesting edge cases in C# using PEXAnalyse Dependency Graph in JavaAnalyse Performance with jvisualvm

Page 15: John Stinson Test Automation In Finance

7) Pick your Arguments

• Teamcity vs Jenkins?• Cucumber Ruby vs Cucumber JVM?• Config Management in SVN or GIT/Stash?• Certain level of unit test Coverage?• Code review process?

Page 16: John Stinson Test Automation In Finance

8) Incentivise Good Behaviour with Metrics

• What are we trying to achieve?

• How many test scenarios automated out of the manual pack?

• Unit test coverage?• Total time taken to test?• Production defects?