Too Dependent on Shared Test Environments? Kick Start Local Workstation Testing!
-
Upload
thoughtworks -
Category
Technology
-
view
731 -
download
0
description
Transcript of Too Dependent on Shared Test Environments? Kick Start Local Workstation Testing!
© ThoughtWorks, 2009
Too Dependent on Shared Test Environments? Kick Start Local Workstation Testing!
Stelios Pantazopoulos
Eric Liu
© ThoughtWorks, 2009
Expectations
• What would you like to get out of this workshop?
• Please write your expectation on a sticky note and place them on the board…
© ThoughtWorks, 2009
Agenda
• Challenges with shared test environments
• Local test builds
– Impact to overall test strategy
– Types of testing suitable for local test builds
– Requirements for local test builds
• Sample application walkthrough
– Local test examples with live application
• Technical
– Application architecture
– Build support
© ThoughtWorks, 2009
Local Dev Sandbox
IsolatedFull Access
Very Few Controls
Integrated Test Environment
SharedRestricted Access
Controlled
Test Workflow
© ThoughtWorks, 2009
Challenges Faced in Shared Test EnvironmentsHas the calculation fix been deployed to the QA environment? I’d like to re-
test it.
Sorry, the deployer is out for the rest of the
afternoon, so we’ll have to wait till tomorrow.
© ThoughtWorks, 2009
Challenges Faced in Shared Test Environments (2)I’d like to re-test the screen which an administrator should be able to get to,
but couldn’t.
The fix to that screen has been deployed, but we’re
still waiting for the administrator role to be added to your user in
Active Directory…
© ThoughtWorks, 2009
Challenges Faced in Shared Test Environments (3)How can I test the behaviour of the
system during the last hour of the day?
We’ll have to setup data and wait till 11pm…
© ThoughtWorks, 2009
Discussion
• What other challenges have you faced with a shared QA environment?
© ThoughtWorks, 2009
Integrated Test Environment
Local Test Sandbox
Local Dev Sandbox
Integrated Test Environment
Test Workflow
© ThoughtWorks, 2009
Integrated Test Environment
Local Test Sandbox
and
Integrated Test Environment
Impact on Overall Test Strategy
• Story Acceptance• User Interface and Usability• System Integration• Security Roles Based Access• Security Penetration• Exploratory• Data Migration• End-to-End Functional Integrated
• Performance/Load/Stress• Failover/Recoverability• Configuration• Production Release Dry Run• Compliance/SOX
© ThoughtWorks, 2009
Story Acceptance
Local Test Sandbox
• Run through each acceptance test scenario to ensure its correctness.
• Ensure that the data set and simulated integration points are complete enough for story acceptance to be done.
Integrated Environment
• Run through each acceptance test scenario to ensure it works in an integrated environment.
• Subject Matter Experts and the Business validate that a story implementation meet their needs.
© ThoughtWorks, 2009
User Interface and Usability
Local Test Sandbox
• Ensure that the look and feel is consistent.
• Ensure that the user interface meets the standards: for example: font size, positioning of screens.
Integrated Environment
• Smoke test to ensure that release did not affect the look and feel.
• Test conformity to user interface standards and that the end to end business processes are intuitive
© ThoughtWorks, 2009
System Integration
Local Test Sandbox
• Simulate flows of data into and out of the system via a simulated version of the integration point.
Integrated Environment
• Run the same test with the actual integration point.
• Test that the system’s interfaces to other internal and external systems work properly in isolation.
© ThoughtWorks, 2009
Security Roles Based Access
Local Test Sandbox
• For each role and capability, ensure system access to functionality and data is correct.
Integrated Environment
• Test the system’s interface to the security infrastructure where end user, their roles and their capabilities are stored and managed.
• Ensure internal and external end users have proper access to the system given their assigned roles and capabilities.
© ThoughtWorks, 2009
Security Penetration
Local Test Sandbox
• URL manipulation, manipulate post parameters, SQL injection attacks and cross site scripting.
Integrated Environment
• Smoke test which includes subset of full test done on local build.
• Ensure the system is safe from an attack to the system from an external party.
© ThoughtWorks, 2009
Exploratory
Local Test Sandbox
• Take advantage of having full control of the local test sandbox (application and data) to do unorthodox tests.
Integrated Environment
• The scope of Exploratory testing is limited due to the controlled nature of the environment.
• Unscripted testing where Business Users, Analysts and Expert Testers with a strong knowledge of the business, exercise the system.
© ThoughtWorks, 2009
Data Migration
Local Test Sandbox
• Focus on the relationships between the data.
• Focus on historical data.
• Witness how the application handles the data.
Integrated Environment
• Most of the testing can be done locally, with a production dry run in this environment
• Validate that data is equivalent between the legacy system and its replacement after the data migration has been performed..
© ThoughtWorks, 2009
End-to-End Functional Integrated
Local Test Sandbox
• Run an end of end business scenario where flows of data between internal and external integration points are simulated.
Integrated Environment
• Same test as local, except internal and external integration points are not simulated, and as close to production as possible.
• Business cycle test run in parallel with external systems. Tests ensure that operations with a cross system impact work properly.
© ThoughtWorks, 2009
Testing that cannot move to Local Test Sandbox
• Performance/Load/Stress
• Failover/Recoverability
• Configuration
• Production Release Dry Run
• Compliance/SOX
© ThoughtWorks, 2009
Requirements for Local Workstation Testing
• Fully-Functional and Self-Contained
• Allow Integration Points to be Locally Controlled
• Allow Complex Aspects to be Locally Controlled
• Example: CSV Files
• Example: Test Support Screens
• Rich Data Set
• Include Realistic Subset of Production Data
• Easy to Install
© ThoughtWorks, 2009
Sample Application Walkthrough
• Application: Movie Event Planner
• Summary: Plan movie events and allow interested parties to sign up. Integrates with external movie showtimes provider.
• Sample Requirements:
– As an organizer, I want to create an event so that others can sign up and go watch a movie together.
– As an attendee, I want to join an event so that I can go watch a movie with others.
– As an attendee, I want to be notified via SMS text message about an upcoming event 2 hours in advance.
– As an administrator, I want to retrieve upcoming movie locations and showtimes from a reliable source so that organizers can plan accurate events.
© ThoughtWorks, 2009
Discussion
• What are some challenges you foresee in testing these requirements?
© ThoughtWorks, 2009
Fire up Local Workstation Test Build!
© ThoughtWorks, 2009
Local Test Example: Story Acceptance
• Verify that I can’t join an event where movie has already started.
© ThoughtWorks, 2009
Local Test Example: Security Roles Based Access
• Verify that an attendee can join an event.
• Verify that an attendee cannot create events.
• Verify that an administrator can retrieve upcoming movie locations and showtimes.
© ThoughtWorks, 2009
Local Test Example: System Integration
• Verify that an attendee gets notified via SMS text message about an upcoming event 2 hours in advance.
• Verify that an administrator can retrieve upcoming movie locations and showtimes from a reliable source so that organizers can plan accurate events.
© ThoughtWorks, 2009
Application
Database
Application Architecture to Support Local Testing
CorporateDirectory
ExternalService
© ThoughtWorks, 2009
Application Architecture to Support Local Testing
• Local database
• Swappable external service implementation
• Swappable security implementation
• Swappable implementations for other complex aspects of system
• Domain layer should be implementation agnostic
– This may already be the case especially if developers are practicing unit testing
• Implementation choices controlled by configuration
© ThoughtWorks, 2009
Application
Database
Application Architecture to Support Testing
CorporateDirectory
ExternalService
Clock
Configuration
System
© ThoughtWorks, 2009
Application
Database
Application Configured to Support Local Testing
Clock
Security Groups
Local Store
Configuration
RichDataSet
External Data
© ThoughtWorks, 2009
Build Support for Local Test Builds
• Identify Good Build Candidates
– Compiles and Passes All Automated Tests
• Configure Application with Local Implementations
• Populate Rich Data Set
• Bundle for Installation and Stamp with Build #
• Publish Build
• Tip: Easily Automated through Continuous Integration
© ThoughtWorks, 2009
Process Improvement Analysis
© ThoughtWorks, 2009
Process Improvement Analysis
© ThoughtWorks, 2009
Review Expectations
© ThoughtWorks, 2009
Questions?
© ThoughtWorks, 2009
Thank you!
• Stelios Pantazopoulos, [email protected]
• Eric Liu, [email protected]