Dynamics 365

FastTrack TechTalk

Test strategy

Ajit Kotwal

Toby James

Adi Vijayashankar

Objectives of this TechTalk✓ Discuss the importance of testing

✓ Provide some recommended practices to define a Test Strategy

✓ Describe some of the key tests and the goals they support

What it doesn’t cover

• A look at testing as aligned to any given implementation

methodology

• An in-depth review of automated test tools, for example, RSAT

• An exhaustive overview of all the different types of tests that

could be performed in a Dynamics 365 F&O implementation

Contents01 Why we test

02 Test strategy

03 Test types

Why we test

Why do we test?

Understand the behavior of the solution

Validate designDrive feedback

Find bugs

Validate requirements

Validate performance

Improve solution quality

Ensure stability and reliability

Improve user experienceEnsure the business can operate safely and

efficiently

Ensure solution still worksImprove code design

Validate data quality

Inform design

Test strategy

Project scope

definitionTest types & phases

Test management

processTest planning

Test strategy

Test Strategy Components

Project scope definition mapped to Test Scope

A clearly defined project scope

helps define test scope

End-to-end “To Be” business

processes provide context for

requirements and test suitability

Many aspects of project scope

help determine the required test

coverage

Business processes

Data – Integrations, Migration, Reporting

Batch processes

Geographical scope

Security (e.g. Project roles)

Complex and custom designs

Non-functional : performance, infra, usability etc

Impact of process changes, Regulatory reqs

Business goals

Project scope definition

Test types & phases

Test management process

Test planning

Test strategy

Solution

Test types and Test PhasesUnit tests

Process tests

Functional tests

End-to-end tests

Performance

tests

Security tests

User acceptance

tests

Regression

tests

Interface tests

Solution

Data tests

A Test Phase can be considered as a set of testing processes executed based on reaching a phase or milestone in a project. For example, at a “sprint end” or “build complete”. Typical test phases are called SIT, UAT etc

A Test Type is not necessarily tied to a milestone and can form part of a test phase or be repeated many times on a project. For example, performance testing. Some test types can also be considered as test phases, depending on the methodology (e.g. cutover testing)

Project scope definition

Test types & phases

Test management process

Test planning

Test strategy

Attributes of the test/test phase

Intended Test Coverage and Source for test cases

Who owns this test type/phase

Project roles and resources needed to prepare,

perform and report on the test type/phase

The tools that can/should be used for this test

Planning considerations

Relationship with other testing types/phases

Project scope definition

Test types & phases

Test management process

Test planning

Test strategy

Objectives of this test

When this test type/phase should be used

Preparation needed to meet the entry criteria

Entry criteria for this test

How the testing should be performed/managed

Exit criteria for this test

Test management

Project scope definition

Test types & phases

Test management process

Test planning

Test strategy

Test planning

Project scope definition

Test types & phases

Test management process

Test planning

Test strategy

Test Strategy – anti patterns

Test Strategy not defined during the initiation phase of the project

Test Strategy is too high-level and generic without consideration to the specific needs of this

project and project solution blueprint

Test strategy is too technical and does not account for the overall business needs

Test strategy directs testing without identifying priority areas or considering the balance of

related business risk

Test strategy does not adequately define the entry/exit criteria, so testing phases start

prematurely and fail to exit in a timely way

3rd party testing organization writing & conducting tests without an adequate understanding of

the business processes or the acceptance criteria, resulting in test cases that add little value

Inadequate attention to planning for testing as an incremental part of the project to ensure

quality is being engineered into the system

Inadequate attention to planning for the true, and proportionate effort for testing resulting in

testing getting “squeezed” when projects face delays/cost overruns

Testing with customer data

Testing throughout the solution lifecycle

Implement PrepareInitiate Operate

Get ready to start Design Build E2E/SIT BAU/Cont. update

Go Live

UAT Deploy

Unit test

Functional test

Process test

Performance test

End-to-end test

Security test

Regression test

UAT

Interface test

Data validation test

Test Strategy• Project scope

definition

• Test phases and test

types

• Test management

process

• Test planning

Security requirements

Performance

requirements

Functional and

Non-functional requirements

Test types

Solution

Different tests support different goalsUnit tests

Process tests

Functional tests

End-to-end tests

Performance tests

Security tests

User acceptance tests

Regression tests

Interface tests

Solution

Data tests

Scenario: Contoso Entertainment System

Contoso Entertainment System is a distributor and installer of home electronic

equipment.

Sales orders are created in Dynamics 365 CE and dual written into Dynamics 365 F&O.

The sales orders are subsequently processed and have some complex sales order validation

scenarios. While some of them are met out of the box, a few extensions are required to validate the sales order.

After the sales orders are validated and confirmed, it is sent to the warehouse for picking

and packing. Mostly individual shipments are prepared. However, in rare cases, consolidated shipments are required for some customers. During the preparation of

shipment, ASN should be sent to the external transportation system. Subsequently, the

order is shipped and invoiced.

Process flow for order-to-cash

INVOICE ORDER RECEIVE PAYMENT

CONFIRM ORDER

PICK ORDER

INDIVIDUAL SHIPMENT

SHIP ORDER

CREATE ORDER

TRANSPORTATION SYSTEM

CONSOLIDATED SHIPMENT

Prepare shipment

WhatIndividual function/code/configAutomated/Manual, Technology facing

WhyValidate low level componentVerify requirements, validate & improve

code design, find/fix defects

WhenBuildIterative, continuous integration

WhoDevelopers

WhereDev environment, build pipeline

ToolsCode toolsSource code management, SysTest

Framework, Azure DevOps, ATL, RSAT

Unit tests

Unit tests - anti-patterns

• Writing low value/redundant tests

• Unit testing is more exploratory in nature rather than well-defined

based on requirements and expected output

• Development for unit testing does not consider performance,

security and other non-functional requirements

• No prioritization on what to unit test

WhatIndividual functionAutomated/manual, Business facing

WhyVerify requirementsVerify configuration and custom code,

validate design, find/fix defects

WhenConfiguration and BuildIterative, continuous integration

WhoFunctional Consultants, Customer

SMEs, Testers

WhereDev environment, test

environment

ToolsTest management toolsRSAT, Azure DevOps

Functional tests

Functional tests - anti-patterns

• Test cases are prepared without consulting with the business and

SMEs and executed only against the gap design

• Testing is not executed on data created by integration/migration

• Testing with dummy data or the same set of data throughout the

project lifecycle

• Test cannot easily be reused or added to other tests to create

extended process or end-to-end tests

WhatIndividual business processAutomated/manual, business facing,

builds on functional tests

WhyValidate business processVerify process with business scenarios

WhenConfiguration and BuildIterative

WhoFunctional Consultants, Customer

SMEs, Testers

WhereTest environment

ToolsTest management and automation RSAT, Azure DevOps

Process tests

Process tests - anti-patterns

• Execute testing without customer data, adequate security setup and

expected configuration.

• Process testing only considers happy path scenarios and ignores

exceptional scenario and edge cases.

• Focus only on customizations, individual features and not the

integrated business process

• Inadequate setup of environments to assist with testing of

integration

• Non-functional requirements are not validated

WhatEnd-to-end business processAutomated/manual, business facing,

builds on process tests

WhyValidate full business processVerify process with business scenarios

WhenThroughout implementation Iterative, pre Go-live test phase

WhoFunctional consultants, Customer

SMEs, Testers

WhereIntegrated test environment

ToolsTest management and automationRSAT, Azure DevOps

End-to-end tests

End-to-end tests - anti-patterns

• Only executed at the end of the build phase, before Go-live

• Not taking key business transactions into account when defining

end-to-end test cases

• Not getting business end users to help with “day-in-the-life” set of

transactions to ensure the business scenarios are representative

• Focus on integrating to external systems, rather than full end-to-

end process

• Non-functional requirements are not validated

WhatNormal, peak, continuous loadBatch jobs, integrated systems, peak

load scenarios, UX scenarios

WhyEnsure production performsCritical processes can scale with load

and growth, fine tune processes

WhenThroughout implementationAligned to other implementation

milestones during initial and late stages

WhoDevelopers, Functional Consultants,

SMEs, Testers, End Users

WhereDedicated perf. test environment, Dev

and Test environments

ToolsUI simulation and dev toolsPerfSDK, Visual Studio, Data Expansion

Tool, DMF, 3rd party

Performance tests

Performance tests - anti-patterns

• Not having clearly defined and agreed performance test objectives

• Leaving all performance testing to a late stage test phase

• Not testing performance of components first and then the whole

Going further… Tutorial, tips/tricks and demonstration

• Finance and Operations: Performance Benchmark for Dynamics 365

• Finance and Operations: Performance Key Patterns and Anti-patterns for Dynamics 365

• Finance and Operations: Performance Troubleshooting Tools for Dynamics 365

• Finance and Operations: Performance Testing Approach

• Select the right environment Tier for the performance benchmark using the Environment

Planning Tech Talk and Environment Planning Docs article

• Task Recorder

• Single-user testing with Task recorder and the Performance SDK

• Multi-user testing with the Performance SDK and a local test controller

• Environment Monitoring and Diagnostics on LCS

• Trace Parser

• SQL QueryStore

WhatEnd user experience of solutionManual, exploratory, business facing,

performed using actual business data

WhyConfirm solution readinessBusiness sign-off, collect end user

feedback, help manage org. change

WhenFinal test cycle before go-livePre go-live checkpoint/milestone

WhoEnd users from the business

WhereDedicated and integrated environment

ToolsTools to track testing and feedback

User acceptance tests

User acceptance tests - anti-patterns

• Not conducted by representative users from the business

• Insufficient ownership of UAT by the business

• All users are assigned System Administrator security role or roles

that do not reflect what will be in production

• Users are not adequately trained on the solution and how their

processes may have changed, leading to false positives

WhatRe-run testsAutomated/Manual, after

code/configuration change

WhyEnsure solution still performs as

expected after change

WhenWhen change is introduced,

before production deployment

WhoTesters, End users, Developers

WherePre-production/Test environment,

Dev environment

ToolsTest management and automationRSAT, DevOps, SysTest Framework, ATL,

Data Task Automation

Regression tests

Determining what to regression test

Technique Pros Cons

Test all

Aim for ~100% test coverage

• Provides comprehensive coverage • Is expensive to setup, run and maintain

• In most cases is not necessary• Is time consuming for manual testing• Is unrealistic in most cases

Prioritize

Test based on business priority

• Ensures all key/critical business scenarios are

tested• Is realistic and practical• Automation can be built over time

• Reduces key risks to business

• Does not guarantee solution is free from regression

Regression exposure

Test based on known change

• Allows for more targeted testing based on

known change• Reduces test scope and effort

• May neglect critical areas to the business

• Difficult to quantify impact based on interpreted change – a minor change to one line of code, may be more impactful than a significant change to code

• Regression can surface in a downstream process – risk that isolating testing just to changes will not surface

other issues

Balanced approach

A combination of techniques

• Ensures critical business processes are tested,

and the impact of specific change is understood and can be planned for

Prioritise automation of your regression test suite

• Change cannot be avoided – configuration, customisation, new features, ongoing

updates…

• Regression testing will be ongoing – efficiency is important

• You need to be able to efficiently and rapidly execute your regression tests

• Use RSAT, Data Task Automation and other tools to build out your regression test suite

during the implementation

• Begin with key business processes, and build the automation over time

• Use your manual test suite to identify and plan test candidates for automation

Regression tests - anti-patterns

• Not planning for regression testing

• Regression testing is not done after configuration changes are

made / changes are made directly to production

• Regression testing is left too late in an update cycle

• Not incorporating automation as a strategy for regression testing

throughout the implementation and post go-live

Other test types

Test type What are we trying to answer

Mock go-live Does my cutover process work and am I confident the production go-live will go

smoothly

Smoke tests Very basic functions work as expected (e.g. login or running reports etc)

Data acceptance tests Did my data migrate correctly and can it be inquired upon, transacted upon and

reported upon

Interface tests Can my interfaces be consumed as intended.

Environment testing Validating that a new environment is fit for purpose

Reporting/BI testing Specific test type to validate reporting/BI including from external systems such

as data warehouses, data lakes etc

Device testing Specific testing to validate shop-floor and warehouse devices such as scanners,

POS, RFID readers etc Can include mobility device testing such as phones,

tablets etc..

Network/Infrastructure

testing

Validation of any relevant underlying networking, fire wall configuration, Wifi,

printers etc

Key takeaways

• Define your test strategy early and prioritise test scope to ensure the right

coverage.

• Your test strategy should consider project scope, test types and test phases, test

planning, and test management.

• Testing should be an activity that occurs continuously, early and throughout the

lifecycle of the solution – A layered and iterative approach to testing focuses on

building quality into the solution.

• Test real scenarios – Real business processes, concurrency, data volumes, batch jobs,

printing, reports, emailing, integrations, workflow, etc.

• Process, end-to-end and performance tests are initiated from the design and

build phase of the project.

• Prioritize automation – Don’t aim to automate everything, build your automation

suite throughout the implementation and overtime after go-live.

Feedback and Q&A

© Copyright Microsoft Corporation. All rights reserved.