Engineering Principle and Practices

by Qingsong Yao

About Me

Senior Tester at SQL Server Team with 8 years of experiences.Lucky to being a great and happy team Grow up through reading, thinking and doingI will show some Engineering Principles and Practices which I collected from other teams.

Agenda

• Continuous Integration• Continuous Delivery• A Good Testing Strategy• Testing in Production• Agile

How’s other team’s Engineering Principle and Practice?

BING Team’s Core Search Principles

Principle 1 - Live Site FirstPrinciple 2 - Leverage Adaptive Planning and ExecutionPrinciple 3 - Optimize for Engineer ProductivityPrinciple 4 - Practice Data Driven Engineering

Leverage Agile Best PracticesFrequent Code DeliveryShort Iterative Engineering CyclesRapid Componentized BuildsContinuous IntegrationUnit Testing Improve service quality and engineering agilityOperate at Near Zero BugsFocused Integration TestingEffective Exploratory Testing

Practice One Continuous Integration

Continuous Integration…is a software development practice where engineers integrate frequently, leading to multiple integrations per day. Each integration is verified by an automated build and test to detect integration errors as quickly as possible.Further Reading:

http://www.martinfowler.com/articles/continuousIntegration.html Continuous Integration in .NET

Principle of Continuous Integration

Maintain a Single Source Repository

Everyone commits to mainline every day

Every Commit should build the mainline on integration machine

Keep the Build fast

Everyone can see what's happening

Introduce to CruiseControl .NetPopular Open Source CC Server Three main components:

CC .Net ServerCC TrayWeb Dashboard

Very stable and easy to be extendedSource Depot Integration.Build different favor of SQL ComponentMsTest, TestShell, LTM

Used for several teams inside SQL and MS

Apollo, DAC, SSM Portal, BigData/Hadoop

CI Using the Cloud for Dev @ GoogleDistributed/parallel build

Every engineer can build all of Google’s code + third party open source code in a matter of minutes Works by constructing the dependency graph than using map/reduce technology

Distributed/parallel testChanges on the code base are continuously tested against all dependent targets once submittedFailures can be tracked down very precisely to the given change which have introduced them

Check out http://google-engtools.blogspot.com/ for details

Practice Two Continuous Delivery

A service/software contains

CodeConfiguration DataInfrastructure

Every change will requires to re-validate the entire service. How can ship fast with low risk and high confidence?

The answer is Continuous Delivery

is a set of practices and principles aimed at Building, Testing, and releasing software faster and more frequently

Further Reading: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

Principles of Continuous Delivery

The process for releasing/deploying software MUST be repeatable and reliable. Automate everything! Done means “released”.If something difficult or painful, do it more often.Keep everything in source controlBuild quality in!Improve continuously.Everybody is responsible for release process

The Benefit of Continuous Delivery

Remove waste, risk and bottlenecks in the process to deliver changes faster and safer.Improved release quality. More collaboration. Test, support, dev and operations work as one delivery team.Predictable and faster turnaround for business requests.Improved Risk Management. Releases to production are reliable, routine “non-events”.This improves confidence, and avoids potentially embarrassing customer impacts.

Core Practices By Mike McGarr

Deployment Pipeline

Deployment Pipeline models the part of your project’s value stream that goes from check-in to release, and then to automate it.

Another Example

DAC I/E Service Deployment Pipeline using CC .Net

Code Change Build

Unit Tests

Functional Tests

Acceptance Tests

TestDeploym

ent

Run test on

deployment

Monitor in

Production

Production

Deployment

Run test on Pre

Production

Pre Producti

on Deployment

SQL Apollo Quality Gate Report

Practice ThreeTesting Strategy

We have large number of BVT, GQL, Perf, Stress tests,

but we need to have a Testing Strategy to fit into the deployment pipeline

"Testing can improve customer satisfaction if

you actually know what the customer thinks is

truly important and test for those subjects.

Making great improvements to an area that

your customer is hardly interested in is a

laudable effort, but won‘t much increase their

idea of how good your product is!"

-Key Questions to ask when improving testing

by Erwin Engelsma

The Testing QuestionsWhat tests I should write?

Understand business requirementUnderstand the design

What tests I should run?Understand the risk and mitigate the riskEach test should have only one goal

How can I have confidence for the quality?

Quality is built in the codeRunning different kind of tests to measure the quality

How can I say I am “Done”?

Agile Testing: A Practical Guide for Testers and Agile Teams

A Couple of Testing StrategiesBalance Functional/Business Face Testing

Write Acceptance Test from the view of Business UserBring User workload in-house.

Push Quality UpstreamDev write Unit and Component TestsRun these tests continuous automatically

Adopt new testing methodologiesTesting in ProductionUsing Crowd Sourcing, Game and Social For TestingExploratory and User Experience Testing

ATDD – Acceptance Test Driven DevelopmentEnables communication between business and developersFor functional specification/testing (not unit testing)Focus on functionality (“what” and not “how”)

Integration of testing activities into design and developmentMinimizes translation from domain to technical language

Fewer bugs injectedFaster to deliver

Three core principles:Where’s the business value?It’s all behaviorEnough is enough

Acceptance Testing Driven Development

Automating Acceptance TestsTool: SpecFlow (http://specflow.org)

.NET version of Cucumber (http://cukes.info)

Feature and ScenarioEvery scenario and scenario variation generates a test caseEach step in a scenario calls a matching “step definition” with test dataDevelopers/Testers write the step definitions

HighlightTests independent to test frameworkTests independent to implementationEasy to design and reuse your test library

ATDD Example

How Google Testing and Release Software

Runtime Analysis and Testing @ Google By Wolfgang Grieskamp

Production Level

Monitoring

Staging Level

A simulation of the production environment with faked

identities etc.

Uses monitoring techniques

Load testing

Integration

Level

Automated testing of

every code change over

the dependency

closure

End-to-End testing with

partial component

isolation

Unit Level

Super-strict component

isolation using e.g.

dependency injection

Extensive use of mock-based

testing

How Google Releases Chrome By “How Google Testing Software”

Dev own quality, not tester.Chrome upgrades every six week, most user are using the latest version. Focus on Unit Test with Java Script Test Framework10 minute Mini test plan, only focus on the area is important and risky.Use Crowd sourcing testing to allow user test beta version and report issues. Roll out of the browser is incrementally.  Collect use behavior to improve Chrome

Practice FourTesting in Production

Testing in Production is NOTRunning all functional testing in productionReducing unit/functional tests and finding production bugs in productionRunning destructive testing in production which aim to kill your serviceTraditional way of “Testing”Run tests on the cloud

Testing in Production isMonitoring your services in productionExperimentation in productionDogfood in ProductionUser scenario execution in production…The Testing Strategies (less about testing tool)

and most importantly, it is an Mind Shift

Monitoring in Production

External MonitoringInternal MonitoringLog AnalysisRunning Synthesis Tests in ProductionRun User Scenario in Production

Experimentation in Production

… is a way we expose users to different experiences in a controlled manner and measure business metrics to determine if we built the right thing or we build the thing right

Practice FiveBeing Agile

How can I be agile?

No single tool/process can enable you

be agile

No single tool/process can block you

be agile

If you are open mind, be motivated,

willing to change, and understand the

principles here, you will be agile.

Building a healthy and happy team is

the key

Conclusion with Protection Services Team ‘sEngineering Methodology

“We want to develop an Engineering Methodology and System which enables continuous and frequent customer value delivery (e.g. more protection) and learning built into, high scale and reliable cloud service, while maintaining high quality and predictable lower cost.” -Nir Caliv

Engineering Methodology Themes

• Continuously and extensively Monitoring and experimental our service in production

Test In Production (TiP)

• Frequent deployments of small changes to production environment (starting with weekly releases)

Continuous Deployment

• Minimize the gap between code development and deployment readiness

Continuous Integration

• Efficient processes to reduce the cost and gaps between product and quality development

Pushing Quality Upstream

• Constant observation, measurement and feedback on processes ,systems and cost

Continuous Improvement