Challenges Engineering Applications for High Performance

Practical Performance Analysthttp://www1.practicalperformanceanalyst.com

• Started mucking around with Open Source 15 years ago

• Fell in love with Gnu/Linux and more importantly the Free Software / Open Source movement

• Travelled across the country advocating the use of Linux and Free Software

• Went to work for Red Hat. Sold & deployed Free Software

• Worked at the Performance R&D Labs at TCS and then moved to Consulting

• Moved to Accenture Technology Consulting and now part of Tech Arch, Performance Engineering

• Founded Practical Performance Analyst (www1.practicalperformanceanalyst.com)

• Revived CMGA (Computer Management Group, Australia)

A few words about me

How Do You Define PerformanceThroughput &

Congestion across the last mile

Poor End User Experience

Client Device Performance

Congestion on the Internet

Infrastructure Performance

Application Performance

Backbone Network Throughput & Performance

Data CenterNetwork

Performance

• All of us have cribbed about poor performing solutions at one point or the other.

• Unfortunately, poor performance seems to be the norm and we’ve all accepted that it’s just how things are. Lets look at some examples:

• Web Store: The web is abound with case studies about online retailers loosing revenue as users turn to competitors offerings. Users today aren’t very tolerant of poor performing websites and with easy access to google finding an alternate offering is no more than a few clicks away. There’s tons of cheap tools available to monitor system these days but engineering systems to performance is a completely different kettle of fish.

• Ticketing / Help Desk Application: You tried reaching out to your internal help desk for support and it seemed like eternity before you got through. But even after you got through to the help desk, the wait continued because the help desk technician just couldn’t get to the data they needed quickly to help you out. The tech team looking into the issue needed insight into the performance bottleneck but the tools they had at their disposal only gave them a siloed view of performance which wasn’t good enough to get to the root cause quickly.

• Batch Job Performance : You’ve had to go back and confess to business that the batch jobs your teams had designed weren’t performing the way you intended them to perform. Un-fortunately, the developers of the batch job hadn’t taken into account the volume of data that the system would be need to tested for. Your development environments happen to be a fraction of your production environments and so was the data store for performance testing.

How Does Poor Performance Manifest Itself

What is SPE

• Definition – Systems Performance Engineering (SPE) is a systematic and quantitative approach for the cost-effective development of software systems to meet stringent Non Functional Requirements i.e. Performance, Capacity, Scalability, Availability, Reliability, etc.

• Systems Performance Engineering is focused on optimal selection of application architecture, design, build and implementation choices with the objective of meeting Non Functional Requirements.

• Systems Performance Engineering can also be defined functionally as the set of tasks or activities that need to be performed across the Software Development Life Cycle (SDLC) to meet the documented Non Functional Requirements.

• Software Performance Engineering is often viewed as the art of building systems that meeting Non Functional requirements within the allocated time frame and budget constraints.

Where Do Programs Generally Go Wrong• Lack of Non Functional Requirements - 99% of programs I’ve worked on have no or very poorly define Non Functional Requirements.

Ask 5 different people on the team their interpretation of on Non Functional and you will end up getting 5 different answers. As a result –• Architects have no guidance on what are the right design choices• Developers don’t have unit performance or tier performance targets• Infrastructure designers have no idea of what specs they need to be building for• Inability to regression test nightly builds against unit performance or tier performance targets • Performance testers have no idea of what and how the various application workloads should be evaluated

• Lack of investment in right capability – Performance isn’t just about response times or server utilization. Engineering performance goes far beyond all of that and it happens to be a way of thinking. Programs rarely invest in the right capability. The fact is most program leads lack an understanding of what is required to build performance into the solution. Functional capability is easy to nail down, performance is rather complicated since it traverses so many domains across the delivery chain.

• Lack of investment in appropriate tools – Tooling is a very complicated subject and performance tools happen to be expensive. Most projects think that Nagios (performance monitoring) and Jmeter (performance testing) is good enough to get them over the line.

• Lack of investment in the required infrastructure – To validate performance you need appropriately sized infrastructure. Testing on infrastructure that is <50% of production and drawing conclusions based on assumptions of linear scalability is quite frequent and a perfect recipe for disaster.

• Lack of funding to design, build, validate and manage performance across the delivery cycle – Performance starts at the requirements gathering stage and continues onto architecture, design, build, test and then capacity planning. Each of these functions requires the right capability across the board. Projects seldom invest in the right capability across the board.

SPE Across the SDLC

Software Development Life Cycle

Functional Requirements Gathering

Architecture & Design

Build Application

System Test,

System Integrated Test & UAT

Deploy Into Production

Performance Engineering Life Cycle

Non Functional Requirements Gathering

Design for Performance &

Performance Modelling

Unit Performance Test &

Code Optimization

Performance Test

Monitoring & Capacity Management

Performance By Design• Requirements Gathering –

• Document Non Functional Requirements • Work with Business & IT to get NFR’s signed off• Decompose NFR targets across various application tiers• Identify & budget for relevant tools for monitoring, diagnostics,

performance testing and capacity planning• Identify & budget relevant capability required to integrate

performance across the SDLC i.e. Performance at Design, Performance at Build, Performance Testing, Diagnostics and Tuning, Monitoring & Capacity Management

• On programs that can afford it, have an owner for Performance across the program

• Design –

• Work with Architecture teams on adopting the relevant design patterns

• PoC design / implementation options and identify potential constraints

• Document Overall Performance Engineering strategy• Document Overall Performance Testing Strategy• Document Overall Capacity Management & Monitoring Strategy

Performance Requirements

Analysis

Performance Modelling &

Capacity Planning

Build & Optimization

Performance Testing

Performance Monitoring

Capacity Management

Performance By Design• Build –

• Translate the NFR’s to tier wise performance targets• Work with development teams and establish process to track regression

of code performance• Enforce unit performance testing of code and work with developers to

track tier wise performance targets• Work with development teams and assist with usage of profiling &

diagnostics tools. Use diagnostics tools to identify potential bottlenecks and optimize code

• Test –

• Build stubs and begin early performance testing• Document workload models and get them signed off• Execute end to end performance testing• Use Diagnostics tools to identify bottlenecks• Analyst results, tune, optimize and re-run

• Go Live -

• Implement system, application, transaction and end user monitoring• Collect relevant business/transaction performance metrics• Forecast & model performance impacts using performance metrics

Performance Requirements

Analysis

Performance Modelling &

Capacity Planning

Build & Optimization

Performance Testing

Performance Monitoring

Capacity Management

SPE Roles Across The SDLCPhase Role Performed

Requirements Gathering Determine Business Volumes & Growth plansDetermine Non Functional Requirements

Design Validate Application ArchitectureValidate Infrastructure ArchitectureValidate Infrastructure DesignDetermine Infrastructure Capacity Requirements

Build & Optimization Set performance targets for developersValidate outcome of Unit Performance TestsRecommend optimization to CodeRecommend optimization to Application DesignRecommend optimization to Infrastructure Design

SIT & UAT Begin validating Application Performance for functionality that’s availableBegin Tier Performance Tests for functionality that’s availableTest, tune & optimize Performance for functionality that has been released

SVT Execute End to End Performance TestsValidate Performance End to EndIdentify applicable bottlenecksUse Diagnostics tools to identify bottlenecks, tune & Optimize application Ensure application meets its Non Functional Requirements Evolve Performance Monitoring Requirements

Pre-Go live Setup Performance Monitors for Network, OS, Application, Business TxnSet alerting for various Performance metricsSetup capture Performance metrics for purposes of Capacity Management

Go-Live & Post Go Live Monitor application for potential breaches in SLA’sIdentify hotspots using low overhead transactional tracing & diagnostics toolsModel application performance and predict capacity impacts for growing business workload

Devops, Continous Integration, Agile, etc.

• Devops, Agile, Continous Integration are paradigms that are have begun to influence development practices around the world. Each of these paradigms have their own strengths and weaknesses.

• Whatever you development paradigms you may adopt for development of your systems you will need to integrate Proactive Performance Management or SPE principles into it

• Un-fortunately most of the bad habits (mentioned in earlier slides) with regards to poor SPE practices are carried into the new paradigms as well. It’s quite rare that we find programs adopting proactive SPE practices into their Agile development processes.

• Agile, Devops, CI are paradigms have a lot of good to offer but don’t forget to integrate Proactive Performance Management practices and more importantly SPE practices into your implementation of those processes.

What Am I Doing To Change The World• Practical Performance Analyst -

• What is PPA - Practical Performance is a community portal that I started work on in 2012. Practical Performance Analyst has grown significantly over the last couple of years with a growing readership base. Our aim is for Practical Performance Analyst to be a open platform for sharing and collaboration on standards, practices and guidelines around Systems Performance Engineering (SPE). Practical Performance Analyst has a growing list of international authors who are experts in their own domains of Performance. Practical Performance Analyst aims to be a hub for all things SPE (Systems Performance Engineering). Drop in and find out.

• Accessing PPA – http://www1.practicalperformanceanalyst.com

• Contributing & Supporting PPA – trevor@practicalperformanceanalyst.com

• Computer Management Group, Australia –

• What is CMGA - Computer Management Group Australia is a professional body that focuses on bringing together professionals from across Australia who have interests in Systems Performance Engineering (SPE). CMG is an international organization of professionals (CMG.org) and CMGA is the Australian Chapter of CMG International. Spent the last 6 months reviving CMGA and getting CMGA going again here in Australia.

• Accessing CMGA – http://www.cmga.org.au

• Joining CMGA – Joining CMGA is quite easy. Hop over to the CMG website and join the group using the Meetup link on the right hand side corner. We use the meetup group to keep in touch and co-ordinate all national events.

What Can You Do To Help

• Learning & Sharing – If you are interested in learning more about Proactive Performance Management & Systems Performance Engineering head off to Practical Performance Analyst (http://www.practicalperformanceanalyst.com). • There’s tons of resources out there on all topics related to SPE. • You’ll also find access to books and tools on all aspects of SPE• Grab your daily dose of SPE news from around the world at PPA

• Contributing & Helping Build the Community – If you are interested in helping building the SPE Community and contribute your knowledge from a Design, Architecture, Build, Optimization, APM, Performance Testing, Capacity Management, Modelling, etc. perspective please drop me an email at trevor@practicalperformanceanalyst.com. We are keen to expand out list of authors and grow our community of readers.

• Networking & Lobbying – Come and join us at Computer Management Group Australia. CMG Australia is a professional organization with a focus on evolving standards, fostering innovation and most importantly advocating adoption of SPE across programs. CMG Australia has various initiatives focused on SPE across industry, academia and research. Come join us and help build appreciation for good development practices.

Thank You

Practical Performance Analysthttp://www1.practicalperformanceanalyst.com

trevor@practicalperformanceanalyst.com