Pov Building Production-Grade Services v1.2...Executive Overview As the application landscape...

Owner: IBM Cloud Status: published Subject: Building Production-Grade Services of 15

Cloud Adoption, Garage Method and Solution Engineering (CAMSE)

CAMSE Points of View

Building Production-Grade Services

As the application landscape transitions towards micro-services, businesses are looking for

guidelines and processes to assure that these services are robust and serviceable.

Version number: 1.2

Final as of: October 2017

Author: Ingo Averdunk ([email protected]) Distinguished Engineer

IBM Cloud


Contents Executive Overview ......................................................................................................... 3

1. Industry Setting and Problem ................................................................................. 4

1.1 12-Factor Applications .................................................................................................................... 4

1.2 Build to Manage .............................................................................................................................. 5

2. Shortcomings of current approach .......................................................................... 6

3. Building Production-grade Services ........................................................................ 7

3.1 Guidelines ....................................................................................................................................... 7

3.2 Automated Service Checks ............................................................................................................. 7

3.3 Service Scorecard ........................................................................................................................... 8

3.4 Service Controls .............................................................................................................................. 8

4. Examples and Proof Points .................................................................................... 9

4.1 IBM Cloud Service Contract ............................................................................................................ 9

4.2 LinkedIn Service Scorecard .......................................................................................................... 13

5. Critical Success Factors ....................................................................................... 14

5.1 Shared Scorecard / Dashboard .................................................................................................... 14

5.2 Consequences .............................................................................................................................. 14

5.3 Capability Focus and Client Impact .............................................................................................. 14

6. Closing .................................................................................................................. 15

6.1 About the Authors ......................................................................................................................... 15

6.2 Additional References ................................................................................................................... 15

6.3 For more Information..................................................................................................................... 15


Executive Overview As the application landscape transitions towards micro-services, businesses are looking for guidelines and processes to assure that these services are robust and serviceable. Several guidelines exist for developers to follow when building modern web-based applications and can be seen as the best practices for creating micro-services. A good example is the Twelve-Factor App manifesto. In addition, guidelines exist that describe how to build more manageable services,

Most of the discussion on building micro-services is targeted to the technical elements. They describe the aspects that micro-services need to adhere to support and leverage dynamic environments, for instance automatic scaling based on load. However, there is little discussion on what it means to build production-ready micro-services. This includes technical aspects, but also organizational, process and cultural elements.

In order to address the shortcomings mentioned below, clear guidelines need to be defined, communicated and enforced. We describe guidelines that help assess if a micro service meets the requirements of being operationally ready and in high quality. These guidelines are automatically checked to remove any ambiguity in interpreting the guidelines. Results of these checks are weighted by relevance and then aggregated towards an overall score, published in scorecards and dashboards.

Like all lawmaking systems, there are consequences to not following the law. If a Service fails to comply with the rules within the Service Contract within the specified time in which those rules take effect, it will result in percussions. At the same time, adhering to the rules should provide a direct incentive in doing so. This approach of pain and gain (also known as “carrot and stick”) has shown positive effects in several organizations.

To show evidence of the approach and its benefits, two examples are provided: The Service Contract in IBM Cloud and the Service Scorecard used within LinkedIn.

The paper closes in describing critical success factors making the approach feasible and scale as the number of services increases.


1. Industry Setting and Problem Today’s application landscape is dramatically changing, from monolithic applications running in traditional data centers towards micro-services running in the cloud. This results in improved agility and velocity, allowing companies to quicker respond to market opportunities, challenges and changes. However, the result is also a large number of loosely coupled components that needs to be managed together to assure availability and performance of services provided to end users.

Enterprises have gained a reasonable handle on developing applications using new approaches like Lean and DevOps, and have started following a Cloud-first approach to releasing new functionality. After their early experience these enterprises transition to the next phase, looking for a more systematic approach to governance, controlling the onboarding of new services and the production-grade management of hundreds to thousands of services, integrated loosely together through an API economy.

Key questions asked

• How do you adopt new services?

• How to assure Quality, Performance, and Availability of end-user facing applications?

• What is the behavior we need to request from our services?

While the focus of this paper is on micro-services, most of the discussion can be applied also to more coarse-grained applications.

1.1 12-Factor Applications

Platform-as-a-service provider Heroku maintains a manifesto of sorts called The Twelve-Factor App. It outlines a methodology for developers to follow when building modern web-based applications and can be seen as the best practices for creating micro-services. The twelve-factor app is a methodology for building software-as-a-service apps that:

• Use declarative formats for setup automation, to minimize time and cost for new developers joining the project;

• Have a clean contract with the underlying operating system, offering maximum portability between execution environments;

• Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration;

• Minimize divergence between development and production, enabling continuous deployment for maximum agility;

• And can scale up without significant changes to tooling, architecture, or development practices. The twelve-factor methodology can be applied to apps written in any programming language, and which use any combination of backing services (database, queue, memory cache, etc.).

1. Codebase - One codebase tracked in revision control, many deploys 2. Dependencies - Explicitly declare and isolate dependencies 3. Configuration - Store configuration in the environment 4. Backing services - Treat backing services as attached resources 5. Build, release, run - Strictly separate build and run stages 6. Processes - Execute the app as one or more stateless processes 7. Port binding - Export services via port binding 8. Concurrency - Scale out via the process model 9. Disposability - Maximize robustness with fast start-up and graceful shutdown 10. Dev/prod parity - Keep development, staging, and production as similar as possible 11. Logs - Treat logs as event streams 12. Admin processes - Run admin/management tasks as one-off processes


1.2 Build to Manage

Continuous Deployment is a key theme in the cloud world, which means that Operations have significant less time to build the required knowledge, and the opportunity to apply this knowledge is much shorter. Therefore, we need a different approach to management. Instead of Operations figuring out their tasks in isolation, Operations works with Development in order to determine how to manage the application. In DevOps, developers already took control of one important aspect of operations: Deployment and Release of the application. However, there are more things developers should do to ease operations.

As organizations are working on building out a sustainable culture, we recognize the need for some simple specific steps to follow to start getting some of the benefits in the short term. To this end, IBM is introducing a new approach to operations which is called “Build to Manage”. It specifies the practice of activities developers can do in order to instrument the application, or provide manageability aspects as part of an application release.

The “Build to Manage” approach includes the following aspects:

- HealthCheck API - Log Format and Catalog - Deployment correlation - Distributed Tracing - Topology Information - Event Format and Catalog - Test Cases and Scripts - Monitoring Configuration - Runbooks - First Failure Data Capture - Documentation

See https://github.com/ibm-cloud-architecture/build-to-manage for a description of the IBM Build-To-Manage approach.


2. Shortcomings of current approach Most of the discussion on building micro-services is targeted to the technical elements. They describe the aspects that micro-services need to adhere to support and leverage dynamic environments, for instance automatic scaling based on load.

However, there is little discussion on what it means to build production-ready micro-services. This includes technical aspects, but also organizational, process and cultural elements. Unless developers have an intrinsic interest in building robust services in high-quality, they tend to fall back to old habits – favoring new functionality over operational (“non-functional”) aspects. Understanding the production needs of a service – security, compliance, resilience, availability, performance, etc. – requires not only the implementation of the technical enablers, but also a high-performing organization and agile, integrated processes.

Even if DevOps is implemented across the organization with teams that have end-to-end responsibility, there is a need to govern at scale across the organization. New operating models such as Site Reliability Engineering (SRE) can become a key enabler to an efficient operating culture. But unless there are well defined rules that development teams have to adhere to, even SRE organizations will be less efficient operating a cloud-enabled landscape at scale.


3. Building Production-grade Services In order to address the shortcomings mentioned below, clear guidelines need to be defined, communicated and enforced.

3.1 Guidelines

The organization needs to clearly state what constitutes service quality. This includes not only technical aspects, but also approaches to embed and sustain the practices throughout the DevOps process.

Examples for technical guidelines:

- HealthCheck API implemented - Service is stateless - Service doesn’t use any local files (logfiles, configuration) - Version currency (i.e. latest version of libraries used) - Topology (i.e. topology information available in topology database) - First Failure Data Capture

Examples for non-technical guidelines and approaches:

- Service owner (“Product Manager”) defined - SRE team defined - Runbooks provided - Documentation: Availability of service documentation such as architecture, key metrics, configuration

and tuning guidelines - Chaotic Testing to assure resiliency of individual services, as well as on the application level - Regular validation of consistent backup and restore capability - Deployment cadence - Rack Diversity

3.2 Automated Service Checks

These guidelines will be checked automatically against every service using a compliance checking framework. There is full transparency what will be checked to assess the service quality. Automated checking resolves any ambiguity in the interpretation of the guidelines (Does it comply? How bad is it?), and provides a clear roadmap to the service owner what to do to improve its grading (Why is it bad? How to fix?). Check scripts also allow the organization to scale with the amount of services, guidelines, and frequency of checking.

The automated checks fetch data from the code repositories, from build or deployment systems or from configuration databases and management systems. Each check returns a standardized result (typically yes/no or a value in a scale from 0 to 100), and ideally a message text providing background / description on the check performed.

Checks should also performed regularly as the service quality will change over time: Good services can turn if not taken care of; bad services can turn good magically. Also, guidelines will evolve over time: new policies are added, existing policies are updated based on new information, some policies may no longer be relevant.

Automated checks builds a model to measure service quality at scale and assist organizations on what to do to improve the quality.


3.3 Service Scorecard

A service scorecard provides an aggregated summary of the quality of all service and shows the results of each check for each service.

Each individual check returns a normalized output and is weighted in accordance to the importance of the requirement. Weighting is driven by the relevance to reliability, if the item causes customer issues or affects the overall experience. Weighted results are then aggregated to a unified score. This normalized view provides a way to measure the production quality for each service and the improvement over time.

The scorecard shows the scores for each service and is published within the company. The transparency is not to blame the development teams, but to provide a clear guidance where improvements are needed most. For each service it shows how to improve the overall score. It also provides a common goal across the company, spanning development and operations. It empowers everyone to improve reliability.

Such a “high score” scorecard also motivates the developers, applying “Gamification” concepts to push service quality.

3.4 Service Controls

Every service has to comply with defined rules to assure its operational readiness. Like all lawmaking systems, there are consequences to not following the law. If a Service fails to comply with the rules within the Service Contract within the specified time in which those rules take effect, it will result in percussions. At the same time, adhering to the rules should provide a direct incentive in doing so. This approach of pain and gain (also known as “carrot and stick”) has shown positive effects in several organizations.

Examples of Service Controls:

- no SRE support for poorly graded services (developers need to provide 24x7 support of services with bad quality)

- A-grade services can deploy 24x7 - No deploy moratorium for A-grade service - Priority build for good services - Dispatch “SWAT” teams for F-grade services: direct support of the development teams to improve

the service quality and their skills. SWAT teams leverage their experience and knowledge in turning bad services to good.

The service score card further helps allocating the SRE headcount to the right services and components.

Note that SRE support does not mean that the Development / DevOps team will not have to participate in incident management activities. A key element of the SRE concept is that excess operational work will overflow to the development team. At the same time, a certain portion of the operations work (for instance 5%) is always shared with the development team.


4. Examples and Proof Points To show evidence of the approach and its benefits, two examples are provided: The Service Contract in IBM Cloud and the Service Scorecard used within LinkedIn.

4.1 IBM Cloud Service Contract

A fundamental principle of IBMs squad and tribe system is that the squad is the primary unit of decision-making. This is intended to greatly empower the squads to adapt to their users, their competitors, and decide how to most effectively work together to deliver and operate their services. The risk to this model is that if the decision-making is completely decentralized, the result can be that different decisions by different squads result in bad outcomes for the business as a whole. To mitigate this risk, the Service Contract was created with the purpose to establish the "laws" that all squads and services must follow.

To determine which aspects should be put into law (and which should not), the following test was used:

If squads were allowed to independently make decisions on a given issue that would

• result in the user seeing those different decisions as a fragmented experience or

• result in IBM being unable to manage the business

then the Service Contract will make a call as to how that decision will be implemented consistently across the squads. It is also important to note that IBM favored consistency over perfection, considering practical concerns about the cost and complexity to implement the decisions contained in the Contract. More than 100 people across organizations helped shape the definition of the IBM Cloud Service Contract.

The Service Contract applies to all Services consumed from the Bluemix catalog, and all supporting services upon which those are built, described as follows:

• User-facing service: this is perhaps the easiest to understand, as it is a service that appears in a catalog for a user to choose. Virtual Servers, Containers, Object Storage, Cloudant, Watson Dialog Services and many others are examples of these types of services.

• Infrastructure service: this type of service is not directly consumable by a user, but it provides capabilities that user-facing services access to deliver their capabilities. Configuration services like etcd would be examples of these types of services. Tools such as Sensu would also be an example of these types of services. They typically are supposed to run always and listen for requests.

• Internal tools: this is something our internal teams use in an ad hoc manner to perform administrative tasks. They do not face the user nor are they used by user-facing services. Tools such as a Vyatta configurator would be an example of this type of service. They behave more like command line functions – executed when needed, have a short lifespan and then exit.

IBM has identified a few key control points so that a simple mechanisms can be introduced in those control points to enable, and where necessary, enforce the elements within the Service Contract. The following picture maps those Control Points visually:


The Build and Deploy control point is a hook that each squad will call at the point where they initiate a build, initiate a deployment, and complete a deploy. It can be used regardless of the choices a given squad has made on their tool chain to make it easier to adapt and consume. This control point will be used to verify that different elements in the Service Contract are in place (notifying the squad of things they may need to correct), verify that deployment is allowed at that time (which might not be the case if a freeze window is in place or the environment is not stable enough to take changes that are not solving the instability), and to automatically post required metrics our business wants to gather around efficiency.

The Compliance Agent control point is an agent all services must embed within their running instances so that we can assure our users that our platform remains compliant against our promises (particularly security) always. It is not enough to fill out a document before going live to guarantee compliance – in the world of continuous delivery we are only as reliable as the last review we performed. The compliance agent will constantly check for various conditions mandated by the Service Contract to ensure they remain in place, and will notify the squads in case an exception is identified so it can be rapidly remediated. Special care was taken in choosing the compliance agent based on a very low and controllable overhead, market proven capabilities, and flexibility to verify a number of different configurations and policy conditions.

The Standard Certified Image control point is intended to preconfigure images that services use to deploy upon. This will simplify the ability for all squads to comply with several elements in the Service Contract without unique work – security configurations, monitoring collectors and other standard tools and artifacts. This control point is not yet part of the Service Contract included as an item that may be introduced later.

Consequences of not following the Service Contract Rules

Like all lawmaking systems, there are consequences to not following the law. If a Service fails to comply with the rules within the Service Contract within the specified time in which those rules take effect, then that Service may be restricted temporarily until the results are achieved. Note that the CTO office and the service squad will make every attempt to collaborate to resolve any problems preventing success prior to taking any of the measures below:

• All production deploy requests will be rejected except those that will put that Service in compliance with the Service Contract

• In cases where significant security exposures are present, that Service could have its runtimes suspended and quarantined


• In cases of persistent noncompliance, that Service may be disabled from onboarding new users and/or lowered to Experimental status in the catalog. In extreme cases the Service could be removed from the catalog entirely.

Throughout this process, guidance and consulting will be available for the Service’s development squads if desired. Once the rules can be shown to be followed, the Service will be restored to its previous state.

With this all said, it is again important to state that these rules are intended to greatly help the experience of the users and help IBM run its business rather than provide a tax for which there is no benefit.

4.1.1 Sections of the Service Contract

Cloud Resource Naming

Cloud Resource Naming (CRN) is a standardized and unique technical name for all resources in the Cloud Platform.

Development Efficiency and Change Metrics and Service Metadata

This includes Service Manifest, Continuous Delivery Metric, Mean Deployment Success Rate, and Change record. There are several metrics that are required in this section, but most will be generated automatically.

Health Metrics

Each Service is responsible for collecting the following data and publishing it to the specified tool or repository.

Metrics required:

• Provisioning (Control Plane) Availability

• Canary Test (Consumption Metric)

• User Errors

• End User Response Time

• High Severity Customer Tickets

Incident Metrics

Each Service is responsible for following the standard procedures and mechanisms here, which will generate the required metrics on behalf of the squads.

Mechanism required:

• Alert Naming Convention

• Requirements for all levels of production availability

Metrics required:

• Mean Time Between Failure (MTBF)

• Mean Time to Acknowledgement (MTTA)

• Mean Time to Restore (MTTR)

• Support Time to Resolution


Root Cause Metrics

Each Service is responsible for following the standard procedures and mechanisms here, which will generate the required metrics on behalf of the squads.

Mechanism required:

• Client Impacting Events

• Incident Management

• External Notification

• Root Cause Analysis Requirements

Metrics required:

• Number of CIEs

• Root Cause Analysis (RCA) Backlog

Security Practices and Configuration

This section describes the security features, compliance requirements and best practices that are to be employed by the service.

Metrics required:

• Assigned Security Focal

• Cloud Security Baseline and Audit Logs

• Secure Development Practices

• Services offering Security Functions

• Access Management

Console UI Integration

All user-facing services that choose to expose a user interface for manipulating their service are expected to integrate with the standard Bluemix console.

Usage and Revenue Metrics

All services (free and paid) need to instrument usage metering on a frequent basis into the Cloud BSS system. Data will be exported into a Segment Data Warehouse and made available for reporting.

Metrics required:

• Active Unique

• Usage Churn

• Usage Growth

• Revenue Churn

• Revenue Growth

• Service monthly recurring revenue (MRR)

• Net Promoter Score (NPS)

Catalog Onboarding

To add or update a service within the Bluemix catalog, squads must follow the Standard Operating Procedure (SOP) system, which requires a set of information to be provided, much of it related to billing definitions.


4.2 LinkedIn Service Scorecard

LinkedIn has implemented a service scorecard assisting the SRE mission to make sure that the services are as fast and as reliable as possible.

A presentation by Daniel Lawrence can be found at the SREcon17 conference website at this link: https://www.usenix.org/conference/srecon17americas/program/presentation/lawrence

Service Scorecard

Results for a given service


5. Critical Success Factors This Point of View document lists many aspects of building production-ready services. In this chapter, we want to highlight and reiterate on some of the critical success factors to make this approach viable and scale in an organization.

5.1 Shared Scorecard / Dashboard

Only if one knows what actually needs improvement, appropriate actions can be taken to address these shortcomings. We suggest automated checks to resolve any ambiguity in interpreting any guidelines. Weighting further enables the business to provide focus to the elements that are most relevant (for reliability, for user experience, etc).

An important aspect is the shared visibility of the results within the company using a scorecard / dashboard. This clearly shows where help is needed most (and more importantly, why something is graded a certain way), and acts as a management instrument driving the right actions.

However, it is important to not misuse the scorecard to blame people. A blameless culture is vital to drive the right behavior in the organization and not promote a culture of fear of punishment and reprimands.

Through gamification concepts like high scores, the system motivates people to go the extra mile in improving their scores. “Hackdays” are a great instrument to pair focused improvements with team-building and fun.

5.2 Consequences

The scorecard mentioned above provide clear guidance on how to get higher grades. But unless there are consequences to the scores, there is a high risk that the right behavior does not take place. We propose a combination of incentives and disincentive, frequently referred as carrot and stick.

On one side there should be tangible benefits for highly-graded services, for instance not imposing a deployment memorandum. On the other side, poorly graded services should result in undesired consequences, for instance the requirement that developers need to operate their services themselves.

See chapter 3.4 for examples.

5.3 Capability Focus and Client Impact

It is important to understand how individual services contributes to the overall user experience and satisfaction. Although technologies like “circuit breakers” and “resiliency” are introduced to reduce the impact of individual failures, the risk that the user is negatively affected by an error continues to exist.

Application and service monitoring produces events or alerts and these are correlated with other events from the environment including network, (cloud) platform updates and maintenance to discern source of an issue before human interaction occurs.

In the case of an alert could negatively affect the availability or user experience, the fact should not just be handled within the operations team. Mature organizations provide transparency to their end-users by notifying (potential) customer impacting events to their user base.

This is particular important where the services are consumed programmatically. The other side will have performed monitoring on their end and may have spotted the impact to its service. Providing transparency will help them separating cause and effect, enabling the consumer to focus on the right activities rather than chasing red herrings.


6. Closing On behalf of IBM, the authors wish you luck in building production-ready micro-services in high quality. Feel free to contact the authors for more information or to provide feedback on this Point of View paper.

6.1 About the Authors

Ingo Averdunk is a Distinguished Engineer in IBM Cloud and leads the Service Management and Operations practice in the Cloud Adoption, Garage Method and Solution Engineering organization. His work has been embraced by IBM clients around the globe. Ingo is based in Munich, Germany.

6.2 Additional References

Production-Ready Microservices Building Standardized Systems across an Engineering Organization By Susan J. Fowler Publisher: O'Reilly Media, 2016

6.3 For more Information

To request a private briefing, email [email protected]. To learn more about IBM’s work with Cloud, please contact your IBM representative or IBM Business Partner, or visit the following websites: https://www.ibm.com/cloud/garage and https://developer.ibm.com/architecture/ We welcome your comments on this article. Please send them to [email protected]. © Copyright IBM Corporation 2017 IBM Software Group Route 100 Somers, NY 10589 U.S.A. Produced in the United States of America February 2016, All Rights Reserved IBM, the IBM logo, ibm.com, IBM Cloud and IBM Cloud Orchestrator are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at: ibm.com/legal/copytrade.shtml Other product, company or service names may be trademarks or service marks of others. References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates.

Pov Building Production-Grade Services v1.2...Executive Overview As the application landscape...

Documents

Transcript of Pov Building Production-Grade Services v1.2...Executive Overview As the application landscape...