Enterprise DevOps - Citihub
Transcript of Enterprise DevOps - Citihub
Integrity.Excellence.Results.
Enterprise DevOps:How to Maintain Segregation of Duties with CICD
www.citihub.com
Contents
3 Introduction
4 Understanding the Context
5 CICD Workow and Core Roles
7 Roles Matrix
8 Control Points
9 How to Maintain Automation
10 10 Conclusion
11 About Citihub Consulting
AuthorErhan Sen, Senior Consultant
Erhan is an innovative senior technology leader with many years of experience in design, development and
end-to-end delivery of large-scale banking and financial systems. He joined Citihub from Capgemini and has been
responsible for formulating, leading and governing enterprise architecture roadmaps, blueprint activities and
associated strategic technology transformations. Erhan has deep expertise in DevOps and CICD automation.
10
Introduction
DefinitionsSegregation of Duties (SoD)
Infrastructure as Code (IaC)
Segregation of duties (also known as
"separation of duties") is the concept of having
more than one person required to complete a
task. In business, this is an internal control
intended to prevent fraud and error. For
Financial Services firms, some regulators
specifically issue firm guidance around the
needneed for SoD in the software delivery lifecycle.
Here are a few examples of references made to
SoD:
o COBIT 5 -> APO01.02
o OSA -> AC-5, AC-6
o ISO 17799 -> 10.1.3, 10.6.1, 10.10.1
o PCI-DSS v2 -> 7.1.2
o NIST 800-53 -> AC-5
o SOX 404, referenced by list of accepted
internal controls
o MAS-TRM -> 7.2.4, 11.0.1.b, F.2.2.a
InfrastructuInfrastructure as Code (IaC) is the process of
managing and provisioning computing
infrastructure (processes, bare-metal servers,
virtual servers, etc.) and their configuration
through machine process-able definition files,
rather than physical hardware configuration or
the use of interactive configuration tools.
•
•
The cost and time-to-market benefits of DevOps and CICD are well advertised but the benefits in helping to manage risk are less
well understood. As discussion develops towards CICD pipelines that are heavily automated and preparing critical code for
deployment in short cycles, the first reaction from risk managers and delivery executives can be one of fear.
In this paper, we explore one of the main fears: that segregation of duties will be compromised.
The practice of integrating, building, testing
and releasing code in short cycles with the help
of automation
ContinuousContinuous Integration: integrating code into a
shared repository frequently (preferably at least
once a day). Each check-in is then verified
through an automated build, allowing teams to
detect problems early
ContinuousContinuous Delivery: Preparing the release
build for production by automated validation
and verification techniques
Note:Note: Although it is out of scope of this article,
the reader should understand that Continuous
Deployment goes one more step further than
Continuous Delivery so that an automatically
verified and prepared release build is also
automatically deployed to production
environments along with any related
infrastructuinfrastructure changes
•
•
•
•
CICD: Continuous Integration (CI) / Continuous Delivery (CD)
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
3
Understanding the Context
Given that the intention of CICD is to automate and speed up
the process of delivering code, there is a genuine risk that a
single individual could be given the ability to develop, compile
and move code from one environment to another. And hence
either maliciously, or by error, introduce risk to a firm’s
production environment. Moreover, a loophole or mistake can
expose confidential information, or cause significant
downtime downtime resulting in reputational or financial loss for the firm.
Although DevOps marketing promotes freedom and
efficiency, it is not the intention of CICD to give a single
developer all this power. The risks can be avoided with the
proper controls in place without compromising too much on
efficiency and automation.
•
•
•
•
Understanding the Risks
Having said that, regulators and auditors want to see proper
implementations which mitigate risks rooting from conflicts of
interest and excessive rights. Apart from employee
entitlements and separation of toxic combinations of roles,
there are other requirements that need to be observed in
DevOps systems in order to complete the whole SoD picture
including:
Proof of proper controls and justification of any type of
change in the system against an approved change
request are fundamental controls from an auditor’s
perspective.
Each action in the process should be designed carefully
so that they can receive, pass and log change
information.
AuditorsAuditors require audit trails throughout the delivery
process, bringing traceability and auditability
requirements into the picture.
Finally, auditors will question the implementation,
currency and integrity of governance rules and policies to
measure adherence to compliance, standards and best
practices.
Understanding the Regulators and Auditors
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
4
Roles
Quality Gate/Check Point/Approval
Operations Admin
DeveloperTester
Peer or Lead Developer
Build Engineer
Infrastructure/Database Admin
DevOps Tool Admin
Coding Guidelines Architecture Standards Risk & Compliance Policies Training Materials
CodeIntegration
SecurityScan
Build
BinaryRepositoryMng
Open SourceManagement
Code Quality& ComplianceScan
Change Management
RequirementsGathering Design Configuration & Environment Management
Deployment
Fault Detection
Incident Management
ProjectPlanning
AnalysisReview
Coding
Task/IssueAssignment
Test CaseDevelopment
Code Review(Peer)
PerformanceTesting
UnitTesting
Verification & Validation
Knowledge Management, Enterprise Governance Continuum & Collaboration
RegressionTesting
UX Testing FunctionalTesting
UI Testing
Release Management & Pipeline Automation
QG1 QG2 QG3 QG4 QG5 QG6
ProjectInitiation & Planning
Analysis DevelopmentIntegration Q&A UAT
Validation & VerificationProduction
Audit RepositoryDevOps Intelligence
Management Dashboards Performance Metric Repository
Plan Deliver Service
Continuous Deployment
Continuous Delivery
Continuous Integration
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
Citihub’s DevOps reference model is designed to bring compliance to regulators’ SoD requirements without sacrificing agility and
speed. The picture below depicts a workflow with quality gates and a distribution of roles which can achieve our goals.
5
CICD Workflow and Core Roles
Developer: Creates or modifies code and is responsible for
requesting the release of their components into production.
Developers should not be able to manipulate a system,
database, test results, DevOps tool configuration or DevOps
process outside of the development environment. Any
environment higher in the stack, such as testing and
production, should be considered off limits for the
development teams’ didevelopment teams’ direct access.
Tester: Develops test cases as well as automated functional
tests. Testers should not have rights to develop or deploy
code, manipulate collected test results, manipulate a system,
database, DevOps tool configuration or DevOps process.
Code Reviewer: As a second pair of eyes to review code to
identify non-compliant or malicious code. Developers should
not be able to review or approve their own code.
BuildBuild Engineer: For each deployable package this role
designs, develops and configures the process which checks
the code out from the version control system and outputs a
deployable package. It is uncommon to have a dedicated
individual performing a build engineer role, but combining the
role with other roles especially with developer role, introduces
risk. The role is critical to ensure standardization and
adheadherence to security, risk and compliance policies. However,
the role has the power to manipulate results, rules and tasks
associated with continuous integration process flow. The
build engineer role should not be given to developers.
Moreover, build engineers should not have write access to
source code repositories. If developer and build engineer
roles are combined, additional compensating controls are
rrequired for example, a second reviewer should be
responsible for review and approval for the build process
configuration.
Infrastructure Engineer: Develops virtual infrastructure
components (infrastructure as code - IaC) that can later be
deployed into production. Combining this role with roles that
have write access to production carries risk in the same way
that allowing developers, with write access to source code,
the ability to deploy code into production. Where possible,
developers should not be given this role and infrastructure
engineersengineers should not be given the ability to deploy into
production. The combination of code development and
system configuration is extremely powerful; therefore, these
roles must be separated. If an organization is small or
separation of these roles will bring high cost, there should be
other controls to be in place for eliminating the mentioned
risks.
DevOps Tools Administrator: Responsible for the
configuration of tools and user role mapping in the DevOps
pipeline. Release process configuration, tool configurations
and governance of rules implemented on tools are managed
by this role. This role should be isolated from any type of
development and infrastructure related activities, as the
combination of being able to produce new code and manage
thethe tools that promote code into production would afford a
single user the ability to compromise your application or
infrastructure.
Operations Engineer: Responsible for configuration
management, provisioning and deployment of assets to
environments. In traditional architectures, they design and
develop scripts to manage operational tasks. In a DevOps set
up which includes an automated integrated deployment
application, this role designs deployment processes and
configures related deployment environments on the tool. This
rolerole should be isolated from any code development related
and DevOps administration activities.
Role Distribution and ResponsibilitiesTo ensure segregation of duties within the release process, roles should be distributed to different parties dividing the accountability
for creation of code and the distribution and installation of that code. The priorities should be to ensure least privilege and prevent,
or at least, detect attempts to promote unsafe or malicious code. Yet, not all roles are critical, more emphasis is placed on roles
which can directly manipulate code, environments and controls. These roles are listed below.
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
6
Roles Matrix
The following figure represents a roles matrix, sometimes called a toxic combination matrix, to show the segregation of duties risks.
Roles
Coding
Code Review
Design Functional Test Case
Design Build Process
Run Build Process
Run Any Test
Attest QA Results
DevOps Tools Configuration
Manual Deployment
Automated Deployment Request
Change/Write Production Systems
Tasks
Developer
Code Reviewer
Tester
Build Engineer
DevOps Tool Administrator
Operations Engineer
Infrastructure Engineer
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
7
Control Points
QG1 – Analysis and Test Case Review: At this stage, the analysis document is evaluated with respect to coverage,
completeness, business rules and test cases. The Business
Analyst Lead, QA Lead or peers can review and approve them
for further implementation.
QG2QG2 – Code Readiness: : Automated Code Review and Peer Review are the two main pillars of this quality gate. Meticulous
machine and/or human inspection of the code (software or
IaC configuration item) to check adherence to coding,
architecture, security and performance guidelines.
Additionally, by collecting unit test results from developer’s
development workbench at this quality gate, this information
cancan be used as evidence to verify a developer code’s
compliance to policies requiring them to testing prior
deployment.
QG3 – Continuous Integration: This quality gate includes performing several series of detailed tests and
measurements. Unit, regression, security, performance,
functional, load and open source compliance tests can be run
and greatly help to achieve DevOps goals. There are two
potential types of actions that can be taken in the event of a
violation or failure, depending on an organization’s policies.
Either the process can be suspended, pending a fix or it can
be continued with a proper waiver by debiting this issue to a
technical debt account. With an extensive control set and a
variety of tests at this quality gate, some organizations may
choose to solely rely on the results here for the decision to
promote code into production.
QG4QG4 - IT QA: This quality gate is present in most enterprises, manual functional, UI and UX tests are performed. This stage
is IT’s last test activity in the whole process.
QG5QG5 - Final Release Assessment: This is the final quality gate before production. It is possible to feed previous
measurements, results, task acknowledgements, approvals,
and waivers via an automated process to the release
manager, team leads and IT managers so that they can judge
the risk of a release.
QG6QG6 – Post Production: After deployment to production environments, production problems are monitored and
collected. Depending on the number and severity of issues,
there is a decision point here to potentially rollback to the last
known healthy state.
Citihub recommends 6 quality gates through the flow of the DevOps delivery pipeline. All 6 quality gates can be automated to some
extent or be fully manual depending on an organization’s maturity, culture, risk appetite and application portfolio.
Quality Gates
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
8
How to Maintain Automation
The mix of manual controls vs. automated controls will be very
dependent on each organization’s maturity, attitude to risk,
regulatory environment and the Software Development
Lifecycle style in use. An optimal balance between ensuring
that risks are fully mitigated and maximizing the benefits of
DevOps principles (faster time to market etc.) will require
buy-in and approval from a number of stakeholders in the
ororganization. Some may argue that manual controls can be
more robust and that a human can catch things that
automation simply cannot; however, automation improves
standardization, controls reliability and can be complimentary
to manual testing.
One of the biggest things that DevOps brings to delivery and
release management is high standardization with automation
yielding strict enforcement of policies—this is highly desired
by management and auditors. Our model, while automating
the process as much as possible, aims not to undermine the
manageability and audit requirements. Especially, at “Quality
Gate 3 – CI”, there is significant automation and verification
datadata collection. CI tool controls the automated chain. Human
intervention can be eliminated, or at least minimized, by CI’s
role and SoD compliance will improve. When incorporating
multiple testing and scanning tools, highly reliable,
dependable and detailed test results can be achieved. It is
even possible to achieve nearly 100% test coverage after
investing in culture changes, tools and unit tests. This set of
contcontrols improves transparency and identification of issues
without relying on human insight in most scenarios. In case an
unacceptable condition is spotted during tests, the delivery
process can be suspended for a remediation by a human
being.
Another area that automation can be deployed to is the
process execution platform by utilizing an application release
automation (ARA) tool. ARA tooling can be deployed to give
extensive data collection throughout the DevOps pipeline.
Configurations cannot be adjusted in production and will
be mirrored throughout the staging environments
simplifying problem management and removing the risk of
erroneous or malicious changes
TheThe latest infrastructure and security code will be built into
each release. Considering the concept of small, frequent
releases, the implication is that patch management
becomes a continuous process rather than something
requiring significant effort, and the risk of relying on
end-of-life components is greatly reduced.
•
•
An ARA tool or a process execution platform would drive the
entire release process to all environments with solid rules and
flows which guarantee compliance. It makes decisions based
on QG results and SDLC metrics to move forward or raise an
issue and request human intervention. It carries change,
release, task and build IDs throughout the pipeline to create
end-to-end traceability, which as we mentioned above is one
ofof the most important audit requirements. When automation is
not possible or a waiver is needed, the tool collects
information electronically and populate the release repository,
thus augmenting valuable audit trails for risk and compliance.
IT managers and internal auditors can enjoy another huge
benefit as an outcome of intensive data collection which gives
a clear visibility of delivery pipeline internals, and provides
invaluable metrics for further possible optimization. invaluable metrics for further possible optimization.
Finally, when CICD and rapid releases are considered in
conjunction with container technology, enterprises have the
opportunity to take a step change improvement in risk
management. If the enterprise adopts the concept of
immutable infrastructure, which is an approach to managing
software deployments wherein components are replaced
rather than change, i.e. an application or services is effectively
re-deployedre-deployed each time a change occurs, the following
additional benefits can be reaped:
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
9
Conclusion
We understand concerns about DevOps and its effects on controlling the environment but
automation can also help to reduce risk. Every organization needs to spend time to fully
understanding the risks of a DevOps approach in their environment weighing the results they
want to achieve in order to fit with its culture, risk appetite and circumstances.
Automation is not evil.
Automation not only provides agility and efficiency, but is also capable of decreasing risk and
increasing security when those requirements are designed into the workflow.
Combine these benefits with the traceability advantages of well-implemented CICD, and it is a
future state worth striving for.
Automated DevOps pipelines enforce standardization
Embedded compliance rules can guarantee adherence to internal and external policies
Integrated code and security scans can fortify IT landscape by identification of
malicious code
• • •
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
10
Contact Us
About Citihub Consulting
EMEA
Richard Hamstead
Moor Place
1 Fore Street
London EC2Y 9DT
+44 800 028 1901+44 800 028 1901
Bellerivestrasse 201
8008 Zurich
+41 44 562 7101
Enterprise DevOps: How to Maintain Segregation of Duties with CICD
Citihub Consulting is a global, independent IT advisory firm with deep domain expertise across every layer of the
technology stack – from business applications and data platforms down to core infrastructure. From IT strategy,
architecture and solution development, through to cost optimisation, risk assessment and implementation – our
trusted experts deliver the right results for your business.
ForFor us consultancy is personal. We have a relentless commitment to great execution, integrity and client success.
We aim to redefine perceptions of our industry and our commitment to delivering the right results for our clients
has never changed, even as the business has grown consistently over the last decades.
For more information, please visit www.citihub.com
North America
Keith Maitland
500 Fifth Ave, Suite 1610
New York, NY 10110
+1 646-780-1150
The Dineen BuildingThe Dineen Building
140 Yonge Street, Suite 200
Toronto, Ontario, M5C 1X6
+1 437 886 8390
Asia Pacific
Steve Rutherford
3 Pickering Street
#01-64
Singapore 048660
+65 3152 2777+65 3152 2777
Flat 406-9, 4/F
Three Pacific Place
1 Queen’s Road East
Hong Kong
+852 8108 2777
11