OpenShift Container Platform 3.9
Installation and Configuration
OpenShift Container Platform 3.9 Installation and Configuration
Last Updated: 2019-12-03
OpenShift Container Platform 3.9 Installation and Configuration
OpenShift Container Platform 3.9 Installation and Configuration
Legal Notice
Copyright © 2019 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA isavailable athttp://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you mustprovide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United Statesand other countries.
Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.
Java ® is a registered trademark of Oracle and/or its affiliates.
XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United Statesand/or other countries.
MySQL ® is a registered trademark of MySQL AB in the United States, the European Union andother countries.
Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by theofficial Joyent Node.js open source or commercial project.
The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marksor trademarks/service marks of the OpenStack Foundation, in the United States and othercountries and are used with the OpenStack Foundation's permission. We are not affiliated with,endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Abstract
OpenShift Installation and Configuration topics cover the basics of installing and configuringOpenShift in your environment. Use these topics for the one-time tasks required to get OpenShiftup and running.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Table of Contents
CHAPTER 1. OVERVIEW
CHAPTER 2. INSTALLING A CLUSTER2.1. PLANNING
2.1.1. Initial Planning2.1.2. Installation Methods2.1.3. Sizing Considerations2.1.4. Environment Scenarios
2.1.4.1. Single Master and Node on One System2.1.4.2. Single Master and Multiple Nodes2.1.4.3. Single Master, Multiple etcd, and Multiple Nodes2.1.4.4. Multiple Masters Using Native HA with Co-located Clustered etcd2.1.4.5. Multiple Masters Using Native HA with External Clustered etcd2.1.4.6. Stand-alone Registry
2.1.5. RPM Versus Containerized2.2. PREREQUISITES
2.2.1. System Requirements2.2.1.1. Red Hat Subscriptions2.2.1.2. Minimum Hardware Requirements2.2.1.3. Production Level Hardware Requirements2.2.1.4. Storage management2.2.1.5. Red Hat Gluster Storage Hardware Requirements2.2.1.6. Optional: Configuring Core Usage2.2.1.7. SELinux2.2.1.8. Red Hat Gluster Storage
Optional: Using OverlayFS2.2.1.9. Security Warning
2.2.2. Environment Requirements2.2.2.1. DNS
2.2.2.1.1. Configuring Hosts to Use DNS2.2.2.1.2. Configuring a DNS Wildcard
2.2.2.2. Network Access2.2.2.2.1. NetworkManager2.2.2.2.2. Configuring firewalld as the firewall2.2.2.2.3. Required Ports
2.2.2.3. Persistent Storage2.2.2.4. Cloud Provider Considerations
2.2.2.4.1. Overriding Detected IP Addresses and Host Names2.2.2.4.2. Post-Installation Configuration for Cloud Providers
2.3. HOST PREPARATION2.3.1. Setting PATH2.3.2. Operating System Requirements2.3.3. Host Registration2.3.4. Installing Base Packages2.3.5. Installing Docker2.3.6. Configuring Docker Storage
2.3.6.1. Configuring OverlayFS2.3.6.2. Configuring Thin Pool Storage2.3.6.3. Reconfiguring Docker Storage2.3.6.4. Enabling Image Signature Support2.3.6.5. Managing Container Logs
21
222222222323232323242425252525252527282929303030313131323334343434373838393939394041
42424343464647
Table of Contents
1
2.3.6.6. Viewing Available Container Logs2.3.6.7. Blocking Local Volume Usage
2.3.7. Ensuring Host Access2.3.8. Setting Proxy Overrides2.3.9. What’s Next?
2.4. INSTALLING ON CONTAINERIZED HOSTS2.4.1. RPM Versus Containerized Installation2.4.2. Install Methods for Containerized Hosts2.4.3. Required Images2.4.4. Starting and Stopping Containers2.4.5. File Paths2.4.6. Storage Requirements2.4.7. Open vSwitch SDN Initialization
2.5. QUICK INSTALLATION2.5.1. Overview2.5.2. Before You Begin2.5.3. Running an Interactive Installation2.5.4. Defining an Installation Configuration File2.5.5. Running an Unattended Installation2.5.6. Verifying the Installation2.5.7. Uninstalling OpenShift Container Platform2.5.8. What’s Next?
2.6. ADVANCED INSTALLATION2.6.1. Overview2.6.2. Before You Begin2.6.3. Configuring Ansible Inventory Files
Image Version Policy2.6.3.1. Configuring Cluster Variables2.6.3.2. Configuring Deployment Type2.6.3.3. Configuring Host Variables2.6.3.4. Configuring Project Parameters2.6.3.5. Configuring Master API Port2.6.3.6. Configuring Cluster Pre-install Checks2.6.3.7. Configuring System Containers
2.6.3.7.1. Running Docker as a System Container2.6.3.7.2. Running etcd as a System Container
2.6.3.8. Configuring a Registry Location2.6.3.9. Configuring a Registry Route2.6.3.10. Configuring the Registry Console2.6.3.11. Configuring Router Sharding2.6.3.12. Configuring Red Hat Gluster Storage Persistent Storage
2.6.3.12.1. Configuring Container-Native Storage2.6.3.12.2. Configuring Container-Ready Storage
2.6.3.13. Configuring an OpenShift Container Registry2.6.3.13.1. Configuring Registry Storage
Option A: NFS Host GroupOption B: External NFS HostOption C: OpenStack PlatformOption D: AWS or Another S3 Storage SolutionOption E: Container-Native StorageOption F: Google Cloud Storage (GCS) bucket on Google Compute Engine (GCE)
2.6.3.14. Configuring Global Proxy Options2.6.3.15. Configuring the Firewall
48484950515151515252535353535354555557575858585859596060666768707072737474757677777778797979798080808181
83
OpenShift Container Platform 3.9 Installation and Configuration
2
2.6.3.16. Configuring Schedulability on Masters2.6.3.17. Configuring Node Host Labels
2.6.3.17.1. Configuring Dedicated Infrastructure Nodes2.6.3.18. Configuring Session Options2.6.3.19. Configuring Custom Certificates2.6.3.20. Configuring Certificate Validity2.6.3.21. Configuring Cluster Metrics
2.6.3.21.1. Configuring Metrics StorageOption A: DynamicOption B: NFS Host GroupOption C: External NFS HostUpgrading or Installing OpenShift Container Platform with NFS
2.6.3.22. Configuring Cluster Logging2.6.3.22.1. Configuring Logging Storage
Option A: DynamicOption B: NFS Host GroupOption C: External NFS HostUpgrading or Installing OpenShift Container Platform with NFS
2.6.3.23. Customizing Service Catalog Options2.6.3.23.1. Configuring the OpenShift Ansible Broker
2.6.3.23.1.1. Configuring Persistent Storage for the OpenShift Ansible Broker2.6.3.23.1.2. Configuring the OpenShift Ansible Broker for Local APB Development
2.6.3.23.2. Configuring the Template Service Broker2.6.3.24. Configuring Web Console Customization
2.6.4. Example Inventory Files2.6.4.1. Single Master Examples
Single Master, Single etcd, and Multiple NodesSingle Master, Multiple etcd, and Multiple Nodes
2.6.4.2. Multiple Masters ExamplesMultiple Masters Using Native HA with External Clustered etcdMultiple Masters Using Native HA with Co-located Clustered etcd
2.6.5. Running the Advanced Installation2.6.5.1. Running the RPM-based Installer2.6.5.2. Running the Containerized Installer
2.6.5.2.1. Running the Installer as a System Container2.6.5.2.2. Running Other Playbooks2.6.5.2.3. Running the Installer as a Docker Container2.6.5.2.4. Running the Installation Playbook for OpenStack
2.6.5.3. Running Individual Component Playbooks2.6.6. Verifying the Installation
Verifying Multiple etcd HostsVerifying Multiple Masters Using HAProxy
2.6.7. Optionally Securing Builds2.6.8. Uninstalling OpenShift Container Platform
2.6.8.1. Uninstalling Nodes2.6.9. Known Issues2.6.10. What’s Next?
2.7. DISCONNECTED INSTALLATION2.7.1. Overview2.7.2. Prerequisites2.7.3. Required Software and Components
2.7.3.1. Syncing Repositories2.7.3.2. Syncing Images
84858586878788888889898990909090919191
92929494949696969799
100102103104105105106106108108110110111111111
112113113114114114115115116
Table of Contents
3
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.7.3.3. Preparing Images for Export2.7.4. Repository Server
2.7.4.1. Placing the Software2.7.5. OpenShift Container Platform Systems
2.7.5.1. Building Your Hosts2.7.5.2. Connecting the Repositories2.7.5.3. Host Preparation
2.7.6. Installing OpenShift Container Platform2.7.6.1. Importing OpenShift Container Platform Component Images2.7.6.2. Running the OpenShift Container Platform Installer2.7.6.3. Creating the Internal Docker Registry
2.7.7. Post-Installation Changes2.7.7.1. Re-tagging S2I Builder Images2.7.7.2. Configuring a Registry Location2.7.7.3. Creating an Administrative User2.7.7.4. Modifying the Security Policies2.7.7.5. Editing the Image Stream Definitions2.7.7.6. Loading the Container Images
2.7.8. Installing a Router2.8. INSTALLING A STAND-ALONE DEPLOYMENT OF OPENSHIFT CONTAINER REGISTRY
2.8.1. About OpenShift Container Registry2.8.2. Minimum Hardware Requirements2.8.3. Supported System Topologies2.8.4. Host Preparation2.8.5. Stand-alone Registry Installation Methods
2.8.5.1. Quick Installation for Stand-alone OpenShift Container Registry2.8.5.2. Advanced Installation for Stand-alone OpenShift Container Registry
CHAPTER 3. SETTING UP THE REGISTRY3.1. REGISTRY OVERVIEW
3.1.1. About the Registry3.1.2. Integrated or Stand-alone Registries
3.2. DEPLOYING A REGISTRY ON EXISTING CLUSTERS3.2.1. Overview3.2.2. Deploying the Registry3.2.3. Deploying the Registry as a DaemonSet3.2.4. Registry Compute Resources3.2.5. Storage for the Registry
3.2.5.1. Production Use3.2.5.1.1. Use Amazon S3 as a Storage Back-end
3.2.5.2. Non-Production Use3.2.6. Enabling the Registry Console
3.2.6.1. Deploying the Registry Console3.2.6.2. Securing the Registry Console3.2.6.3. Troubleshooting the Registry Console
3.2.6.3.1. Debug Mode3.2.6.3.2. Display SSL Certificate Path
3.3. ACCESSING THE REGISTRY3.3.1. Viewing Logs3.3.2. File Storage3.3.3. Accessing the Registry Directly
3.3.3.1. User Prerequisites3.3.3.2. Logging in to the Registry
119120121121121121122122122122123123123123124125125126126127127127128128128128130
133133133133133133133134134134135135136137137138139139139140140140142142143
OpenShift Container Platform 3.9 Installation and Configuration
4
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.3.3. Pushing and Pulling Images3.3.4. Accessing Registry Metrics
3.4. SECURING AND EXPOSING THE REGISTRY3.4.1. Overview3.4.2. Manually Securing the Registry3.4.3. Manually Exposing a Secure Registry3.4.4. Manually Exposing a Non-Secure Registry
3.5. EXTENDED REGISTRY CONFIGURATION3.5.1. Maintaining the Registry IP Address3.5.2. Whitelisting Docker Registries3.5.3. Setting the Registry Hostname3.5.4. Overriding the Registry Configuration3.5.5. Registry Configuration Reference
3.5.5.1. Log3.5.5.2. Hooks3.5.5.3. Storage3.5.5.4. Auth3.5.5.5. Middleware
3.5.5.5.1. S3 Driver Configuration3.5.5.5.2. CloudFront Middleware3.5.5.5.3. Overriding Middleware Configuration Options3.5.5.5.4. Image Pullthrough3.5.5.5.5. Manifest Schema v2 Support
3.5.5.6. OpenShift3.5.5.7. Reporting3.5.5.8. HTTP3.5.5.9. Notifications3.5.5.10. Redis3.5.5.11. Health3.5.5.12. Proxy3.5.5.13. Cache
3.6. KNOWN ISSUES3.6.1. Overview3.6.2. Image Push Errors with Scaled Registry Using Shared NFS Volume3.6.3. Pull of Internally Managed Image Fails with "not found" Error3.6.4. Image Push Fails with "500 Internal Server Error" on S3 Storage3.6.5. Image Pruning Fails
CHAPTER 4. SETTING UP A ROUTER4.1. ROUTER OVERVIEW
4.1.1. About Routers4.1.2. Router Service Account
4.1.2.1. Permission to Access Labels4.2. USING THE DEFAULT HAPROXY ROUTER
4.2.1. Overview4.2.2. Creating a Router4.2.3. Other Basic Router Commands4.2.4. Filtering Routes to Specific Routers4.2.5. HAProxy Strict SNI4.2.6. TLS Cipher Suites4.2.7. Highly-Available Routers4.2.8. Customizing the Router Service Ports4.2.9. Working With Multiple Routers
143144145145145148150151151152153153155155156156157157158159160160161
162163163163164164164164165165165165166166
168168168168168169169170170171172172172172173
Table of Contents
5
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.10. Adding a Node Selector to a Deployment Configuration4.2.11. Using Router Shards
4.2.11.1. Creating Router Shards4.2.11.2. Modifying Router Shards
4.2.12. Finding the Host Name of the Router4.2.13. Customizing the Default Routing Subdomain4.2.14. Forcing Route Host Names to a Custom Routing Subdomain4.2.15. Using Wildcard Certificates4.2.16. Manually Redeploy Certificates4.2.17. Using Secured Routes4.2.18. Using Wildcard Routes (for a Subdomain)4.2.19. Using the Container Network Stack4.2.20. Exposing Router Metrics4.2.21. ARP Cache Tuning for Large-scale Clusters4.2.22. Protecting Against DDoS Attacks
4.3. DEPLOYING A CUSTOMIZED HAPROXY ROUTER4.3.1. Overview4.3.2. Obtaining the Router Configuration Template4.3.3. Modifying the Router Configuration Template
4.3.3.1. Background4.3.3.2. Go Template Actions4.3.3.3. Router Provided Information4.3.3.4. Annotations4.3.3.5. Environment Variables4.3.3.6. Example Usage
4.3.4. Using a ConfigMap to Replace the Router Configuration Template4.3.5. Using Stick Tables4.3.6. Rebuilding Your Router
4.4. CONFIGURING THE HAPROXY ROUTER TO USE THE PROXY PROTOCOL4.4.1. Overview4.4.2. Why Use the PROXY Protocol?4.4.3. Using the PROXY Protocol
4.5. USING THE F5 ROUTER PLUG-IN4.5.1. Overview4.5.2. Prerequisites and Supportability
4.5.2.1. Configuring the Virtual Servers4.5.3. Deploying the F5 Router4.5.4. F5 Router Partition Paths4.5.5. Setting Up F5 Native Integration
CHAPTER 5. DEPLOYING RED HAT CLOUDFORMS5.1. DEPLOYING {MGMT-APP} ON OPENSHIFT CONTAINER PLATFORM
5.1.1. Introduction5.2. REQUIREMENTS FOR RED HAT CLOUDFORMS ON OPENSHIFT CONTAINER PLATFORM5.3. CONFIGURING ROLE VARIABLES
5.3.1. Overview5.3.2. General Variables5.3.3. Customizing Template Parameters5.3.4. Database Variables
5.3.4.1. Containerized (Podified) Database5.3.4.2. External Database
5.3.5. Storage Class Variables5.3.5.1. NFS (Default)
173173176178179180180181181
182183189189191
192193193194194194194195
200200201
202203205205205205206210210210211212213214
216216216217218218218219219219219
220221
OpenShift Container Platform 3.9 Installation and Configuration
6
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.5.2. NFS External5.3.5.3. Cloud Provider5.3.5.4. Preconfigured (Advanced)
5.4. RUNNING THE INSTALLER5.4.1. Deploying Red Hat CloudForms During or After OpenShift Container Platform Installation5.4.2. Example Inventory Files
5.4.2.1. All Defaults5.4.2.2. External NFS Storage5.4.2.3. Override PV Sizes5.4.2.4. Override Memory Requirements5.4.2.5. External PostgreSQL Database
5.5. ENABLING CONTAINER PROVIDER INTEGRATION5.5.1. Adding a Single Container Provider
5.5.1.1. Adding Manually5.5.1.2. Adding Automatically
5.5.2. Multiple Container Providers5.5.2.1. Preparing the Script
5.5.2.1.1. Example5.5.2.2. Running the Playbook
5.5.3. Refreshing Providers5.6. UNINSTALLING RED HAT CLOUDFORMS
5.6.1. Running the Uninstall Playbook5.6.2. Troubleshooting
CHAPTER 6. MASTER AND NODE CONFIGURATION6.1. INSTALLATION DEPENDENCIES6.2. CONFIGURING MASTERS AND NODES6.3. MAKING CONFIGURATION CHANGES USING ANSIBLE
6.3.1. Using the htpasswd commmand6.4. MAKING MANUAL CONFIGURATION CHANGES6.5. MASTER CONFIGURATION FILES
6.5.1. Admission Control Configuration6.5.2. Asset Configuration6.5.3. Authentication and Authorization Configuration6.5.4. Controller Configuration6.5.5. etcd Configuration6.5.6. Grant Configuration6.5.7. Image Configuration6.5.8. Image Policy Configuration6.5.9. Kubernetes Master Configuration6.5.10. Network Configuration6.5.11. OAuth Authentication Configuration6.5.12. Project Configuration6.5.13. Scheduler Configuration6.5.14. Security Allocator Configuration6.5.15. Service Account Configuration6.5.16. Serving Information Configuration6.5.17. Volume Configuration6.5.18. Basic Audit6.5.19. Advanced Audit6.5.20. Specifying TLS ciphers for etcd
6.6. NODE CONFIGURATION FILES6.6.1. Pod and Node Configuration
221222222222222223223223223224224224224225225225225226227227227227228
229229229229231
233233233234236236236238238239239240241
243244245245246247248249252254255
Table of Contents
7
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.6.2. Docker Configuration6.6.3. Setting Node Queries per Second (QPS) Limits and Burst Values6.6.4. Parallel Image Pulls with Docker 1.9+
6.7. PASSWORDS AND OTHER SENSITIVE DATA6.8. CREATING NEW CONFIGURATION FILES6.9. LAUNCHING SERVERS USING CONFIGURATION FILES6.10. CONFIGURING LOGGING LEVELS6.11. RESTARTING OPENSHIFT CONTAINER PLATFORM SERVICES
CHAPTER 7. OPENSHIFT ANSIBLE BROKER CONFIGURATION7.1. OVERVIEW7.2. MODIFYING THE OPENSHIFT ANSIBLE BROKER CONFIGURATION7.3. REGISTRY CONFIGURATION
7.3.1. Production or Development7.3.2. Storing Registry Credentials7.3.3. Mock Registry7.3.4. Dockerhub Registry7.3.5. APB Filtering7.3.6. Local OpenShift Container Registry7.3.7. Red Hat Container Catalog Registry7.3.8. ISV Registry7.3.9. Multiple Registries
7.4. DAO CONFIGURATION7.5. LOG CONFIGURATION7.6. OPENSHIFT CONFIGURATION7.7. BROKER CONFIGURATION7.8. SECRETS CONFIGURATION7.9. RUNNING BEHIND A PROXY
7.9.1. Registry Adapter Whitelists7.9.2. Configuring the Broker Behind a Proxy Using Ansible7.9.3. Configuring the Broker Behind a Proxy Manually7.9.4. Setting Proxy Environment Variables in Pods
CHAPTER 8. ADDING HOSTS TO AN EXISTING CLUSTER8.1. OVERVIEW8.2. ADDING HOSTS USING THE QUICK INSTALLER TOOL8.3. ADDING HOSTS
Procedure8.4. ADDING ETCD HOSTS TO EXISTING CLUSTER8.5. REPLACING EXISTING MASTERS WITH ETCD COLOCATED8.6. MIGRATING THE NODES
CHAPTER 9. LOADING THE DEFAULT IMAGE STREAMS AND TEMPLATES9.1. OVERVIEW9.2. OFFERINGS BY SUBSCRIPTION TYPE
9.2.1. OpenShift Container Platform Subscription9.2.2. xPaaS Middleware Add-on Subscriptions
9.3. BEFORE YOU BEGIN9.4. PREREQUISITES9.5. CREATING IMAGE STREAMS FOR OPENSHIFT CONTAINER PLATFORM IMAGES9.6. CREATING IMAGE STREAMS FOR XPAAS MIDDLEWARE IMAGES9.7. CREATING DATABASE SERVICE TEMPLATES9.8. CREATING INSTANT APP AND QUICKSTART TEMPLATES9.9. WHAT’S NEXT?
255256256256257258258262
264264265265266267269269269270271271271
272272272273274274274275275276
277277277278278280281282
284284284284285285285286286286287288
OpenShift Container Platform 3.9 Installation and Configuration
8
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
CHAPTER 10. CONFIGURING CUSTOM CERTIFICATES10.1. OVERVIEW10.2. CONFIGURING A CERTIFICATE CHAIN10.3. CONFIGURING CUSTOM CERTIFICATES DURING INSTALLATION10.4. CONFIGURING CUSTOM CERTIFICATES FOR THE WEB CONSOLE OR CLI10.5. CONFIGURING A CUSTOM MASTER HOST CERTIFICATE10.6. CONFIGURING A CUSTOM WILDCARD CERTIFICATE FOR THE DEFAULT ROUTER10.7. CONFIGURING A CUSTOM CERTIFICATE FOR THE IMAGE REGISTRY10.8. CONFIGURING A CUSTOM CERTIFICATE FOR A LOAD BALANCER10.9. RETROFIT CUSTOM CERTIFICATES INTO A CLUSTER
10.9.1. Retrofit Custom Master Certificates into a Cluster10.9.2. Retrofit Custom Router Certificates into a Cluster
10.10. USING CUSTOM CERTIFICATES WITH OTHER COMPONENTS
CHAPTER 11. REDEPLOYING CERTIFICATES11.1. OVERVIEW11.2. CHECKING CERTIFICATE EXPIRATIONS
11.2.1. Role Variables11.2.2. Running Certificate Expiration Playbooks
Other Example Playbooks11.2.3. Output Formats
HTML ReportJSON Report
11.3. REDEPLOYING CERTIFICATES11.3.1. Redeploying All Certificates Using the Current OpenShift Container Platform and etcd CA11.3.2. Redeploying a New or Custom OpenShift Container Platform CA11.3.3. Redeploying a New etcd CA11.3.4. Redeploying Master Certificates Only11.3.5. Redeploying etcd Certificates Only11.3.6. Redeploying Node Certificates Only11.3.7. Redeploying Registry or Router Certificates Only
11.3.7.1. Redeploying Registry Certificates Only11.3.7.2. Redeploying Router Certificates Only
11.3.8. Redeploying Custom Registry or Router Certificates11.3.8.1. Redeploying Registry Certificates Manually11.3.8.2. Redeploying Router Certificates Manually
CHAPTER 12. CONFIGURING AUTHENTICATION AND USER AGENT12.1. OVERVIEW12.2. IDENTITY PROVIDER PARAMETERS12.3. CONFIGURING IDENTITY PROVIDERS
12.3.1. Configuring identity providers with Ansible12.3.2. Configuring identity providers in the master configuration file12.3.3. Configuring an identity provider or method
12.3.3.1. Manually provisioning a user when using the lookup mapping method12.3.4. Allow all12.3.5. Deny all12.3.6. HTPasswd12.3.7. Keystone
12.3.7.1. Configuring authentication on the master12.3.7.2. Creating Users with Keystone Authentication12.3.7.3. Verifying Users
12.3.8. LDAP authentication
289289289289290291292293294295295295296
297297297297298299299299299300301301302303303304304304304304304306
309309309310311311312312313313314315316317318318
Table of Contents
9
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.3.9. Basic authentication (remote)12.3.9.1. Configuring authentication on the master12.3.9.2. Troubleshooting
12.3.10. Request header12.3.11. GitHub
12.3.11.1. Registering the application on GitHub12.3.11.2. Configuring authentication on the master12.3.11.3. Creating users with GitHub authentication12.3.11.4. Verifying users
12.3.12. GitLab12.3.13. Google12.3.14. OpenID connect
12.4. TOKEN OPTIONS12.5. GRANT OPTIONS12.6. SESSION OPTIONS12.7. PREVENTING CLI VERSION MISMATCH WITH USER AGENT
CHAPTER 13. SYNCING GROUPS WITH LDAP13.1. OVERVIEW13.2. CONFIGURING LDAP SYNC
13.2.1. LDAP Client Configuration13.2.2. LDAP Query Definition13.2.3. User-Defined Name Mapping
13.3. RUNNING LDAP SYNC13.4. RUNNING A GROUP PRUNING JOB13.5. SYNC EXAMPLES
13.5.1. RFC 230713.5.1.1. RFC2307 with User-Defined Name Mappings
13.5.2. RFC 2307 with User-Defined Error Tolerances13.5.3. Active Directory13.5.4. Augmented Active Directory
13.6. NESTED MEMBERSHIP SYNC EXAMPLE13.7. LDAP SYNC CONFIGURATION SPECIFICATION
13.7.1. v1.LDAPSyncConfig13.7.2. v1.StringSource13.7.3. v1.LDAPQuery13.7.4. v1.RFC2307Config13.7.5. v1.ActiveDirectoryConfig13.7.6. v1.AugmentedActiveDirectoryConfig
CHAPTER 14. CONFIGURING LDAP FAILOVER14.1. PREREQUISITES FOR CONFIGURING BASIC REMOTE AUTHENTICATION14.2. GENERATING AND SHARING CERTIFICATES WITH THE REMOTE BASIC AUTHENTICATION SERVER
14.3. CONFIGURING SSSD FOR LDAP FAILOVER14.4. CONFIGURING APACHE TO USE SSSD14.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM TO USE SSSD AS THE BASIC REMOTEAUTHENTICATION SERVER
CHAPTER 15. CONFIGURING THE SDN15.1. OVERVIEW15.2. AVAILABLE SDN PROVIDERS
Installing VMware NSX-T (™) on OpenShift Container Platform15.3. CONFIGURING THE POD NETWORK WITH ANSIBLE
321322324324332332332335335335336337340341341
342
345345345345346347347348348349351353355357360363363365366367368369
371371
371372374
377
379379379379379
OpenShift Container Platform 3.9 Installation and Configuration
10
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15.4. CONFIGURING THE POD NETWORK ON MASTERS15.5. CONFIGURING THE POD NETWORK ON NODES15.6. MIGRATING BETWEEN SDN PLUG-INS
15.6.1. Migrating from ovs-multitenant to ovs-networkpolicy15.7. EXTERNAL ACCESS TO THE CLUSTER NETWORK15.8. USING FLANNEL
CHAPTER 16. CONFIGURING NUAGE SDN16.1. NUAGE SDN AND OPENSHIFT CONTAINER PLATFORM16.2. DEVELOPER WORKFLOW16.3. OPERATIONS WORKFLOW16.4. INSTALLATION
CHAPTER 17. CONFIGURING FOR AMAZON WEB SERVICES (AWS)17.1. OVERVIEW17.2. PERMISSIONS17.3. CONFIGURING A SECURITY GROUP
17.3.1. Overriding Detected IP Addresses and Host Names17.4. CONFIGURING AWS VARIABLES17.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM FOR AWS
17.5.1. Configuring OpenShift Container Platform for AWS with Ansible17.5.2. Manually Configuring OpenShift Container Platform Masters for AWS17.5.3. Manually Configuring OpenShift Container Platform Nodes for AWS17.5.4. Manually Setting Key-Value Access Pairs
17.6. APPLYING CONFIGURATION CHANGES17.7. LABELING CLUSTERS FOR AWS
17.7.1. Resources That Need Tags17.7.2. Tagging an Existing Cluster
CHAPTER 18. CONFIGURING FOR OPENSTACK18.1. OVERVIEW18.2. PERMISSIONS18.3. CONFIGURING A SECURITY GROUP18.4. CONFIGURING OPENSTACK VARIABLES18.5. CONFIGURING OPENSHIFT CONTAINER PLATFORM MASTERS FOR OPENSTACK
18.5.1. Configuring OpenShift Container Platform for OpenStack with Ansible18.5.2. Manually Configuring OpenShift Container Platform Masters for OpenStack18.5.3. Manually Configuring OpenShift Container Platform Nodes for OpenStack18.5.4. Installing OpenShift Container Platform by Using an Ansible Playbook
18.6. APPLYING CONFIGURATION CHANGES
CHAPTER 19. CONFIGURING FOR GCE19.1. OVERVIEW19.2. PERMISSIONS19.3. CONFIGURING MASTERS
19.3.1. Configuring OpenShift Container Platform Masters for GCE with Ansible19.3.2. Manually Configuring OpenShift Container Platform Masters for GCE
19.4. CONFIGURING NODES19.5. CONFIGURING MULTIZONE SUPPORT IN A GCE DEPLOYMENT19.6. APPLYING CONFIGURATION CHANGES
CHAPTER 20. CONFIGURING FOR AZURE20.1. OVERVIEW20.2. PERMISSIONS
380381381382383383
386386386386386
389389389389390391
392392393393394394394395395
397397397397398398398399400400401
402402402402402403404404404
406406406
Table of Contents
11
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20.3. PREREQUISITES20.4. THE AZURE CONFIGURATION FILE20.5. CONFIGURING MASTERS20.6. CONFIGURING NODES20.7. APPLYING CONFIGURATION CHANGES
CHAPTER 21. CONFIGURING FOR VMWARE VSPHERE21.1. OVERVIEW21.2. ENABLING VMWARE VSPHERE CLOUD PROVIDER21.3. CONFIGURING OPENSHIFT CONTAINER PLATFORM FOR VSPHERE USING ANSIBLE21.4. THE VMWARE VSPHERE CONFIGURATION FILE21.5. CONFIGURING MASTERS21.6. CONFIGURING NODES21.7. APPLYING CONFIGURATION CHANGES21.8. BACKUP OF PERSISTENT VOLUMES
CHAPTER 22. CONFIGURING FOR LOCAL VOLUME22.1. OVERVIEW
22.1.1. Enable Local Volumes22.1.2. Mount Local Volumes22.1.3. Configure Local Provisioner22.1.4. Deploy Local Provisioner22.1.5. Adding New Devices
CHAPTER 23. CONFIGURING PERSISTENT VOLUME CLAIM PROTECTION23.1. OVERVIEW
23.1.1. Enable PVC Protection
CHAPTER 24. CONFIGURING PERSISTENT STORAGE24.1. OVERVIEW24.2. PERSISTENT STORAGE USING NFS
24.2.1. Overview24.2.2. Provisioning24.2.3. Enforcing Disk Quotas24.2.4. NFS Volume Security
24.2.4.1. Group IDs24.2.4.2. User IDs24.2.4.3. SELinux24.2.4.4. Export Settings
24.2.5. Reclaiming Resources24.2.6. Automation24.2.7. Additional Configuration and Troubleshooting
24.3. PERSISTENT STORAGE USING RED HAT GLUSTER STORAGE24.3.1. Overview
24.3.1.1. Container-Native Storage24.3.1.2. Container-Ready Storage24.3.1.3. Standalone Red Hat Gluster Storage24.3.1.4. GlusterFS Volumes24.3.1.5. gluster-block Volumes24.3.1.6. Gluster S3 Storage
24.3.2. Considerations24.3.2.1. Software Prerequisites24.3.2.2. Hardware Requirements24.3.2.3. Storage Sizing
406406407408408
410410410412413414414415415
416416416417417418419
421421421
422422422422423424424425426427427428429429429429430430430431431
432432432432433
OpenShift Container Platform 3.9 Installation and Configuration
12
24.3.2.4. Volume Operation Behaviors24.3.2.5. Volume Security
24.3.2.5.1. POSIX Permissions24.3.2.5.2. SELinux
24.3.3. Support Requirements24.3.4. Installation
24.3.4.1. Container-Ready Storage: Installing Red Hat Gluster Storage Nodes24.3.4.2. Using the Advanced Installer
24.3.4.2.1. Example: Basic Container-Native Storage Installation24.3.4.2.2. Example: Basic Container-Ready Storage Installation24.3.4.2.3. Example: Container-Native Storage with an Integrated OpenShift Container Registry24.3.4.2.4. Example: Container-Native Storage for OpenShift Logging and Metrics24.3.4.2.5. Example: Container-Native Storage for Applications, Registry, Logging, and Metrics24.3.4.2.6. Example: Container-Ready Storage for Applications, Registry, Logging, and Metrics
24.3.5. Uninstall Container-Native Storage24.3.6. Provisioning
24.3.6.1. Static Provisioning24.3.6.2. Dynamic Provisioning
24.4. PERSISTENT STORAGE USING OPENSTACK CINDER24.4.1. Overview24.4.2. Provisioning Cinder PVs
24.4.2.1. Creating the Persistent Volume24.4.2.2. Cinder PV format24.4.2.3. Cinder volume security
24.5. PERSISTENT STORAGE USING CEPH RADOS BLOCK DEVICE (RBD)24.5.1. Overview24.5.2. Provisioning
24.5.2.1. Creating the Ceph Secret24.5.2.2. Creating the Persistent Volume
24.5.3. Ceph Volume Security24.6. PERSISTENT STORAGE USING AWS ELASTIC BLOCK STORE
24.6.1. Overview24.6.2. Provisioning
24.6.2.1. Creating the Persistent Volume24.6.2.2. Volume Format24.6.2.3. Maximum Number of EBS Volumes on a Node
24.7. PERSISTENT STORAGE USING GCE PERSISTENT DISK24.7.1. Overview24.7.2. Provisioning
24.7.2.1. Creating the Persistent Volume24.7.2.2. Volume Format
24.8. PERSISTENT STORAGE USING ISCSI24.8.1. Overview24.8.2. Provisioning
24.8.2.1. Enforcing Disk Quotas24.8.2.2. iSCSI Volume Security24.8.2.3. iSCSI Multipathing24.8.2.4. iSCSI Custom Initiator IQN
24.9. PERSISTENT STORAGE USING FIBRE CHANNEL24.9.1. Overview24.9.2. Provisioning
24.9.2.1. Enforcing Disk Quotas24.9.2.2. Fibre Channel Volume Security
434434434435435436436436438438439440441
443445446446449450450450451452452453453454454454456457457457458459459459459460460461461461461
462462462463463463464464464
Table of Contents
13
24.10. PERSISTENT STORAGE USING AZURE DISK24.10.1. Overview24.10.2. Prerequisites24.10.3. Provisioning24.10.4. Configuring Azure Disk for regional cloud
24.10.4.1. Creating the Persistent Volume24.10.4.2. Volume Format
24.11. PERSISTENT STORAGE USING AZURE FILE24.11.1. Overview24.11.2. Before you begin24.11.3. Example configuration files24.11.4. Configuring Azure File for regional cloud24.11.5. Creating the PV24.11.6. Creating the Azure Storage Account secret
24.12. PERSISTENT STORAGE USING FLEXVOLUME PLUG-INS24.12.1. Overview24.12.2. FlexVolume drivers
24.12.2.1. FlexVolume drivers with master-initiated attach/detach24.12.2.2. FlexVolume drivers without master-initiated attach/detach
24.12.3. Installing FlexVolume drivers24.12.4. Consuming storage using FlexVolume drivers
24.13. USING VMWARE VSPHERE VOLUMES FOR PERSISTENT STORAGE24.13.1. Overview
Prerequisites24.13.2. Provisioning VMware vSphere volumes
24.13.2.1. Creating persistent volumes24.13.2.2. Formatting VMware vSphere volumes
24.14. PERSISTENT STORAGE USING LOCAL VOLUME24.14.1. Overview24.14.2. Provisioning24.14.3. Creating Local Persistent Volume Claim24.14.4. Feature Status
24.15. DYNAMIC PROVISIONING AND CREATING STORAGE CLASSES24.15.1. Overview24.15.2. Available dynamically provisioned plug-ins24.15.3. Defining a StorageClass
24.15.3.1. Basic StorageClass object definition24.15.3.2. StorageClass annotations24.15.3.3. OpenStack Cinder object definition24.15.3.4. AWS ElasticBlockStore (EBS) object definition24.15.3.5. GCE PersistentDisk (gcePD) object definition24.15.3.6. GlusterFS object definition24.15.3.7. Ceph RBD object definition24.15.3.8. Trident object definition24.15.3.9. VMware vSphere object definition24.15.3.10. Azure Disk object definition
24.15.4. Changing the default StorageClass24.15.5. Additional information and examples
24.16. VOLUME SECURITY24.16.1. Overview24.16.2. SCCs, Defaults, and Allowed Ranges24.16.3. Supplemental Groups24.16.4. fsGroup
465465465465465466467467467467469469469470471471471472475476476477477477478478479479479480480480481481481
483483483484484485485487488488488489490490490491
494497
OpenShift Container Platform 3.9 Installation and Configuration
14
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24.16.5. User IDs24.16.6. SELinux Options
24.17. SELECTOR-LABEL VOLUME BINDING24.17.1. Overview24.17.2. Motivation24.17.3. Deployment
24.17.3.1. Prerequisites24.17.3.2. Define the Persistent Volume and Claim24.17.3.3. Deploy the Persistent Volume and Claim
24.18. ENABLING CONTROLLER-MANAGED ATTACHMENT AND DETACHMENT24.18.1. Overview24.18.2. Determining What Is Managing Attachment and Detachment24.18.3. Configuring Nodes to Enable Controller-managed Attachment and Detachment
24.19. PERSISTENT VOLUME SNAPSHOTS24.19.1. Overview24.19.2. Features24.19.3. Installation and Setup
24.19.3.1. Starting the External Controller and Provisioner24.19.3.2. Managing Snapshot Users
24.19.4. Lifecycle of a Volume Snapshot and Volume Snapshot Data24.19.4.1. Persistent Volume Claim and Persistent Volume
24.19.4.1.1. Snapshot Promoter24.19.4.2. Create Snapshot24.19.4.3. Restore Snapshot24.19.4.4. Delete Snapshot
CHAPTER 25. PERSISTENT STORAGE EXAMPLES25.1. OVERVIEW25.2. SHARING AN NFS MOUNT ACROSS TWO PERSISTENT VOLUME CLAIMS
25.2.1. Overview25.2.2. Creating the Persistent Volume25.2.3. Creating the Persistent Volume Claim25.2.4. Ensuring NFS Volume Access25.2.5. Creating the Pod25.2.6. Creating an Additional Pod to Reference the Same PVC
25.3. COMPLETE EXAMPLE USING CEPH RBD25.3.1. Overview25.3.2. Installing the ceph-common Package25.3.3. Creating the Ceph Secret25.3.4. Creating the Persistent Volume25.3.5. Creating the Persistent Volume Claim25.3.6. Creating the Pod25.3.7. Defining Group and Owner IDs (Optional)25.3.8. Setting ceph-user-secret as Default for Projects
25.4. USING CEPH RBD FOR DYNAMIC PROVISIONING25.4.1. Overview25.4.2. Creating a pool for dynamic volumes25.4.3. Using an existing Ceph cluster for dynamic persistent storage25.4.4. Setting ceph-user-secret as the default for projects
25.5. COMPLETE EXAMPLE USING GLUSTERFS25.5.1. Overview25.5.2. Prerequisites25.5.3. Static Provisioning
499501
503503503503503504505505505505506506506507507507510511511511511512513
514514514514514515516517521
523523523523524525526527527528528528529532532533533534
Table of Contents
15
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25.5.4. Using the Storage25.6. COMPLETE EXAMPLE USING GLUSTERFS FOR DYNAMIC PROVISIONING
25.6.1. Overview25.6.2. Prerequisites25.6.3. Dynamic Provisioning25.6.4. Using the Storage
25.7. MOUNTING VOLUMES ON PRIVILEGED PODS25.7.1. Overview25.7.2. Prerequisites25.7.3. Creating the Persistent Volume25.7.4. Creating a Regular User25.7.5. Creating the Persistent Volume Claim25.7.6. Verifying the Setup
25.7.6.1. Checking the Pod SCC25.7.6.2. Verifying the Mount
25.8. SWITCHING AN INTEGRATED OPENSHIFT CONTAINER REGISTRY TO GLUSTERFS25.8.1. Overview25.8.2. Prerequisites25.8.3. Manually Provision the GlusterFS PersistentVolumeClaim25.8.4. Attach the PersistentVolumeClaim to the Registry
25.9. BINDING PERSISTENT VOLUMES BY LABELS25.9.1. Overview
25.9.1.1. Assumptions25.9.2. Defining Specifications
25.9.2.1. Persistent Volume with Labels25.9.2.2. Persistent Volume Claim with Selectors25.9.2.3. Volume Endpoints25.9.2.4. Deploy the PV, PVC, and Endpoints
25.10. USING STORAGE CLASSES FOR DYNAMIC PROVISIONING25.10.1. Overview25.10.2. Scenario 1: Basic Dynamic Provisioning with Two Types of StorageClasses25.10.3. Scenario 2: How to enable Default StorageClass behavior for a Cluster
25.11. USING STORAGE CLASSES FOR EXISTING LEGACY STORAGE25.11.1. Overview
25.11.1.1. Scenario 1: Link StorageClass to existing Persistent Volume with Legacy Data25.12. CONFIGURING AZURE BLOB STORAGE FOR INTEGRATED DOCKER REGISTRY
25.12.1. Overview25.12.2. Before You Begin25.12.3. Overriding Registry Configuration
CHAPTER 26. WORKING WITH HTTP PROXIES26.1. OVERVIEW26.2. CONFIGURING NO_PROXY26.3. CONFIGURING HOSTS FOR PROXIES26.4. CONFIGURING HOSTS FOR PROXIES USING ANSIBLE26.5. PROXYING DOCKER PULL26.6. USING MAVEN BEHIND A PROXY26.7. CONFIGURING S2I BUILDS FOR PROXIES26.8. CONFIGURING DEFAULT TEMPLATES FOR PROXIES26.9. SETTING PROXY ENVIRONMENT VARIABLES IN PODS26.10. GIT REPOSITORY ACCESS
CHAPTER 27. CONFIGURING GLOBAL BUILD DEFAULTS AND OVERRIDES
537538539539539540542542543543543544545545545545545545546549549549550550550551551551
552552552555559559559561561561562
564564564565566566567567567568568
569
OpenShift Container Platform 3.9 Installation and Configuration
16
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27.1. OVERVIEW27.2. SETTING GLOBAL BUILD DEFAULTS
27.2.1. Configuring Global Build Defaults with Ansible27.2.2. Manually Setting Global Build Defaults
27.3. SETTING GLOBAL BUILD OVERRIDES27.3.1. Configuring Global Build Overrides with Ansible27.3.2. Manually Setting Global Build Overrides
CHAPTER 28. CONFIGURING PIPELINE EXECUTION28.1. OVERVIEW28.2. OPENSHIFT JENKINS CLIENT PLUGIN28.3. OPENSHIFT JENKINS SYNC PLUGIN
CHAPTER 29. CONFIGURING ROUTE TIMEOUTS
CHAPTER 30. CONFIGURING NATIVE CONTAINER ROUTING30.1. NETWORK OVERVIEW30.2. CONFIGURE NATIVE CONTAINER ROUTING30.3. SETTING UP A NODE FOR CONTAINER NETWORKING30.4. SETTING UP A ROUTER FOR CONTAINER NETWORKING
CHAPTER 31. ROUTING FROM EDGE LOAD BALANCERS31.1. OVERVIEW31.2. INCLUDING THE LOAD BALANCER IN THE SDN31.3. ESTABLISHING A TUNNEL USING A RAMP NODE
31.3.1. Configuring a Highly-Available Ramp Node
CHAPTER 32. AGGREGATING CONTAINER LOGS32.1. OVERVIEW32.2. PRE-DEPLOYMENT CONFIGURATION32.3. SPECIFYING LOGGING ANSIBLE VARIABLES32.4. DEPLOYING THE EFK STACK32.5. UNDERSTANDING AND ADJUSTING THE DEPLOYMENT
32.5.1. Ops Cluster32.5.2. Elasticsearch
32.5.2.1. Persistent Elasticsearch Storage32.5.2.1.1. Using NFS as a persistent volume32.5.2.1.2. Using NFS as local storage32.5.2.1.3. Changing the Scale of Elasticsearch32.5.2.1.4. Expose Elasticsearch as a Route
32.5.3. Fluentd32.5.4. Kibana32.5.5. Curator
32.5.5.1. Creating the Curator Configuration32.6. CLEANUP32.7. TROUBLESHOOTING KIBANA32.8. SENDING LOGS TO AN EXTERNAL ELASTICSEARCH INSTANCE32.9. SENDING LOGS TO AN EXTERNAL SYSLOG SERVER32.10. PERFORMING ADMINISTRATIVE ELASTICSEARCH OPERATIONS32.11. REDEPLOYING EFK CERTIFICATES32.12. CHANGING THE AGGREGATED LOGGING DRIVER32.13. MANUAL ELASTICSEARCH ROLLOUTS
32.13.1. Performing an Elasticsearch Rolling Cluster Restart32.13.2. Performing an Elasticsearch Full Cluster Restart
569569570571
572572573
575575576576
578
579579579580580
581581581581
584
585585585586595596596596598599601602602603614615617618618
620620623624625626627628
Table of Contents
17
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
CHAPTER 33. AGGREGATE LOGGING SIZING GUIDELINES33.1. OVERVIEW33.2. INSTALLATION
33.2.1. Large Clusters33.3. SYSTEMD-JOURNALD AND RSYSLOG33.4. SCALING UP EFK LOGGING33.5. STORAGE CONSIDERATIONS
CHAPTER 34. ENABLING CLUSTER METRICS34.1. OVERVIEW34.2. BEFORE YOU BEGIN34.3. METRICS PROJECT34.4. METRICS DATA STORAGE
34.4.1. Persistent Storage34.4.2. Capacity Planning for Cluster Metrics
Known Issues and Limitations34.4.3. Non-Persistent Storage
34.5. METRICS ANSIBLE ROLE34.5.1. Specifying Metrics Ansible Variables34.5.2. Using Secrets
34.5.2.1. Providing Your Own Certificates34.6. DEPLOYING THE METRIC COMPONENTS
34.6.1. Metrics Diagnostics34.7. SETTING THE METRICS PUBLIC URL34.8. ACCESSING HAWKULAR METRICS DIRECTLY
34.8.1. OpenShift Container Platform Projects and Hawkular Tenants34.8.2. Authorization
34.9. SCALING OPENSHIFT CONTAINER PLATFORM CLUSTER METRICS PODS34.10. INTEGRATION WITH AGGREGATED LOGGING34.11. CLEANUP34.12. PROMETHEUS ON OPENSHIFT CONTAINER PLATFORM
34.12.1. Setting Prometheus Role Variables34.12.2. Deploying Prometheus Using Ansible Installer
34.12.2.1. Additional Methods for Deploying Prometheus34.12.2.2. Accessing the Prometheus Web UI34.12.2.3. Configuring Prometheus for OpenShift Container Platform
34.12.3. OpenShift Container Platform Metrics via Prometheus34.12.3.1. Current Metrics
34.12.4. Undeploying Prometheus
CHAPTER 35. CUSTOMIZING THE WEB CONSOLE35.1. OVERVIEW35.2. LOADING EXTENSION SCRIPTS AND STYLESHEETS
35.2.1. Setting Extension Properties35.3. EXTENSION OPTION FOR EXTERNAL LOGGING SOLUTIONS35.4. CUSTOMIZING AND DISABLING THE GUIDED TOUR35.5. CUSTOMIZING DOCUMENTATION LINKS35.6. CUSTOMIZING THE LOGO35.7. CUSTOMIZING THE MEMBERSHIP WHITELIST35.8. CHANGING LINKS TO DOCUMENTATION35.9. ADDING OR CHANGING LINKS TO DOWNLOAD THE CLI
35.9.1. Customizing the About Page35.10. CONFIGURING NAVIGATION MENUS
630630630632632633633
635635635635635636636638638638639643643643644644645645646646646646647647648649649649650650652
653653653654655655655655656656656657658
OpenShift Container Platform 3.9 Installation and Configuration
18
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35.10.1. Top Navigation Dropdown Menus35.10.2. Application Launcher35.10.3. System Status Badge35.10.4. Project Left Navigation
35.11. CONFIGURING FEATURED APPLICATIONS35.12. CONFIGURING CATALOG CATEGORIES35.13. CONFIGURING QUOTA NOTIFICATION MESSAGES35.14. CONFIGURING THE CREATE FROM URL NAMESPACE WHITELIST35.15. DISABLING THE COPY LOGIN COMMAND
35.15.1. Enabling Wildcard Routes35.16. CUSTOMIZING THE LOGIN PAGE
35.16.1. Example Usage35.17. CUSTOMIZING THE OAUTH ERROR PAGE35.18. CHANGING THE LOGOUT URL35.19. CONFIGURING WEB CONSOLE CUSTOMIZATIONS WITH ANSIBLE35.20. CHANGING THE WEB CONSOLE URL PORT AND CERTIFICATES
CHAPTER 36. DEPLOYING EXTERNAL PERSISTENT VOLUME PROVISIONERS36.1. OVERVIEW36.2. BEFORE YOU BEGIN
36.2.1. External Provisioners Ansible Role36.2.2. External Provisioners Ansible Variables36.2.3. AWS EFS Provisioner Ansible Variables
36.3. DEPLOYING THE PROVISIONERS36.3.1. Deploying the AWS EFS Provisioner
36.3.1.1. AWS EFS Object Definition36.4. CLEANUP
658659659660661
662663664664664664665665665666667
668668668668668669670670670671
Table of Contents
19
OpenShift Container Platform 3.9 Installation and Configuration
20
https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/developer_guide/#dev-guide-secrets
CHAPTER 1. OVERVIEWOpenShift Container Platform Installation and Configuration topics cover the basics of installing andconfiguring OpenShift Container Platform in your environment. Configuration, management, andlogging are also covered. Use these topics for the one-time tasks required to quickly set up yourOpenShift Container Platform environment and configure it based on your organizational needs.
For day to day cluster administration tasks, see Cluster Administration .
CHAPTER 1. OVERVIEW
21
https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#admin-guide-index
CHAPTER 2. INSTALLING A CLUSTER
2.1. PLANNING
2.1.1. Initial Planning
For production environments, several factors influence installation. Consider the following questions asyou read through the documentation:
Which installation method do you want to use? The Installation Methods section provides someinformation about the quick and advanced installation methods.
How many pods are required in your cluster? The Sizing Considerations section provides limitsfor nodes and pods so you can calculate how large your environment needs to be.
How many hosts do you require in the cluster? The Environment Scenarios section providesmultiple examples of Single Master and Multiple Master configurations.
Is high availability required? High availability is recommended for fault tolerance. In this situation,you might aim to use the Multiple Masters Using Native HA example as a basis for yourenvironment.
Which installation type do you want to use: RPM or containerized? Both installations provide aworking OpenShift Container Platform environment, but you might have a preference for aparticular method of installing, managing, and updating your services.
Which identity provider do you use for authentication? If you already use a supported identityprovider, it is a best practice to configure OpenShift Container Platform to use that identityprovider during advanced installation.
Is my installation supported if integrating with other technologies? See the OpenShift ContainerPlatform Tested Integrations for a list of tested integrations.
2.1.2. Installation Methods
IMPORTANT
As of OpenShift Container Platform 3.9, the quick installation method is deprecated. In afuture release, it will be removed completely. In addition, using the quick installer toupgrade from version 3.7 to 3.9 is not supported.
Both the quick and advanced installation methods are supported for development and productionenvironments. If you want to quickly get OpenShift Container Platform up and running to try out for thefirst time, use the quick installer and let the interactive CLI guide you through the configuration optionsrelevant to your environment.
For the most control over your cluster’s configuration, you can use the advanced installation method.This method is particularly suited if you are already familiar with Ansible. However, following along withthe OpenShift Container Platform documentation should equip you with enough information to reliablydeploy your cluster and continue to manage its configuration post-deployment using the providedAnsible playbooks directly.
If you install initially using the quick installer, you can always further tweak your cluster’s configurationand adjust the number of hosts in the cluster using the same installer tool. If you wanted to later switch
OpenShift Container Platform 3.9 Installation and Configuration
22
https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#admin-guide-high-availabilityhttps://access.redhat.com/articles/2176281
to using the advanced method, you can create an inventory file for your configuration and carry on thatway.
2.1.3. Sizing Considerations
Determine how many nodes and pods you require for your OpenShift Container Platform cluster.Cluster scalability correlates to the number of pods in a cluster environment. That number influences theother numbers in your setup. See Cluster Limits for the latest limits for objects in OpenShift ContainerPlatform.
2.1.4. Environment Scenarios
This section outlines different examples of scenarios for your OpenShift Container Platformenvironment. Use these scenarios as a basis for planning your own OpenShift Container Platform cluster,based on your sizing needs.
NOTE
Moving from a single master cluster to multiple masters after installation is notsupported.
For information on updating labels, see Updating Labels on Nodes.
2.1.4.1. Single Master and Node on One System
OpenShift Container Platform can be installed on a single system for a development environment only.An all-in-one environment is not considered a production environment.
2.1.4.2. Single Master and Multiple Nodes
The following table describes an example environment for a single master (with etcd installed on thesame host) and two nodes:
Host Name Infrastructure Component to Install
master.example.com Master, etcd, and node
node1.example.com Node
node2.example.com
2.1.4.3. Single Master, Multiple etcd, and Multiple Nodes
The following table describes an example environment for a single master, three etcd hosts, and twonodes:
Host Name Infrastructure Component to Install
master.example.com Master and node
CHAPTER 2. INSTALLING A CLUSTER
23
https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/scaling_and_performance_guide/#scaling-performance-cluster-limitshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#updating-labels-on-nodeshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#node
etcd1.example.com etcd
etcd2.example.com
etcd3.example.com
node1.example.com Node
node2.example.com
Host Name Infrastructure Component to Install
2.1.4.4. Multiple Masters Using Native HA with Co-located Clustered etcd
The following describes an example environment for three masters with co-located clustered etcd, oneHAProxy load balancer, and two nodes using the native HA method:
Host Name Infrastructure Component to Install
master1.example.com Master (clustered using native HA) and node andclustered etcd
master2.example.com
master3.example.com
lb.example.com HAProxy to load balance API master endpoints
node1.example.com Node
node2.example.com
2.1.4.5. Multiple Masters Using Native HA with External Clustered etcd
The following describes an example environment for three masters, one HAProxy load balancer, threeexternal clustered etcd hosts, and two nodes using the native HA method:
Host Name Infrastructure Component to Install
master1.example.com Master (clustered using native HA) and node
master2.example.com
master3.example.com
lb.example.com HAProxy to load balance API master endpoints
OpenShift Container Platform 3.9 Installation and Configuration
24
https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#node
etcd1.example.com Clustered etcd
etcd2.example.com
etcd3.example.com
node1.example.com Node
node2.example.com
Host Name Infrastructure Component to Install
2.1.4.6. Stand-alone Registry
You can also install OpenShift Container Platform to act as a stand-alone registry using the OpenShiftContainer Platform’s integrated registry. See Installing a Stand-alone Registry for details on thisscenario.
2.1.5. RPM Versus Containerized
An RPM installation installs all services through package management and configures services to runwithin the same user space, while a containerized installation installs services using container images andruns separate services in individual containers.
See the Installing on Containerized Hosts topic for more details on configuring your installation to usecontainerized services.
2.2. PREREQUISITES
2.2.1. System Requirements
The following sections identify the hardware specifications and system-level requirements of all hostswithin your OpenShift Container Platform environment.
2.2.1.1. Red Hat Subscriptions
You must have an active OpenShift Container Platform subscription on your Red Hat account toproceed. If you do not, contact your sales representative for more information.
2.2.1.2. Minimum Hardware Requirements
The system requirements vary per host type:
CHAPTER 2. INSTALLING A CLUSTER
25
MastersPhysical or virtual system, or an instance running on a public or private IaaS.
Base OS: RHEL 7.3 or later with the "Minimal" installation option and the latestpackages from the Extras channel, or RHEL Atomic Host 7.4.5 or later.
Minimum 4 vCPU (additional are strongly recommended).
Minimum 16 GB RAM (additional memory is strongly recommended, especially if etcdis co-located on masters).
Minimum 40 GB hard disk space for the file system containing /var/.
Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.
Minimum 1 GB hard disk space for the file system containing the system’s temporary
directory.
Masters with a co-located etcd require a minimum of 4 cores. 2 core systems will notwork.
NodesPhysical or virtual system, or an instance running on a public or private IaaS.
Base OS: link:RHEL 7.3 or later with "Minimal" installation option, or RHEL Atomic Host7.4.5 or later.
NetworkManager 1.0 or later.
1 vCPU.
Minimum 8 GB RAM.
Minimum 15 GB hard disk space for the file system containing /var/.
Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.
Minimum 1 GB hard disk space for the file system containing the system’s temporary
directory.
An additional minimum 15 GB unallocated space per system running containers forDocker’s storage back end; see Configuring Docker Storage. Additional space mightbe required, depending on the size and number of containers that run on the node.
ExternaletcdNodes
Minimum 20 GB hard disk space for etcd data.
See the Hardware Recommendations section of the CoreOS etcd documentation forinformation how to properly size your etcd nodes.
Currently, OpenShift Container Platform stores image, build, and deploymentmetadata in etcd. You must periodically prune old resources. If you are planning toleverage a large number of these resources, place etcd on machines with largeamounts of memory and fast SSD drives.
OpenShift Container Platform 3.9 Installation and Configuration
26
https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#masterhttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/installation_guide/indexhttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/installation_and_configuration_guide/https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/#nodehttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/installation_guide/indexhttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/installation_and_configuration_guide/https://github.com/coreos/etcd/blob/master/Documentation/op-guide/hardware.md#hardware-recommendationshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#admin-guide-pruning-resources
AnsibleController
The host that you run the Ansible playbook on must have at least 75MiB of free memory perhost in the inventory.
Meeting the /var/ file system sizing requirements in RHEL Atomic Host requires making changesto the default configuration. See Managing Storage with Docker-formatted Containers for instructionson configuring this during or after installation.
The system’s temporary directory is determined according to the rules defined in the tempfilemodule in Python’s standard library.
IMPORTANT
OpenShift Container Platform only supports servers with x86_64 architecture.
You must configure storage for each system that runs a container daemon. For containerizedinstallations, you need storage on masters. Also, by default, the web console is run in containers onmasters, and storage is needed on masters to run the web console. Containers are run on nodes, sostorage is always required on the nodes. The size of storage depends on workload, number ofcontainers, the size of the containers being run, and the containers' storage requirements. Containerizedetcd also needs container storage configured.
2.2.1.3. Production Level Hardware Requirements
Test or sample environments function with the minimum requirements. For production environments,the following recommendations apply:
Master Hosts
In a highly available OpenShift Container Platform cluster with external etcd, a master host shouldhave, in addition to the minimum requirements in the table above, 1 CPU core and 1.5 GB of memoryfor each 1000 pods. Therefore, the recommended size of a master host in an OpenShift ContainerPlatform cluster of 2000 pods would be the minimum requirements of 2 CPU cores and 16 GB ofRAM, plus 2 CPU cores and 3 GB of RAM, totaling 4 CPU cores and 19 GB of RAM.
A minimum of three etcd hosts and a load-balancer between the master hosts are required.
See Recommended Practices for OpenShift Container Platform Master Hosts for performanceguidance.
Node Hosts
The size of a node host depends on the expected size of its workload. As an OpenShift ContainerPlatform cluster administrator, you will need to calculate the expected workload, then add about 10percent for overhead. For production environments, allocate enough resources so that a node hostfailure does not affect your maximum capacity.
For more information, see Sizing Considerations and Cluster Limits.
IMPORTANT
Oversubscribing the physical resources on a node affects resource guarantees theKubernetes scheduler makes during pod placement. Learn what measures you can taketo avoid memory swapping.
CHAPTER 2. INSTALLING A CLUSTER
27
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/managing_containers/managing_storage_with_docker_formatted_containershttps://docs.python.org/2/library/tempfile.html#tempfile.tempdirhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/scaling_and_performance_guide/#scaling-performance-capacity-host-practices-masterhttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/scaling_and_performance_guide/#scaling-performance-cluster-limitshttps://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cluster_administration/#disabling-swap-memory
2.2.1.4. Storage management
Table 2.1. The main directories to which OpenShift Container Platform components write data
Directory Notes Sizing Expected Growth
/var/lib/openshift Used for etcd storageonly when in singlemaster mode and etcd isembedded in theatomic-openshift-master process.
Less than 10GB. Will grow slowly with theenvironment. Onlystoring metadata.
/var/lib/etcd Used for etcd storagewhen in Multi-Mastermode or when etcd ismade standalone by anadministrator.
Less than 20 GB. Will grow slowly with theenvironment. Onlystoring metadata.
/var/lib/docker When the run time isdocker, this is the mountpoint. Storage used foractive containerruntimes (includingpods) and storage oflocal images (not usedfor registry storage).Mount point should bemanaged by docker-storage rather thanmanually.
50 GB for a Node with16 GB memory.
Additional 20-25 GB forevery additional 8 GB ofmemory.
Growth is limited by thecapacity for runningcontainers.
/var/lib/containers When the run time isCRI-O, this is the mountpoint. Storage used foractive containerruntimes (includingpods) and storage oflocal images (not usedfor registry storage).
50 GB for a Node with16 GB memory.
Additional 20-25 GB forevery additional 8 GB ofmemory.
Growth limited bycapacity for runningcontainers
/var/lib/origin/openshift.local.volumes
Ephemeral volumestorage for pods. Thisincludes anythingexternal that is mountedinto a container atruntime. Includesenvironment variables,kube secrets, and datavolumes not backed bypersistent storage PVs.
Varies Minimal if pods requiringstorage are usingpersistent volumes. Ifusing ephemeralstorage, this can growquickly.
OpenShift Container Platform 3.9 Installation and Configuration
28
/var/log Log files for allcomponents.
10 to 30 GB. Log files can growquickly; size can bemanaged by growingdisks or managed usinglog rotate.
Directory Notes Sizing Expected Growth
2.2.1.5. Red Hat Gluster Storage Hardware Requirements
Any nodes used in a Container-Native Storage or Container-Ready Storage cluster are consideredstorage nodes. Storage nodes can be grouped into distinct cluster groups, though a single node can notbe in multiple groups. For each group of storage nodes:
A minimum of three storage nodes per group is required.
Each storage node must have a minimum of 8 GB of RAM. This is to allow running the Red HatGluster Storage pods, as well as other applications and the underlying operating system.
Each GlusterFS volume also consumes memory on every storage node in its storage cluster,which is about 30 MB. The total amount of RAM should be determined based on how manyconcurrent volumes are desired or anticipated.
Each storage node must have at least one raw block device with no present data or metadata.These block devices will be used in their entirety for GlusterFS storage. Make sure the followingare not present:
Partition tables (GPT or MSDOS)
Filesystems or residual filesystem signatures
LVM2 signatures of former Volume Groups and Logical Volumes
LVM2 metadata of LVM2 physical volumes
If in doubt, wipefs -a should clear any of the above.
IMPORTANT
It is recommended to plan for two clusters: one dedicated to storage for infrastructureapplications (such as an OpenShift Container Registry) and one dedicated to storage forgeneral applications. This would require a total of six storage nodes. Thisrecommendation is made to avoid potential impacts on performance in I/O and volumecreation.
2.2.1.6. Optional: Configuring Core Usage
By default, OpenShift Container Platform masters and nodes use all available cores in the system theyrun on. You can choose the number of cores you want OpenShift Container Platform to use by settingthe GOMAXPROCS environment variable. See the Go Language documentation for more information,including how the GOMAXPROCS environment variable works.
CHAPTER 2. INSTALLING A CLUSTER
29
https://golang.org/pkg/runtime/#GOMAXPROCS
For example, run the following before starting the server to make OpenShift Container Platform onlyrun on one core:
# export GOMAXPROCS=1
2.2.1.7. SELinux
Security-Enhanced Linux (SELinux) must be enabled on all of the servers before installing OpenShiftContainer Platform or the installer will fail. Also, configure SELINUX=enforcing and SELINUXTYPE=targeted in the /etc/selinux/config file:
# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=enforcing# SELINUXTYPE= can take one of these three values:# targeted - Targeted processes are protected,# minimum - Modification of targeted policy. Only selected processes are protected.# mls - Multi Level Security protection.SELINUXTYPE=targeted
2.2.1.8. Red Hat Gluster Storage
To access GlusterFS volumes, the mount.glusterfs command must be available on all schedulablenodes. For RPM-based systems, the glusterfs-fuse package must be installed:
# yum install glusterfs-fuse
This package comes installed on every RHEL system. However, it is recommended to update to thelatest available version from Red Hat Gluster Storage. To do this, the following RPM repository must beenabled:
# subscription-manager repos --enable=rh-gluster-3-client-for-rhel-7-server-rpms
If glusterfs-fuse is already installed on the nodes, ensure that the latest version is installed:
# yum update glusterfs-fuse
Optional: Using OverlayFSOverlayFS is a union file system that allows you to overlay one file system on top of another.
As of Red Hat Enterprise Linux 7.4, you have the option to configure your OpenShift Container Platformenvironment to use OverlayFS. The overlay2 graph driver is fully supported in addition to the older overlay driver. However, Red Hat recommends using overlay2 instead of overlay, because of its speedand simple implementation.
Comparing the Overlay Versus Overlay2 Graph Drivers has more information about the overlay andoverlay2 drivers.
See the Overlay Graph Driver section of the Atomic Host documentation for instructions on how toenable the overlay2 graph driver for the Docker service.
OpenShift Container Platform 3.9 Installation and Configuration
30
https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/scaling_and_performance_guide/#comparing-overlay-graph-drivershttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/managing_containers/#using_the_overlay_graph_driver
2.2.1.9. Security Warning
OpenShift Container Platform runs containers on hosts in the cluster, and in some cases, such as buildoperations and the registry service, it does so using privileged containers. Furthermore, those containersaccess the hosts' Docker daemon and perform docker build and docker push operations. As such,cluster administrators should be aware of the inherent security risks associated with performing docker run operations on arbitrary images as they effectively have root access. This is particularly relevant for docker build operations.
Exposure to harmful containers can be limited by assigning specific builds to nodes so that any exposureis limited to those nodes. To do this, see the Assigning Builds to Specific Nodes section of theDeveloper Guide. For cluster administrators, see the Configuring Global Build Defaults and Overridessection of the Installation and Configuration Guide.
You can also use security context constraints to control the actions that a pod can perform and what ithas the ability to access. For instructions on how to enable images to run with USER in the Dockerfile,see Managing Security Context Constraints (requires a user with cluster-admin privileges).
For more information, see these articles:
http://opensource.com/business/14/7/docker-security-selinux
https://docs.docker.com/engine/security/security/
2.2.2. Environment Requirements
The following section defines the requirements of the environment containing your OpenShiftContainer Platform configuration. This includes networking considerations and access to externalservices, such as Git repository access, storage, and cloud infrastructure providers.
2.2.2.1. DNS
OpenShift Container Platform requires a fully functional DNS server in the environment. This is ideally aseparate host running DNS software and can provide name resolution to hosts and containers runningon the platform.
IMPORTANT
Adding entries into the /etc/hosts file on each host is not enough. This file is not copiedinto containers running on the platform.
Key components of OpenShift Container Platform run themselves inside of containers and use thefollowing process for name resolution:
1. By default, containers receive their DNS configuration file (/etc/resolv.conf) from their host.
2. OpenShift Container Platform then inserts one DNS value into the pods (above the node’snameserver values). That value is defined in the /etc/origin/node/node-config.yaml file by thednsIP parameter, which by default is set to th
Top Related