IOS XR7 Data Sheet...Cisco IOS XR’s lockstep evolution The Cisco® Internetwork Operating System...

© 2019 Cisco and/or its affiliates. All rights reserved. of 40

IOS XR7 Data Sheet

Data sheet

Cisco public


Contents

Cisco IOS XR’s lockstep evolution 4

Designing for the future 4

Introducing IOS XR7 5

Trustworthy 7

IOS XR RPMs 16

ZTP for access deployments 18

Powerful new IOS XR install 24

IOS XR 7 packaging and install 24

Packages and SMUs in IOS XR7 25

Basic installation commands with IOS XR7 26

Configuring a remote repository 27

IOS XR7 install show commands 27

One-touch install 27

Zero-Touch Provisioning (ZTP) APIs 31

Service layer APIs 31

Open Forwarding Abstraction (OFA) API 32

Hardware root of trust 33

TAm certificates and keys 36

Secure boot 37

Signed RPMs 38

Trust at runtime (IMA) 38

Trust reports and visualization 39

Cisco environmental sustainability 40

Cisco Capital 40


IOS XR7: Simple, modern, trustworthy.

Understanding the challenges ahead Service Providers (SP), across the world, are gearing up to face the imminent surge in subscribers and content-driven

traffic volume in their networks over the coming decade. This is due, in part, to the increase in video and application

content consumption over existing networks, driven by mobile application platforms and the rise of several digital

content providers.

Internet traffic has seen a compounded annual growth rate of 30 percent or higher over the last 5 years. It is anticipated

that by 2022:

● IP traffic will increase to 396 exabytes per month

● 1/3 of the Internet traffic will be in metropolitan service areas

● More than 28 billion devices and connections will be online

● 48 percent of all connections will be video-capable1

The forthcoming release of 5G enabled devices will continue the insatiable demand for bandwidth and network

performance through better speeds and reduced latency. We will see new types of traffic generated from connected

devices, online gaming platforms, video conferencing platforms, which will result in a massive increase in the number of

devices deployed in the network. To keep pace with this growth, network providers need to focus on simple, secure,

consistent, and highly automated operations in all areas of the network.

1 Cisco Visual Networking Index, 2018


Cisco IOS XR’s lockstep evolution

The Cisco® Internetwork Operating System (IOS) XR Software has closely anticipated and consistently evolved to meet

these technological transitions. While IOS XR brings with it the best-in-class routing protocols and features, the continued

focus on intent-based transport technologies such as segment routing and Ethernet Virtual Private Networking (EVPN),

has made IOS XR the leading choice for web-scale and large-scale service providers across network segments.

It is equally important to recognize the gradual transition towards scalable, automated operational workflows in SP

networks that is driving vendor network operating systems across the industry to implement open, extensible software

architectures with model-driven APIs at every layer of the network stack. With Release 6.x, IOS XR took its first steps in

this direction by moving from a 32-bit QNX operating system to a 64-bit Linux operating system with a strong focus on

operational features that could help meet the requirements of the industry. To that effect, the support for YANG models

for programmatic provisioning and high-scale, real-time telemetry data, along with highly performant service layer APIs

at lower layers of the stack for custom protocols and controllers, greatly elevated the capabilities of IOS XR to integrate

with desired workflows and tools.

The introduction of Zero-Touch Provisioning (ZTP) capabilities in IOS XR to address Day 0 deployment requirements was

another feather in its cap with a significant uptake of ZTP workflows over the course of 4 years since its introduction. ZTP

enables operators to get routers running IOS XR provisioned in their networks without touching the router console and

without any manual intervention. This reduces “truck offload to service activation” time and also reduces the need to

send expensive Subject Matter Experts (SMEs) to the field to configure devices manually. The amount of time and money

saved using ZTP during initial rollouts directly impacts the bottom line, enabling greater investment in network

optimizations and designs that can help generate more revenue. As we look forward to the penetration of IOS XR into

access networks, ZTP will prove to be a critical building block to help reduce Operational Expenses (OpEx) while scaling

out networks to meet the demands of 5G and associated technologies.

The evolution didn’t stop there. As IOS XR opened up its underlying Linux environment for direct access to end users, a

whole host of scripting languages (Bash, Python, Golang, C++) and configuration management tools (Ansible, Puppet,

Chef, etc.) became available for use. With the support for containers (LXC and Docker), operators could now package up

their favorite Linux tools or implement custom agents and protocols and run them right on the box. This design can help

SPs take advantage of the superior capabilities of the IOS XR network stack in conjunction with the native Linux

environment to further reduce operational complexity for managing installed network devices.

Designing for the future

Where do we go from here? The first step is to build upon all the operational enhancements that IOS XR brought in with

Release 6.x. Automation at scale with high performance and extensible APIs will continue to be in high demand from SPs.

Additional investments in streaming telemetry at scale, Linux-style workflows, and integrations with industry standard

tools, must continue to drive further evolutions.

There are clear challenges to address as the size and scale of networks continue to increase. Operations have to become

more streamlined and a lot simpler. It is necessary to make automation easier to accomplish at every stage of

deployment—from Day 0 ZTP and Day 1 provisioning and monitoring to Day N service manipulation and management.

Concerns with network device security continue to mount as new attack vectors emerge each year that compromise

device integrity. As the boundaries of SP networks expand further, particularly in access deployments, Man-In-The-

Middle (MITM) attacks will become a lot more prevalent—trying to take over network devices or provisioning services to

infiltrate and compromise networks and consequently the data they carry. Automated provisioning of devices without a

secure out-of-band management network, often outside a trusted Network Operations Center (NOC), increases the

attack surface of the network. Greater focus on creating a “trustworthy” network device that implements secure


workflows for automated provisioning, validates all installed artifacts, and monitors running processes to weed out

attacks must be high priority.

With the announcement of the Cisco 8000 series platforms enabling highly scalable, 400G-optimized routers, IOS XR

continues to expand its portfolio of supported platforms. By running on Virtual Network Function (VNF) solutions using

the XRv9000 virtual router, the NCS 5000, NCS 5500 series, the ASR 9000 series and disaggregated offerings on Qualified

third-party hardware; IOS XR7 offers the most comprehensive portfolio of platforms and solutions in the industry,

enabling consistent operations and feature capability across all segments of the service provider network.

Introducing IOS XR7

With Release 7 of IOS XR, the architecture evolves to address these requirements head-on. The IOS XR7 architecture

focuses on three primary tenets:

Simple

Easier to provision, automate, and integrate with a wide variety of workflows.

1. Simpler and leaner architecture: For newer hardware platforms on IOS XR7, there is no admin plane and no

system containers, making it easier than ever before to manage the system through off-the-shelf tools and

scripting solutions.

2. Simpler operations: With Linux-style workflows and integrations, enabling the use of scalable configuration

management tools (Ansible, Puppet, Chef) and support for standard Linux applications on-box as binaries or

containers (LXC, Docker).

3. Simpler, secure Day 0 rollouts: With powerful and secure zero-touch capabilities based on RFC 8572 to help

enable secure device onboarding through template-driven ZTP scripts based on YANG-modeled transactions

between IOS XR routers and bootstrap servers.

4. Simpler software delivery and deployment: Through artifacts called Golden Installation Software Objects

(GISO) to combine custom scripts, applications, packages, and files into a deployable ISO artifact. Individual

IOS XR components and patches are delivered through Router Performance Modules (RPMs).


5. Powerful new design of IOS XR install: Built on top of DNF (Dandified YUM) and extended to support

modular chassis systems and YANG-modeled APIs, IOS XR install allows operators to manage the lifecycle of

IOS XR RPMs, native Linux RPMs, and GISO installations while supporting real-time telemetry notifications of

the install process.

Modern

Model-driven APIs at every layer of the stack. Industry-leading support for transport technologies.

1. YANG-modeled management layer APIs: To automate device provisioning and management. These

models include native IOS XR YANG models and OpenConfig models.

2. Streaming telemetry capabilities: For cadence-based or event-driven monitoring of data derived from

YANG-modeled paths in the manageability layer over gRPC, TCP, or UDP as transport.

3. Service Layer (SL) and Open Forwarding Abstraction (OFA) APIs: Service layer APIs are highly performant

APIs at the common network infrastructure layer of IOS XR (RIB, Label Switch Database, BFD, L2, etc.) to

enable controllers and custom protocols to manipulate routes in IOS XR RIB and create label-switched paths

on the fly, for example. The OFA API is a model-driven API on top of the ASIC SDK of the hardware platform

to enable high performance access to the lowest layer of the network stack either directly or through

modeled abstractions such as P4 runtime.

4. Segment routing and EVPN: Segment routing and EVPN have been the mainstays of IOS XR-based

transport deployments with focus on simplicity, scale, and programmatic extensibility. IOS XR7 continues

focus on these technologies with further support for SR Flex-Algo, SRV6, and more. EVPN drives the next

level of simplicity by offering a unified control plane protocol (BGP) for all service types, including Layer 2

VPN and Layer 3 VPN services.

5. Zero-touch APIs: Exhaustive APIs for zero-touch provisioning to enable Day 0 automation. This includes

Command Line Interface (CLI) automation APIs through Bash and Python libraries on-box. With IOS XR7,

operators will also be able to use the on-box netconf client in the ZTP Python library to bring up the system

through YANG-modeled XML inputs, further enabling network teams to transition their systems from CLI to

YANG models when ready.


Trustworthy

Secure, trusted workflows are only possible with a trustworthy Network Operating System (NOS) running on a

trustworthy network device. With IOS XR7, this is achieved by ensuring:

1. Trust starting in the hardware: A secure non-tamperable Trust Anchor Module (TAm) houses known good

values of the hardware components, along with keys and certificates rooted to Cisco. These are utilized to

verify components of the hardware during BIOS boot.

2. Secure boot: Trust is reinforced by validating parts of the network OS (IOS XR7) through a secure boot

process. This is enabled by validating the signatures of the bootloader and kernel modules against the keys in

TAm, before starting the OS.

3. Trust at runtime: Trust is maintained at runtime by enforcing IMA (Integrity Measurement Architecture)

appraisal checks against all runtime processes, from it to subsequent processes launched either by IOS XR or

third-party applications and logging all discrepancies.

4. Signed RPMs: Trust is enforced for all IOS XR RPMs and third-party application RPMs by validating

signatures on these RPMs before installation, based on the keys in TAm.

5. Ability to visualize and report on trust: Remote attestation capabilities are available to visualize and report

on all the trust verification operations performed by the system during the lifecycle of the device.

As we elaborate on these concepts further, it should become apparent that IOS XR continues to evolve to meet the needs

of service providers as they look to address the challenges of the future.


Understanding the 64-bit IOS XR architecture The 64-bit IOS XR architecture was introduced with IOS XR Release 6.X as IOS XR transitioned from a 32-bit, QNX-based

operating system to a 64-bit, Linux-based operating system.

It consists of the following parts:

● Host (Hypervisor) layer: This is the underlying 64-bit WRL7 Linux distribution layer that comprises the kernel

(version 3.14) and a root filesystem that spawns the libvirt and Docker daemons. The system containers (XR

control plane LXC and Calvados admin plane LXC) are launched by default using the libvirt daemon. The libvirt

daemon can also be utilized by network operators to launch their own applications on the host layer. The Docker

daemon is used for third-party applications only.

● XR control plane LXC: The XR control plane LXC houses IOS XR control plane processes (such as all of XR’s

features, APIs, protocols, etc.). Further, an operator can choose to run applications compiled for WRL7 natively

inside the XR control plane LXC.

● Admin plane (Calvados) LXC: The admin plane (or Calvados) LXC is a default system container and is the first

container that is launched on the host layer post boot. Once the admin plane container is up, it then launches the

XR control plane container. The admin plane container is responsible for the lifecycle of the XR control plane

container.

● Optional third-party LXC/Docker: Network operators can utilize the virsh and Docker clients in IOS XR control

plane LXC shell to launch their own custom LXC or Docker container on the host layer. This provides the

opportunity to compile and package a custom Linux application into a container with a Linux distribution of

choice and then deploy it on the router.


Maintaining compatibility for existing platforms The above architecture enables unique capabilities for 64-bit IOS XR with the admin plane and control plane separation as

well as In Service Software Upgrade (ISSU) and multitenancy use cases. Over the last few years, SPs have been moving to

more resilient network designs with built-in redundancy in different parts of the network, along with support for real-time

path selection and failover capabilities through controllers, segment routing, and similar features. Consequently, the need

for multitenancy and ISSU as features on standalone routers has reduced over time.

Owing to this, existing hardware platforms supported by IOS XR such as NCS5500, NCS1000, and ASR9000 will continue

to support the above architecture to maintain compatibility for existing deployment operations, even as most of the new

capabilities of IOS XR7 are enabled on top.

The new hardware platforms, namely the Cisco 8000 and the NCS540 will be introduced with a simpler modified

architecture to further simplify operations and introduce new capabilities. We describe this architecture below.

IOS XR7: Simpler, leaner, modern

Evolving a simpler architecture for the future IOS XR7 is a modern network OS built on top of the WindRiver Linux 9 (WRL9) Linux distribution using Linux kernel

version 4.8.28 as shown below:

RP/0/RP0/CPU0:ios#

RP/0/RP0/CPU0:ios#

RP/0/RP0/CPU0:ios#bash

Sun Jul 28 16:56:13.577 UTC

[ios:~]$

[ios:~]$uname -a

Linux ios 4.8.28-WR9.0.0.16_cgl #1 SMP Sun Jun 23 19:01:53 UTC 2019 x86_64 GNU/Linux

[ios:~]$

Simplicity and ease of use is one of the core principles behind IOS XR7. This simplicity manifests itself in various forms.

One of the major changes in IOS XR7 compared to IOS XR 6.X is the complete removal of the admin plane on the newer

hardware platforms, namely the Cisco 8000 and NCS540. This implies that the isolated XR control plane and admin plane


container setup that IOS XR 6.X supported (as described in the previous section) is no longer required. Consequently, the

XR7 software architecture is shown below:

● The IOS XR control plane processes run natively on the host.

● There is a single Linux shell for the whole system (the host layer).

● The libvirt and Docker daemons along with the respective virsh and Docker clients are all running in the same

environment (host shell).

● With the removal of the admin plane, all the system- and environment-level configurations and action capabilities

now move to XR.

Admin mode deprecated on Cisco 8000 and NCS540 platforms with IOS XR7:

For example, on the Cisco 8000 platform, issuing an “admin” command will throw the following “deprecation” warning:

RP/0/RP0/CPU0:ios#

RP/0/RP0/CPU0:ios#admin

Sun Jul 28 07:41:32.553 UTC

WARNING: Admin mode has been deprecated. Please consult the Command Line Interface Guide.

RP/0/RP0/CPU0:ios#

Admin mode show commands:

The typical admin mode CLI show commands such as “show environment power” or “show environment fan” have now

been moved to the XR CLI.

Admin mode exec commands:

The admin mode exec commands that typically had system-level significance have also been transitioned to the XR CLI,

making them even easier to automate. For example, the command to reload the box in IOS XR 6.X on the NCS5500 or

ASR9000 boxes is “hw-module location all reload” from the admin CLI shell. With IOS XR7 on the Cisco 8000 platform, the

command becomes “reload location all.”

Simpler operations: supporting Linux workflows

Over the last decade, SP network operations have evolved to use Linux community automation workflow installation

techniques that originated within the server world. This evolution started in the web scale service provider networks,


where the companies focused on creating networks that are provisioned, managed and manipulated using community

driven or home-grown automation tools, with almost no manual intervention.

These workflows have percolated into Day0 rollouts as well. Converting a potentially weeks long manual process of

device onboarding into a matter of hours based on the level of automation at Day-0. These workflows fall under the

category of ZTP (Zero-Touch Provisioning) which is discussed at great length in a subsequent section.

The tools utilized by SPs have seen drastic changes: moving away from Telnet, expect-style, CLI-based scripts, to more

deterministic, model-driven, ssh-based configuration management tools. These tools provide improved scalability,

reliability and are often multi-vendor compatible.

IOS XR has consistently evolved to remain effective and offers application-hosting and Packet/IO capabilities which

enable Linux applications and tools to run on-box as native binaries or containers (LXC/Docker) and allow these

applications to send and receive traffic while running on Fixed or Modular Chassis platforms. Additionally, capabilities

such as ZTP, gRPC based APIs (Service-Layer, Yang modeled APIs), Telemetry were enabled because of the functioning

Linux environment.


Automating large-scale deployments Today, if we look at the ratio of the number of network devices to network admin/operators in web deployments, the

ratio is already at 3000:1 and increasing consistently. This implies that Linux-style automation capabilities are only going

to become more prevalent as deployments evolve. In the next few years, a lot of the traditional service providers will also

adopt techniques that the web and large-scale service providers are leveraging. As a result, network operating systems

must evolve to meet these requirements.

For the last 5 to 6 years, IOS XR has been consistently evolving to meet these requirements. With IOS XR 6.X, application

hosting and packet/IO capabilities were introduced, enabling Linux applications and tools to run on-box as native binaries

or containers (LXC/Docker) and allowing these applications to send and receive traffic while running on fixed or modular

chassis platforms.

Further, capabilities such as ZTP, gRPC-based APIs (service layer, YANG-modeled APIs), and telemetry were enabled

because of the functioning Linux environment.

With IOS XR7, we take this even further. Let’s consider the critical components:

Architectural components of Linux packet/IO Interfaces in the Linux kernel: All the interfaces discovered by the IOS XR stack across Route Processors (RPs) and Line

Cards (LCs) are programmed into the kernel of the active RP in IOS XR7. These interfaces allow Linux applications to run

unhindered and behave as if they are running on a normal Linux system. This is crucial to ensure that off-the-shelf Linux

applications can be run on or against IOS XR, allowing operators to bring their existing tools and automate deployments

with IOS XR.

Routes in the Linux kernel: In the initial releases of IOS XR7, only the basic routes required to allow applications to

send/receive traffic will be programmed into the kernel. We plan to introduce enhanced capabilities in the near future.

The Linux network stack that is part of the kernel is used by normal Linux applications to send/receive packets. To allow

the Linux network stack to plumb into and use the XR network stack, we program basic routes into the Linux kernel.

1. Default route: The default route sends traffic destined to unknown subnets out of the kernel to the IOS XR

Routing Information Base (RIB) / Forwarding Information Base (FIB) for routing purposes.

2. Local/connected routes: These are the routes associated with the subnet configured on interfaces, such as

the management ports or production/data ports.

VRFs in the Linux kernel: VRFs configured in IOS XR (through CLI or YANG APIs) will be automatically synced to the

kernel. In the kernel, these VRFs will appear as network namespaces (netns). With this capability it is possible to isolate

Linux applications/processes into specific VRFs like an out-of-band management VRF and open up sockets or

send/receive traffic only on interfaces in that VRF.

When synced to the Linux kernel, every VRF is programmed as a netns with the same name as a VRF but with the string

“vrf-” prefixed to it. The default VRF in IOS XR has the name “default.” So, in the Linux kernel, this gets programmed as

“vrf-default.” Similarly, for a custom VRF called “blue,” the corresponding netns created will be “vrf-blue.”


Ability to open Linux sockets: Since the required interfaces and routes already appear in the kernel, nothing prevents

applications from opening up sockets (TCP/ User Datagram Ports (UDP)). The IOS XR infrastructure takes care of

programming these socket entries and distributes the information to line cards as well. As a result, any packet received on

a line card interface can be punted to the active RP’s kernel to send the packet to the application opening the socket.

Ability to send/receive traffic: Combining the ability to open sockets and leveraging the routes in the kernel, Linux

applications are able to send and receive traffic uninhibited.

NEW!: Supporting Linux-managed interfaces

One of the newest capabilities in IOS XR7 is the capability to make an interface of IOS XR completely “Linux managed.”

By default, all interfaces in IOS XR are managed via IOS XR configuration (CLI/YANG-API), and the attributes of the

interface (IP address, Multi-Tenant Unit (MTU), state) are inherited from the corresponding configuration and state of the

interface in XR.

A “Linux-managed” interface on the other hand, as the name suggests, is managed by Linux. For such an interface, all its

attributes and its state can be managed by using typical Linux commands (iproute2 utilities, ifconfig, etc.), and these

Linux-based inputs become the source of truth for the IOS XR-reflected interface.

In other words, one or more of the interfaces can be configured to be Linux managed, and standard automation tools

used on Linux servers can be used to manage interfaces in IOS XR. This is a big step towards making the system behave

like a Linux box operationally while enabling IOS XR network stack capabilities on top.

Troubleshooting tools: Packet capture

One of the most significant enhancements brought about by packet/IO infrastructure in IOS XR7 is the ability to capture

IOS XR control plane traffic using tcpdump and pcap libraries.

This is a huge step forward in helping operators and automated tools easily leverage tcpdump and its options to capture

and debug control plane issues on the system. The utility to use is packaged into the XR shell as “xr_tcpdump” and is

derived from the standard tcpdump and pcap library, so it supports the same options as tcpdump. Operators can now use

this utility to capture the packets on interfaces in the IOS XR kernel, exactly the same way as one would on a typical Linux

box.

Note: Data plane traffic is not supported using xr_tcpdump in the Linux shell.

This is going to be a significant step in IOS XR’s journey as it brings the feature-rich capabilities of IOS XR on top of a

standard Linux stack, allowing operators to use Linux workflows and automation tools to start managing IOS XR nodes

and troubleshoot them using standard Linux techniques such as tcpdump (shown above) as well.


Flexible software delivery (Golden ISO, RPMs)

Golden ISO (GISO)

With IOS XR7, the software can be delivered in a variety of different formats to fit the operational needs of a network

operator. The idea was to take a base binary artifact (an ISO) for the Linux distribution being used on the server fleet,

open it up, add packages and configuration files that will be used as a base across all the servers, and create a new ISO out

of the combination. This combined ISO is called the “Golden ISO” since it represents the “Golden” configuration to be

used for the server machines. Following the creation of the Golden ISO, servers would be PXE booted to get all of them

to the base “Golden” state before configuration management tools would take over to set up role-based environments on

the newly booted devices.

With IOS XR7, there will be several new capabilities that will be introduced to the Golden ISO build process. The evolution

of the GISO process is shown below:

With IOS XR7, the GISO build process will enable the following artifacts to be added to the base ISO:

1. IOS XR functionality RPMs: RPMs of different optional components in IOS XR, like BGP, OSPF, ISIS, Telnet

etc.

2. Base Configuration: A valid IOS XR configuration file can be added to the Golden ISO.

3. ztp.ini file: With IOS XR7, a new type of file will be introduced for IOS XR ZTP workflows. The ztp.ini file is

meant to allow certain default settings to be set for the ZTP process enabling a user to influence the ZTP

state machine through supported knobs.


4. Third Party application RPMs: Third party Linux applications will be supported in the Golden ISO build

process for IOS XR. For this purpose, Linux applications must be compiled for the underlying WindRiver Linux

9 (WRL9) distribution, packaged into an RPM and the RPM MUST BE SIGNED to allow the IOS XR install

process to install the RPM post boot.

5. Secure ZTP artifacts: We describe the secure ZTP artifacts in greater detail in a subsequent section. The

basic requirement for secure ZTP to function is to have an owner (customer/network-operator) certificate on

the box when it boots. A TLS client certificate (optional) can also be packaged based on the workflow desired

with Secure ZTP. The Secure ZTP enforcement flags are settings that the user may specify to modify the

state machine of the ZTP process to either enforce Secure ZTP or enable backward-compatible classic ZTP

workflows.

The build tool for Golden ISOs for IOS XR is open-source and can be found here: https://github.com/IOSXR/gisobuild.

Deploying IOS XR7 Golden ISO (GISO) images:

Once a GISO image has been created off-box, it can be deployed on a supported platform using various techniques as

shown below:

● Deploying GISO using iPXE: All barebone Cisco platforms (without an installed image), compatible with IOS XR,

support iPXE as a default bootloader. iPXE enables operators with an out-of-band management network to utilize

a DHCP server to point to the location of a GISO that may be downloaded by the platform executing iPXE. To

understand iPXE further, take a look here: https://xrdocs.io/device-lifecycle/tutorials/2016-07-27-ipxe-deep-dive/.

● Deploy GISO using USB boot: Another technique used quite often by service providers, particularly in access

networks, is to bootstrap devices in the field using a USB that contains a burnt image. A GISO image containing

the required artifacts can be burnt to a USB much like any other IOS XR image. Once done, the box must be

powered on and rebooted to select USB boot from the bootloader menu to install the GISO. This workflow is

shown below.

● GISO install using IOS XR install replace: One of the key enhancements brought into IOS XR install as part of

IOS XR 6.X was to enable the replacement of the existing software on the system with another image by simply

issuing an install replace in the XR CLI and pointing to the image location. This process activates the new software

by simply replacing the existing software with the base ISO + RPMs + supported files contained in the new image.

If the XR “install replace” command is pointed to a GISO compiled by the operator, then it will replace the existing

image with the GISO’s contents. This is shown below.

https://github.com/IOSXR/gisobuild

https://xrdocs.io/device-lifecycle/tutorials/2016-07-27-ipxe-deep-dive/


IOS XR RPMs

With IOS XR7, all the IOS XR components are fundamentally RPMs. These RPMs can be:

● Optional RPMs: Can be removed from the system or upgraded with just a process restart without any device

reload. A set of components such as BGP and ISIS are optional RPMs. Based on deployments and use cases, more

components within the system will become optional RPMs over time.

● Mandatory RPMs: Can be upgraded individually but may require a device reload. These RPMs cannot be removed

from the system.

These RPMs use semantic versioning for each release of IOS XR, greatly simplifying the delivery of new features and bug

fixes, both delivered as one or more upgraded RPMs. Consequently, with IOS XR7 on the new hardware platforms,

Software Maintenance Upgrade (SMUs) are no longer needed. Instead, bug fixes simply are an upgraded version of

existing RPMs on the system.

Simpler, secure Day 0 rollouts through Secure ZTP

With IOS XR7, Zero-Touch Provisioning (ZTP) in XR will be enhanced to support the requirements described in RFC 8572

(https://tools.ietf.org/html/rfc8572). This is what we refer to as “Secure ZTP.”

Zero-touch provisioning, as described earlier, is an important piece of the puzzle for most service providers looking to

automate Day 0 provisioning of routers to help reduce OpEx associated with SMEs and staging facilities used today. Let’s

break down the importance of ZTP for SPs, particularly access networks where the requirements deployment and

security problems are quite acute.

https://tools.ietf.org/html/rfc8572


Understanding ZTP requirements for access deployments

With XR7, IOS XR is all set to become available on access platforms such as the NCS540. Existing 1RU and 2RU platforms

such as NCS5501 and NCS5502, which already support IOS XR, are also applicable to use cases typically seen in access

deployments.

In most access deployments, network operators receive their purchased routers and usually ferry them to prestaging

facilities as part of the first “truck” rollout. At the prestaging facility, the devices are typically configured manually with

the help of SMEs to place bootstrap configurations on them. These preconfigured boxes are then shipped out to the

installation site, often using third-party installers to simply set up and power on the devices that were prestaged. This

constitutes the second “truck” rollout. This may not be the exact workflow for all network operators but is fairly

representative of the workflows in use across a majority of them.

The use of competent SMEs at prestaging facilities, the multiple truck rollouts, the postinstallation rollouts, and

corrections in case of errors in the bootstrap configurations contribute to the consistently rising OpEx (operational

expenses) that most large-scale service providers have to incur. As the number of devices in these deployments continue

to grow to meet consumer demands and newer 5G architectures, it is imperative to find techniques to reduce the OpEx

costs as much as possible.

Some of the concerns associated with these deployments are:

● Can the use of SMEs (typically at prestaging facilities or in the field to implement postinstallation corrections) be

reduced to curb OpEx

● Can prestaging facilities themselves be avoided? (ideal goal)

● A lot of these devices go out into a field where there is no secure NOC to host these devices. They may be

deployed by the side of the road or on top of telephone poles or in co-located insecure NOCs. Trust and security

are two important parameters to consider when deploying devices in access networks.


ZTP for access deployments

The scale at which zero-touch onboarding of devices in the access network must happen should be evident from the

figure below.

Owing to this, IOS XR7 ZTP is designed to address the following requirements:

1. Tree-based buildouts: To be topology agnostic, tree-based buildouts are necessary. There are no out-of-

band management networks to work with. So, every device in the network does not have L2 connectivity to a

DHCP server. This implies that an already-provisioned device must act as a DHCP relay for the next device in

the tree. This is the logic that needs to be used in the ZTP script that runs on each device—transform the

device into a DHCP relay for the connected devices downstream.


2. Security is critical: Since access devices typically go into insecure locations, it is necessary to establish trust

during the onboarding process. At a higher level, this translates into the following requirements:


3. L2 reachability through VLAN discovery: Often, the initial reachability services such as DHCP servers and

web servers may only be reachable across a layer-2 metro Ethernet cloud, for example, in mobile backhaul

topologies. Data (production) ports need to be utilized for ZTP in these networks due to absence of out-of-

band management networks. For these reasons, VLAN discovery techniques will be implemented by IOS XR7

ZTP to discover the VLAN in use across the layer-2 cloud and properly tag the DHCP request with the correct

VLAN, post discovery, to proceed with ZTP.

Support for RFC 8572 – Secure ZTP Considering the goals described above, IOS XR7 implements concepts from RFC-8572 to enable Secure ZTP capabilities in

IOS XR. This support is three-fold:

1. Support for the Secure ZTP YANG model to implement interactions between the router and a RESTCONF-

based ZTP server to download artifacts (config, scripts) to automatically provision the router.

2. Support for a Secure Golden ISO that can package an owner certificate (associated with the network

operator) to verify signatures on downloaded artifacts and validate the network domain that the device is

trying to join based on responses from the ZTP server. The Secure GISO also supports adding an optional TLS

Client certificate to help establish a TLS connection to the ZTP server to secure and encrypt subsequent

interactions.

3. An on-box infrastructure to relay a non-tamperable Secure Unique Identifier (SUDI), in other words, a

combination of a certificate and serial number to the ZTP server, and to provide a secure location on the

router to store owner certificates and keys.

The network infrastructure required to deploy IOS XR7 Secure ZTP is shown below. The ZTP server (also called bootstrap

server) follows the guidelines of RFC 8572 to implement a RESTCONF server to communicate with the device, validate the

device, and respond back with signed artifacts to provision the device.


The basic components of this architecture and workflow are:

1. Device SUDI in Trust Anchor Module (TAm): To uniquely identify a router as a Cisco device, all IOS XR7-

supported platforms will be shipped with a nontamperable Trust Anchor Module (TAm) in the hardware. The

subsequent section on device trust and security goes into further detail on the working of the TAm and the

various keys and certificates stored within. For ZTP we are particularly interested in the Secure Unique Device

Identifier (SUDI) certificate that is rooted to Cisco-CA, which contains a unique public key along with a serial

number and is used as a cryptographically secure identification of the device in its interaction with external

services.


2. Secure Golden ISO image: A Secure GISO image is created by using the normal GISO techniques described

in the earlier section. The network operator packages a certificate (called the owner certificate that is rooted

to a CA of their choice) into the GISO image. In addition, an optional TLS client certificate may also be

packaged into the image for TLS connections to the ZTP server.

3. SZTP DHCP options: To discover the ZTP server that hosts the artifacts to be downloaded by the router, the

router must learn the redirect URL to the ZTP server from some source. This source could be DHCP, DNS, or

USB based. Typically, a DHCP server would be used to provide an “untrusted” redirect URL to a

bootstrap/ZTP server using DHCPv4 option 143 or DHCPv6 option 136 (both options defined as SZTP DHCP

options in RFC 8572).

4. Bootstrap server/ZTP server: The ZTP server consists of a RESTCONF server that will interact using the

SZTP YANG model with the router to deliver config/script artifacts to it.

Note: As part of the IOS XR7 ecosystem, the Cisco Crosswork Network Automation platform (https://developer.cisco.com/docs/crosswork/) will provide the ZTP server capabilities as a supported service from Cisco, enabling Day 0 automation with dashboards, scripts, and template management and more.

5. SZTP YANG model: RFC 8572 defines a YANG model (see RFC 8572: https://tools.ietf.org/html/rfc8572)for

model-driven interactions between the router and the ZTP server. Of particular interest is the initial request

made by the router to the ZTP server, which contains router-specific information such as the hardware (HW)

model, OS name, version, and an optional nonce. The response from the ZTP server encodes the onboarding

information that would be fetched by the device to either reimage or provision itself.

https://developer.cisco.com/docs/crosswork/

https://tools.ietf.org/html/rfc8572


Secure ZTP: Understanding the trust model

The premise of Secure ZTP is that three types of validations must occur for a fully trusted device to onboard on to an

operator’s network:

1. Router/client validation: The ZTP server must validate the SUDI certificate offered by the device in the initial

YANG request.

2. Server validation: For this purpose, the owner certificate that is loaded onto the box as part of the Secure

GISO image is validated against an “ownership voucher” or domain certificate delivered to the device either

during the GISO installation or during interactions with the ZTP server. This helps validate that the router is

joining the correct network “domain.”

3. Artifact validation: The artifacts specified in the onboarding information in the YANG response from the

ZTP server can identify 4 types of artifacts:

a. Boot image: The boot image URL is specified in the onboarding information. The boot image artifact

downloaded from this URL must be a separately signed artifact that is securely booted on the device

using the principles of secure boot laid out in the UEFI specification 2.3.1. With Cisco hardware

platforms, we go a step further and enable hardware-based secure boot to protect from rootkit attacks

that can compromise normal UEFI secure boot.

b. Preconfiguration script: This is a base64 encoded script (Bash, Python, etc.) that is provided in the

onboarding information in the ZTP server YANG response. This script is typically used to do

housekeeping tasks on the system before applying a configuration, like installing required IOS XR

packages utilized by the config.

c. Configuration: This is a standard XR CLI configuration or a valid XR YANG-modeled configuration in

XML that may be merged or replaced based on the action specified in the onboarding information.

d. Postconfiguration script: The postconfiguration script is similar to the preconfiguration script but is

used to handle the final setup of the router such as creation of SSH keys, connecting and registering to

Day 1 monitoring and provisioning solutions, and spinning up on-box applications.

It should be fairly obvious from the breakdown of the Secure ZTP capabilities of IOS XR7 that IOS XR7 devices are

perfectly suited to enable secure, trusted, Day 0 rollouts at high scale for web and large-scale service providers. The

access networks, in particular, will greatly benefit from these workflows, significantly reducing OpEx and the time to

deploy IOS XR7 routers such as the NCS540, Cisco 8000, and others, irrespective of the topology.

https://uefi.org/sites/default/files/resources/UEFI_Secure_Boot_in_Modern_Computer_Security_Solutions_2013.pdf


Powerful new IOS XR install

Modern network operating systems need modern solutions for software management.

IOS XR 7 packaging and install

Design principles:

IOS XR7 employs a completely fresh approach to the design of IOS XR install. The basic principles behind it are:

1. Conform to Linux norms: Provide a user experience for software installation and upgrade that "looks and

feels standard and Linux-like," integrating extra functionality needed to satisfy carrier class requirements

using standard solutions in the Linux ecosystem wherever possible and extending the capabilities where

needed for XR use cases.

2. Implement a modular architecture: Implement an architecture that can be adapted for different form

factors and use cases from fixed single-node systems, through modular systems, to large clusters.

3. Provide automatable interfaces: Provide programmatic northbound interfaces that are ready for use in

automations.

Further, IOS XR7 install handles installation of ALL packages on the system (XR RPMs or Linux RPMs), enabling users to

manage the lifecycle of XR packages and custom applications in a unified manner.

High-level architecture:

On newer platforms (Cisco 8000, NCS540) running IOS XR7, the software running on each node (Route Processor [RP],

Line Card [LC]) uses the WindRiver Linux 9 (WRL9) distribution, which is an RPM-based distribution.

For this reason, IOS XR7 install utilizes the latest architecture for RPM package management, which originated in the

Fedora community, called DNF. DNF or Dandified yum is the next-generation version of yum. It roughly maintains CLI

compatibility with yum and defines a strict API for extensions and plugins.

Since DNF as a package manager is capable of handling packages on a single Linux node, IOS XR7 extends the capabilities

of DNF to handle installation and package management requests for a modular system (consisting of multiple RPs and

LCs).

https://fedoraproject.org/wiki/DNF

https://fedoraproject.org/wiki/Yum


Packages and SMUs in IOS XR7

Before we dive into the user interface for IOS XR7 install, let’s look at the types of packages that will be supported by the

install process.

1. IOS XR7 package RPMs: All components in IOS XR7 are fundamentally RPMs. However, these RPMs can be

classified into two types:

a. Optional package RPMs: A number of IOS XR packages (features and protocols such as BGP, Multicast,

ISIS, Telnet, etc.) will be published as RPMs with a clear dependency tree in their metadata fields. As

explained in the earlier section describing software delivery in IOS XR7, a list of packages in IOS XR has

now become optional packages (in other words, they can be removed from the system). This allows

operators to remove these components if their configuration does not utilize them. As IOS XR7 matures

further, we intend to isolate even more components into individual RPMs, allowing operators to run

thinner variants of XR with just the amount of software they need, and nothing more. Optional package

RPMs can be installed or upgraded without requiring a reload of the system. Only a process restart is

required.

b. Mandatory package RPMs: These are components that are still RPMs but utilized by a number of

higher-level optional components as well as infrastructure components as a common entity. Hence these

RPMs cannot be removed from the system, but they can be upgraded or patched.

Mandatory package RPMs may require a reload post an upgrade. By default, they are part of the base

ISO image for IOS XR7.

2. Bootable IOS XR7 images: The install process can accept a bootable IOS XR7 image as a source of packages

(RPMs) and files (certs, settings, configs). This can be an image published on https://software.cisco.com or a

custom Golden ISO (see earlier section explaining Golden ISO as a delivery format for packages, special files,

certificates/keys, etc.). The install process will use the supplied image and activate the software in the image

as if the system was booted with the ISO.

https://software.cisco.com/


IOS XR7 install user interface and workflows A typical environment to help work with IOS XR7 install is shown below:

The Linux server serves two purposes in this topology:

● It serves up an RPM repository that hosts a set of optional RPMs that are not part of the base image on the router.

The list of RPMs on the server is shown in the figure.

● A Python telemetry collector to set up and receive event-driven syslog telemetry data to monitor the install

operations on the router.

Basic installation commands with IOS XR7

Install command Function

install source [repo name| repo url] package-name This is a one-touch command to install and apply packages on the

system. The new software is configured to be booted for the next

reboot only.

Install rollback [id] [noprompt] [commit] This is a one-touch command to roll back to a known snapshot ID.

install commit This command marks the end of the transaction and is used to

indicate that the operator is happy with the updated software and

the bootloader’s default menu entry is updated to boot the new

software (instead of the snapshot taken at the beginning of the

transaction).

https://github.com/IOSXR/telemetry-grpc-collectors/blob/master/clients/python/telemetry_client_json.py


Configuring a remote repository

With IOS XR7, the remote repository can be set up as part of the configuration. This is one way of installing packages from

an RPM repository. This methodology is similar to how typical Linux systems are set up—the DNF or yum repositories are

configured before running package management commands.

Note: It is also possible to set up a local DNF repository on the router itself or simply point to a set of RPMs (copied

over) in a local directory on the router to install the required components.

IOS XR7 install show commands

Show install active: This show command dumps the currently active IOS XR packages on the system. These are the

RPMs that have been already installed or were part of the base image when the platform booted.

Show install cached: This output lists down ALL the RPM packages added to the RPM database. This includes IOS XR

packages as well as the Linux (WRL9) RPMs.

Show install available: This command first checks the set of installed IOS XR packages on the system and then sends a

request to all configured repositories (like repo1 configured above). If there are packages in the repos that are not in the

current RPM database, then these additional packages are dumped in the output.

One-touch install

As explained earlier, one of the install exec commands is “install source.” This command invokes DNF on each node on the

system, takes care of dependency management, pulls in all the required packages, and activates the required software by

either restarting a process or reloading the box as the need may be.

Viewing install progress through telemetry

Throughout the install process above, structured syslog events will be sent to the telemetry collector, as and when

different install events occur. It is very useful to monitor the install operations during maintenance windows or

automation. Some sample data is shown below:

{

"encoding_path": "Cisco-IOS XR-infra-syslog-oper:syslog/messages/message",

"subscription_id_str": "1",

"collection_start_time": "1564462273170",

"msg_timestamp": "1564462273182",

"collection_end_time": "1564462273182",

"node_id_str": "ios",

"data_json": [

{

"keys": [

{

"message-id": 189

}

],

"timestamp": "1564462273178",


"content": {

"category": "INFRA",

"card-type": "RP",

"time-stamp": "1564462273000",

"group": "INSTALL",

"time-of-day": "Jul 30 04:51:13.164 UTC: ",

"time-zone": "UTC",

"text": "instorch[194]: %INFRA-INSTALL-6-ACTION_BEGIN : Compound install

operation 1.1 started \n",

"process-name": "instorch",

"node-name": "0/RP0/CPU0",

"message-name": "ACTION_BEGIN",

"severity": "message-severity-informational"

}

}

],

"collection_id": "3"

}

"time-zone": "UTC",

"text": "instorch[194]: %INFRA-INSTALL-6-ACTION_COMPLETE : Apply by process

restart 1.1 complete \n",



############################################################ SNIP OUTPUT

################################################################## "group":

"INSTALL",

"time-of-day": "Jul 30 04:53:19.408 UTC: ",

"time-zone": "UTC",

"text": "instorch[194]: %INFRA-INSTALL-6-ACTION_COMPLETE : Compound install

operation 1.1 complete \n",



"message-name": "ACTION_COMPLETE",

"severity": "message-severity-informational"

}

}

],

"collection_id": "9"

}

The process name “instorch” (install orchestrator) is associated with the messages sent by the install process to syslog.

This can be a very useful technique to get real-time events associated with install operations and monitor them for errors.


Rollbacks and snapshots:

One of the most useful capabilities that IOS XR7 brings to the table is the safety of rollbacks to previous snapshots

representing different commit stages in the install process.

● Whenever any install process is initiated in the IOS XR7, a backup snapshot of the root file system is taken, and

the snapshot is stored on disk. Further the bootloader is updated, so that the snapshot is booted by default.

● If the install process fails at any stage, the system automatically reverts to the snapshot just created.

● When one issues an “install commit,” the bootloader is updated so as to point to the new root file system as the

default boot entry. The old root file system is saved as a snapshot with an ID.

● If one desires, it is possible to rollback to an earlier snapshot by using the “install rollback” command.

IOS XR7: Modern, powerful, and extensible

IOS XR has consistently focused on building programmatic and model-driven interaction points within its stack to provide

automatable interfaces that enable easier deployment and management of network devices. This design allows the

capabilities of IOS XR to be easily extended through powerful integrations with external tools, applications and

controllers.

But it is hard to pigeon-hole these capabilities into a conversation about APIs alone. As technologies such as Segment-

Routing and EVPN have evolved, Network operators have leveraged them to transition their legacy transport networks

from protocol-heavy deployments (MPLS, RSVP-TE, Layer2 and Layer3 VPN deployments) to simpler, leaner solutions

built on these technologies, often in tandem with external controllers and planning tools.


It is therefore important to understand these capabilities in the context of evolving Network Operations. Typically, we can

classify network operations into 3 categories:

● Day0: The initial rollout of devices, automated deployments through ZTP, registration with Day1 management

and monitoring solutions. Zero-Touch provisioning APIs and infrastructure play a key role here.

● Day1: Role-based Device configuration in the network, bring-up of centralized management solutions and

controllers, mass-deployment of applications and policies using scalable configuration management tools and

automation. Model-Driven Yang-based APIs for device provisioning, model-driven Telemetry capabilities for

device monitoring, Segment-routing and EVPN solutions to roll-out services and policies in the network are the

key elements utilized in Day-1 operations.

● DayN: Manipulate state in the network using real-time network and application information, remediate network

degradation issues, raise alarms and schedule maintenance windows, analyze network telemetry data and

patterns to evolve the network design and meet future challenges. Custom controllers and applications that

respond to network and application events, leveraging high-performance lower-layer APIs such as Service-Layer

and Open Forwarding abstraction (OFA) APIs, feedback loops implemented through telemetry and device-

provisioning APIs built on YANG are the typical components that drive innovation in DayN solutions.

Having understood how these APIs and technologies align with different operational paradigms described above, let’s

break down the IOS XR7 network stack to understand where these capabilities lie.

Shown below are the different layers of the IOS XR7 stack with corresponding capabilities identified:


● Manageability layer: This layer exposes the Command Line Interface (CLI) and YANG-modeled APIs for

features and protocols in the network application layer. These YANG models are used for device provisioning and

streaming telemetry. ZTP APIs are built on top of CLI and YANG APIs in this layer. This layer leverages an

internal “database” within IOS XR called SysDB.

● Network application/protocol layer: This layer contains protocols (BGP, ISIS, OSPF, etc.) and features such as

L2VPN, L3VPN, etc. This layer interacts with SYSDB to store the operational and configuration state of the

protocols and features. The higher-layer components of segment routing and EVPN fall into this layer.

● Network infrastructure/service adaptation layer: Typically consists of components such as the RIB, label switch

database, BFD, network interface handler, and APIs for controllers/agents. The API provided on top of this layer

for end users is called the service layer API. The infrastructure components of all XR control plane protocols

(BGP, ISIS, OSPF, etc.), segment routing, and EVPN get mapped to this layer.

● Hardware abstraction layer (OFA): This layer interacts with the ASIC SDK (provided by the ASIC vendor) and

handles programming of the data plane based on RIB state or LSD (label switch database) state. The Open

Forwarding Abstraction (OFA) API abstracts the capabilities of this layer for external applications.

● Platform integration layer: Typically consists of kernel and/or userland drivers for devices such as fans, leds,

sensors, etc. The integration of these device-specific drivers with the higher layers of the software stack happens

at this layer. It is this layer that allows a software stack to extract information (for example, temperature) from

sensors and influence component state (for example, fan speed).

Manageability layer: YANG models, device provisioning, telemetry

The manageability layer houses the internal configuration and operational state database of XR called SysDB. This is the

foundation of the YANG models that are exposed for automation through external tools over protocols such as Netconf

or gRPC. Further, IOS XR supports native (XR-specific) YANG models as well as OpenConfig YANG models for device

provisioning and management. These YANG models are published for every release of XR on GitHub here.

IOS XR telemetry enables cadence-based and event-based streaming of structured operational state data from the device

over TCP, UDP, and gRPC as transports and GPB or JSON as encoding formats. The proto files for gRPC client collectors

for IOS XR telemetry per release of XR can be found here.

Further, IOS XR7 also supports OpenConfig techniques for device management with support for gNOI and gNMI.

Zero-Touch Provisioning (ZTP) APIs

Zero-touch provisioning APIs are available as Bash and Python libraries on-box. These APIs can be leveraged to perform

all CLI operations (configuration, show commands, exec commands) as well as YANG-based RPC calls (config, oper,

action) to provision the system. These APIs can be leveraged by ZTP scripts as part of Day 0 operations or for device

automation through on-box scripts for Day 1 and beyond. More details on ZTP libraries can be found here.

Service layer APIs

For most automation use cases, the manageability layer that provides the CLI, YANG models, and streaming telemetry

capabilities is adequate. However, over the last few years, we have seen a growing reliance in web-scale and large-scale

service provider networks on off-box controllers or on-box agents that extract away the state machine of a traditional

protocol or feature and marry their operation to the requirements of a specific set of applications on the network.

These agents/controllers require high-performance access to the lowest layer of the network stack (RIB, label switch

database [LSD], BFD events, interface events, etc.), and the API that exposes this functionality is called the service layer

https://github.com/YangModels/yang/tree/master/vendor/cisco/xr

https://github.com/ios-xr/model-driven-telemetry/tree/master/protos

https://github.com/openconfig/gnoi

https://github.com/openconfig/gnmi

https://developer.cisco.com/learning/modules/iosxr-cli-automation


API. These APIs are accessible over gRPC and are model-driven with the models described using the GPB Interface

Descriptor Language (IDL). These models and sample clients can be found on GitHub here.

More details on service layer APIs can be found on xrdocs here: https://xrdocs.io/cisco-service-layer/

Open Forwarding Abstraction (OFA) API

The Open Forwarding Abstraction (OFA) API is a model-driven API built to abstract underlying hardware SDKs, enabling

applications and controllers to manipulate hardware state directly over a performant channel by providing a consistent

interface to program against, irrespective of the hardware platform.

Internally, this API provides flexibility and rapid enablement of IOS XR across multiple data planes and hardware

platforms, allowing the IOS XR portfolio to grow further.

The OFA API can also be leveraged as an abstraction to expose capabilities such as P4-runtime on top of non-P4-

compatible hardware, enabling unique use cases where client applications and controllers are designed for a specific

abstraction layer.

The OFA API is also designed to be model-driven and available over gRPC as a transport option. This API will be released

for public consumption soon.

IOS XR7: Trustworthy

In IOS XR7, a running theme around any discussion of device or NOS security is the idea of Trust. The goal of IOS XR7 and

its compatible hardware is to build a “Trustworthy Platform” where device and OS integrity are as important as the idea

of security. In fact, it is important to note that no device can ever be considered secure – this is an ideal state; but not

necessarily achievable. A level of Trust can be measured, verified and audited.

It is important to articulate the primary concerns associated with network device security:

● How does an operator ensure that the router is running only the software that was intended?

● How does one know that a router hasn’t been physically altered since Cisco built it?

It is possible to address these concerns by immediately building security controls such as enabling only signed software to

install and run or having known-good-values for the hardware components to match and verify during boot. Taking it a

step further and ensuring that security controls are backed by tamper-proof hardware that stores certificates and keys

and the known-good-values to use for validation helps provide complete validation. Attestation capabilities allow the

verification results to be visualized and reported which can prove the ability to trust the component and network.

https://github.com/Cisco-Service-Layer/service-layer-objmodel

https://xrdocs.io/cisco-service-layer/


To address these concerns and understand the Trust paradigm within IOS XR7 and its associated platforms, we need to

understand the chain of Trust for a typical network device.

Hardware root of trust

The concept is simple. If we trace back the boot process, then pushing the root of trust as further back as possible ensures

the highest level of trust in the system.

Typically, there are well-known attacks and vulnerabilities that compromise the runtime OS, kernel, and bootloader

(GRUB).

Secure boot as a concept is laid out in the UEFI specification 2.3.1 that expects “trust” to begin at the BIOS stage,

subsequently checking the bootloader, kernel, and then the OS. But in the last few years, we’ve seen UEFI rootkits in the

wild that can compromise the BIOS, thereby jeopardizing the secure boot process. Further, the scope of security issues

for Intel ME and AMT, both BIOS extensions on Intel hardware, is significantly increasing. Tons of research has been

already presented about problems in the UEFI firmware ecosystem and how relatively easy it is to deliver and install

implant/rootkit.

This implies that to properly secure the boot process, validation checks have to begin much earlier in the boot cycle, even

before the BIOS boots. With IOS XR7 platforms, we do exactly that. Cisco hardware platforms execute what we call a

“hardware-anchored secure boot.”


http://blog.ptsecurity.com/2018/10/intel-me-manufacturing-mode-macbook.html?source=post_page---------------------------

https://embedi.org/resources/the-adventure-of-the-final-intel-amt-problem/?source=post_page---------------------------


Cisco secure boot versus UEFI secure boot Cisco secure boot is differentiated by anchoring the secure boot process in hardware, thus providing the most robust

security. It is robust because hardware modification is difficult, expensive, and not easy to conceal, even if the hacker has

physical possession of the device.


This process creates a “chain of trust” from microloader through bootloader to the operating system, establishing the

authenticity of the software. If any digital signature check fails, the Cisco device will not let that software boot, so that

malicious code will not run.

Cisco Trust Anchor Module (TAm) vs industry-standard TPM When the Cisco hardware-anchored secure boot has authenticated the software as genuine Cisco software in a Cisco

device with the TAm, the operating system then queries the TAm to verify that the hardware is authentic. It does this by

cryptographically checking the TAm for a Secure Unique Device Identifier (SUDI) that could have come only from Cisco.

The SUDI is permanently programmed into the TAm and logged by Cisco during Cisco’s closed, secured, and audited

manufacturing processes. This programming provides strong supply chain security, which is particularly important for

embedded systems like routers and switches.

In contrast, although the industry-standard TPM has similar functions to those of the TAm, it is not typically permanently

programmed during manufacture with a unique device identifier and is left to the end user. This process requires user

intervention and development, which does provide flexibility; however, the risk of supply chain modifications is also

greater.


TAm certificates and keys

The basic setup of TAm is shown alongside. It is useful to understand the functions of keys and certificates in TAm to see

how each component in the boot process becomes trustworthy—from BIOS, bootloader, kernel, OS, and RPMs to

application processes.

These keys (PK, KEK, DB/DBx) are defined in the UEFI Secure Boot specification.

Cisco TAm follows the same specification.

In particular, the following keys are unique to the Cisco TAm implementation:

● <platform-family> key: This is a public key certificate that is common to all types of platforms belonging to a

single family (for example, Cisco 8000). This key is used to sign the bootloader (GRUB), initrd, kernel, and

kernel modules.

● RPM key: This key is used to sign RPMs.

● IMA public key certificate: Used to sign hash of binaries that will launch processes at runtime.

● LDWM key: This key is used to sign system boot firmware (BIOS) and Field-Programmable Gate Array (FPGA)

images. Following the LDWM draft, these keys have a smaller footprint than RSA keys.

● SUDI (Secure Unique Device Identifier): SUDI refers to an asymmetric key pair and an associated certificate. The

certificate contains the device UDI (PID + serial number).


https://tools.ietf.org/html/draft-mcgrew-hash-sigs-00


● Attestation Key (AIK): It is an RSA key pair and is used to sign data held within the TAm itself. It cannot be used

to sign data outside the TAm, making it very hard to spoof a “TAm quote” (high-integrity reports of the

trustworthiness of integrity measurements). In addition, AIK is also used to attest PCR state (state generated in

TAm post each verification process in a boot sequence). The retrieval of the TAm quote and PCR quote is called

attestation.

Note: Even though the UEFI specification does not enforce it, Cisco TAm design enforces that all keys in TAm will be

part of a trust chain that can be traced back to the Cisco public root certificate.

All the keys in TAm are installed by Cisco during the manufacturing process.

Secure boot

As explained above, Cisco platforms execute a hardware-anchored secure boot. The hardware root of trust is enabled

through the use of the tamper-proof module called TAm that hosts the set of keys used to verify different stages of the

boot process and beyond. The basic secure boot flow is described below:

BIOS launch and verification:

1. Cisco public keys in TAm (PK, KEK, DB) are used to verify signatures during the initial boot process. At

powerup, a microloader in the TAm first verifies the digital signature of the BIOS using the LDWM key in

TAm.

2. Once verified, BIOS executes verification of the hardware against Known Good Values (KGVs) of the

hardware inside the database in TAm. These known good values were programmed by manufacturing. In the

case there is a failure, then the failure is logged.

Bootloader launch and verification:

3. Next, the BIOS verifies the digital signature of bootloader using the <platform-family> key in TAm DB.

Kernel, initrd, grub-config verification:

4. Once bootloader is verified, bootloader is launched by BIOS. Bootloader then takes help of BIOS to verify

kernel, initrd, and grub-config.

5. Each verification operation is logged. Initrd is then expanded to create the root file system.

Kernel modules verification:

6. Kernel is launched and the required keys (PK, KEK, IMA, RPM) are loaded into the kernel keyrings.

7. Kernel then verifies the kernel modules, and the results are logged.

OS process boot:

8. IMA, which is used to validate signatures at runtime, is launched with appropriate IMA policy to validate the

init process.


XR RPM installation

9. IOS XR install process now runs to install IOS XR RPMs that are part of the image.

10. The IOS XR install process uses the RPM key loaded up into the kernel keyring from TAm to verify the

signatures on all RPMs before installing them. The results of RPM verification are logged.

XR process launch

11. Finally, XR processes are launched and each process is subject to the IMA policy checks to verify signatures on

their hashes before launch.

Signed RPMs

Each XR RPM in IOS XR7 is signed using the RPM key in the TAm. This ensures that only signed RPMs can be installed on

the system using IOS XR7 install (existing and new platforms). This also implies that any third-party applications that are

compiled for the underlying Linux distribution of IOS XR7 must be signed before installation. For this purpose, operators

can load up custom certificates onto the system and sign their applications using the custom certificates, enabling a great

flexibility in the management lifecycle of these applications.

Trust at runtime (IMA)

Integrity Measurement Architecture (IMA) is a process intended to provide assurance that executables preparing to load

have not been modified from their original form. It is important to be able to ensure that the system has not been

exposed to tampered code.

While all IOS XR binaries and processes would be set up for IMA by default, a network operator is responsible for signing

their third-party application binaries and loading up the signed hash on the system before attempting to run a third-party

process.

There are two parts to the IMA process:

● IMA measurements/logging: IMA maintains a runtime measurement list, and in Cisco platforms it is anchored in

the hardware-trusted platform module (TAm). IMA also maintains an aggregate integrity value over this list and

pushes it into TAm. The benefit of storing the aggregate integrity value in the TAm is that the measurement list

cannot be compromised by any software attack, without being detectable.

Hence, on a trusted boot system, IMA can be used to attest to the system’s runtime integrity.

● IMA appraisal: The IMA appraisal extension adds local integrity validation and enforcement of the IMA

measurement against a "good" value stored as an extended attribute. The validation process checks for file data

integrity and also verifies the digital signature, thereby providing authenticity.

IMA appraisal would normally block applications if the appraisal failed—this is the enforcement mode.

Note: In the initial releases, IOS XR7 will simply log the failure post validation instead of blocking the app. This is to help

operators get used to packaging their custom processes and applications and running it on the IOS XR system, before

moving towards enforcement.


Trust reports and visualization

The system takes a number of cryptographic measurements at various points in its life: hardware integrity, bootup,

software execution, software install and upgrade, IPC integrity, etc. Visibility into the measurements made on each node

in the device is integral to achieving trust in the system.

Remote attestation is the process of querying the TAm on each node in the device for the measurements taken on that

node and providing visibility post the query.

The high-level flow of a network device integrity attestation process is shown below:


Cisco environmental sustainability

Information about Cisco’s environmental sustainability policies and initiatives for our products, solutions, operations, and

extended operations or supply chain is provided in the “Environment Sustainability” section of Cisco’s Corporate Social

Responsibility (CSR) Report.

Reference links to information about key environmental sustainability topics (mentioned in the “Environment

Sustainability” section of the CSR Report) are provided in the following table:

Sustainability topic Reference

Information on product material content laws and regulations Materials

Information on electronic waste laws and regulations, including products, batteries, and packaging WEEE compliance

Cisco makes the packaging data available for informational purposes only. It may not reflect the most current legal

developments, and Cisco does not represent, warrant, or guarantee that it is complete, accurate, or up to date. This

information is subject to change without notice.

Cisco Capital

Flexible payment solutions to help you achieve your objectives

Cisco Capital makes it easier to get the right technology to achieve your objectives, enable business transformation

and help you stay competitive. We can help you reduce the total cost of ownership, conserve capital, and accelerate

growth. In more than 100 countries, our flexible payment solutions can help you acquire hardware, software, services and

complementary third-party equipment in easy, predictable payments. Learn more.

Printed in USA C78-743014-00 11/19

https://www-1.compliance2product.com/c2p/getAttachment.do?code=YM6Y0yThdO6Wj1FxxYPYfUG2dtFkTeFWGpzLRO8tcURFEifUCRV403Tq2ZMWP6Ai

https://www-1.compliance2product.com/c2p/getAttachment.do?code=YM6Y0yThdO6Wj1FxxYPYfUG2dtFkTeFWGpzLRO8tcURFEifUCRV403Tq2ZMWP6Ai

https://www.cisco.com/c/en/us/about/product-innovation-stewardship/materials.html

https://www.cisco.com/c/en/us/about/product-innovation-stewardship/product-recycling/weee-compliance.html

https://www.cisco.com/go/financing

IOS XR7 Data Sheet...Cisco IOS XR’s lockstep evolution The Cisco® Internetwork Operating System...

Documents

Transcript of IOS XR7 Data Sheet...Cisco IOS XR’s lockstep evolution The Cisco® Internetwork Operating System...