PCI Passthrough with Red Hat OpenStack...
Transcript of PCI Passthrough with Red Hat OpenStack...
![Page 1: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/1.jpg)
PCI Passthrough with Red Hat
OpenStack Platform
Ken Savich
Sr OpenStack Solution Architect, Red Hat2016-12-6
![Page 2: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/2.jpg)
About Me
- Sr OpenStack Solution Architect
- with Red Hat since January 2016
- previously engineering at Rackspace - 9 years
- I love ramen.
![Page 3: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/3.jpg)
пройти через
![Page 4: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/4.jpg)
Ken Savich
Agenda
- Background
- The story
- Technical details
- Scripting
- Results
- Applications
![Page 5: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/5.jpg)
Ken Savich
Red Hat OpenStack Platform 9
- 100% Community based
- Enterprise proven
- Automated updates and upgrades
- Co-engineered performance
![Page 6: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/6.jpg)
ADD NAME (View > Master > Slide master)
Background
![Page 7: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/7.jpg)
Ken Savich
Why PCI Passthrough?
- Performance
- Exclusive access
- No hypervisor support
- Domain isolation
- Security
![Page 8: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/8.jpg)
Ken Savich
device emulation
![Page 9: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/9.jpg)
Ken Savich
device passthrough
![Page 10: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/10.jpg)
ADD NAME (View > Master > Slide master)
The Story
![Page 11: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/11.jpg)
Ken Savich
- Cisco- Nvidia- Petrel- Red Hat
![Page 12: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/12.jpg)
ADD NAME (View > Master > Slide master)
![Page 13: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/13.jpg)
ADD NAME (View > Master > Slide master)
Requirements
- Red Hat OpenStack
- Self-contained infrastructure
- Windows 7 guests w/Petrel software
- Native guest access to Nvidia Tesla m60 grid cards
- Performance!
![Page 14: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/14.jpg)
ADD NAME (View > Master > Slide master)
Technical Details
![Page 15: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/15.jpg)
ADD NAME (View > Master > Slide master)
Requirements
- RHEL 7.3 - kernel < 4.1
- Virtualization Technology - bios & kernel
- vmx (intel)
- svm (amd)
- Red Hat OpenStack Platform 8 or 9
- A PCI card with native drivers for your guest
$ egrep `(vmx|svm)` /proc/cpuinfo
![Page 16: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/16.jpg)
ADD NAME (View > Master > Slide master)
Procedure
- Turn off hyper-threading- Find your pci device- Configure your Compute node(s)- Configure your Controller node(s)- Create a flavor- Launch and instance- Test test test!
![Page 17: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/17.jpg)
ADD NAME (View > Master > Slide master)
PCI Passthrough - first steps
- Get your packages $ sudo yum install pciutils gcc kernel-devel kernel-debug kernel-headers -y
- blacklist any competing drivers/etc/default/grub: GRUB_CMDLINE_LINUX=... rd.driver.blacklist=nouveau
$ sudo echo "blacklist nouveau" > /etc/modprobe.d/nvidia-blacklist.conf
$ sudo echo "blacklist nvidia" >> /etc/modprobe.d/nvidia-blacklist.conf
- turn on IOMMU/etc/default/grub: GRUB_CMDLINE_LINUX=... intel_iommu=on
![Page 18: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/18.jpg)
ADD NAME (View > Master > Slide master)
PCI Passthrough - now we’re feeling it
- Find your device$ sudo lspci |grep -i nvidia
05:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)
05:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
- Find the pci id of your device$ sudo lspci -n|grep 05:00.0|awk '{print $3}'
10de:13c2
- Tell grub the PCI stub ID/etc/default/grub: GRUB_CMDLINE_LINUX=... pci-stub.ids=10de:13c2
![Page 19: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/19.jpg)
ADD NAME (View > Master > Slide master)
PCI Passthrough - bounce it
- Update grub$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
- Configure kernel hugepages# mkdir /hugetlbfs
# echo "2048" > /proc/sys/vm/nr_hugepages
# echo "vm.nr_hugepages=2048" > /etc/modprobe.d/hugepages.conf
# echo "hugetlbfs /hugetlbfs hugetlbfs defaults 0 0" >> /etc/fstab
- REBOOT!!!
![Page 20: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/20.jpg)
ADD NAME (View > Master > Slide master)
PCI Passthrough - Nova - Compute Node
- Find your vendor and product codes# lspci -nn|grep -P '^(?=.*nvidia)(?=.*vga)'
05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
- Whitelist your device in nova.conf/etc/nova/nova.conf:
pci_passthrough_whitelist = [{"vendor_id": "10de", "product_id": "13c2"}]
- Restart nova service$ sudo systemctl restart openstack-nova-compute.service
![Page 21: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/21.jpg)
ADD NAME (View > Master > Slide master)
PCI Passthrough - Nova - Controller
- Add the filter class to the scheduler/etc/nova/nova.conf:
scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter
- Make sure PciPassthroughFilter is in the default list/etc/nova/nova.conf:
scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter
![Page 22: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/22.jpg)
ADD NAME (View > Master > Slide master)
PCI Passthrough - Nova - Controller
- Add an alias for your device/etc/nova/nova.conf
pci_alias = {"vendor_id":"10de", "product_id":"13c2", "name":"nvidia"}
- Restart nova services# systemctl restart openstack-nova-api.service openstack-nova-scheduler.service \
openstack-nova-conductor.service
![Page 23: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/23.jpg)
ADD NAME (View > Master > Slide master)
Off to the races...
- Create a flavor$ nova flavor-key m1.large set "pci_passthrough:alias"="nvidia:2"
- Launch an instance$ nova boot --image rhel7 --key_name test --flavor m1.large mypci
- Install native drivers in your guest
![Page 24: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/24.jpg)
ADD NAME (View > Master > Slide master)
Common Problems
- PCIe reset
- Virtio drivers
- Display virtualization
- Live migration
![Page 25: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/25.jpg)
ADD NAME (View > Master > Slide master)
Divider SlideScripting
![Page 26: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/26.jpg)
ADD NAME (View > Master > Slide master)
Scripting
- goo.gl/RsceFv
- Requires RHEL 7.x
- Nvidia specific
- Must be run on compute nodes
![Page 27: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/27.jpg)
ADD NAME (View > Master > Slide master)
Per-node configuration with Tripelo
- Interfaces
OS::TripleO::ControllerExtraConfigPre: Controller node additional configuration
OS::TripleO::ComputeExtraConfigPre: Compute node additional configuration
- for examples, see:
/usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml
![Page 28: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/28.jpg)
ADD NAME (View > Master > Slide master)
Results
![Page 29: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/29.jpg)
ADD NAME (View > Master > Slide master)
Benchmarking
- Windows 7 instance- Red Hat virtIO drivers
- Furmark = 60+ fps
![Page 30: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/30.jpg)
ADD NAME (View > Master > Slide master)
Applications
![Page 31: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/31.jpg)
ADD NAME (View > Master > Slide master)
Possibilities!
- HPC applications- Video encoding- Audio pre/post- NFV
- SR-IOV for VNF- Gaming
![Page 32: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/32.jpg)
ADD NAME (View > Master > Slide master)
Try it for yourself, you’ll like it
- Get Red Hat OpenStack 9 here:- https://access.redhat.com/products/red-hat-openstack-platform/evaluation
● Provides a full-featured 60 day entitlement.● Provides self-service support.● Offers three 2-socket server subscriptions - enough to run a small 3-node test
cluster.● Is not intended for production use.
Let me know how it goes: [email protected]
![Page 33: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/33.jpg)
ADD NAME (View > Master > Slide master)
References
- Red Hat PCI Passthrough Documentation:
- http://goo.gl/Ea9KLp
- OpenStack PCI Passthrough Documentation:
- https://wiki.openstack.org/wiki/Pci_passthrough
![Page 34: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/34.jpg)
ADD NAME (View > Master > Slide master)
Questions
![Page 35: PCI Passthrough with Red Hat OpenStack Platformschd.ws/hosted_files/openstackmountainwest2016/8f/pci-passthrough… · PCI Passthrough with Red Hat OpenStack Platform Ken Savich Sr](https://reader031.fdocuments.net/reader031/viewer/2022022513/5aea7d397f8b9ab24d8d7301/html5/thumbnails/35.jpg)
ADD NAME (View > Master > Slide master)
Divider SlideThank you