QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix
-
Upload
yan-vugenfirer -
Category
Technology
-
view
1.662 -
download
0
description
Transcript of QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix
![Page 1: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/1.jpg)
QEMU development and testing automation using MS HCK
Anton Nayshtut, [email protected] Yan Vugenfirer, [email protected]
Daynix Computing LTD
![Page 2: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/2.jpg)
Daynix Computing LTD
Agenda
What is WHQL/HCK?
Introducing VirtHCK
Using VirtHCK
Call to action
�2
![Page 3: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/3.jpg)
Daynix Computing LTD
What is HCK/WHQL?The Windows certification program (previously known as the Windows Logo Program) for devices and system
A comprehensive device\system test toolkit by Microsoft
Devices which pass HCK are certified and their binaries are digitally signed
�3
![Page 4: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/4.jpg)
Daynix Computing LTD
HCK test kit
Introduced with Windows 8 (previous kit know as WLK)
Predefined test suites for each device type
Ability to add your own tests for additional testing outside of the certification process
Reproducible scenarios
�4
![Page 5: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/5.jpg)
Daynix Computing LTD
Benefits for QEMU and Linux community
QEMU device testing (not only virtio)
Host subsystems testing, for example networking
Extensive coverage for defined device categories
Regression testing
Stress testing
�5
![Page 6: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/6.jpg)
Daynix Computing LTD
HCK PCI compliance testDevice under test compliance to PCI spec
Applicable for:
New QEMU chipsets
PCIe devices
Next generation of virtio devices
Existing PCI devices
�6
![Page 7: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/7.jpg)
Daynix Computing LTD
HCK NDIS (network) tester (1)
Comprehensive networking test suite
Send/receive,
Stress tests (MPE)
Offload, VLAN, priority, packet filtering
etc
Not once used to test the robustness of the host networking stack
�7
![Page 8: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/8.jpg)
Daynix Computing LTD
HCK NDIS (network) tester (2)Applicable for
QEMU network devices
New devices
New features
Host network components
Bridge
Open vSwitch
TAP
Kernel�8
![Page 9: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/9.jpg)
Daynix Computing LTD
HCK Windows OS tests
Device path exerciser
Identifies drivers that do not correctly handle the different IO calls
Driver verifier
Captures deadlocks, memory leaks, memory corruption
Verified correctness of kernel API usage
�9
![Page 10: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/10.jpg)
Daynix Computing LTD
HCK power management tests
The test cycling through different sleep states
There are variations of IO and PNP (disable/enable) during, before and after power transitions
Applicable for:
QEMU Devices power management support
SeaBIOS
�10
![Page 11: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/11.jpg)
Daynix Computing LTD
Our past experienceTransfer hangs with vhost.
Offload/VLAN related corner cases with network devices (virtio-net, vmxnet3, vmxnet).
vmxnet3 and pvscsi in QEMU
Open vSwitch out of order issues
RSC (Windows GRO)
PCI compliance
�11
![Page 12: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/12.jpg)
Daynix Computing LTD
HCK - great, but frightening
Cumbersome setup requirements
Complex network configuration
Hard to replicate setups
Setup conflicts
No power management tests automation support
�12
![Page 13: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/13.jpg)
Daynix Computing LTD
VirtHCK
�13
![Page 14: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/14.jpg)
Daynix Computing LTD
Introducing VirtHCK
Framework for automating HCK test execution
Based on easily configurable host and guest scripts
Creates isolated virtual network environments
Allows running different HCK scenarios in automated manner
OSS, distributed under BSD license
�14
![Page 15: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/15.jpg)
Daynix Computing LTD
Repository and WiKi
Get VirtHCK from https://github.com/daynix/bricklets.git
VirtHCK wiki with steps for deployment and usage: https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation
�15
![Page 16: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/16.jpg)
Daynix Computing LTD
HCK issues solved by VirtHCK
Setup replication
Multiple setups sharing the same host
Setup isolation solves the following issues:
HCK controller controls all HCK clients in the network
Inability to use VM templates without manual VM reconfiguration
Automatic network topology setup during network device tests
�16
![Page 17: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/17.jpg)
Daynix Computing LTD
Setup types supported
Network (virtio-net, vmxnet3, vmxnet)
Storage (virtio-block, virtio-scsi, pvscsi)
Simple PCI (virtio-serial, virtio-balloon)
Easily extendable by small changes in configuration files
�17
![Page 18: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/18.jpg)
Daynix Computing LTD
Typical HCK deployment
WHQL control network
Control NIC
Test Client
HCK Controller
Control NIC - e1000
Control NIC
Test Client
Control NIC
Test Client
�18
![Page 19: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/19.jpg)
Daynix Computing LTD
VirtHCK setup
Disconnected from outside world network during test run
Isolated NICs under test network
Isolated WHQL control network
A different number of test clients can run according to test type
�19
![Page 20: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/20.jpg)
Daynix Computing LTD
VirtHCK setup for storage and other devices
WHQL control bridge
“World” bridge Physical NIC
Control NIC - e1000
Test VM 1
Studio VM
Outside world NIC
Control NIC - e1000
�20
![Page 21: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/21.jpg)
Daynix Computing LTD
VirtHCK setup for storage and other devices - running
WHQL control bridge
“World” bridge Physical NIC
Control NIC - e1000
Test VM 1
Studio VM
Outside world NIC
Control NIC - e1000
X�21
![Page 22: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/22.jpg)
Daynix Computing LTD
VirtHCK setup for NICs
Test bridge
WHQL control bridge
“World” bridge Physical NIC
Control NIC - e1000
Test VM 1
NIC under test
Control NIC - e1000
Test VM 2
NIC under test
Studio VM
Outside world NIC
Control NIC - e1000
�22
![Page 23: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/23.jpg)
Daynix Computing LTD
VirtHCK setup for NICs - running
Test bridge
WHQL control bridge
“World” bridge Physical NIC
Control NIC - e1000
Test VM 1
NIC under test
Control NIC - e1000
Test VM 2
NIC under test
Studio VM
Outside world NIC
Control NIC - e1000
X�23
![Page 24: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/24.jpg)
Daynix Computing LTD
VirtHCK - quick startClone VirtHCK from github
Create VM images in /images directory (studio and test client)
Update STUDIO_IMAGE/CLIENT1_IMAGE/CLIENT2_IMAGE variables in hck_setup.cfg
Run VirtHCK - “sudo ./VirtHCK/hck.sh”
Install Microsoft HCK Studio and Clients software according to Microsoft documentation Windows HCK Getting Started and following Checklist for a new client VM guidelines.
�24
![Page 25: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/25.jpg)
Daynix Computing LTD
Directory treeVirtHCK/ - Directory of all VirtHCK scripts, supposed to run in Linux environment
VirtHCK/guest_tools/ - Directory for scripts required for Windows guests
Images directory where VM's images should be located
�25
![Page 26: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/26.jpg)
Daynix Computing LTD
VirtHCK walkthrough - host
Configuration: network, VMs etc
Host scripts
Images
�26
![Page 27: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/27.jpg)
Daynix Computing LTD
hck_setup.cfg
VirtHCK setup configuration file
Environment configuration
VM start up parameters
Configured accordingly to required HCK setup type
Allows multiple setups on one host
�27
![Page 28: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/28.jpg)
Daynix Computing LTD
Running multiple setups
Use UNIQ_ID to configure multiple setups
UNIQ_ID will be used as a base for creation of:
VNC or Spice ports
Bridges names
Network interface names, MAC addresses and etc.
�28
![Page 29: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/29.jpg)
Daynix Computing LTD
Test device types
Network
Storage (also boot)
Balloon
Serial
More can be added
�29
![Page 30: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/30.jpg)
Daynix Computing LTD
Remote access
VNC or Spice
Spice is needed for power management tests
�30
![Page 31: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/31.jpg)
Daynix Computing LTD
Network infrastructure
Linux bridge or Open vSwitch can be used
�31
![Page 32: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/32.jpg)
Daynix Computing LTD
Images
Configure
Studio VM image
Test VMs images
�32
![Page 33: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/33.jpg)
Daynix Computing LTD
VM resources
Number of vCPUs
Memory
NICs for control channels
�33
![Page 34: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/34.jpg)
Daynix Computing LTD
Extra parametersQEMU binary
VHOST, snapshots, cache mode for fast install
ISOs for guest installation
BIOS
�34
![Page 35: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/35.jpg)
Daynix Computing LTD
Configuration summary
�35
![Page 36: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/36.jpg)
Daynix Computing LTD
Host scripts - hck.sh
Main script to run HCK enviroment
Dumps configuration
Creates network configuration for HCK setup
Runs VMs in loop (crucial during PM tests)
Cleans up all the configurations on exit
�36
![Page 37: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/37.jpg)
Daynix Computing LTD
VM execution scripts
Executed by hck.sh
run_hck_studio.sh - starts up Studio VM
run_hck_client.sh -starts up Client VM
Configuration according hck_setup.cfg
�37
![Page 38: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/38.jpg)
Daynix Computing LTD
Network scripts
Passed as parameter to QEMU to add network interface to appropriate bridge
hck_ctrl_bridge_ifup.sh
hck_test_bridge_ifup.sh
hck_world_bridge_ifup.sh
�38
![Page 39: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/39.jpg)
Daynix Computing LTD
Guests configuration
�39
![Page 40: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/40.jpg)
Daynix Computing LTD
Studio VM
Setup HCK according to Microsoft’s documentation
Follow the checklist - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation#wiki-Checklist_for_a_new_studio_VM:
Do not run HCK studio directly
Use run_hck_studio.bat or run_hck_studio_8.1.bat located in guest_tools
The above scripts will disconnect the HCK Studio VM from outside network
�40
![Page 41: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/41.jpg)
Daynix Computing LTD
Test client VMs
Follow the checklist for test client - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation#wiki-Checklist_for_a_new_client_VM:
Install HCK client according to Microsoft’s documentation from \\hck-studio\HCKInstall\Client\setup.exe
�41
![Page 42: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/42.jpg)
Daynix Computing LTD�42
![Page 43: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/43.jpg)
Daynix Computing LTD
What’s nextHCK automation on the guest
SVVP configuration
Xen support (other hypervisors?)
Adding more device types
Using external configuration
Using libvirt
More guest scripts to automate configuration of test clients
�43
![Page 44: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/44.jpg)
Daynix Computing LTD
HCK automation on the guest
Goal:
Run full test cycle for specific device type without user interaction.
�44
![Page 45: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/45.jpg)
Daynix Computing LTD
Adding more device types
Goal
Test all the devices in QEMU and not only virtio (PCI bridges, USB controllers and etc)
�45
![Page 46: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/46.jpg)
Daynix Computing LTD
More plans
Using external configuration
Multiple setups on one host without replicating VirtHCK and images
Use libvirt
Arbitrary number of test clients on one setup
Xen support
�46
![Page 47: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/47.jpg)
Daynix Computing LTD
Call to action
Test QEMU devices with VirtHCK (not only virtio devices)
Contribute
�47
![Page 48: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/48.jpg)
Q&A�48
![Page 49: QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix](https://reader033.fdocuments.net/reader033/viewer/2022050817/55615627d8b42aa20d8b533b/html5/thumbnails/49.jpg)
Daynix Computing LTD
LinksWiKi - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation
Repository - https://github.com/daynix/bricklets.git
HCK kit download - http://msdn.microsoft.com/en-us/windows/hardware/hh833788.aspx
Windows HCK Getting Started - http://msdn.microsoft.com/en-us/library/windows/hardware/jj123537.aspx
Daynix - www.daynix.com
�49