Troubleshooting Virtual Machine Manager 2012 SP1John MessecProgram Manager

Kurt SchererProgram Manager

Amit VirmaniSoftware Design Engineer

Madhu JujareSoftware Design Engineer

Session Objectives and TakeawaysOverview of VMM Distributed ArchitectureSetting Up Tracing and LoggingGeneral Troubleshooting TipsTroubleshooting:• Host Communication Issues• Storage• Software-Defined Networks

Where To Go For More Information and Help

Tools and Tips

General Troubleshooting and Common Issues• Ensure all updates are installed• Application and System Event logs on involved

systems• Ensure servers can communicate• Installation logs• Incorrect RunAs Accounts

• E.g. – Provisioning storage, RunAs account must be local administrator on a Windows-based file server

Microsoft Baseline Configuration Analyzer 2.0

ETW Tracing – Capture a TraceTracing With Just The Tools On Box!Logman.exe

• Discover ETW Providers (things for which you can collect traces)

• Create a data collector for VMM

• Start and stop the collector

C:\Logman.exe query providers

C:\Logman.exe create trace MyVMMTrace –o trace.etl -p Microsoft-VirtualMachineManager-Debug

C:\Logman.exe start MyVMMTraceC:\Logman.exe stop MyVMMTrace

ETW Tracing – Reading The Logs• Trace logs are in a binary format, not human-readable• Another on-the-box tool

• Creates human-readable trace file from the .etl file

C:\Netsh trace convert trace.etl

Demo: Grabbing traces with LogMan and NetShJohn Messec

Multi-Service, Multi-Machine Tracing

MPS Reporting Tool• GUI Interface,

allows trace collection from many machines

• Creates zipped archive of .etl traces to send to support

Analyzing VMM TracesConvert from binary .etl to human-readable using Netsh or TraceViewer.exe

Log is chronological top-to-bottomWork upwards from the bottom to identify issuesSearch for error code of issue if availableTraces are asynchronous; other traces may be mixed in with the error traces

Demo: Multi-Machine Tracing and MPS Reporting ToolJohn Messec

General Troubleshooting Tips• Try repro’ing in Powershell• Does the issue repro independent of VMM (e.g.

directly in Hyper-V)• Dig In To Jobs View to pinpoint the exact step• Look at trace logs (bottom up)• Configure circular 1GB logs for hard-to-reproduce

issues

Troubleshooting VMM Agent Communications

VMM Deployment OverviewVMM Server

VMM Client

VMM Library

Hyper-V Host

Host Agent

VM

Library Agent

Library Share

Guest Agent

Powershell

WCF

SQL Server

AD

O.N

et

WinRM

WinRMVMM Engine

BITS

Common Issues• WinRM

• WinRM enabled on all systems?• Firewall/proxy letting communication through?• Ports configured correctly?• Permissions?

• Adding Hosts• Ensure credentials used are in local

Administrators group• Ensure Admin$ share is accessible from VMM

Server

• Domain Group Policy Conflict• CredSSP – disabled by Group Policy

Demo: Troubleshooting Agent CommunicationsJohn Messec

Troubleshooting Storage

Evolving Storage Management

VM migration automation using NPIV automation or VDS HW providers

VMM 2007 VMM 2008VMM 2008 R2 SP1

SMI-S based discovery, management and automation of block storage

VMM 2012

Discovery, management and automation of block and file storage using Windows Server 2012

VMM 2012 SP1

Private Cloud Storage Management

Allocation and Assignment

Standards Based Management

VM, Host, and Cluster Storage Management

Extensive SAN and NAS Support

Scalable Provisioning

Storage Monitoring

Array Onboarding

Storage Classification

Rapid ProvisioningBlock and File

Storage Management End to End

Mapping

Insight Flexibility Automation

SAN Migration

VMM Server

Discovery of portals and targetsLog on & log off

  

FC Array / ISCSI Target

LUN LUN LUN LUNSAN admin tool 

Host

HostSMAPI/

VDSHBA WMI API

Cluster WMI

MS iSCSI initiator

HostHost

Host

FC/SAS HBA DiscoveryPort to LUN mappings

• Cluster storage discovery• Add/remove cluster disk

resource• Convert to CSV

SC 2012 SP1 VMM – Storage Architecture

SMI-S/SMP Provider  

• Enumerate/Rescan• Create

Partition/Format• Mount/UnMount• Volume to Disk

mapping• Disk to LUN mapping

Storage Management Service

Storage Management ServiceDiscovery of Array, Pool and LUNS

LUN create, snapshot, cloneMask and Unmask SMAPI

Windows Server 2012 – Storage Management API

Management

Applications

Storage Managemen

t APIs

Windows Storage Management API (Windows PowerShell and WMI)

Pass-through API for extensibility (WMI)

Storage Array

Storage Management Provider Interface

Non-Microsoft management application

System Center 2012 SP1 - VMM

File Server Manager(File and Storage Services role)

Vendor-provided SMI-S provider (CIMXML or WMI)

Storage Array

Vendor-provided transport and/or protocol

SMP: Windows Standards Based Storage

Management Service

SMP: Vendor-provided WMI-based provider

SMP: Storage Spaces (no pass-through API support)

SM I-S SAN/NAS Proxy

Provider

SM I-S SANw/ Embedded

Provider

SM I-S NASProxy Provider

WindowsScale-out File Server

SM API

WMI

SM P:SM I-S

StorageService

SM PPartner Provider

SM P:Spaces

WSMAN/WinRM

WSMan/WinRM

Hyper-VStandalone Hyper-V

Cluster

NAS Device

SAN/NAS Device

CIMXML/WMI

CIMXMLCIMXML

SAN Device

Proprietary

Proprietary Proprietary

VMM Server

WSMan/WinRM

SM I-S SAN Proxy

Provider

SANDevice

Proprietary

FC/iSCSI

(some providersrequire this

CIMXMLWMI

VMVM

Library Server

WSMan/WinRM

Troubleshooting Storage: Trace collection

• VMM Server• VMM logs• Traces high level API calls issued from VMM. This could be SMAPI, Host SMAPI, Cluster WMI,

FC HBA API, ISCSI API etc. for each work flow.• SMAPI / Storage WMI logs• Traces provider specific api’s issued for each SMAPI request. For SMI provider you will see

SMI-S calls. • CIM-XML logs• Traces each CIM-XML request/response from storage service to SMI provider and vice versa.

http://blogs.technet.com/b/hectorl/archive/2011/12/08/scvmm-2012-collecting-storage-related-traces.aspx

• Host/Cluster• VMM Agent logs• Host SMAPI logs• VDS logs• Cluster WMI and Cluster logs• MS ISCSI Initiator logs• HBA API logs

Demo: Troubleshooting StorageAmit Virmani

TroubleshootingSoftware-Defined Networks

Logical Networks• Make network virtualization possible between hosts• Bridge between physical to virtual• Definition of network sites

• VLANs, IP subnets, mappings – assumption of connectivity• Allocate host groups

• Allowing VMs to use network virtualization• IP Pools

• Required for network virtualization• Provider Address (PA)

Logical Switches• Consists of port profiles• Span multiple hosts• Capability definitions for host NICs• Cannot create without Uplink and vNIC Port Profiles

• Can be created during switch creation

Native Port Profile• Uplink Port Profiles:

• Host NIC configurations• Configure Teaming and Load Balancing• Assign logical networks• Hosts must support networks to assign switch to host

• vNIC Port Profiles:• vNIC configurations• Combination of security settings, bandwidth limits, and performance settings• Bandwidth settings not used with SR-IOV• Defaults available

• Group profiles with classifications

VM Network• Not a fabric component• No isolation – Pre-SP1

• VMs directly connect to logical network

• Isolation• VMs have their own address space• No external connection by default

Connectivity of VM Networks• Requires gateway device• ‘No connectivity’ during creation unless gateway

devices exist• IP pools

• Subnet was defined during creation• Ranges can be divided into pools• One address is reserved for virtual switch

ArchitectureVM

VM Network

Logical network

Logical Switch(Hyper-V Host)

VM Network

Logical Switch(Hyper-V Host)

VM VM VM

Physical network

vNIC Port Profile

Network Virtualization / Gateway

Network Site Definition

Uplink Port Profile

Cloud Deployment• VM must be connected to a VM network• VM network must be connected to Logical network• Logical network must be associated with the cloud

Demo: Software-Defined NetworksKurt Scherer

SummaryOverview of VMM Distributed ArchitectureSetting Up Tracing and LoggingGeneral Troubleshooting TipsTroubleshooting:• Host Communication Issues• Storage• Software-Defined Networks

Evaluation

Complete your session evaluations today and enter to win prizes daily. Provide your feedback at a CommNet kiosk or log on at www.2013mms.com.Upon submission you will receive instant notification if you have won a prize. Prize pickup is at the Information Desk located in Attendee Services in the Mandalay Bay Foyer. Entry details can be found on the MMS website.

We want to hear from you!

Resources

http://channel9.msdn.com/Events

Access MMS Online to view session recordings after the event.

Additional Resources• How To Report Issues

• QFE Requests Prioritized by Product Team

• VMM Wiki• General Troubleshooting

• VMM Error Codes• http://social.technet.microsoft.com/wiki/contents/articles/4906.system-center-2012-virtual-

machine-manager-vmm-error-codes.aspx

• VMM 2012 SP1 KB Articles• http://

social.technet.microsoft.com/wiki/contents/articles/15364.list-of-public-microsoft-support-knowledge-base-articles-for-system-center-2012-virtual-machine-manager-service-pack-1.aspx

Related Sessions

SD-B201 Implementing Common Scenarios in VMM: Services and Service

Templates

IM-B203 Building UI Add–Ins For System Center 2012 SP1 Virtual Machine

Manager

IM-B308 System Center 2012 SP1 Virtual Machine Manager: Managing

Large Datacenters

IM-B310 Troubleshooting Common System Center 2012 SP1 Virtual Machine

Manager Issues

WS-B312 How to Design and Configure Networking in VMM and HyperV

(Part 1 of 2)

WS-B313 How to Design and Configure Networking in VMM and HyperV

(Part 2 of 2)

WS-B314 Implementing Common Storage Scenarios in Windows Server

2012 with SCVMM

WS-B327 Software Defined Networking with WS2012, SC2012 SP1 and

Partner Solutions

Appendix

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.