AutomatingProblemAnalysisandTriage

SashaGoldshtein@goldshtn

ProductionDebugging

Requirements

• Obtainactionableinformationaboutcrashesanderrors

• Obtainaccurateperformanceinformation

Limitations

• Can’tinstallVisualStudio

• Can’tsuspendproductionservers

• Can’trunintrusivetools

IntheDevOpsProcess…

Automaticbuild(CI)

Automaticdeployment

(CD)

Automaticmonitoring

Automaticerrortriageandanalysis

Automaticremediation

DumpFiles

DumpFiles

• Auserdump isasnapshotofarunningprocess• Akerneldump isasnapshotoftheentiresystem• Dumpfilesareusefulforpost-mortemdiagnosticsandforproductiondebugging• Anytimeyoucan’tattachandstartlivedebugging,adumpmighthelp

LimitationsofDumpFiles

• Adumpfileisastaticsnapshot• Youcan’tdebugadump,justanalyzeit• Sometimesareproisrequired(ormorethanonerepro)

• Sometimesseveraldumpsmustbecompared

TaxonomyofDumps

• Crashdumps aredumpsgeneratedwhenanapplicationcrashes• Hangdumps aredumpsgeneratedon-demandataspecificmoment• Thesearejustnames;thecontentsofthedumpfilesarethesame!

GeneratingaHangDump

• TaskManager,right-clickandchoose“CreateDumpFile”• Createsadumpin%LOCALAPPDATA%\Temp

Procdump

• Sysinternalsutilityforcreatingdumps• Examples:

Procdump -ma app.exe app.dmpProcdump -ma -h app.exe hang.dmpProcdump -ma -e app.exe crash.dmpProcdump -ma -c 90 app.exe cpu.dmpProcdump -m 1000 -n 5 -s 600 -ma app.exe

WindowsErrorReporting

• WERcancreatedumpsautomatically• HKLM\Software\Microsoft\Windows\WindowsErrorReporting\LocalDumps

• Canbeapplication-specific,notsystem-wide

DebugDiag

• Microsofttoolformonitoringanddumpgeneration• VerysuitableforASP.NET• Dumpanalysiscomponentincluded

DebuggingSymbols

• Debuggingsymbolslinkruntimememoryaddressestofunctionnames,sourcefilenamesandlinenumbers• PDBfiles• Requiredforproperdebugginganddumpanalysis

SymbolsforMicrosoftBinaries• Microsofthasapublicsymbolserver withPDBfilesforMicrosoftbinaries• Configure_NT_SYMBOL_PATHenvironmentvariable

setx _NT_SYMBOL_PATH srv*C:\symbols*http://msdl.microsoft.com/download/symbols

OpeningDumpFiles

• VisualStudiocanopendumpfiles• For.NET,CLR4.0+andVS2010+required

OpeningDumpFiles

• WinDbg isafreelightweightdebugger• Nointrinsic.NETsupport,buthasSOSextension

!analyze -v (CLR4.0+).loadby sos clr!printexception!clrstack

AutomaticDumpAnalysis

BasicAutomation• RunWinDbg automaticallyonabunchoffilesandlogitsoutput:

@echo offfor %%f in (.\*.dmp) do (echo Launching analysis of file %%f...start "Analyzing %%f" "C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe" -z %%f -c ".logopen%%f.log; !analyze -v; .logclose; qd"

)

BasicAutomation• Parsetheresultsforinterestingtokens:

for %%f in (.\*.dmp.log) do (echo In file %%f:findstr "EXCEPTION_MESSAGE MANAGED_OBJECT_NAME" %%f

)

ClrMD

• Text-basedanalysisofdebuggercommandoutputisveryfragileandlimited• ClrMD isa.NETlibraryforanalyzingdumpfiles(andrunningprocesses)• ManagedAPIforthe.NETdebuggingruntime(“SOS”)• DistributedthroughNuGet (search“ClrMD”)• Open-sourceonGitHubhttps://github.com/Microsoft/clrmd

• Alreadyactivelyusedtosimplify.NETdiagnostics• PerfView• msos https://github.com/goldshtn/msos• NetExt https://netext.codeplex.com

ClrMDBasicClasses

DataTarget

ClrRuntime ClrRuntime

ClrHeap ClrThread

ClrType ClrType ClrThread

mscordacwks.dll

• Manageddumpanalysisrequiresmscordacwks.dllmatchingtheCLRversion• ItcanbeautomaticallydownloadedfromtheMicrosoftsymbolserverinmostcases

ConnectingtoaTarget• Attachtoaprocessoropenadump:

DataTarget target = DataTarget.LoadCrashDump(@"dump.dmp");target.AppendSymbolPath("srv*C:\symbols*http://msdl.microsoft.com/download/symbols");

var runtime = target.CreateRuntime(target.ClrVersions[0].TryDownloadDac());

BasicExceptionTriageforeach (var thread in runtime.Threads){

var e = thread.CurrentException;if (e != null){Console.WriteLine("Thread {0}", thread.ManagedThreadId);Console.WriteLine("\t{0} - {1}", e.Type.Name, e.Message);

foreach (var frame in e.StackTrace)Console.WriteLine("\t" + frame.DisplayString);

}}

InspectingtheHeap• Enumerateallheapobjectsandstatistics• Findspecificobjects• InspectGCinformation(roots,finalizationqueues,etc.)

ClrHeapEnumerateObjectsGetObjectTypeEnumerateRoots

ClrTypeGetSizeEnumerateRefsOfObjectGetFieldValue

WaitInformation• Threadshavealistofblockingobjects,whichhaveownerthreads• Waitanalysisanddeadlockdetectionismadepossible

ClrThreadBlockingObjects

BlockingObjectReasonObjectHasSingleOwnerOwner/OwnersWaiters

Summary

• AutomaticdumpanalysisisherewithClrMD• Potentialforamazingtoolsandworkflowsthatenabletrueautomaticmonitoring,triage,andanalysis• IfyouwerescaredofWinDbg inthepast,wehavebettertoolsnow!

Thankyou!SashaGoldshtein

@goldshtn