AutomatingProblemAnalysisandTriage
SashaGoldshtein@goldshtn
ProductionDebugging
Requirements
• Obtainactionableinformationaboutcrashesanderrors
• Obtainaccurateperformanceinformation
Limitations
• Can’tinstallVisualStudio
• Can’tsuspendproductionservers
• Can’trunintrusivetools
IntheDevOpsProcess…
Automaticbuild(CI)
Automaticdeployment
(CD)
Automaticmonitoring
Automaticerrortriageandanalysis
Automaticremediation
DumpFiles
DumpFiles
• Auserdump isasnapshotofarunningprocess• Akerneldump isasnapshotoftheentiresystem• Dumpfilesareusefulforpost-mortemdiagnosticsandforproductiondebugging• Anytimeyoucan’tattachandstartlivedebugging,adumpmighthelp
LimitationsofDumpFiles
• Adumpfileisastaticsnapshot• Youcan’tdebugadump,justanalyzeit• Sometimesareproisrequired(ormorethanonerepro)
• Sometimesseveraldumpsmustbecompared
TaxonomyofDumps
• Crashdumps aredumpsgeneratedwhenanapplicationcrashes• Hangdumps aredumpsgeneratedon-demandataspecificmoment• Thesearejustnames;thecontentsofthedumpfilesarethesame!
GeneratingaHangDump
• TaskManager,right-clickandchoose“CreateDumpFile”• Createsadumpin%LOCALAPPDATA%\Temp
Procdump
• Sysinternalsutilityforcreatingdumps• Examples:
Procdump -ma app.exe app.dmpProcdump -ma -h app.exe hang.dmpProcdump -ma -e app.exe crash.dmpProcdump -ma -c 90 app.exe cpu.dmpProcdump -m 1000 -n 5 -s 600 -ma app.exe
WindowsErrorReporting
• WERcancreatedumpsautomatically• HKLM\Software\Microsoft\Windows\WindowsErrorReporting\LocalDumps
• Canbeapplication-specific,notsystem-wide
DebugDiag
• Microsofttoolformonitoringanddumpgeneration• VerysuitableforASP.NET• Dumpanalysiscomponentincluded
DebuggingSymbols
• Debuggingsymbolslinkruntimememoryaddressestofunctionnames,sourcefilenamesandlinenumbers• PDBfiles• Requiredforproperdebugginganddumpanalysis
SymbolsforMicrosoftBinaries• Microsofthasapublicsymbolserver withPDBfilesforMicrosoftbinaries• Configure_NT_SYMBOL_PATHenvironmentvariable
setx _NT_SYMBOL_PATH srv*C:\symbols*http://msdl.microsoft.com/download/symbols
OpeningDumpFiles
• VisualStudiocanopendumpfiles• For.NET,CLR4.0+andVS2010+required
OpeningDumpFiles
• WinDbg isafreelightweightdebugger• Nointrinsic.NETsupport,buthasSOSextension
!analyze -v (CLR4.0+).loadby sos clr!printexception!clrstack
AutomaticDumpAnalysis
BasicAutomation• RunWinDbg automaticallyonabunchoffilesandlogitsoutput:
@echo offfor %%f in (.\*.dmp) do (echo Launching analysis of file %%f...start "Analyzing %%f" "C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe" -z %%f -c ".logopen%%f.log; !analyze -v; .logclose; qd"
)
BasicAutomation• Parsetheresultsforinterestingtokens:
for %%f in (.\*.dmp.log) do (echo In file %%f:findstr "EXCEPTION_MESSAGE MANAGED_OBJECT_NAME" %%f
)
ClrMD
• Text-basedanalysisofdebuggercommandoutputisveryfragileandlimited• ClrMD isa.NETlibraryforanalyzingdumpfiles(andrunningprocesses)• ManagedAPIforthe.NETdebuggingruntime(“SOS”)• DistributedthroughNuGet (search“ClrMD”)• Open-sourceonGitHubhttps://github.com/Microsoft/clrmd
• Alreadyactivelyusedtosimplify.NETdiagnostics• PerfView• msos https://github.com/goldshtn/msos• NetExt https://netext.codeplex.com
ClrMDBasicClasses
DataTarget
ClrRuntime ClrRuntime
ClrHeap ClrThread
ClrType ClrType ClrThread
mscordacwks.dll
• Manageddumpanalysisrequiresmscordacwks.dllmatchingtheCLRversion• ItcanbeautomaticallydownloadedfromtheMicrosoftsymbolserverinmostcases
ConnectingtoaTarget• Attachtoaprocessoropenadump:
DataTarget target = DataTarget.LoadCrashDump(@"dump.dmp");target.AppendSymbolPath("srv*C:\symbols*http://msdl.microsoft.com/download/symbols");
var runtime = target.CreateRuntime(target.ClrVersions[0].TryDownloadDac());
BasicExceptionTriageforeach (var thread in runtime.Threads){
var e = thread.CurrentException;if (e != null){Console.WriteLine("Thread {0}", thread.ManagedThreadId);Console.WriteLine("\t{0} - {1}", e.Type.Name, e.Message);
foreach (var frame in e.StackTrace)Console.WriteLine("\t" + frame.DisplayString);
}}
InspectingtheHeap• Enumerateallheapobjectsandstatistics• Findspecificobjects• InspectGCinformation(roots,finalizationqueues,etc.)
ClrHeapEnumerateObjectsGetObjectTypeEnumerateRoots
ClrTypeGetSizeEnumerateRefsOfObjectGetFieldValue
WaitInformation• Threadshavealistofblockingobjects,whichhaveownerthreads• Waitanalysisanddeadlockdetectionismadepossible
ClrThreadBlockingObjects
BlockingObjectReasonObjectHasSingleOwnerOwner/OwnersWaiters
Summary
• AutomaticdumpanalysisisherewithClrMD• Potentialforamazingtoolsandworkflowsthatenabletrueautomaticmonitoring,triage,andanalysis• IfyouwerescaredofWinDbg inthepast,wehavebettertoolsnow!
Thankyou!SashaGoldshtein
@goldshtn
Top Related