Dealing With Leaks

29
© 2003 Mark Bartosik, www.leakbrowser.com Dealing with Leaks

Transcript of Dealing With Leaks

Page 1: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Dealing with Leaks

Page 2: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

What is a resource leak ?

• Repeated allocation of a resource without timely deallocation

Probable resources:• memory

virtual memory, non-paged pool memory

• address space• OS objects / handles

files, threads, events, semaphores, sockets, windows, timers, etc.

• disk space• application level / non OS recourses e.g. database records

Page 3: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Unreferenced resources

int global = 20;

void foo(){

int b = 22;HANDLE h = CreateEvent(....);

int * int_ptr = int(1000); int * array_ptr = new int[1024];}

Page 4: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Unreferenced heap memoryProcess / User memory

static storage (fixed size)

0x38C

22

21

stackgrows ondem and

stack (one per thread)

1000

? ar? n mod

1024 e lements(4096 bytes)

heap

0xDE ADBEAF

0x17E D54230xD EA DBEAF

0x1 7E D5423

heapgrows ondem and

20

Kerne l / O S memory

0x80101234

0x80101DC0 event ob ject0x801013580x38C

0x388

0x390

int global = 20;

void foo(){ int b = 22; HANDLE h = CreateEvent(...); int * int_ptr = int(1000); int * array_ptr = new int[1024];

execution is here}

int main(){ int a = 21; foo();

return EXIT_SUCCESS;}

Page 5: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Unreferenced heap memoryProcess / User memory

static storage (fixed size)

21

stackgrows ondem and

stack (one per thread)

1000

? ar? n mod

1024 e lements(4096 bytes)

heap

0xD EA DBEAF

0x1 7E D5423

heapgrows ondem and

20

Kerne l / O S memory

0x80101234

0x80101DC0 event ob ject0x801013580x38C

0x388

0x390

int global = 20;

void foo(){ int b = 22; HANDLE h = CreateEvent(...); int * int_ptr = int(1000); int * array_ptr = new int[1024];

}

int main(){ int a = 21; foo(); execution is here return EXIT_SUCCESS;}

Page 6: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Unbounded resource consumptionstd::map<unsigned, transaction_t> transactions;

void do_transaction(P1_t p1, P2_t p2)

{

transactions[id] = transaction_t(p1,p2);

}

Page 7: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

C++, avoiding the problem

• encapsulate ownership within objects or smart pointersbstr_t CComPtr shared_ptr auto_ptr scoped_ptr auto_handle std::string, avoid BSTR, char*

• only use raw pointers to implement tightly encapsulated objects,e.g. when implementing shared_ptr

• never use raw pointers to transfer ownership• never use raw pointers with business logic

exception: indication optional objects - use of NULL• Use a garbage collector• Problem:

No language enforcement -- trusts the programmer

Page 8: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

VB, avoiding the problem

• Stay within Visual Basic 5/6 environment (avoid C / OS APIs).

• Release objects (with events) in “unload” methods

set some_object = nothing

• Forms reference objects (with events) and the objects can reference the form. This is a cyclic reference.

• VB7 is implemented on .NET only and garbage collected.

Page 9: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Java, .NET, avoiding the problem

• Stay within JVM or .NET VM (avoid C / OS APIs).You have no destructors, and finalizers are restrictive.

• Cyclic references -- no problem they are garbage collected.

• Don’t get lax, watch out for unbounded resource consumption.

Page 10: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Garbage collection - the solution

Process / U se r memory

static sto rage (fixed size)

stackgrows ondem and

stack (one per thread)

stuff

managedheap

0x17E D 5423

0x17E D5423

heapgrows ondem and

17ED78AC

stuff

0x17E D78A C

stuff

garbage

garbage

• pause the entire program

• find every object that can be reached from the static storage or thread stacks

• mark those objects with an “age”

• delete all older objects, in any order you like

Page 11: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Garbage collection - the solution

Process / U se r memory

static sto rage (fixed size)

stackgrows ondem and

stack (one per thread)

stuff

managedheap

0x17E D 5423

0x17E D5423

17ED78AC

stuff

0x17E D78A C

stuff

• pause the entire program

• find every object that can be reached from the static storage or thread stacks

• mark those objects with an “age”

• delete all older objects, in any order you like

Page 12: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Garbage collection - the penalties• Pause the entire program!

When?For how long?

• Find EVERY reachable object?Page in every object from disk!Must run in physical memory to be efficient.Working set will be same as used memory!

• Delete all the garbage, in what order?Any order!So how do I test deterministically?So I cannot use finalizers?

• These may or may not be acceptable compromises• Remember it only solves half the problem!

Page 13: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Workarounds• Watch dogs

Who watches the watch dog?What can the watch dog do? Bark!

• COM+Memory and lifetime limits can be configured for COM servers

• Heap limitsMaximum heap size can be configured

• WRITE GOOD CODE!• FIND THE LEAKS!

Page 14: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Available tools

• Bounds checker• Purify• Heap Agent• Great Circle• Lint• Parasoft• C++ debug heap

Page 15: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Typical restrictions

• Crashes itself on large scale programs• Crashes large scale programs• Only reports unreferenced memory not unbounded allocations• Only reports malloc, free etc• Does not report OS API leaks like SysAllocString• Only supports C++• Requires recompilation• Reports many false positives• Requires many hours to run• Grinds target program to a halt

Page 16: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

perfmon

Page 17: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Identifying the problem

• Monitor resourcesperfmon “process private bytes”

“process handle count”“process thread count”

taskmgrVM Size, Handle count, Thread count, GDI Objects

• Exercise the program• Coverage test -- does it leak?• Use case tests -- when does it leak (most)?

Page 18: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Quantifying the problem

reso

urce

usa

ge

time and/or number of use cases

custom eracceptab lerestart tim e

sim ilar to MT BF

m ax acceptab lereso urce usage

• Must stay out of the red

• Must consistently reach the green, without crashing

• The restart period might be... 1 hour 1 working day 1 working week 1 year 1000 transactions never!

• Max resources might be... 100 KB 2 GB

Page 19: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Needle in a haystack• 3 processes• 250 component DLLs• 3500 source files• 1,500,000 lines of source code• 100 KB of leaks per transaction• exercising 15% of the code

great only 225,000 lines of source to check!• 10,000 allocations per transaction• caused by probably no more than half a dozen lines of code

I have hidden a few leaks in the 1,500,000 lines of D3000 code.We are going to find them NOW!

Page 20: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Ignore one time initialization

reso

urce

usa

ge

time and/or number of use cases

custom eracceptab le

restart tim esim ilar to M T BF

m ax acceptab leresource usage

without one-timein itia lization leaks

problem isthe slope

with one-timein itia lization leaks

• Ignore the first few transactions

• They might cause one-time initialization leaks, but who cares?

Page 21: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Cycle and observe everything

• Choose a small prime number

• Perform this number of transactions or cycles

• Track all allocations and deallocations

reso

urce

s

cycles / transactions

cycle

leaks percycle

cycles toignore

cycles tomonitor (7) cycles to

continue

tota

lle

aks

Page 22: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Cycle again observing deallocationsstrtype a = b;

strtype & strtype::operator=(const strtype & rhs){ if (m_len < rhs.m_len) { char * temp = new char[rhs.m_len]; delete m_p; m_p = temp; } m_len = rhs.m_len; memmove(m_p, rhs.m_p, m_len); return *this;}

• Allocations made in the last cycle, may not be deleted until the next

• Perform additional cycles, only tracking deallocations

Page 23: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Leak Browser learns

• Now generate a report of the leak• Leak Browser has just “learnt”, what you are interesting in• Next time Leak Browser will examine these few areas of code in

greater detail (requiring more CPU)

• Clear the leaks• Repeat 7 monitoring cycles• Repeat some additional cycles• Regenerate the report

Page 24: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Not everything is a leak

• Background tasks may provide “background noise”for example, incoming news or price updates

• Caches may delay some deallocations• Queues and logs may include records of the last 1000

transactions

Page 25: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Capabilities

• Resource profiling - finding the bloat• Fast over 10,000 allocations per second can be tracked• Tracks about 300 key APIs, not just malloc & free• Finds both unreferenced leaks and unbounded leaks• Does not require code recompilation• Is not intrusive with large applications• Remote installation and debugging• It works!

Page 26: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

How does it work?• Traps key APIs• API is redirected

into Leak Browser stub

• Stack trace is recorded with allocation details

• In-memory database is maintained

• If no PDB further instrumentation can be done

Core OS

C/C++ runtime

C/C++ runtime

W in32

Your code

Page 27: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Challenges• Must be thread safe but...

– Cannot hold a thread lock and call the OS– Cannot allocate memory– Restrictions on using thread local storage

(it may not be allocated yet, or it may have been deallocated)

• Has only micro-seconds to trace the stack– Standard Microsoft algorithm takes seconds– Cannot afford to call the OS

• Must cope with optimized code– for stack tracing, and instrumentation– must modify hundreds or thousands of functions without side effects

Page 28: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Limitations

• Need to be able to cycle the application• No true garbage detection (yet)• .NET interop only (so far)• No JVM support - none planned• Slow display -- too much HTML• Occasional crashes, but more reliable than the rest

Page 29: Dealing With Leaks

© 2003 Mark Bartosik, www.leakbrowser.com

Summary

• Use garbage collection where appropriate• Use best practice for the programming language of choice• Monitor resource usage during development• Are you within your customer acceptable restart period?• If not

– Leak Browserwww.leakbrowser.com