Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent,...

70
Satisfy Your Technical Curiosity Satisfy Your Technical Satisfy Your Technical Curiosity Curiosity 27, 28 & 29 March 2007 27, 28 & 29 March 2007 International Convention International Convention Center (ICC) Center (ICC) Ghent, Belgium Ghent, Belgium

Transcript of Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent,...

Page 1: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Satisfy Your Technical Curiosity

Satisfy Your Technical CuriositySatisfy Your Technical Curiosity

27, 28 & 29 March 200727, 28 & 29 March 2007International Convention Center (ICC)International Convention Center (ICC)

Ghent, BelgiumGhent, Belgium

Page 2: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Satisfy Your Technical Curiosity

Essentials Abilities of the ArchitectEssentials Abilities of the ArchitectSecuritySecurity

Ron JacobsRon JacobsArchitect EvangelistArchitect Evangelist

http://www.ronjacobs.comhttp://www.ronjacobs.com

Page 3: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

What does an architect need to know about security?

How can I systematically manage security at the architecture and design stage?

What security basics do I need to know?

Page 4: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

What does an architect need to know about security?

How can I systematically manage security at the architecture and design stage?

What security basics do I need to know?

Page 5: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Security

Precautions taken to keep somebody or something safe from crime, attack, or danger

Protection against attack from without or subversion from within

Page 6: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Be able to define security requirements Be able to define security requirements

Understand security risks and countermeasuresUnderstand security risks and countermeasures

Be able to complete a threat modelBe able to complete a threat model

Be familiar with industry standards related to securityBe familiar with industry standards related to security

Page 7: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

ThreatThreat

CountermeasureCountermeasure

Masada Israel, 73 AD Masada Israel, 73 AD Roman General Lucius Flavius SilvaRoman General Lucius Flavius SilvaDefeats a band of ZealotsDefeats a band of Zealots

Page 8: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

What does an architect need to know about security?

How can I systematically manage security at the architecture and design stage?

What security basics do I need to know?

Page 9: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

AssetsAssets are the things an are the things an attacker wants to take from attacker wants to take from youyou

ThreatsThreats are the ways in which are the ways in which the attacker will try to get at the attacker will try to get at your assetsyour assets

Mitigations Mitigations are the ways you are the ways you block the attacker from getting block the attacker from getting the assetsthe assets

Vulnerabilities Vulnerabilities are unmitigated are unmitigated threatsthreats

Threat Models Threat Models are an assessment of the are an assessment of the Assets, Threats, Mitigations and Assets, Threats, Mitigations and Vulnerabilities of the system you are Vulnerabilities of the system you are building or have builtbuilding or have built

Page 10: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

AssetsAssets are more than money… are more than money…

Reputation & Customer ConfidenceReputation & Customer Confidence

Confidential DataConfidential Data

Processor, Storage, Processor, Storage, BandwidthBandwidth

AvailabilityAvailability

PerformancePerformance

Page 11: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Security User Stories

• Describes something the bad guy wants to do (a threat)

• Short and to the point

• Written by the user in non-technical language

As an attackerI want to <attack>So that <crime>

By <method>

Page 12: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Security User Stories

• As an attacker

• I want to obtain credentials

• So that I can plunder bank accounts

• By tricking users into logging into my bogus site with a Phishing mail

Page 13: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Satisfy Your Technical Curiosity

Burglary exposes Burglary exposes millions of veterans millions of veterans to identity theftto identity theftThose affected by loss of Those affected by loss of Social Security numbers, Social Security numbers, other data should be on other data should be on guard, government warns.guard, government warns.By Christopher Lee, Steve VogelBy Christopher Lee, Steve VogelTHE WASHINGTON POSTTHE WASHINGTON POSTTuesday, May 23, 2006Tuesday, May 23, 2006

Burglary exposes Burglary exposes millions of veterans millions of veterans to identity theftto identity theftThose affected by loss of Those affected by loss of Social Security numbers, Social Security numbers, other data should be on other data should be on guard, government warns.guard, government warns.By Christopher Lee, Steve VogelBy Christopher Lee, Steve VogelTHE WASHINGTON POSTTHE WASHINGTON POSTTuesday, May 23, 2006Tuesday, May 23, 2006

“…“…According to a police report, someone pried According to a police report, someone pried open a window to the employee's home open a window to the employee's home between 10:30 a.m. and 4:45 p.m. The burglar between 10:30 a.m. and 4:45 p.m. The burglar took a laptop, external drive and some coins…took a laptop, external drive and some coins…A career data analyst, who was not A career data analyst, who was not authorized to take the information homeauthorized to take the information home, , has been put on administrative leavehas been put on administrative leave pending the outcome of investigations by the pending the outcome of investigations by the FBI, local police and the VA inspector general, FBI, local police and the VA inspector general, Nicholson said. He would not identify the Nicholson said. He would not identify the employee…“employee…“

“…“…According to a police report, someone pried According to a police report, someone pried open a window to the employee's home open a window to the employee's home between 10:30 a.m. and 4:45 p.m. The burglar between 10:30 a.m. and 4:45 p.m. The burglar took a laptop, external drive and some coins…took a laptop, external drive and some coins…A career data analyst, who was not A career data analyst, who was not authorized to take the information homeauthorized to take the information home, , has been put on administrative leavehas been put on administrative leave pending the outcome of investigations by the pending the outcome of investigations by the FBI, local police and the VA inspector general, FBI, local police and the VA inspector general, Nicholson said. He would not identify the Nicholson said. He would not identify the employee…“employee…“

Page 14: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

What does an architect need to know about security?

How can I systematically manage security at the architecture and design stage?

What security basics do I need to know?

Page 15: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Basic Security ConceptsBasic Security Concepts

• Reduce Attack Surface

• Defense In Depth

• Least Privilege

• Fail to Secure Mode

Page 16: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Attack SurfaceAttack Surface

• The “Attack Surface” is the sum of the ways in which an attacker can get at you

• Smaller Attack Surface is better

Which one has the Smaller attack surface?

Page 17: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Understand Your Attack SurfaceUnderstand Your Attack Surface

• Networking protocols that are enabled by default

• Network Endpoints

• Code that auto-starts or will execute when accessed

• Examples: Services, daemons, ISAPI filters and applications, SOAP services, and Web roots

• Reusable components

• ActiveX controls, COM objects, and .NET Framework assemblies, especially those marked with the AllowParticallyTrustedCallersAttribute)

• Process identities for all the code you run

• User accounts installed

Page 18: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Reducing Attack SurfaceReducing Attack Surface

Service: Autostart SYSTEM

TCP/UDP

TCP/UDP

TCP/UDP

Page 19: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Reducing Attack SurfaceReducing Attack Surface

Service: Autostart SYSTEM

Turn off less-used ports

TCP/UDP

TCP/UDP

TCP/UDP

Page 20: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Reducing Attack SurfaceReducing Attack Surface

Service: Autostart SYSTEM

Turn off UDP connections

TCP/UDP

Page 21: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Reducing Attack SurfaceReducing Attack Surface

Service: Autostart SYSTEM

Restrict requeststo subnet/IP range

TCP only

Page 22: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Reducing Attack SurfaceReducing Attack Surface

Service: Autostart SYSTEM

Authenticate connections

TCP only

Page 23: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Reducing Attack SurfaceReducing Attack Surface

Service: Manual NetService

TCP only

Lower privilegeTurn feature off

Page 24: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Reducing Attack SurfaceReducing Attack Surface

Service: Manual NetService

Harden ACLs on data store

TCP only

Everyone (Full Control)Everyone (Full Control)Everyone (Full Control)Everyone (Full Control)

Admin (Full Control)Admin (Full Control)Everyone (Read)Everyone (Read)Service (RW)Service (RW)

Admin (Full Control)Admin (Full Control)Everyone (Read)Everyone (Read)Service (RW)Service (RW)

Page 25: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Basic Security ConceptsBasic Security Concepts

• Reduce Attack Surface

• Defense In Depth

• Least Privilege

• Fail to Secure Mode

Page 26: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Defense In DepthDefense In Depth

• Don’t count on one line of defense for everything

• What if the attacker penetrates that defense?

• Contain the damage

• An example – Nuclear Plants

• “Multiple redundant safety systems. Nuclear plants are designed according to a "defense in depth" philosophy that requires redundant, diverse, reliable safety systems. Two or more safety systems perform key functions independently, such that, if one fails, there is always another to back it up, providing continuous protection. “

• - Nuclear Energy Institute

Page 27: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Defense in DepthDefense in Depth (MS03-007) (MS03-007)Windows Server 2003 UnaffectedWindows Server 2003 Unaffected

The underlying DLL The underlying DLL (NTDLL.DLL) not (NTDLL.DLL) not vulnerablevulnerable

The underlying DLL The underlying DLL (NTDLL.DLL) not (NTDLL.DLL) not vulnerablevulnerable

Code made more conservative during Security PushCode made more conservative during Security PushCode made more conservative during Security PushCode made more conservative during Security Push

EvenEven if it was running if it was runningEvenEven if it was running if it was running IIS 6.0 doesn’t have WebDAV enabled by defaultIIS 6.0 doesn’t have WebDAV enabled by defaultIIS 6.0 doesn’t have WebDAV enabled by defaultIIS 6.0 doesn’t have WebDAV enabled by default

EvenEven if it did have if it did have WebDAV enabledWebDAV enabledEvenEven if it did have if it did have WebDAV enabledWebDAV enabled

Maximum URL length in IIS 6.0 is 16kb by Maximum URL length in IIS 6.0 is 16kb by default (>64kb needed) default (>64kb needed) Maximum URL length in IIS 6.0 is 16kb by Maximum URL length in IIS 6.0 is 16kb by default (>64kb needed) default (>64kb needed)

EvenEven if it was vulnerable if it was vulnerableEvenEven if it was vulnerable if it was vulnerable IIS 6.0 not running by default on IIS 6.0 not running by default on Windows Server 2003Windows Server 2003IIS 6.0 not running by default on IIS 6.0 not running by default on Windows Server 2003Windows Server 2003

EvenEven if it there was an if it there was an exploitable buffer exploitable buffer overrunoverrun

EvenEven if it there was an if it there was an exploitable buffer exploitable buffer overrunoverrun

Would have occurred in Would have occurred in w3wp.exew3wp.exe which is which is now running as ‘network service’now running as ‘network service’Would have occurred in Would have occurred in w3wp.exew3wp.exe which is which is now running as ‘network service’now running as ‘network service’

EvenEven if the buffer was if the buffer was large enoughlarge enoughEvenEven if the buffer was if the buffer was large enoughlarge enough

Process halts rather than executes malicious code, Process halts rather than executes malicious code, due to buffer-overrun detection code (-GS)due to buffer-overrun detection code (-GS)Process halts rather than executes malicious code, Process halts rather than executes malicious code, due to buffer-overrun detection code (-GS)due to buffer-overrun detection code (-GS)

Microsoft Security Bulletin MS03-007Unchecked Buffer In Windows Component Could Cause Server Compromise

(815021)Originally posted: March 17, 2003

Impact of vulnerability: Run code of attacker's choice

Maximum Severity Rating: Critical

Affected Software: • Microsoft Windows NT 4.0 • Microsoft Windows 2000 • Microsoft Windows XP

Not Affected Software:• Microsoft Windows Server 2003

Microsoft Security Bulletin MS03-007Unchecked Buffer In Windows Component Could Cause Server Compromise

(815021)Originally posted: March 17, 2003

Impact of vulnerability: Run code of attacker's choice

Maximum Severity Rating: Critical

Affected Software: • Microsoft Windows NT 4.0 • Microsoft Windows 2000 • Microsoft Windows XP

Not Affected Software:• Microsoft Windows Server 2003

Page 28: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Basic Security ConceptsBasic Security Concepts

• Reduce Attack Surface

• Defense In Depth

• Least Privilege

• Fail to Secure Mode

Page 29: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Least PrivilegeLeast Privilege

• A defense in depth measure

• Code should run with only the permissions it requires

• Attackers can only do whatever the code was already allowed to do

• Recommendations

• Use least privilege accounts

• Use code access security

• Write Apps that non-admins can actually use

Page 30: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Basic Security ConceptsBasic Security Concepts

• Reduce Attack Surface

• Defense In Depth

• Least Privilege

• Fail to Secure Mode

Page 31: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Fail To Secure ModeFail To Secure Mode

• Watch out for exceptions

• Never initialize variables to success resultsFunction Authenticate(UserID As String, Password As String) Dim Authenticated As Boolean = True Try Dim conn As New SqlConnection(connString) conn.Open() Dim cmd As New SqlCommand("SELECT Count(*) FROM Users …”) Dim count As Integer count = cmd.ExecuteScalar() Authenticated = (count = 1) Catch ex As Exception MessageBox.Show("Error logging in " + ex.Message) End Try Return AuthenticatedEnd Function

Danger!!Assumes Success

Authenticated flag may

still be true here

Authenticated As Boolean = TrueAuthenticated As Boolean = True

Catch ex As ExceptionCatch ex As Exception

Page 32: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Architects Must

• Understand security terminology and best practices

• Pay attention to what is happening in the industry

• Instill security thinking throughout the application lifecycle

• Ensure that the team has an up to date threat model

• Ensure that the team has operational procedures that will ensure ongoing security

Page 33: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Satisfy Your Technical Curiosity

Essentials of the Architect and Essentials of the Architect and ArchitectureArchitectureAvailabilityAvailability

Ron JacobsRon JacobsArchitect EvangelistArchitect Evangelist

http://www.ronjacobs.comhttp://www.ronjacobs.com

Page 34: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

How do you define availability?

How do I define the availability requirements of the system?

How do I architect a system with the "right" level of availability?

Page 35: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

How do you define availability?

How do I define the availability requirements of the system?

How do I architect a system with the "right" level of availability?

Page 36: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Security

Precautions taken to keep somebody or something safe from crime, attack, or danger

Protection against attack from without or subversion from within

Availability means the system is Availability means the system is open for businessopen for business

Business for a retail store means Business for a retail store means Customers are browsing and buyingCustomers are browsing and buying

Most retail stores have planned downtimeMost retail stores have planned downtimefor holidays, inventory or just close during for holidays, inventory or just close during off-peak hours like late night or early morningoff-peak hours like late night or early morning

Page 37: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

• Availability can be expressed numerically as the percentage of the time that a service is available for use.

• Percentage of availability = (total elapsed time – sum of downtime)/total elapsed time

0 100 200 300 400 500 600

99.999%

99.99%

99.9%

5 minutes5 minutes

8 hours, 45 minutes8 hours, 45 minutes

53 minutes53 minutes

Page 38: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

How do you define availability?

How do I define the availability requirements of the system?

How do I architect a system with the "right" level of availability?

Page 39: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Dimensions of Availability

FunctionalityFunctionality

PerformancePerformanceData AccuracyData Accuracy

Does the Does the system do what system do what it is supposed it is supposed to do?to do?

Is the data Is the data provided by provided by the system the system accurate and accurate and complete?complete?

Does the Does the system system function within function within the acceptable the acceptable performance performance criteria?criteria?

Page 40: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Service Level Agreements

• Define what you mean by available

• The system is available when

• The home page displays within 2 seconds when you navigate to the URL

• You can add items to the shopping cart in 1 second or less

• You can purchase items in your shopping cart using a credit card in 15 seconds or less

• Your definition should betestable with automatedtools or third party vendors

Page 41: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

How do you define availability?

How do I define the availability requirements of the system?

How do I architect a system with the "right" level of availability?

Page 42: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Component RedundancyEliminates single point of failure

Active / Active configurationExample: Web Farm

Active / PassiveExample: Cluster of SQL Servers

Y1

Y3

Y2 ZX

Y1

Y3

Y2 Load Balancer

Y1

Y3

Y2 Load Balancer

Use High Availability PatternsUse High Availability Patterns

Page 43: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Sequential DependencySequential Dependency

• Components connected is a chain, relying on the previous component for availability

• The total availability is always lower than the availability of the weakest link

Server 1Server 1 Server 2Server 2 Server 3Server 3

Availability (A)= AAvailability (A)= AS1S1*A*AS2S2*A*AS3S3

Page 44: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Sequential Dependency ExampleSequential Dependency Example

• Availability = Database * Network * Web Server * Desktop

• Availability = 98% * 98% * 97.5% * 96% = 89.89%

• Total Infrastructure Availability = 89.89%

98%98%

Database ServerDatabase Server

98%98%

NetworkNetwork

97.5%97.5%

Web ServerWeb Server

96%96%

DesktopDesktop

Page 45: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Redundant Dependency ExampleRedundant Dependency Example

• Database Availability= 1 – ((1 – 0.98) * (1 – 0.98)) = 99.99%

• Web Server Availability = 1- ((1-0.975)*(1-0.95)) = 99.87%

• Availability = Database * Network * Server * Workstation

• Availability = 0.99 * 0.98 * 0.99 * 0.96 = 0.9169

• Total Infrastructure Availability = 92.20%

• Total availability is higher than the availability of the individual links

98%98%

NetworkNetwork

97.5%97.5%

Web ServersWeb Servers

96%96%

DesktopDesktop

98%98%

Database ServersDatabase Servers

98%98%95%95%

Page 46: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Availability Requires PeopleAvailability Requires People

• People are the biggest cause of downtime

• Organization - ensure skills are available or on call when required

• Procedures - Operators need correctly documented, tested and maintained procedures

Page 47: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Availability Procedures

• Startup

• Shutdown

• Disable

• Restart

• Troubleshooting

• SLA Monitoring

• Patching / Updating

• Provision / De-Provision Users

Page 48: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Architects Must

• Define what “Available” means for the system and a means for measuring it

• Work with the stakeholders to craft an SLA

• Architect solutions for diagnoses and recovery of the system

Page 49: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Satisfy Your Technical Curiosity

Essentials of the Architect and Essentials of the Architect and ArchitectureArchitecturePerformancePerformance

Ron JacobsRon JacobsArchitect EvangelistArchitect Evangelist

http://www.ronjacobs.comhttp://www.ronjacobs.com

Page 50: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

How is performance a risk for my solution?

What are the myths of performance?

How do I engineer for performance?

Page 51: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

How is performance a risk for my solution?

What are the myths of performance?

How do I engineer for performance?

Page 52: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Every project has risksEvery project has risks

PeoplePeople

ScheduleSchedule

RequirementsRequirements

How are you How are you managing themanaging theperformanceperformanceand scalability and scalability risk of your risk of your solution?solution?

Page 53: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Failure to Manage Risk

• 1999 UK Passport agency builds system to automate passport issuance process

• Backlog of 500,000 passports builds up

• Cost of passport processing rose dramatically

• 2001 UK Public Records office puts census data online

• System designed for a peak of 1.2M users per day

• In first month, system had 1.2M users per hour

• System crashed and had to be redesigned in a 6 month effort to increase performance and scalability

• Average cost of failed project in a 2002 KPMG study – $12.6 million dollars

Page 54: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

How is performance a risk for my solution?

What are the myths of performance?

How do I engineer for performance?

Page 55: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Code first, fix laterCode first, fix later

Gold PlatingGold Plating

Massive Massive HardwareHardware

Page 56: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

How is performance a risk for my solution?

What are the myths of performance?

How do I engineer for performance?

Page 57: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Engineering For PerformanceEngineering For Performance

• Build performance and scalability thinking in the development lifecycle

• Define your objectives

• Measure against your objectives

When You measure what you are speaking about, and express it in numbers, you know something about it; but when You cannot

express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science.

- Lord Kelvin (William Thomson)

Page 58: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Performance ModelingPerformance Modeling

• A structured and repeatable approach to modeling the performance of your software

• Similar to “Threat Modeling” in security

• Begins during the early phases of your application design

• Continues throughout the application lifecycle

• Consists of

• A document that captures your performance requirements

• A process to incrementally define and capture the information that helps the teams working on your solution to focus on using, capturing, and sharing the correct information.

Page 59: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Performance modeling ProcessPerformance modeling Process

Critical Scenarios

•Have specific performance expectations or requirements.

Significant Scenarios

•Do not have specific performance objectives

•May impact other critical scenarios.

• Look for scenarios which

•Run in parallel to a performance critical scenario

•Frequently executed

•Account for a high percentage of system use

•Consume significant system resources

1. Identify Key Scenarios1. Identify Key Scenarios

2. Identify Workloads2. Identify Workloads

3. Identify Performance Objectives3. Identify Performance Objectives

4. Identify Processing Steps4. Identify Processing Steps

5. Allocate Budget5. Allocate Budget

6. Evaluate6. Evaluate

7. Validate7. Validate

Itera

te

Page 60: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Performance modeling ProcessPerformance modeling Process

• Workload is usually derived from marketing data

•Total users

•Concurrently active users

•Data volumes

•Transaction volumes and transaction mix

• Identify how this workload applies to an individual scenario

•Support 100 concurrent users browsing

•Support 10 concurrent users placing orders.

1. Identify Key Scenarios1. Identify Key Scenarios

2. Identify Workloads2. Identify Workloads

3. Identify Performance Objectives3. Identify Performance Objectives

4. Identify Processing Steps4. Identify Processing Steps

5. Allocate Budget5. Allocate Budget

6. Evaluate6. Evaluate

7. Validate7. Validate

Itera

te

Page 61: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Performance modeling ProcessPerformance modeling Process

• Performance and scalability goals should be defined as non-functional or operational requirements

• Requirements should be based on previously identified workload

• Consider the following:

•Service level agreements

•Response times

•Projected growth

•Lifetime of your application

1. Identify Key Scenarios1. Identify Key Scenarios

2. Identify Workloads2. Identify Workloads

3. Identify Performance Objectives3. Identify Performance Objectives

4. Identify Processing Steps4. Identify Processing Steps

5. Allocate Budget5. Allocate Budget

6. Evaluate6. Evaluate

7. Validate7. Validate

Itera

te

Page 62: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Define Your ObjectivesDefine Your Objectives

• Performance and scalability goals should be defined as non-functional or operational requirements

• Requirements should be based on expected use of the system

• Compare to previous versions or similar systems

Metric Definition Measured By Impacts

Throughput How Many? Requests per second

Number of servers

Response Time How Fast? Client latency Customer Satisfaction

Resource Util. How Much? % of resource Hardware/ Network

Page 63: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Define Your ObjectivesDefine Your Objectives

• Objectives must be SMART

• S – Specific

• M – Measurable

• A – Achievable

• R – Results Oriented

• T – Time Specific

"application must run fast" “Page should load quickly"

"3 second response time on home page with 100 concurrent users and < 70% CPU" "25 journal updates posted per second with 500 concurrent users and < 70% CPU"

"If You cannot measure it, You cannot improve

it.“-Lord Kelvin

Page 64: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Build an objectiveBuild an objective

Scenario Response Time

Throughput Workload Resource Utilization

Browse Home page

Client latency 3 seconds

50 requests per second

100 concurrent users

< 60% CPU Utilization

Search Catalog

Client latency 5 seconds

10 requests per second

100 concurrent users

< 60% CPU Utilization

Page 65: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Performance modeling ProcessPerformance modeling Process• Identify the steps that must take

place to complete a scenario

• Use cases, sequence diagrams, flowcharts etc. all provide useful input

• Helps you to know where to instrument your code later

• Start at a high level, don’t go to low

1. Identify Key Scenarios1. Identify Key Scenarios

2. Identify Workloads2. Identify Workloads

3. Identify Performance Objectives3. Identify Performance Objectives

4. Identify Processing Steps4. Identify Processing Steps

5. Allocate Budget5. Allocate Budget

6. Evaluate6. Evaluate

7. Validate7. Validate

Itera

te

Page 66: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Performance modeling ProcessPerformance modeling Process• Use your performance baseline to

measure how much time each processing step is taking

• If you are not meeting your target budget the time among the processing steps

1. Identify Key Scenarios1. Identify Key Scenarios

2. Identify Workloads2. Identify Workloads

3. Identify Performance Objectives3. Identify Performance Objectives

4. Identify Processing Steps4. Identify Processing Steps

5. Allocate Budget5. Allocate Budget

6. Evaluate6. Evaluate

7. Validate7. Validate

Itera

te

Page 67: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Performance modeling ProcessPerformance modeling Process• Run automated test scenarios and

evaluate the performance against objectives

• As much as possible, tests must be repeatable throughout application lifecycle

1. Identify Key Scenarios1. Identify Key Scenarios

2. Identify Workloads2. Identify Workloads

3. Identify Performance Objectives3. Identify Performance Objectives

4. Identify Processing Steps4. Identify Processing Steps

5. Allocate Budget5. Allocate Budget

6. Evaluate6. Evaluate

7. Validate7. Validate

Itera

te

Page 68: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Performance modeling ProcessPerformance modeling Process• Check your results against

performance objectives

• Leave yourself a margin early in the project to avoid early performance optimization

• As you progress toward completion allow less margin

1. Identify Key Scenarios1. Identify Key Scenarios

2. Identify Workloads2. Identify Workloads

3. Identify Performance Objectives3. Identify Performance Objectives

4. Identify Processing Steps4. Identify Processing Steps

5. Allocate Budget5. Allocate Budget

6. Evaluate6. Evaluate

7. Validate7. Validate

Itera

te

Page 69: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.

Architects Must

• Clearly define the performance objectives of the system

• Create a plan for measuring the performance of the system that is easy and repeatable

• Monitor the solution throughout the lifecycle to insure that performance risks are adequately managed

Page 70: Satisfy Your Technical Curiosity 27, 28 & 29 March 2007 International Convention Center (ICC) Ghent, Belgium.