The Blackboard Reference Architecture: Advanced Performance Management

Stephen Feldman (sfeldman@blackboard.com)

Director Blackboard Performance Engineering

07/10/07

2

Who is this presentation geared for?

• Directors of Technology and Infrastructure

• Key Decision Makers over technology purchase and implementation decisions

• Key Stakeholders in the operational management of the system

• Curious bystanders who walked into the session by accident.

3

• Learn what are reference architectures.

• Understand why reference architectures are important for enterprise application deployments.

• Introduction to initial components of the Blackboard Reference Architecture

• Short overview of user performance management.

What will you attain from this presentation?

4

• Introduction to Reference Architectures• The Blackboard Reference Architecture• Key Components

– Platform Infrastructure– Web/Application Delivery and Management– Advanced Storage Architectures– Monitoring and Management– User Performance Management– Behavior Modeling (Web Analytics)

Presentation agenda

5

Where does your institution fit?

Level 1 Level 2 Level 3 Level 4 Level 5

Reactive and

Exploratory

MonitorAnd

Instrument

PerformanceOptimizing

BusinessOptimizing

ProcessOptimizing

6

• An architectural set of best of practices for system deployment.

• Begin with a holistic view of the system and narrow down to each essential component.

• Each component must add value to the overall architecture.– Without the component the system lacks the

quintessential definition of the “Ideal Architecture”

What is a Reference Architecture?

7

• Consider it a blue print for success.– Remember that buildings are not constructed of blue

prints, but rather are designed from blue prints.– This is not a pie-in-the-sky idea, but a vision towards

enterprise performance optimization.

• Provide an overview of what an enterprise application requires to achieve Level 5 in the Performance Maturity Model.

• Without it, your institution will struggle to ascend beyond Level 2: Monitoring and Management.

Why is the Reference Architecture important to you?

8

What is the Blackboard Reference Architecture?

Federated ApplicationsEnterprise SearchOther WSI ...

Application Layer

Enterprise Storage

· Optimization· Backup· Recovery· Growth

Analysis

SNMP

ManagementMonitoring

Integration

Even

t-Driven

M

gm

t.A

dvan

ced

Rep

ortin

gB

ehavio

r M

od

el Stu

dies

SIS & Back office

B2 Partners

Campus Systems

Email

Publishers

Directory Svcs.

SSO

Portals

SMS & MobileCam

paig

n

Mg

mt.

Database Layer

Blackboard Reference Architecture

User Experience

Virtualization

MonitoringManagementClustering

Load Balancing

Beyond ServicesBlackboardInstitution

AccessSecurityIdentify

9

Platform Infrastructure• Focusing on a few major players

– Hardware Perspective• Dell

– Linux/Windows on Intel– Linux/Windows on AMD

• Sun Microsystems– Solaris on Niagra/T1– Solaris on Ultra Sparc IV– Linux/Windows on AMD– Linux/Windows on Intel

– Platform OS• Red Hat Linux• Windows 2003• Solaris 10

– RDBMS• Oracle 10G and Oracle 10G RAC• SQL Server 2005

10

Dell Multi-Purpose Reference Architecture

11

Sun Multi-Purpose Reference ArchitectureOracle and Oracle RAC on SUN

Reference Architecture

Gig-E Switch(separate subnet)

Gig-E Switch Private Interconnect(separate subnet)

Public Network

Storage Network

Oracle Cluster Interconnect Network

Attempt to bond interconnect NICs

Sun T1000 6 to 8-core

Sun T2000 6 to 8-core

Sun T1000 6 to 8-core

Sun T2000 6 to 8-core

Sun T1000 6 to 8-core

Sun T2000 6 to 8-core

Gig-E Switch

Application Tier

Database Tier Virtual IP addresses

Netscaler7000Citrix Netscaler 7000

NetApp FAS Appliance

Sun Modular Blade Units (Niagara: T1)

Storage Tier

12

Alternative Performance Configurations: Multi-Home Clustering

AS Node 1

AS Node n

AS Node 3

AS Node 2

Apache Web Server

Single Physical Server

HTTP Traffic

Vista Admin Node

Vista Node n

Vista Node 2

Vista Node 1

Hardware Load Balancer

Single Physical Server

HTTP Traffic

Academic Suite Vista/CE Learning System

13

Alternative Performance Configurations: Distributed Load-Balancing and Clustering

Physical Server n

Physical Server 3

Physical Server 1

Physical Server 2

Vista Admin Node

Vista Node n

Vista Node 2

Vista Node 1

Hardware Load Balancer

HTTP Traffic

Hardware Load Balancer

AS Instance

Apache

Server 1

AS Instance

Apache

Server 2

AS Instance

Apache

Server 3

AS Instance

Apache

Server n

HTTP Traffic

Academic Suite Vista/CE Learning System

14

Alternative Performance Configurations: Server Virtualization (Linux/Windows)

Virtual M

achine

Virtual M

achine

Virtual M

achine

Virtual M

achine

2x2 Physical Server

Physical Memory Pool

vCP

U

vCP

U

vCP

U

System Hypervisor Kernel

Network Switch

CPU Core CPU Core CPU Core CPU CoreNetwork Attached

StorageVirtual Interfaces Virtual InterfacesVirtual InterfacesVirtual Interfaces

vCP

U

vCP

UvC

PU

vCP

U

vCP

U

Re

served

RA

M

Re

served

RA

M

Re

served

RA

M

Re

served

RA

M

Physical Interfaces

15

Alternative Performance Configurations: Server Virtualization (Solaris)

16

Web/Application Delivery and Management

Security

Caching

Compression

SSL Acceleration

Web Servers

Application Servers

DB Servers

Servers Add

Up Quickly

Variety of Point

Products

Chatty Protocols,

Long Hauls,High Latency

17

Citrix NetScaler: Reference Architecture Component

Security

Caching

Compression

SSL Acceleration

Web Servers

Application Servers

DB Servers

Reduces Load on Backend Servers

Eliminates Multiple

Inefficient Point Products

Reduces Bandwidth Required

18

Storage architectures

• Ideal architectures should support multi-protocol– Network File Storage (NFS)– Fiber-Channel SAN– IP/SAN (ISCI)– Common Internet File System (CIFS)

• High-Performing (I/O per second)– Read/Write Performance– Data Replication

• Handle growth and change– Least interruption

• Storage Utilization– Flexibility in mixing spindle type and speed

19

Monitoring and Management• One of the most critical components in the Reference Architecture.

– Without monitoring and management tools in place it would be the equivalent of walking in a pitch-black room with no lights.

– Visibility is essential with enterprise applications.• Issues can be highlighted and observed in real-time• Enterprise architectures are large…M & M tools can help make the problem more

manageable. • M & M tools assist with going back in time to understand patterns,

trends and key events.• Automate problem detection and notification — you don’t have to ask

about current status• M &M tools assist with the quantification of problems.

– Provide critical diagnostic information about why something happened.• Most M & M tools make the data digestible to stakeholders of the

system who are often not as technical as you are.

20

Monitoring and Management: Example Artifact

21

User Performance Management

UserExperience

PlatformHealth

Infrastructure NetworksApplications

• Assumes platform health is the only indicator of user performance or app availability

Impact

Verify

Escalate

Alert

22

User Performance Management startswith user experience

UserExperience

PlatformHealth

Infrastructure NetworksApplications

Detect

Investigate

Localize

Resolve

23

Where User Performance Management fits• UPM the missing piece• Without it:

– No way to measure end user experience

– No immediate detection of user errors

– No problem reproduction– No view of what really

happens in production

Users

Systems

OperationsMarketing

UserPerformanceManagement

• Incidents• Service levels

Synthetic testingReachability, baselining,CDN, competitors, load

Web analyticsConversion, SEO,

traffic, campaign ROI

Platform MgmtDevices, functions,agents, databases

Synthetic testingReachability, baselining,CDN, competitors, load

Platform MgmtDevices, functions,agents, databases

Web analyticsConversion, SEO,

traffic, campaign ROI

$

24

Device monitoring: Watching the infrastructure

• Less relation to application availability

• Vital for troubleshooting and localization

• Will show “hard down” errors– But good sites are redundant anyway

• Correlation between a metric (CPU, RAM) and performance degradation shows where to add capacity

25

Web Analytics: Looking at user behavior

• Studies the behavior of Web site visitors

• Help marketing teams support business objectives

• Report on effectiveness of campaigns

• Analyze visitor demographics

• Doesn’t look at the impact of performance on user behavior

26

Synthetic testing: Checking it yourself

• Local or outside

• Same test each time

• Excellent for network baselining when you can’t control end-user connection

• Use to check if a region or function is down for everyone

• Limited usefulness for problem re-creation

27

User performance management: See what’s happening

• See what your users are actually doing

• Monitor usage while in production

• Get real-time visibility

• Re-produce problems for troubleshooting

• Prioritize problems based on the impact to actual users

28

The technology: Real user monitoring

• Detect individual incidents • Capture, alert, localize & escalate• Store detailed user performance data• Share with other IT systems• Measure capacity at different loads• Report on subscribers, regions, etc.

29

Web Analytics: Behavior Modeling

• Look into the past to see the future.• Web Analytics is the analysis of how users behave on

the system.• While built-in product features are helpful to understand

user behavior, analyzing log data can provide more meaningful statistics.

• Goal is to understand the following:– Where are your users spending their time?– How long are they spending?– Where are they coming from?– Where are they going?– What patterns do they exhibit when they are interacting with the

application?– What types of errors or failures are they experiencing?

30

Want More?

• To view my resources and references for this presentation, visit

www.scholar.com• Simply click “Advanced Search” and

search by s_feldman and tag: ‘bbworld07’ and ‘refarch’