Criticality of Accurate Detection in the Automated Patch

Management Process

Chris Andrew

VP Product Management

Agenda

• Hackers Turn Up the Heat

• What Is Automated Patch Detection?

• Accurate Patch Detection Exists

• Best Practices in Automated Patch Management

• Technology Demonstration

• Dealing with Real-World Challenges

• Q&A

Hackers Turn Up the Heat

Failed Windows XP Upgrade Downs 60,000 UK Government PCs

http://www.eweek.com/article2/0,1759,1732672,00.asp By John Lettice/ The Register - special to eWEEK.com

Most of the desktop computers in the UK's Department for Work and Pensions were paralyzed for four days on Monday, when a failed upgrade took them offline. The outage, covering 75 percent to 80 percent of the DWP's 80,000 PCs, is one of the largest in the UK government's not entirely impressive IT history. And possibly one of the most costly. According to staff reports, the outage occurred on Monday afternoon, disconnecting staff e-mail, benefits processing, and Internet and intranet connectivity. According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, "upgrade."

Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS was going to apply a patch to these. Unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 percent of the Win2K desktops. This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that [Microsoft Corp.] consultants have been flown in from the U.S. to clear up the mess." EDS is also thought to be flying in fire brigades.

If these claims are true, the DWP could face grave difficulties in rolling all of its machines back to their previous, working state. Staff from Microsoft and EDS are reported to have been working around the clock to dig the department out of the pit, while speaking on the "Today" program Friday morning, a spokeswoman amusingly insisted that the department's systems had not in fact fallen over. They were working; it was merely the case that "80 percent of desktop computers are not connecting through to the mainframe systems."

So that's cleared that up then. She added that the emergency payments system was "working perfectly." The emergency system appears to have kicked in on Wednesday, and the department was preparing a press release on the matter Thursday. There was no sign of it when this story was published.

Reports coming in on Friday however suggest that at least some of the DWP's systems are coming back online

Note: Rating is on a scale of 1 – 5 where 1 is “not at all important” and 5 is “extremely important

Base = 1,378 – 1,394

Data: Secure Enterprise Security Deployment Survey, October 2004

Q: Rate the relative risk of the following:

Current Climate in Vulnerability Management

Current Climate in Vulnerability Management

• The age of zero day exploits is upon us… Only worse– Slammer patch to exploit 6 months– Welchia patch to exploit 26 days– Sasser patch to exploit 11 Days– ADODB stream exploit in the wild for weeks prior

to patch, months for effective patch– JPEG processor exploit ADODB stream exploit in

the wild for weeks prior to patch, months for effective patch

– NetDDE exploits are in the wild– Many exploits are known for months before a

patch is available

Worms vs. Viruses

Top 10 Malcode Reported by Sophos Antivirus

0

2

4

6

8

10

Co

un

t

Virus 10 9 6 4 3 2

Worm 0 1 4 6 7 8

1998 1999 2000 2001 2002 2003

A Continuous Cycle of Infection

0

10

20

30

40

50

60

70

12/2

7/20

03

1/3/

2004

1/10

/200

4

1/17

/200

4

1/24

/200

4

1/31

/200

4

2/7/

2004

2/14

/200

4

2/21

/200

4

2/28

/200

4

3/6/

2004

3/13

/200

4

3/20

/200

4

3/27

/200

4

4/3/

2004

4/10

/200

4

4/17

/200

4

4/24

/200

4

5/1/

2004

5/8/

2004

5/15

/200

4

5/22

/200

4

5/29

/200

4

6/5/

2004

6/12

/200

4

6/19

/200

4

6/26

/200

4

7/3/

2004

Sasser

CodeRed

Nachi

Blaster

Most InfoSec Organizations are Overwhelmed

Base = 1,395

Data: Secure Enterprise Security Deployment Survey, October 2004

Current Climate in Vulnerability Management

What is Automated Patch Detection?

Patch Management – Mitigating Risk

Network-related Security Risks

Lost Revenue Criminal/Statutory Fines/Penalties Service Level

Legal Liability Shareholder/Customer Confidence Competition

Enterprise Network Vulnerabilities

Intrusion Incorrect / Unauthorized Configurations

Reliability Unauthorized Services / Software / Content

PatchLink’s Patch & Vulnerability Approach:

Detect Deploy DefendConfiguration Status

Unauthorized Elements

Immediate Remediation

Administrative Control

Continuously Monitor

Automated Response

Professional Services

3D

Patch Management Market Drivers

• Increasing security “incidents” – Steady growth from 2000 to 2003

• Increasing Patch Counts, Including Apps

– Microsoft has released roughly 1.38 patches per week since January 2002, all products included

• Incomplete Patch Deployments – Over 90% of the security exploits are

carried out through vulnerabilities for which there are known patches.

• Poor Processes– During a 6-12 month period,

approximately 20% of machines become “unpatched”

• Not Addressed by Software Giants– < 5% of organizations have a

“satisfactory automated patch management solution”

CERT Coordination Center

Reported Incidents (in thousands)

Sources: Microsoft and CERT Coordination Center Data

Key factors that create patching obstacles:

• Limited time to satisfactorily test patches & inability to keep up with pace of current attacks

• Complex, heterogeneous networks & remote users

• Philosophical opposition to patching & how often

• Lack of security practices & standard

• No sense of urgency

• Think that firewalls & anti-virus are enough

• “It’s the software company’s problem”

The “Problem” with Patching

Patch Management is NOT a task!

It is an ongoing, necessary vulnerability management process that requires rigorous testing & continuous

auditing to establish baseline security policies.

“Most administrators unable to keep pace with the barrage of security alerts coming out at the pace of about one every

two to three days.”

“Automation is the only effective solution.”

~David Tschanz, MCP Magazine, August 2003

Important Reminder…

Accurate Patch Detection Exists

the Patch Development Kit

Introducing Patch Development Kit

• Every business runs something special– In-house custom developed software– Legacy applications – Not generally used

• May need pre-release or private patches– Microsoft early release and BETA software– Specific pre-release fixes given by vendor(s)

• Company specific Anti-patches– Get rid of stuff you DON’T want! KaZaA, AV, MP3

Now you can patch or uninstallanything with PDK…

Rolling Your Own Remediation

• Detection Patch– Name must start with “Detect …”– Establishes existence of a given product version– Impact = Critical

• Software / Hardware Patch – Version specific patch, pre-requisite to detect patch– Impact = Critical through Informational

• Software Installation / Removal– Indicate OS that it works on– Impact = Software

Types of Patches

Detect Windows 2000

Detect Service Pack 2

Detect Service Pack 3

Detect Office 2000

Detect WinZip

MSO2-020

MSO2-020

MSO3-007

MSO3-026

MSO3-007

MSO3-026

…

…

…

The Pre-requisite Tree

• Report Properties– Basic information, vendor URLs, ID, hyperlinks

• Patch Signature(s)– Registry fingerprint– File fingerprint

• Patch Package(s)– Content files and directories– Package scripting

What Info Is In A Patch

• Title - Generic name • Identifier - Vendor Q# or id

• Release Date - Original vendor ship

• Hyperlink - URL for more info

• Vendor - Original Author

• Impact - Critical, Software, etc

• Status - Beta, Active, etc

• Description - 3000 character limit

Report Properties

• Signature uniquely Identifies ONE patch – Usually requires multiple fingerprints– May also pre-req a “Detect” signature– File properties: inspection or use a tool– Registry information: RegSpy or inspection

• BOOLEAN result – TRUE => computer has the patch

• One package per signature– The “fix” for not having the patch

• Always one package per signature

Patch Signature

• Quickly Add Content– Drag & Drop from your desktop– Move to a macro directory (eg: %TEMP%)– EXEs, Setup Program files, data, etc…

• Scripting Options– VBS … most commonly used– JavaScript– Command Line

• Working Directory

– Place where the script or program is run from

Patch Package

1. Pre-Script Executes - Used to cleanup the target computer- Rarely used by PatchLink

2. All files downloaded via HTTP

3. Files are copied to target location 

4. Command Line Executes- Use if you’re a BAT file aficionado

5. Post-Script Executes- Install the patch, prompt the user, etc.

Sequence of Patch Delivery

• Build an Update Server for TESTING!• Run just YOUR patch report

– DAGENT scan will be considerably faster• Turn debugging ON at the agent

– See any / all errors in your signature(s)• Test Detect XYZ first

– Needs to return TRUE/FALSE correctly!• Validate your VBS script before using

– If it doesn’t work at a cmd line…

Patch Testing 101

• File Information– Most common fingerprint type

• Registry Information– Windows only fingerprint

• System Information• Patch Version• Expression

– Used primarily with UNIX

Fingerprint Types

• Determine your fingerprint using:– REGSPY– SnapShot Utility– Rational tools– Vendor documentation– Etc.

• Build a good basic OS image– Quick and easy to recreate– Always starting “fresh” again…

How to Fingerprint

1. Determine fingerprint for detection patch2. Create your Detect Patch3. Verify Detect Patch operates correctly4. Hide the Detect Patch

=> SAVE YOUR PATCH !!!

5. Determine fingerprint for update patch6. Create your Patch, pre-req Detect Patch7. Test patched / not patched test

=> SAVE YOUR PATCH !!!

TEST, TEST …. & TEST AGAIN

Recipe Card

• Win9X / WinNT differences– Registry key differences– File location differences

• Multiple Pre-requisites– May need A and B or C– Consider using an expression

• Didn’t wait for detection to finish…– Strip out as many reports as possible!

Common Pitfalls

Detect WINZIP

WinZIP Latest Version Patch

CodeRed / NIMDA Patch Structure

Export to File

*DEMO* Defining Fingerprints

*DEMO* Package Scripting

Best Practices in Automated Patch

Detection & Deployment

• The CORRECT approach…– locate only the systems that need the patch– test on the control group first– then in limited production– then roll out en-masse– finally enforce using a policy

• When an emergency strikes, is it OK to push?– avoid red tape when needed

• Reporting & patch management– audit all parts of the organization– security team establishes quarterly baselines– management team tracks to 100% patch completion

Deploying Patches

ResearchResearch• Investigate• Assess Impact• Dependencies• Identify Targets• H/W and S/W Requirements

PlanPlan• Schedule

• Resources• Personnel

• Develop Scripts

TestTest• Develop Test Plan• Configure• Install • Test and Validate• Verify Requirements

PilotPilot• Repeat: x times

• Login• copy• Install• Reboot• Verify• Test

RolloutRollout• Repeat: y times

• Login• copy• Install• Reboot• Verify• Test

• Support

MonitorMonitor•Enforce/Validate •Subscriptions:

• Vendors• Newsgroups

• Web Searches• Vendor• Assoc• CERT/NIST etc.

• Prelim Assess

Patch Management Process

London

Rancho Cordova, CA

Berlin

Alpharetta,GA

$500 CACHE

EXISTING CACHE

$500 CACHE

• Simple one server design

• Cache acceleration

Server

WAN

Centralized Approach

Server

London

Rancho Cordova, CA

Berlin

Alpharetta,GA

Server

Server

Server

• One server at each site

• Reports pulled across sites

• Admin page links it all up…

WAN

Decentralized Approach

• Senior executive support– Protecting infotech assets must be management priority

• Standardized patch management policies, procedures, & tools– Develop PM policies and use tools that meet organizations’

infrastructure requirements• Dedicated resources & clearly assigned

responsibilities to PM process• Current technology inventory

– Effective PM tools must be able to inventory all aspects of IT infrastructure

Critical Elements for Effective Patch Management

• Identification of relevant vulnerabilities & patches for all system inventory

• Risk assessment – When do I patch and how often?

– Costs associated with patching some systems verses others

• Testing – Ensuring security patches don’t crash complex,

enterprise systems

• Distribution of patches to all users (incl. remote) – Not patching certain systems can come back to haunt you

• Monitoring through network & host vulnerability scanning

Critical Elements for Effective Patch Management

• Use an automated system for analyzing & deploying patches

• Apply patches on an “as needed” basis

• Use a planned approach, grouping systems by department, location, etc.

• Patch across all operating systems

• Develop a solid change control process

• Thoroughly test all patches before deploying

• Match test lab & production server configurations

• Plan for proactive, scheduled maintenance

Recommended Best Practices

Automated Patch Technology

Demonstration

SCAN

ISOLATE

FIXZero-day? Patch?

Policy?Virus?

Spyware?

• New layer of simplicity and automation• Grouping of Computers• Mandatory Patch Policies• Hours of Operation

• Across multiple platforms:• Agent for Sun Solaris• Agent for RedHat Linux• Agent for Novell NetWare

Never drop Patches again!

• Export to CSV File– Patched vs. Not Patched Report– Baseline Compliance Report– Inventory Audit Report

• Graphical Network Assessment– Network assessment– More trend graphs to follow…

• Application Reporting interface– Summary level reports from one server

• Pull data directly from SQL

Proving you got the job done…Proving you got the job done…

Dealing with Real-World Challenges

While You Were Sleeping…

• WAKE ON LAN Solution• Broadcast the magic packet over UDP• Plug-in for product

-OR-

• Magic Packet utility

• Wake one agent• Wake an entire group

Setting Policy & Owner Automatically

• Automatically enroll computer(s) in a group– By Name Mask– By IP Address Range– Etc…

• Runs as a service on PLUS server• Assign correct administrative owner• Deploys all required baseline patches

Reaching the Road Warriors

• FASTPATCH Solution• Locate best distribution point that is available

– WINS Resolvable– TRACERT Hop Count– SCLIENT Distance in mSec

• Service or System Task • Offers redundancy for WAN Distribution

Auditing the Enterprise

• Pull patch reports across multiple PLUS servers• Show enterprise wide compliance• Customizable report headers, footers• Pre-canned reports• Extensible RAD project

– NASA solution for multiple space centers

Establishing a Secure Enclave

Zero Day Vulnerability = No Patch

Scanner Integration Block Diagram

Upload Utility(Reads From Scanner

Integration Point and writes to PatchLink Scanner

Integration Point)

Scanner Integration Point

(Fixed API for reading Systems and

Vulnerabilities data.)

PatchLink Scanner Integration

Point(.NET Web Service: Fixed API fro writing

Systems and Vulnerabilities data.)

Scanner ISV

Application Programmer Interface: Developed by the Independent Software Vendor

Scanner ISV/ Integrator

Upload Utility that is ether developed by the Independent Software Vendor and integrated into their product or a systems Integrator as a middle ware product.

PatchLink

Application Programmer Interface: Developed by PatchLink.

Building a Custom “Patch”

Why PATCHLINK UPDATE?

1. Works in all network configs & complete heterogeneous support.

2. Comprehensive point solution. Works in any type of security architecture. Existing scanners, AV, intrusion detection, & other security apps & devices co-exist seamlessly. No need to change the base security configuration.

3. Worms often recur in networks due to offline obstacles. Will automatically and continuously detect, deploy, and disinfect all machines in one pass, whether online or offline.

Thousands of Customers - Millions of nodes protected worldwide!

Large Business

…

Government

…

Financial

…

Education

…

QQ&&AA

Electronic copy of PPT & Top40.doc