Home-Grown DB2 Monitoring and Related Tools

1

May 10, 2007 09:20 a.m. – 10:20 a.m.

Platform: DB2 for Linux, Unix and Windows

Case Study:Home-Grown DB2 Monitoring

and Related Tools

Jayesh ThakrarThe Warranty Group

Session: D12

2

2

Topics1. Presentation Objectives

2. DB2 Monitoring -Needs and Gap Analysis

3. Buy versus Build Solution

4. Discuss a Home-Grown Solution

3

3





4

4

Presentation Objectives• Empower / enable you to -

Perform DB2 monitoring needs & gap analysisSurvey & compare available packaged solutionsDecide between buy versus build

• Walk-through a sample home-grown solution -Objectives & needsDesign and packagingDeployment

• Highlight DB2 LUW built-in monitoring features

5

5





6

6

Needs Analysis - Background• Evaluate Current Environment -

Hardware (servers), software and applicationsList of RDBMS platforms & versions to support(If supporting more than just DB2 LUW)Count of databases & platforms that need monitoring(e.g. 100 Production, 60% of Test & UAT, 20% of Dev)

• Evaluate scope of responsibilities -Databases or “application systems”System resources (database, storage, etc.)Backup and recoveryChange control, security, etc.

7

7

Needs Analysis - Operational• Non-technical

People, skill-sets & day-to-day needsProcess and change management rigorsOrganizational structure(Development & production DBAs, OnCall, etc.)Peer groups for “tiers above and below”Formal / informal service level agreementsBudget and resource allocationThe “unspoken / implicit” needs & objectives

Promote standardizationPromote rigor, structure and change controlPromote documentationPromote “visibility” into infrastructure & environments

8

8

Needs Analysis - Technical• Monitor what ? (Document details & specifics)

RDBMS and databasesDatabase jobs and batchesDatabase resourcesSystem resources

• Communication, reporting, security and auditingFor status = ok/healthyFor status = problem

• Integration needsAlerting, systems management, escalation, etc.

9

9

“Gaps” = Needs - Current• Categorize the “needs”

Needs that are absolutely necessaryNeeds that serve as job aids, tools and automation“Good to have”

• Categorize the gapsCritical to your operations and service deliveryCritical to “bottom line”Critical to corporate alignment

• Determine timeframe and budget to meet the “gaps”

10

10





11

11

Before you “Buy” or “Build”• Corporate introspection and survey

Corporate standardsSolutions available or used in-houseBest of breed versus “comprehensive solution”Budgeting, cost allocation, etc

• “Dove-tail” monitoring solution into -Corporate objectives and prioritiesCorporate / enterprise application needs

• Prepare TCO (Total Cost of Ownership) and ROI (Return On Investment)

• Marshall executive sponsorship

12

12

Buy versus Build Considerations -• Consider “realreal” cost, time & benefits for both !!!!

• Buy -Cost = buy + maintenance + infrastructure

+ training + time +…Time = procurement + customization + training +deployment + operational procedures +…..Benefits = jump-start (time savings) + “proven solution” +support + best practices + documentation + …..Personal observations –Cost = usually 1.5 x original estimates (or more) Time = usually 2.0 x original estimates (or more)

13

13

Buy versus Build Considerations -• Build -

Cost = Time + time + time +….People + people + people +……

Time = Proof-of-concept (build & throw-away) +Actual development

(sufficient rigor and project management controls needed)Other needs and considerations -

Risks Support and maintenanceDocumentationCorporate standardsOther

14

14

Surveying available solutions• Features

Create matrix (absolutely needed features and others)

Include – Infrastructure and other needs, monitoring solution foot-print, scalability, manageability, security

• Cost

• “Freebies” included e.g. -Training and implementationBest practices

• OtherBest of breed versus “comprehensive & integrated”Product and vendor quality and viabilityProduct and vendor ranking & support

15

15





16

16

Why We Chose Home-Grown Solution• Needs

24 x 7 monitoring (no previous monitoring in place)

Quick, effective, proven solution for DBAs only

Good to have - integration (HP Openview + Netbackup)(Netbackup = Veritas Netbackup from Symantec Corp)

• Cost

• Skill-set and existing scripts

• Spare time !Database tasks completed prior to application tasks

• Challenge & Opportunity

17

17

Home Grown Solution Topics

1. Overview and Key Features

2. In-Depth Look

3. Toolkit Deployment

4. Next Steps…..

18

18

Home Grown Solution


2. In-Depth Look


4. Next Steps…..

19

19

Overview• Scripts to monitor database availability and capacity

• Scripts for 30-60% of regular, time-consuming, DBA administration & maintenance tasks

• Saves money & time and helps productivityCost (license, training and deployment)Time (procurement, testing, deployment, day-to-day tasks)Integration (HP OpenView, Netbackup)

• Tried and tested at Fortune 500 Financial Institution

Data Center Outsourcing Company

The Warranty Group

20

20

Key Features• Implemented using shell scripts

• Support for multiple DB2 instances and databases

• Individual configuration for each database

• Modular and easily configurable

• Email (or any set of actions) on exception

• Scripts tested on AIX, Solaris and RedHat Linux

• Single script for comprehensive monitoring

• Single script for DB2 admin & maintenance tasks

• Other misc. scripts

21

21

Key Features – HP Openview Integration

Alerts from DB2 Monitoring appear on HP OpenView Console and message browser windows

22

22

Toolkit Contents• opc_db2_mon.ksh - Availability and capacity

monitoring script• db2utils.ksh - General utilities script• monitor-system.ksh – Ad hoc OS and DB2

performance data collection script• 7 SQL scripts – Performance monitoring related• db2_discovery.sql – Comprehensive HTML report• 15 crontab entry scripts – Sample deployment• import-audit-data.ksh - Audit data extract, load &

purge• README.txt – Readme for db2utils.ksh

23

23

Home Grown Solution


2. In-Depth Look


4. Next Steps…..

24

24

In-Depth Look Availability and Capacity Monitoring

Performance Monitoring

Administration Utilities

Security and Audit

25

25

Availability & Capacity Monitoring• Implemented by - opc_db2_mon.ksh• Single, standalone, light-weight script• Monitoring of -

Instance availability (db2sysc)DB2 TCP/IP Listener availability (db2tcpcm)Database availability (connectivity)Tablespace states (normal)Filesystem space (> 85% used)

• Action on exception –HP OpenView event & email

26

26

1. Check for $HOME/sqllib/db2profile

2. Check for $HOME/.profile

3. Check for db2sysc and db2tcpcm processes

4. Check for filesystemfree space (> 15% free)

5. Check for database and tablespace states

6. Check for available log space (> 64 MB)

7. Connect to the database

Availability & Capacity Monitoring

27

27

Performance Monitoring• Implemented as - 7 SQL scripts• Not for ad hoc performance trouble-shooting• Data collection for profiling and trending• Uses DB2 v8 snapshot functions

(monitor switches ON at instance level)• Daily data collection –

5 weeks retention and automatic purge of older data• Cumulative & differential values for metrics

(e.g. cumulative and differential logical reads)• Collects data across all partitions

28

28

Administration Utilities• Implemented by – db2utils.ksh• Database Backup

Online or offline, full or incrementalTo disk, tape or third-party software

• Configuration Backupdb & dbm cfgdb & node directory

• Delete old “objects”log archives backups (on disk or Netbackup)Files

29

29

Security and Audit• Implemented by – import-audit-data.ksh• Standard “db2audit” data extract and import into

database (recommend dedicated database)• Actual data collected as per your audit configuration• Audit data configuration options in –

“DB2 Administration Guide – Implementation”• Interactive script – run on a periodic basis

- Stops auditing (momentarily)- Extracts audit data- Imports into audit database- Prunes DB2 audit data files

30

30

Home Grown Solution


2. In-Depth Look


4. Next Steps…..

31

31

Toolkit Contents (repeated) -• opc_db2_mon.ksh - Availability and capacity

monitoring script• db2utils.ksh - General utilities script • monitor-system.ksh – Ad hoc monitoring &

performance script• 7 SQL scripts – Performance monitoring related• db2_discovery.sql – Comprehensive HTML report• 15 crontab entry scripts – Sample deployment• import-audit-data.ksh - Audit data extract, load &

purge• README.txt – Instructions and other details

32

32

Deployment – 1..• Copy and untar the tar file • Untar creates directory –

DB2INSTANCE_HOME/scripts• Become familiar with the contents of the directory• Read README.txt

33

33

Deployment – 1..2..• Copy instance-database.env to

<instance>-<database>.env

• Edit each .env file

• Sample <instance>-<database>.env file#-------------- Start of Customization --------------. $HOME/.profileexport SCRIPTS_DIR="/home/db2inst1/scripts"export AUDIT_LOG_DIR="/home/db2inst1/scripts"export REPORTS_DIR="/home/db2inst1/scripts/reports"export OLD_BACKUP_RETENTION=7export DB_BACKUP_DIR="/applications/Database/backups/bocmsdev"export NOTIFICATION_EMAIL=“[email protected]"export alert_level_1_email=“[email protected]"export alert_level_2_email=“[email protected]”export alert_level_3_email=“[email protected]"export APPLICATION_NAME=“Finance Data Warehouse(Development)"#---------------- End of Customization --------------

34

34

Deployment – 1..2..3..• Edit performance data setup scripts, if necessary

(DB2INSTANCE_HOME/scripts/db2mon)

• Run the performance data setup scripts01-db2mon_create_tablespace.sql

02-db2mon_create_tables.sql

03-db2mon_get_snapshot_data.sql

04-db2mon_get_object_data.sql

35

35

Deployment – 1..2..3..4..• Edit monitoring script, if necessary

(scripts/opc_db2_mon.ksh)

• Create /etc/db2.txt with entries for all databases

across all instances

• Create monitored object(s) in HP OpenView or

appropriate Systems Management tool

• Validate exception action

• Provide the script to admin (root) for monitoring

36

36

Deployment – 1..2..3..4..5..• Test db2utils.ksh and the monitoring scripts

(see 01-*.ksh through 15*.ksh in sample crontab.txt)

• Create crontab entries (scripts/crontab.txt for sample)###### Edit History# 12/22/2006 (Jayesh Thakrar) - Created initial crontab entries######03 * * * * $HOME/scripts/01-check-filesystem-space.ksh >/dev/null21 * * * * $HOME/scripts/01b-check-database-availability.ksh >/dev/null03 22 28 * * $HOME/scripts/03-db2-rotate-diag-log-files.ksh >/dev/null41 22 28 * * $HOME/scripts/04-db2utils-rotate-audit-log-file.ksh >/dev/null01 05 * * 0 $HOME/scripts/06-db2-runstats-all.ksh >/dev/null13 16 * * 0 $HOME/scripts/07-db2-reorgchk.ksh >/dev/null27 17 * * 0 $HOME/scripts/08-db2-schema-extract.ksh >/dev/null03 22 * * * $HOME/scripts/09-db2-delete-old-archived-logs.ksh >/dev/null31 18 * * 3,6 $HOME/scripts/10b-db2-delete-old-backups.ksh >/dev/null31 19 * * 3,6 $HOME/scripts/10-db2-online-full-backup.ksh >/dev/null16 18 * * * $HOME/scripts/11-db2-daily-snapshots.ksh >/dev/null22 23 * * * $HOME/scripts/12-db2mon-get-snapshots.ksh >/dev/null32 23 * * 0 $HOME/scripts/13-db2mon-get-object-data.ksh >/dev/null42 23 * * * $HOME/scripts/14-db2mon-analyze-snapshot-data.ksh >/dev/null52 23 * * 0 $HOME/scripts/15-db2mon-purge-old-data.ksh >/dev/null23 06 * * * /usr/bin/crontab -l > $HOME/scripts/crontab-l-$USER-$HOSTNAME.txt

37

37

Deployment Ready –Take-off !!

Deployment – 1..2..3..4..5..go

38

38

Home Grown Solution


2. In-Depth Look


4. Next Steps…..

39

39

Possible Future Enhancements• Use db2pd instead of snapshots (Why ? - lightweight)

• Monitor local and remote databases (authentication info ?)

• Single repository database for all monitored databases

• Configurable threshold monitoring and alerting

• Collect operating system performance data

• Automated analysis and feedback on capacity planning

• Port to Java, Perl or Python AND Windows

• GUI front-end (HTML, web-services, XML?)

40

40

Comparison with DB2’s built-in Health Monitor

• Health monitor available out of the box• Health monitor depends upon the availability of DAS

(DB2 Administration Server)• DB2 fault-monitor also available out of the box –

however occasional problems (performance/CPU, respawning rapidly, etc.)

• Health monitor does not maintain historic data• Health monitor interval not configurable• Toolkit scripts customizable and extensible

41

41

Questions?

42

42

Jayesh ThakrarThe Warranty Group

[email protected]

Session: D12

Case Study: Home-Grown DB2 Monitoring and Related Tools

Home-Grown DB2 Monitoring and Related Tools

Documents

Transcript of Home-Grown DB2 Monitoring and Related Tools