CON6018 SriVuyyuru at OOW 2015 Final

8/17/2019 CON6018 SriVuyyuru at OOW 2015 Final

1/33

CON6018 - Best practices while maintaining large Oracle

Case study of a Big Box Retailer

Sri Vuyyuru, Senior Consultant

Oracle Open World 2015


2/33

Agenda

• About BIAS

• Case study background

• Problems faced with Large Oracle footprints

• Best practices Managing Large Oracle footprints

–

Patching Management – Data Purging strategy

– Proactive vs. Reactive


3/33

About BIAS Corporation

• Founded in 2000

•

Distinguished Oracle Leader – Technology Momentum Award

– Portal Blazer Award

– Titan Award – Red Stack + HW Momentum Awards

– Excellence in Innovation Award

• Management Team is Ex-Oracle

• Location(s): Headquartered in Atlanta; Regional office in Washington D.C.;

Offshore – Hyderabad and Chennai, India • ~250 employees with 10+ years of Oracle experience on average

• Inc.500|5000 Fastest Growing Private Company in the U.S. for the 6thTime

• Voted Best Place to work in Atlanta for 2nd year

• 33 Oracle Specializations spanning the entire stack

Who We re…


4/33

Oracle created the OPN Specialized Program to showcase the Oracle partners who have achieved expertise in Oracle product aspecialization status through competency development, business results, expertise and proven success. BIAS is proud to be speOracle products, which include the following:


5/33

Problems faced with Large Oracle footprints

• Compliance

• Incident Alerting vs. Long Term Incident Resolution• Monitoring Overload

• Monitoring like its 1999 (shell and cron)

• Communication and coordination among various

teams

• Prioritization across various application teams


6/33

• Business Impact

– Availability – Security

– Risk

– Operating costs

– Man power

– SLAs

– Efficiency

Problems faced with Large Oracle footprints Busines


7/33

Day One Common issues

•

Environment Summary -> Total of 70production and 150 non productiondatabases

• Time spent on Month # 1 : 100 to 110 hrs

per week• Total # of alerts : About 350-400 per week

• Inefficient way of database monitoring


8/33

Day One Common issues

Category Avg # of Hours spent per w

Tablespace Monitoring 40

Datafile sizing 20

Multiple Monitoring systems 20

ASMLIB Change Management 15

Database manual start/stop 10

This workload collectively represents Two and a Half FTE’s!


9/33

Best Practices: Tablespace Monitoring improv

• Issue: Lot of time spent on troubleshooting tablespalerts due to a minor mistake in the monitoring scr

• How a small change in script could make a huge din saving time & money

•

Alert!!!host1:orcl:Cluster Database : Critical : TablespaceSYSTEM is 100 percent full


10/33


SELECT F.TABLESPACE_NAME,

TO_CHAR ((T.TOTAL_SPACE - F.FREE_SPACE),'999,999') "USED_MB",

TO_CHAR (T.TOTAL_SPACE, '999,999') "TOTAL_MB",TO_CHAR ((ROUND (((T.TOTAL_SPACE - F.FREE_SPACE)/T.TOTAL_SPACE)*100)),'999') PERCENT_USED

FROM (SELECT TABLESPACE_NAME, ROUND (SUM (BLOCKS*(SELECT VALUE/1024

FROM V$PARAMETER

WHERE NAME = 'db_block_size')/1024)) FREE_SPACE

FROM DBA_FREE_SPACE

GROUP BY TABLESPACE_NAME) F,

(SELECT TABLESPACE_NAME, ROUND (SUM (BYTES/1048576)) TOTAL_SPACE

FROM DBA_DATA_FILESGROUP BY TABLESPACE_NAME) T

WHERE F.TABLESPACE_NAME = T.TABLESPACE_NAME

AND F.TABLESPACE_NAME='SYSTEM';

TABLESPACE_NAME USED_MB TOTAL_MB PERCENT_USED

------------------------------------------------------------------------------------------------------

SYSTEM 1,382 1,390 99

EXAMPLE


11/33


SELECT F.TABLESPACE_NAME,TO_CHAR ((T.ALLOC_MB - F.FREE_SPACE),'999,999') "USED_MB",

TO_CHAR (T.TOTAL_SPACE, '999,999') "TOTAL_MB",TO_CHAR ((ROUND (((T.ALLOC_MB - F.FREE_SPACE)/T.TOTAL_SPACE)*100)),'999') PERCENT_USEDFROM (SELECT TABLESPACE_NAME, ROUND (SUM (BLOCKS*(SELECT VALUE/1024FROM V$PARAMETERWHERE NAME = 'db_block_size')/1024)) FREE_SPACEFROM DBA_FREE_SPACEGROUP BY TABLESPACE_NAME) F,(SELECT TABLESPACE_NAME, SUM(BYTES)/1048576 "ALLOC_MB", ROUND (SUM ((CASE WHEN AUTOEXTENSIBLE = 'YES' T

GREATEST(BYTES, MAXBYTES) ELSE BYTES END)/1048576)) TOTAL_SPACEFROM DBA_DATA_FILES

GROUP BY TABLESPACE_NAME) TWHERE F.TABLESPACE_NAME = T.TABLESPACE_NAME

AND F.TABLESPACE_NAME='SYSTEM';

TABLESPACE_NAME USED_MB TOTAL_MB PERCENT_USED------------------------------------------------------------------------------------------------------SYSTEM 1,382 32,768 4

EXAMPLE


12/33

• Resolution: Deployment of upgraded script had cut down t

monitoring time by more than 50%• Business Impact : Less operating costs and less resource

for this monitoring moving forward


EXAMPLE


13/33

Best Practices: Datafile sizing strategy

• Issue: Improper sizing of data files leads to increased stora

for the company• Incorrect sizing during deployment of new databases

• “Lets get done with it” attitude

• Impacts other databases residing on the same server


14/33

•

Inefficient method of sizing a datafile• Autoextensible – YES vs. NO


EXAMPLE 1


15/33

•

Efficient method of sizing a datafile

Tip : Set the Maxsize to an optimum number using ‘ Autoextenoption


EXAMPLE 2


16/33

Resolution:

• Sizing - Generic vs. Application Specific

• OEM can be helpful – Use of Information Publisher growth reporegular intervals

Business Impact :

• This approach reduces additional storage costs and improves e

database monitoring



17/33

Best Practices: Leveraging OEM

• Issue: Too many monitoring systems, Improper

handling, and bad monitoring strategy for OracleEnterprise Manager (OEM)

• DBA team spending unnecessary time fieldingsame alert multiple times


18/33

Resolution:

• Importance of OEM

• Host and Database validation

• Migration of crontab scripts

• Improve metric thresholds and review often

• Business Impact : This provides a more efficientway of monitoring large oracle footprints

Best Practices: Leveraging OEM


19/33

Best Practices: ASMLIB Change Management

• Issue: Availability issues for RAC databases after update okernel versions

• About ASMLIB drivers

• Role of ASMLIB with respect to Linux

• Business Impact

– Production – Non Production

• Improvements with Oracle Linux 6


20/33

Deep Dive: ASMLIB Issue Existing workflow


21/33

Deep Dive: ASMLIB Issue Proposed workflow

Simplified process streamlines the upgrade process and red


22/33

• Resolution: Avoidance of multiple parties and cutting downfor the databases to be available for the application teams

• The oracleasm kernel driver is built into the Unbreakable EKernel for Oracle Linux 6 and does not need to be installe

• Business Impact : This approach helps to maintain SLAs amanual intervention from DBA team

Deep Dive: ASMLIB Issue Proposed workflow


23/33

Best Practice: Automate Database Startup and Shutd

• Issue: Databases unavailable for longer durations after boservers

• RAC vs. Non-RAC databases

• Manual intervention was needed for some databases

• Setup of scripts on standalone databases


24/33


• Some caveats

–

dbstart, dbshut and dbora scripts should bealways executable

– Environment variables to be set properly

– Don’t forget to set the right entries in oratab

– Be cautious while editing the startup/shutdownscripts

• Resolution: After fixing the scripts, we haveimproved the availability of databases and thusreducing operating costs as well


25/33


• Oracle restart – available starting from 11.2

Dbstart and dbshut scripts until 11g

Oracle Restart feature starting from11gR2 and beyond


26/33


• Oracle Restart feature starting from 11gR2 – Benefits

–

No scripts to deal with – CRSCTL – easy to use

– Order of startup and shutdowns can be handled


27/33

Patching Management: Compliance

• Database servers not up to date with the latest patch setupdates

• Security has been a serious concern in the last few years

• PSU – Not just fixing security vulnerabilities, but also incredatabase efficiency

•

Instance specific issues, functionality issues, regression teperformed


28/33

• Successful management of DB patching activity – Be Com

perform Quarterly PSU Patching

• Business Impact : Reduces the risk of attacks from externand thereby increases security on database front

Patching Management: Compliance


29/33


30/33

Summary Takeaways

Category Business Objective Achieved

Tablespace Monitoring Decrease in operating costs and r

Datafile sizing Reducing storage costs

Multiple Monitoring systems Improves efficiency of database m

ASMLIB change management Maintain SLAs and reduce manua

Database manual start/stop Increase high availability

Patching Management Reduce risk and improves security

Data purging strategy Decreases storage costs


31/33

Month 6: Best Practices Implemented

Category Avg # of Hours spent per we

Tablespace Monitoring 20

Datafile sizing 5

Multiple Monitoring systems 5

ASMLIB Change Management 5

Database manual start/stop 5

FTE’s Reduced from 2 ½ to 1!


32/33

Month 6: Best Practices Implemented

• Time spent on Month # 6 : 40 to 45 hrs per week (Reduce

to 110 hrs per week)• Total # of alerts : About 80-100 per week (Reduced from 3

per week)

• Increased company's performance and profitability

•

Strategy – Proactive vs. Reactive


33/33

QUESTIONS

Contact info:

Sri Vuyyuru

Email: [email protected]

LinkedIn:

www.linkedin.com/in/srivuyyuru

Work: +1 770-685-6283

Cell: +1 404-398-5360
mailto:[email protected]://www.linkedin.com/in/srivuyyuruhttp://www.linkedin.com/in/srivuyyurumailto:[email protected]

CON6018 SriVuyyuru at OOW 2015 Final

Documents

Transcript of CON6018 SriVuyyuru at OOW 2015 Final