1

CDF Taking Stock ‘08

By Anil Kumar By Anil Kumar

CD/LSCS/DBI/DBACD/LSCS/DBI/DBA

July 16, 2008July 16, 2008

2

Infrastructure

Space Usage & Capacity Planning

Backup & Recovery

Accomplishments

What’s Next

Concerns

Q & A

CDF Taking Stock ’08 - Outline

3

Machine Usage Type OS Oraclebzora1 Prd - Online SunFire V440, 4x1.2GHz, 16Gb 5.9 64bit 10.2.0.3 64bit b0dau36 Dev/Int - Online SunE450, 4x400MHz, 4Gb 5.9 64bit 10.2.0.3 64bit

fcdfora4 Int/Prd - Off Sun-Fire V880 8x900MHz, 32Gb 5.9 64bit 10.2.0.3 64bit fcdfora1 Dev/Int - Off SunE4500, 8x400MHz, 2.5Gb 5.9 64bit 10.2.0.3 64bit

fcdfora6 Prd Replica Dell PE6650, 8x2GHz, 8Gb RHEL3 10.2.0.3 fcdfra5 Dev Replica Dell2650, 4x2.7GHz, 2Gb RHEL3 10.2.0.3

CDF Taking Stock ’08 - Infrastructure

4

CDF Taking Stock ’08 – Space Usage

5

CDF Taking Stock ’08 – SAM Space Usage

SAM Space Usage

020000400006000080000

1000006/

9/20

02

10/9

/200

2

2/9/

2003

6/9/

2003

10/9

/200

3

2/9/

2004

6/9/

2004

10/9

/200

4

2/9/

2005

6/9/

2005

10/9

/200

5

2/9/

2006

6/9/

2006

10/9

/200

6

2/9/

2007

6/9/

2007

10/9

/200

7

2/9/

2008

6/9/

2008

Date

Size

(MB

)

Series1

6

ON-LINE Growth: 80-90GB /Year

Next two years CDF online expected growth 170G. 145G is available.

OFF-LINE

Growth: 50-60GB/Year Next two years CDF offline

expected growth is 120G. 612G available.

Note: SAM growth : 25-30G

CDF On-line Space Usage

0100000200000300000400000500000

12/1

9/2

000

6/1

9/2

001

12/1

9/2

001

6/1

9/2

002

12/1

9/2

002

6/1

9/2

003

12/1

9/2

003

6/1

9/2

004

12/1

9/2

004

6/1

9/2

005

12/1

9/2

005

6/1

9/2

006

12/1

9/2

006

6/1

9/2

007

12/1

9/2

007

6/1

9/2

008

Date

Siz

e(M

B)

Series1

CDF Taking Stock ’08 – Capacity Planning

CDF Offline Space Usage

050000

100000150000200000250000300000

12/2

4/2

000

6/2

4/2

001

12/2

4/2

001

6/2

4/2

002

12/2

4/2

002

6/2

4/2

003

12/2

4/2

003

6/2

4/2

004

12/2

4/2

004

6/2

4/2

005

12/2

4/2

005

6/2

4/2

006

12/2

4/2

006

6/2

4/2

007

12/2

4/2

007

6/2

4/2

008

Date

Siz

e(M

B)

Series1

7

Applic. Machine Capacity Used Rman Capacity Rman Used

CDF Online bzora1 800G 655G 939G 500G*

CDF Offline fcdfora4 1.3T 688G 3.5T 3.0T**

CDF Replica fcdfora6 1.2T 466GNo RMAN implementation

* One Copy of RMAN and archives (24GB in size on average) ** Two Copies of RMAN on NAS and the NAS is shared with other systems.

CDF Taking Stock ’08 – Backup Space Used

8

cdfonprd Daily, 5 days of archives. Allocated 939GB, Used 300GB Back-up time -> 2 Hrs RMAN backup to SAN. Also One local copy. Backups to dcache daily. No Export.

cdfofprd Daily, 4 days of archives Allocated 3.5TB, Used 2.8TB (2 copies, 342GB each), Back-up time: 4Hrs

15Mins Both copies of rman to NAS. No local copy. Backup to dcache daily.

Daily Export of SAM and Offline Apps (e.g. Books, Filecatalog and Frontier)

cdfreplica No RMAN Backup and No Export since it is replica. In case of failure, replication will be reinstantiated.

CDF Taking Stock ’08 – Backups Details

9

CDF Online Production 100% uptime CDF Offline Production 99+% uptime. ( 150 Minutes maintenance, not down but

dealing with index corruption for CachedFiles table during cleanup activity ) CDF Replica 100% uptime.

Upgraded the Oracle databases to version 10.2.0.3

OS and DB Security Patches up-to-date

Tuning of CDF online databases. (Redo tuning, Backup to SAN, API Tuning) Performed targeted monitoring/benchmark tests.

Move RMAN to NON-LOCAL Disks – Decrease Contention – Permanent Tests with SUN -> Suspected Disk Contention. New bigger disks & Distribute I/O Report on many untuned queries, API modifications to tune queries.

Deployment of tuned slow control code is pending.

Proof of product for ASO ( Advanced Security Option)

Deployed Six SAM Schama cuts

CDF Taking Stock ’08 - Accomplishments

10

Disk Capacity Upgrade on cdf online prd database. This will also help to distribute the I/O across multiple mount points and hence tuning of I/O Replacing 12 146GB with 12 300GB, ~900GB additional space, 136G dedicated

space for archives. – Planned for August Implementation.

Upgrade the Oracle databases to version 10.2.0.4 OR 11g

ASO ( Advanced Security Option) Deployment. Db on fcdfora5 was used for proof of product. More tests will be performed on b0dau36 and fcdfora1.

May consider moving cdfofdev/cdfofint databases to 3PAR storage since 2-3 incidences of block corruption on existing disks.

CDF Taking Stock ’08 - Next

11

EXPEDITE the MIGRATION of PRE-SCALE Schema to Production. Risk: Production data is hosted on dev system. No Backup. Risk of losing data in case

of failure.

Disks on b0dau36 is getting old, hard to find, not supported by SUN. Need to consider getting new storage for b0dau36.

Deputy CDF DB Liaison ?

Enforce the Integration tests before deploying the schema/APIs in production.

Single point of failure for SAM and DFC applications.

There are still externally authenticated accounts on cdfonline. Need to be converted to password.

Some of CDF application Data Models are not in designer.

CDF Taking Stock ’08 - Concerns

12

Q & A

CDF Taking Stock ’08 – Q & A