DATA CENTERS INTEGRATION PLANNING TEAM

Data Centers Integration Planning Team Report Last printed 9/2/05 10:58 AM

DATA CENTERS INTEGRATION PLANNING TEAM SEPTEMBER 2, 2005 Draft compiled by Patrick J. Lynch, Data Centers IPT Leader, from contributions by team members (see Appendix A). EXECUTIVE SUMMARY OF RECOMMENDATIONS Create a single organization with a unified reporting structure, to be called “Production Services,” composed of the current 36-person ITS Data Center organization (Database Administration, Systems, Production Services), combined with the 8-person ITS-Med Production Services unit, plus one database administrator from ITS-Med Information Systems. (See the proposed organization chart in this report for details). General recommendations for the integrated group include:

Clarifying responsibilities of Systems and those of Data Center Operations, particularly as ITS-Med staff who currently perform both systems and operations functions are integrated into the new unified reporting structure.

Creating an integrated staff orientation and cross-training program to familiarize the medical and central staffs with each other’s applications, procedures, and physical plant.

Including Oracle DBA work within one DBA group with cross-trained DBAs supporting both central and medical databases, applications and developers. Transfer all server system administration responsibilities to Systems group.

Structure Production Services to play a central role for Infrastructure Services in:

Scheduling, tracking and communicating core infrastructure changes (e.g., making sure the firewall upgrade doesn’t happen the same weekend as a major application upgrade).

Implement formal incident tracking with root cause analysis, MTF and MTR tracking for key applications, publishing of metrics. Create integrated, campus-wide capacity planning: tracking application performance and growth as it impacts TSM, firewalls, and the

network. Create a formal integration planning team for the eventual reduction of the number of Data Centers, and merging of the current Data

Center staffing, policies, and procedures. Design and implement an integrated disaster recovery program for Systems, Database Administration, and all Data Centers.

Data Centers Integration Planning Team Report

PROPOSED INTEGRATED UNIT MISSION AND DEFINITION ITS Production Services would be a unit of Information Technology Services, responsible for the operation and support of all Yale University Data Centers, and the database administration and systems administration for all University systems currently in production within the ITS Data Centers. Specific missions for each of the three major organizations within the proposed Production Systems unit: Systems The Systems group installs, tests and maintains software products and operating systems for all mainframe, Unix, and NT servers that are located in the university Data Centers. The Systems group provides technical support and consulting to Operations and other staff in ITS on a variety of projects and topics as requested. Systems installs the operating system software and associated products for all new servers that go into production, and provides support for all ITS Data Centers on a 7-day 24-hour basis as required. Database Administration The Database Administration group installs, tests and maintains vendor-supplied relational database management software and application systems for all administrative Unix and NT servers located in the ITS Data Centers. The group provides support to the Administrative Systems group, the ITS Helpdesk, and Data Center Services group who operate the ITS Data Centers. Database Administration provides support for all ITS Data Centers on a 7-day 24-hour basis as required. Data Center Services (formerly called “Production Services”) Host and provide technical support for servers and databases running critical University applications. The hosting service is in secure, environmentally controlled Data Centers with 7x24 monitoring and guaranteed off-shift technical support. Technical staff support a wide range of operating systems, database applications and layered software. We are committed to running secure, supported, stable databases and servers whose performance exceeds customers’ requirements.

Target Service Levels Cost Drivers Funding ModelsDependencies &

Collaborating Units

ITS S

taff

FAS F

aculty

Med

Ctr

Fac

ulty

Oth

er F

aculty

Cen

tral

Sta

ff

Med

Ct

Sta

ff

FAS S

tuden

ts

Med

Ctr

Stu

den

ts

Oth

er S

tuden

ts

Oth

er

# Clients FTEs

1. Central DBAs, Data Center, Systems DBAs Systems Ops*

FOR ALL SERVICES: General collaboration with Security, Technology and Planning, and Network

Services.

2. Email

99.999% Reliability 99.99% Availability, Low TCO, High Performance 20,000 NA 1.8

See Footnote

# servers/equipment, complexity full cost recovery SO, CA, EU, C X X X X X X X

3. TSM

99.999% Reliability 99.99% Availability, Low TCO, High Performance 3,700 NA 1

See Footnote

# servers/equipment, complexity full cost recovery SO, CA, EU, C, DT X X X X X X

4. Active Directory

99.999% Reliability 99.99% Availability, Low TCO, High Performance 3,000-5,000 NA 0.20

See Footnote

# servers/equipment, complexity full cost recovery SO X X X X

5. www.yale.edu and related servers

99.999% Reliability 99.99% Availability, Low TCO, High Performance lots NA 0.15

See Footnote

# servers/equipment, complexity full cost recovery AMT, SO, EU X X X X X X X

6.

Finance, HR, Payroll Applications -- incl. Oracle Applications, Data

Warehouse/Brio Portal, Kronos, Resumix, Web Methods, Coeus, OFA

and more

99.999% Reliability 99.99% Availability, Low TCO, High Performance 2,000-4,000 8.5 6.55

See Footnote

# servers/equipment, complexity full cost recovery

HD, SO, CA, DT, EU, Admin Sys, C X X X X

7.

SFAS Applications -- incl. Banner, GSPS, Ranger Loan, Online Course Information/Selection/Evaluation,

CBord Odyssey PCS (Dining), Imaging

99.999% Reliability 99.99% Availability, Low TCO, High Performance 5,000-10,000 1 1

See Footnote

# servers/equipment, complexity full cost recovery SO, Admin Sys, C X X X X X X

8.AMT Applications -- Sakai/Classes

v2, Classes v1, Jive, uPortal

99.999% Reliability 99.99% Availability, Low TCO, High Performance 5,000-10,000 0.10 0.40

See Footnote

# servers/equipment, complexity full cost recovery AMT, SO X X X X

10. BERT and Benefactor

99.999% Reliability 99.99% Availability, Low TCO, High Performance 1,000? 0.15 0.40

See Footnote

# servers/equipment, complexity full cost recovery Admin Sys, SO X X

12. FAMIS

99.999% Reliability 99.99% Availability, Low TCO, High Performance 800-900? NA 0.25

See Footnote

# servers/equipment, complexity full cost recovery Facilities, SO X

12.YUHS Applications -- incl. IDX, All

Scripts, more

99.999% Reliability 99.99% Availability, Low TCO, High Performance 800-900? NA 0.25

See Footnote

# servers/equipment, complexity full cost recovery YUHS, Admin Sys, SO X

Current Major Applications (Pre-Integration)

Inventory of Primary Services, Applications, and Activities

CENTRAL DATA CENTER AND SYSTEMS GROUP

Service, Application, Activity

Clients (Check those who apply)

Key Metrics

12. Miscellaneous Other Applications

99.999% Reliability 99.99% Availability, Low TCO, High Performance lots 0.25

See Footnote

# servers/equipment, complexity full cost recovery Admin Sys, SO X X X X X X X

*Note: Operators' time is distributed across all servers and all supported applications in the Data

Center.

1. Central Data Center

2.

Management of Data Center facilities -- UPS/electrical, HVAC,

security

Adequate power and cooling, with capacity for

growth

350+ servers and related equipment

(disk/tape/firewalls)

# servers/equipment, complexity full cost recovery Facilities X X X X X X X

3.

24*7 monitoring application availability, troubleshooting and follow-up for applications and

servers 100% availability% availability,

MTF, MTR# servers/equipment,

complexity full cost recoverySystems/DBAs/Admin.

Systems/AMT X X X X X X X

4.

Equipment procurement, inventory, setup, installation,

startup/shutdown procedures,etc.Installation complete by

target deadline.

Elapsed time to install and

setup servers# servers/equipment,

complexity full cost recoverySystems/DBAs/Admin.

Systems/AMT X X X

5.

Monitor nightly TSM backups, restart backups as needed, vault

tapes 100% backups complete

% backups complete within

window# servers/equipment,

complexity full cost recovery Systems/DBAs X X X

6.

Respond to hardware failures, diagnostics, contact and follow up

with vendor.Maximize Mean Time to

Resolution (MTR) MTR# servers/equipment,

complexity full cost recovery Systems/DBAs X

7.Manage weekly Iron Mountain off-

site backups for select servers 100% backups complete

% backups complete,

elapsed time# servers/equipment,

complexity full cost recovery TSM Admin X

8.

Monitor payroll processing, manage printing of payroll checks and advice

depositsProcessing completes

within timeframe allowed

Elapsed time for payroll processing

# servers/equipment, complexity full cost recovery

Systems/DBAs/Admin Systems/Payroll Dept. X X

10. Central Systems

12.

Install and configure servers and layered software (TSM, BB, Argent, NAV, Apache, Tomcat) and ready for

production (filesystems, users, backup, security, procedures,

monitoring)

System available to application programmers by designated deadline, robust, reliable and high

performing system.

Deadline met, % availability,

cust sat# servers/equipment,

complexity full cost recoveryOperations/DBAs/Admin

Systems/AMT/T&P X

Current Major Services, Applications, Activities (Pre-Integration)

13.Troubleshoot and resolve problems

(hardware, software, other).100% availability within

SLA levels

Maximize MTF, Minimize MTR, % availability,



Systems/AMT X

14.

Support application developers and database adminstrators in installing and configuring complex application

environments.

Rapid response, resources available as

needed, ablility to get it right the first time




Systems/AMT X

15.

Manage Storage Area Networks (SAN), EMC Symmetrix, IBM Shark,

EquaLogic iSCSI

Provision LUNs rapidly and correctly, advise customers as needed,

optimize SAN performance


cust sat

TB storage, # hosts connected, IO throughput,

complexity of technology full cost recovery Operations X

16.

Manage TSM Central Backup servers, tape libraries and daily

processing.

All backups complete within window, client backups optimized to reduce app/desktop downtime, efficient processing practices

Maximize MTF, Minimize MTR,

% availability,cust

sat

TB storage, # hosts connected, IO throughput,

complexity of technology full cost recovery Operations X X X

Manage Email infrastructure supporting 18-20,000 users.

Complex assortment of servers running open source sendmail and Imp (web mail), spam assassin,

clamAV.Robust, reliable, high

performing


fast performance,

cust sat

volume of email including SPAM, TB

storage, IO throughput,

complexity, evolving technology full cost recovery

Operations/Support/AMT/CIO X X X X X X X X

Manage Yale's Active Directory infrastructure, including WINs.

Multiple, redundant domain controllers with directory replication

and complex roles distributed across servers.

Robust, reliable, high performing


fast performance,

cust sat

# users, specialized departmental requirements,

evolving technology full cost recovery Windows Gurus/Operations X X X X X X X X X X

17.Strategic Planning: track emerging technologies and advise customers

Reliable, supportable technologies, reduced

TCO

Equipment expenditure per customer, trend over time, cust

sat# servers/equipment,

complexity full cost recovery Admin Systems/AMT X X

18.Lifecycle equipment management

and capacity planning

Reliable, supportable technologies, reduced

TCO

Equipment expenditure per customer, trend over time, cust


complexity full cost recovery DBAs/Admin Systems/AMT X X

19. Performance monitoring and tuning

High Performing applications, partnership with appln programmers on performance issues

System statistics re:

resource consumption,


complexity full cost recovery DBAs X X

20. Disaster Recovery Planning

Identify Priority A applications, update

plans as needed, annual tabletop test, workload move tests scheduled

with the customer

Balance DR requirements

with TCO, auditors, cust


complexity full cost recovery DBAs/Admin System X X X

Inventory of Primary Services, Applications, and ActivitiesClients (Check those who apply)

ITS-MED DATA CENTER AND PRODUCTION SYSTEMS GROUP

Target Service Levels

1-Highest Level5-Lowest Level* = formal SLA

Cost Drivers

E-environmentH-hardwareS-softwareF-FTEM-maintenanceSC-scopeA-All

Funding Models

Periodicity - Monthly

Dependencies & Collaborating Units

ITS S

taff

FAS F

aculty

Med

Ctr

Fac

ulty

Oth

er F

aculty

Cen

tral

Sta

ff

Med

Ct

Sta

ff

FAS S

tuden

ts

Med

Ctr

Stu

den

ts

Oth

er S

tuden

ts

Oth

er -

YM

G

Clients FTEs

1. ITS Med Production Services DBAs Ops Systems

1. IDX 1 * 1800 4.5 A YMG Pays CA, SO, HD X X X X X

Medicode

Exec Reporting

Patient Service

Patient Online

ICWeb

2. MEDDM1 - Datawarehouse 2 650 3.25 A Departments Pay CA, SO, HD X X

CRS

IDX

Miscellaneous Applications

3. StemSoft 2 50 0.25 A Dept Pays CA, SO, HD X X

4. Remedy 1 20 1 A CATS Pays CA, SO, HD X

5. FMT (SRCIA) 1 1 A Financial Office Pays CA, SO, HD X

6. YSM1 - OLTP System 1 925 7.25 A Departments Pay CA, SO, HD X X X X X Transplant System Training Management HIC Miscellaneous Applications BMS Security System7. Empac - Facilities 1 165 1 A Dept Pays HD,SANE X8. Document Imaging 2 500 0.35 Dept Pays DT, Dept Liasions X

IDX Financial Ops Dermpath Obgyn IACUAC

9. YCMI 2 * 100 0.25 A Dept Pays SO, SANE X X10. Tivoli 2 1400 1 A EU - optional CA, DT X X X X11. Oracle Application Servers 1 1500 0.25 A Departments Pay CA,SO.HD X X X X X12. File Services 1 0.2 A Eu, Dept - Optional CA,DT X X X X13. Animal Care 1 211 0.5 A Dept Pays HD X X14. Human Resrouce Medical Sch. 1 371 1.5 A Dept Pays CA, HD X15. Antivirus/Spyware 4 7000 0.2 A EU, Automatic DT,HD X X X X X X X16. Email 1 * 7000 3.6 A EU, Automatic CA,HD X X X X X Student

Staff17. Infrastructure 1-4 2 SANE,CA,HD X X X X X

WINS PDC/BDC DFS CAS License SUS FTP Servers Big Brother

VPN Netvigil

Current Major Services, Applications, Activities (Pre-Integration)

Service, Application, Activity Key Metrics

SAN Admin PPP18. Data Center Management Internal X X X X X HVAC & Service Contract 1 * DC X X X X X Power utilization/tracking 1 DC X X X X X Access to Data Center 2 DC X X X X X IIS Monitoring Services 3* DC X X X X X 24x7 Big Brother / Pager 1 DC X X X X X Network Services 1 DC X X X X X Maintenance Contracts ITA X X X X X Hardware 1 * X X X X X Software 1 * X X X X X Cleaning 4 * X X X X X Inventory of Equipment 3 DC X X X X X Outage Notification Process 2 X X X X X Turnover Procedures 3 X X X X X Installation of New Hardware 3 DC X X X X X Off site backup (archive one) 2 * ITA X X X X X Server Hosting Services C X X X X X Internal 2 * X X X X X External (departments) 2 * X X X X X HelpDesk Support for IDX 1 YMG X X X X X Purchasing ITA X X X X X Supplies 4 X X X X X Hardware 3 X X X X X Software 3 X X X X X Purchase Approval 2 X X X X X Budget Services to IS & SANE X X X X X Hardware 4 X X X X X Software 4 X X X X X MTCE Contract Support 2 X X X X X Cross Training 3 DC X X X X X

Keys:HD = Help DeskSO = Security OfficeCA = Client AccessDT = DesktopEU = End UserDC = Data Center CostsITA= IT All - charged back to other IT DepartmentsC = Customer paysFTE count includes IS developers plus one DBA. DBA FTE % breakdown as follows:

.07 System Admin

.08 Database Admin

.005 Hardware supportFTE count includes Data Center Staff, one IS developer and one DBA. DBA FTE% breakdown as follows:

.05 System Admin

.02 Database Admin

Target Service Levels Cost Drivers Funding ModelsDependencies &

Collaborating Units

ITS S

taff

FAS F

aculty

Med

Ctr

Fac

ulty

Oth

er F

aculty

Cen

tral

Sta

ff

Med

Ct

Sta

ff

FAS S

tuden

ts

Med

Ctr

Stu

den

ts

Oth

er S

tuden

ts

Oth

er

Clients FTEs

0. DATABASE ADMINISTRATION

DATABASE ADMINISTRATIONInventory of Primary Services, Applications, and Activities

Clients (Check those who apply)

Service, Application, Activity Key Metrics

Current Major Services,

1. Major Services

2.

Management/Support of Production Applications (Student, Faculty, Administrative) which includes monitoring (Veritas, scripts, Session Alerts), tuning, troubleshooting, code migrations (Kintana, manual and Yale owned systems). This may require coordination amongst many groups; Admin Systems, Systems, Operations, Data Network Services and Security/Firewall groups.

100% availability during non-maintenance windows

complexity, importance of application full cost recovery

DBAs/Systems/ Sysadmins/Admin Systems/DNO/Security x x x x x x x x x x

3.

Based on application requirements, provide backup and recovery, and disaster recovery capabilities using; SQL*Backtrack, scripts, OS/EMC copies, etc 100% accurate

size and number of databases, requirements of apps full cost recovery DBAs/Systems x x x x x x x x x x

4.

Database/Environment Creation/Cloning using; SQL*Backtrack, scripts, OS/EMC

complete when customer requires

size and complexity of environment full cost recovery DBAs/Sysadmins/Systems x x x

5.

Core Technology upgrades/patching; includes but not limited to: Oracle products, Web Application Software, 3rd party Applications, Yale owned/developed systems

based on vendor and user availability and requirements

number of patches and environments to patch full cost recovery DBAs x x x x x x x x x x

6.Assist and Advise Developers and their users as needed

complexity, importance of application full cost recovery

DBAs/Sysadmins/Developers/Users x x x

7.

Research new versions or replacements of existing software/applications and make suggestions to customer as needed

complexity, importance of application full cost recovery DBAs/Systems/Developers x x x x x x x x x x

8.

Define and enforce standards for databases, database objects and code strive for 100% accuracy general full cost recovery

DBAs/Developers/Sysadmins x x

9.

Install, upgrade, integrate and troubleshoot 3rd Party Application Software which includes but is not limited to; Oracle, Autosys, Classes, COEUS, Envision, Hyperion/Brio, Jive, Kintana/Object Migrator, Kronos, Load Runner, SQL Server, Nettracker, Odyssey PCS, OFA, Oracle Financials, Perfect Tracker, Pinnacle, Ranger, Remedy, Resumix, RT, Sakai, SCT/Banner, SQL*Backtrack, Veritas, Web Methods, EMPAC, StemSoft, BlackBoard, SRCIA (FMT) as required

complexity, importance of application full cost recovery DBAs/Developers/Systems x x x x x x x x x x

12.

Provide Database Environment Load Test capabilities in order to achieve/maintain service levels when requested general full cost recovery DBAs/Developers/Users x x

13.Administer DBA group work/activities/libraries/reports as required general full cost recovery Sysadmin

14.Specifications/Ordering of Hardware to house Databases when requested general full cost recovery

DBA/Admin Systems/Systems x x x x x

15.

Installation and maintenance of OS and related operating systems software as needed general full cost recovery

DBA/Admin Systems/Systems x x x x x

16.

Installation and maintenance of Web Server software; ie Apache, Tomcat when requested general full cost recovery

DBA/Admin Systems/Systems x x x x x x x x x x

17. Maintenance of ITS DBA Website as needed general full cost recovery Sysadmin x x x x x

14. Would be a Systems responsibility x x x x x

15. Would be a Systems responsibility x x x x x

16.

Would continue to be a shared responsibility depending on the needs of the application software x x x x x x x x x x

17.Could stay as is or go under web group x x x x x

Proposed Changes In Services (correspondong to the services

Role Essential Duties Skills & abilitiesNumber of staff needed

Database Administration Manager • Supervise, coach, train database administrators and system administrators• Provide production support• Liason between Database Administration and other technical groups (systems, operations, developers and functional users)• Manage upgrades, security and functional patch applications, migration of code to many Oracle databases on many machines• Manage work that gets sent into our department and disperse it to appropriate personnel• Production support; monitoring of databases, setting and enforcer of standards• Keep up with technology and my staff (conferences, classes, books, etc)

* Communicate and maintain ability to retain focus on "the big picture" of a project and its key deliverables.* Define and managed the short term and long term strategy of the DBA group.* Address impact issues to management.* Define task and activities for the DBA group during development life cycle and production cycle.* Drive the future direction of DBA group.* Define/architect/design of the databases.* Gathers information to define strategy and requirements for release of the application and databases.* Manages team projects and deadlines. * Ensures that the team is trained.* Provide cross training for related jobs to ensure that the DBA group has coverage. * Can provide appropriate support for employees. * Helps each team member with priority planning. * Manages Technical Database Specialists* Hires and trains new staff. * Develops key performance measures for team. * Improves processes procedures in an attempt to operate efficiently. * Communicate with functional and development teams* Multi-task

1Database Administrators • Provide production support

• Liason between Database Administration and other technical groups (systems, operations, developers and functional users)• Manage upgrades, security and functional patch applications, migration of code to many Oracle databases on many machines• Production support; monitoring of databases, setting and enforcer of standards• Keep up with technology (conferences, classes, books, etc)* R&D new products* work with many developers in different periods of develepment; performance, design, troubleshooting, etc

A thorough understanding of database systems and the commands/utilitites including but not limited to SQL and PL/SQL and the ability to apply this knowledge in an independent way. A thorough understanding of relational concepts and modeling techniques. Comprehensive knowledge of backup and recovery scenarios, PC and server processes and operations and the ability to make recommendations. Comprehensive knowledge of one or many operating systems and the ability to script. Interpersonal and communication skills. Able to train others through technical writing and oral presentations. Ability to work individually and within a team. Instructs, guides, and evaluates the work of other technical staff. Performs complex problem solving. Develops solutions to technical problems referred by other staff members. Develops training sessions for the user community. Manages production databases to ensure a continuous high level of performance and availability. Contributes to the management and enhancement of the RDBMS environment, including database standards and procedures, database tool support and training.

7

Database Administrators-Central Group

DATA CENTERS STAFF ROLES

Sysadmins * Interact effectively with Application developers and the DBA group.* Ability to manage multiple systems, very good organizational skills.* Have the ability to self manage tasks of the application administration.* Application Security: To be able to identify the proper forms displayed for the defined application roles. Monitor various security reports.* Identify the user profile options and processing requirements.* Define and Maintain Concurrent Managers to Accomplish Application On-line and Batch Requirements.* Configure Peripherals within the Applications.* Maintenance of DBA activities* Maintenance ofCD/Book library * Maintenance of Space reports* Budget Office OFA administration* Maintenance of DBA Website* Testing of Microsoft Patch Testing in F&A database environments* Administration of Brio Scheduler environment* Session Alert maintenance

* Service-minded and self-motivated.* Ability to work in teams including those which cross organizational boundaries. * Ability to work independently to accomplish goals and also as a team member to be able to incorporate the tasks of an individual into a larger team specification.* Troubleshoot problems.* Excellent communication and organizational skills. Analytical skills. * The ability to learn technology and procedures rapidly.* Understanding of basic computer client server components.* Participate in meetings with central and departmental administrative users to determine system design.* Assist in developing system requirements and programming specifications.* Experience with Oracle Applicatons

3

Systems Roles-Central

Role Essential Duties Skills & abilities

Number of staff needed

Manager, Systems

* Supervise, coach and develop Unix and Windows systems programmers.* Align staff to support changing workloads and to transition to new technology.* Coordinate support, trouble-shooting and planning for applications.* Identify new software, hardware and/or processes that will improve application delivery.* Provide strategic planning and guidance for complex technical projects, including major capacity upgrades. * Assist staff with setting task priorities and coordinating day-to-day work.

* Detailed knowledge of multiple operating systems and of advanced computer technology. * Demonstrated interpersonal, project management and managerial skills.* Excellent oral and written communication skills.

1 Manager

Systems Programmer * Install, configure, document and maintain operating systems and associated software.* Consult and collaborate with vendors and with staff in other ITS departments on issues and problems.* Monitor and address problems with performance, reliability and security.* Design and implement tools for maintenance, monitoring and capacity planning.* Track developing technology and recommend implementation where appropriate.

* In depth knowledge of relevant operating systems and modern systems administration practices (e.g., automated install and configuration management).* Knowledge of host based security best practices and tools.* Proficiency in relevant administrative scripting languages.* Creative problem solving and debugging skills.* Ability to work independently on complex and/or multiple projects.* Excellent communication and organizational skills.

4 AIX/Linux3 Windows/AD2 Email1+bkup TSM

Data Center Roles-Central Data Center

Role Essential Duties Skills & abilitiesNumber of

staff needed1. All Operations Staff-see attached job interview(Manager,Tech. Coordinator, Shift Supervisor, Shift Coordinator, Operator, Office Assistant III)

Monitor and maintain Data Center temperature and humidity controls. Maintain cleanenviorment which includes vacuuming components. Ensure all visitors sign in and out onsign- in sheet.

Notify Physical Plant Controlls Center and/or Liebert when necessary to remedy problems.Knowledge of visitors' purposes.

All staff perform all duties listed

1. All Operations Staff-see attached job interview(Manager,Tech. Coordinator, Shift Supervisor, Shift Coordinator, Operator, Office Assistant III)

Continuous monitoring of Liebert UPS, TSM, Big Brother, Rover and Argent Guardian.How to use and apply information given by these applications. All staff perform all duties listed


Proper use of procedures supplied by Systems Staff. Check to see that everything that isbrought up is monitored.

Maintaining manuals to ensure current version in use. Precise attention to details All staff perform all duties listed


Run TSM backup jobs and ship the monthly tapes off site and store MKSYSB tapes is otherlocal Data Center.

Close monitoring of TSM jobs. Keeping records of off site tape shipments. All staff perform all duties listed


Create job schedules on Excel, follow schedules nightly to run jobs. Secondary verificationof all schedules each morning.

Knowledge of autosys and all jobs' requirements to complete successfully. All staff perform all duties listed


Report machine failures; take appropriate action to remedy problems. All service calls arelogged and status is reported in the shift Turnover report.

Contact vendors, arrange all aspects of repairs, notify Systems staff. All staff perform all duties listed


Coordinate necessary events regarding planned or unplanned outages. Contact vendors, arrange all aspects of maintenance outages, notify Systems staff. All staff perform all duties listed


Report daily events to next shift; send report to opinfo-l subscribers daily. Follow up onon Pending issues to ensure all problems are resolved.

Technical written communication skills. All staff perform all duties listed


Take action on requests from systems staff - e-mail, phone, change forms, etc. All staff perform all duties listed


Take installation projects to completion - including power,space, network requirementsand make updates to all documentation accordingly.

All staff perform all duties listed


Print payroll, verify counts and re-verify when pickup takes place with Payroll staff. Troubleshoot printer problems. All staff perform all duties listed

ITS-Med Data Center Staff Roles

Role Essential Duties Skills & abilitiesNumber of staff needed

Operator Batch processing for IDX, including claims, billing, scheduling and Typing skills, troubleshooting skills, PC skills, good interpersonal skills 2specialized interfaces. Knowledge of AIX & IDXTelephone support for IDX passwords and printer operation Knowledge of YMG business operationsMonitoring of multiple systems.

Manager of Operations Supervision of operators & Jr. System Admistrator for YMG services Troubleshooting & problem solving skills, Good communication skills 1Production scheduling development & setup Knowledge of AIX & IDX & MPEiXSecurity officer for YMG applications Knowledge of YMG business operationsWork with YMG & vendors testing for upgrades and problem resolution Supervisory skillsFormulate and implement operations procedures Good communication skills with clients, vendors and staffSee that IDX processing is done correctly and on time Attention to detail

System Administrator Configuring, ordering and installation of hardware Knowledge of hardware 2Installation and configuration of OS and utility software such as Knowledge of operating systemsantivirus and spyware Good communication skillsInstallation of selected applications as requested Networking skillsTesting and troubleshooting of hardware, software and some apps. Availability 24 x 7Work with 3rd party vendors for hardware and softwareBackup for operations staff

Sr. Systems Software Engineer Tivoli support, including OS, hardware and application Knowledge of hardware 2Script writing Knowledge of operating systemsManagement of SAN technology Good communication skillsTesting, review and installation of new software Knowledge of TSM and SANCordinate and implement patches and upgrades Script writing using a variety of languages

Strong knowledge of application workflow

Associate Director Budget planning Knowledge of budgeting and finance 1Setting goals and objectives for department Knowledge of Yale's business procticesBackup for Mgr of Operations and security applications Good communication skillsContract negotiations with vendors Good supervisory skillsDevelop SLAs for client services Good understanding and working knowledge of applicationsResponsibility for datacenter and backup datacenter security Up to date knowledge of hardware, software and datacenter infrastructureStaff supervision Marketing skillsCollaborate with Executive Director of YMG based on SLA agreementPurchasing approvalStrong bussiness relationship with YNHHMarketing applications to Yale community

SERVICES COMPARISON SUMMARY—CENTRAL AND MEDICAL AREAS

Notes:

ITS- ITS-Service Component Central Med Notes

Full Cost Recovery:Costs vary based upon service options selected X X Plant Operations:Environmental Controls (temperature, humidity) X XFire Alert & Suppression System X XPhysical Security & Access Control System X XUPS X XBack-Up Generator X ITS-Med - Primary data center onlyPlant Maintenance X XRedundant cooling capacity X X 2 30 Ton Units in main data center only

Client Support:Business Hours On-Site Support X X 7:00 AM - 8:00 PM, Mon - FriNight and Weekend On-Site Support (24*7) XOff-Hours On-Call Support - 24 x 7 x 365 X X Client Access (for departmental servers) X X ITS-Med - by appointment during off-hoursPost updates to Systems Status Website X

Security:Multi tier firewalled subnets XVPN Server X XVLAN X XGuest Sign-In Procedure X X

Enterprise System Management: Operating System Administration X XHardware System Administration X XHardware Procurement & Receiving X XHardware Maintenance Contracts X XWeb Server Administration X XLayered product support X eg Tomcat, MS Sql Server, MySql, etc.Operations Management X XData Back-Up & Restore Service X X See detailed Server Back-Up Services SummarySystem Monitoring Service - Interactive X X System Monitoring & Paging Service - Automated X XVendor Management Support X XLifecycle systems capacity planning XChange Management XTrouble Ticket Tracking (RT) XPerformance Monitoring and Tuning XCisco Content Services Switch (CSS) Load Balancing X X For IDX FlowCast applications onlyStorage Area Network (SAN) with remote mirror XStorage Area Network (SAN) snapshot copies X

Departmental System Management:Operating System Administration X Hardware System Administration XHardware Procurement & Receiving XHardware Maintenance Contracts XWeb Server Administration XOperations Management XData Back-Up & Restore Service X See detailed Server/Client Back-Up Services SummarySystem Monitoring Service - Interactive XSystem Monitoring & Paging Service - Automated XVendor Management Support XApplication Hosting XDatabase Hosting XApplication & Database Software Licensing XCentralized File Sharing (Work-group computing) XFTP Service XDocument Imaging Service X

Departmental Services are services offered to individual academic and adminstrative departments that do not have systems that are supported as part of "Enterprise system Management" services.


Notes:



Video Media Service XE-Sign Service XCitrix Server Services XDepartmental Server Hardware Space Rental Program XDepartmental Server Hardware Leasing Program X Departmental Server Software System Administration Program XOff-site Tape Rotation Service X

Disaster Response:Primary and secondary site data centers X X Failover to dev/test servers at 2nd site XOff-site Back-Up Tape Rotation X X ITS-Med - for systems not using TivoliIron Mountain Off-site Backups XArchive One Off-site Backups X

Data Center DBA Services Summary

Notes:


Full Cost Recovery:Costs vary based upon service options selected X X Client Support:Business Hours On-Site Support X X Mon-Fri 7:30am - 5:30pmOff-Hours On-Call Support - 24 x 7 x 365 X X Session Alerts in Oracle Financials, DW X 24hrs/day 7day/wk

Security Patching/Testing:Oracle Software; iAS, RDBMS, etc X X as neededTesting of MS Critical Updates Patches for F&A X by requestOS specific security patches - for DB hosts X as needed for AIX and Linux hostsRequest/Install up-to-date Verisign Security Certificates X Oracle and IIS web servers

3rd Party Software Maintenance/Upgrades doesn't include Yale developed softwareOracle RDBMS, iAS, Developer, Designer, Reports, etc X XOracle Names/Oracle Internet Directory XAutosys XClasses XCOEUS X XEnvision XHyperion/Brio XJive XKintana/Object Migrator XKronos Time Keeper XMicrosoft SQL Server X XNettracker XOdyssey PCS/Dining Hall SoftwareOFA XOracle Financials (hr, pr, gl, ap, po, ar, fa, etc) XPerfect Tracker X XPinnacle XRanger XRemedy X XResumix X


Notes:



RT XSakai XSCT/Banner XSQL*Backtrack (BMC) XVeritas/Precise XWeb Methods XEMPAC XStemSoft XBlackBoard XSRCIA (FMT) XApache XTomcat X

Production Code MigrationKintana XManual systems X XMS Access Program X custom in-house move to production program

MonitoringOfferings

Veritas Performance Monitoring Xadhoc use, scheduled alerts, historical reportinginlcudes; database, webservers, client, etc

Oracle Enterprise Manager XTablespace Usage, Chaining, etc X scripts run twice daily through crontab, monthly ts reportsOracle Applications Specific X jserv monitoring, number of users/hrCustomized Web Applications X http server system usage metrics and module metricsAIX/Linux Error reports, disc space X scripts run twice daily

MiscellaneousOracle CD/Book Cataloging XMaintenance of DBA Web Site XApplication Security Administration X user account setup via custom security system for ITS-MED apps

System BackupsBMC SQL*Backtrack X hot, cold, exports, archive log maintenanceOracle Exports X X by requestOracle Database Copies/Duplication (OS or EMC) X X by requestOS customized scripts X hot, cold, exports, archive log maintenance, alert log maintenanceOracle object recovery X X by request

Database Hosting Services create, maintain, and upgrade databases for Yale clients outside ITSYCMI (10 databases) X

OS System Admin Services for DB Hosts AIX, Linux, and Windows servers for Oracle and Sql Server hostsHardware and OS installations, maintenance, and upgrades XMonitoring and resolving hardware issues XProgramming as needed for system processes X

Application Developer SupportAssist and advise IS application development staff X help developers on a variety of complex technical issues

Renamed from“Data Center”

Renamed from“Production Services”

Exact structure and reportingof one ormore positions to be determined

Rossible new sub-grouppending Core Services IPT report

Possible change advocated bya few IPT members;

see Appendix A

DirectorJoseph Paolillo

INFRASTRUCTURESERVICES

Director

PRODUCTIONSERVICES

Director

TECHNOLOGY& PLANNING

Manager

DATABASEADMINISTRATION

Director

IT SECURITYSERVICES

Director

NETWORKSERVICES

Manager

SYSTEMSADMINISTRATION

Manager

DATABASEADMINISTRATION

ORACLE

Database Administrat ion Group Emai l Technical Group Systems Administrat ion Group Data Center Operat ions Group YMG Ser vices

ORACLE

ORACLE

ORACLE

ORACLE

ORACLE

EMAIL

TSM/UNIX/LINUX

EMAIL

EMAIL (Open)

.5 FTE

TSM/UNIX/LINUX

DATACENTER SERVICES

UNIX/LINUX

UNIX/LINUX

UNIX/LINUX

UNIX/LINUX

WINDOWS

WINDOWS

WINDOWS

Sys. Admin.

Sys. Admin.DBA

DBA

DBA

DBA

DBA

DBA

DATA CENTER

DATA CENTER

DATA CENTER

DATA CENTER

DATA CENTER

DATA CENTER

Supervisor

ITS-Med staff

ITS staff

Open position

Each box represents a full-timestaff member except as noted.

KEY

Supervisor

DATA CENTER

Office Support

DATA CENTER

Technical Coord.

Supervisor

Computer Oper.

Computer Oper.

IDX for YMG

IDX for YMG

IDX for YMG

IDX for YMG

System Analyst

Oper. Manager

Computer Oper.

Computer Oper.

Computer Oper.

DATA CENTER

Computer Oper.

DATA CENTER

DATA CENTER

DATA CENTER

Shift Coord.

Shift Coord.

Shift Coord.

DATA CENTER(Open)

Shift Coord.

ORACLE

APPLICATIONS

P. Lynch, 8-16-2005

APPLICATIONS

APPLICATIONS

DBA

Sys. Admin.

Sys. Admin. Sys. Admin.

Sys. Admin.

Sys. Admin.

Sys. Admin.

Sys. Admin.Sys. Admin. Sys. Admin.

Sys. Admin. Sys. Admin.

Sys. Admin.

LINUX (Open)

.5 FTE

Sys. Admin.

Sys. Admin.

Sys. Admin.

PRODUCTIONS SYSTEMS ORGANIZATION

Possible consolidationof email technical support,

pending Core ServicesIPT report?

WIN/UNIX/LINUX

[Management TBD]


QUICK WINS General wins Increased communications and team work between ITS-MED and Central IT. Cross-training

• Cross training of ITS-Med and Central ITS production services. (Possibly done by spending 1 day a week at other location so you can “wear the other IT shop’s shoes.”).

• Training all relevant staff in HIPAA security and privacy regulations and policies. • Cross-train TSM system administrators to provide secondary TSM support for each other. • Provide central operations staff with access and training to allow vendor and support staff access to YSM Data Center off-shift, to

assist with monitoring and supporting applications and servers off-shift, to assist with TSM backups and TSM server support off-shift. Coordination and common standards and practices

• Schedule coordination. Scheduling, tracking and communicating core infrastructure changes (e.g., making sure a firewall upgrade doesn’t happen the same weekend as a major application upgrade).

• Formal incident tracking with root cause analysis, MTF and MTR tracking for key applications, publishing of metrics. • Capacity planning: tracking application performance and growth as it impacts TSM, firewalls, network. • Disaster recovery planning. • Auditor readiness. • Review NetVigil capabilities and consider for use in monitoring of central apps/servers. • Review automated scripts in use at Med and consider for use in mgt of central apps/servers. • Review hardware and software maintenance agreements to identify areas of overlap and leverage. • Review equipment life cycle analyses, identify equipment likely to be decommissioned and equipment likely to be purchased for any

savings. Database Administration

Share on-call support • Become acquainted with each other’s databases/technologies so that on-call support can be shared amongst team.

Integration of backups, monitoring, and production code migration

• Identify things we do the same and differences in order to determine possible integration where possible in the areas of backups, monitoring, production code migration.

Systems Administration

• TSM technical expertise can be consolidated. • Knowledge sharing on system administration best practices can begin immediately. • ITS Central can learn from and potentially implement Data Center automation and monitoring used by ITS-Med. • Combine our SAV Ant-Virus services.


Data Center Services

• Off-site backup storage vendor. • Consolidate COEUS systems. • Combine the ordering of Data Center supplies (e.g. tape, paper etc.). • Cleaning: Having the same vendor clean both Data Centers. • Cable management: Adopt same physical practices. • Since the Central Data Center is manned 7/24, consolidate and centralize on-call and help desk (pager) coverage

information/practices.

• Joint planning effort for the design and building of a new main Data Center: o New hardware that might be purchased should be housed in a new Data Center location. The central Data Center would need

to plan for housing high performance clusters that require large power loads and dense air-conditioning. o Possibly turn the ITS-Med Data Center into a backup Data Center. o If possible, standardize on the same hardware brands and models (e.g. 3494, LTO, Shark, etc.). o Consolidate and/or reorganize BigBrother systems, including help files and on-call links. o Inventory- Central Data Center to look at Medical Data Centers ‘Computer Service System’ program for tracking hardware. Is

it a better method than using Excel?

Develop common standards for: o Data center infrastructure and base services (ie supplies, power, cabling). o Hardware purchases, combine license and maintenance contracts. o Use of Big Brother for all servers. o Common hardware purchase standards. o Tracking the hardware and software we support in the Data Centers. o Managing service contracts. o Design new Data Center for conventional, high-density, and cluster computing.


IMMEDIATE ISSUES OR BARRIERS TO INTEGRATION General concerns

Continuity of services to all clients, at current service levels, has to be the number one priority. As soon as is practical we should make the policy, personnel, and training changes necessary to get beyond the psychological and

geographic separations of “central” versus “medical” operations so that we begin to behave as a truly integrated single unit, albeit with widely scattered physical locations for the next few years.

There is a strong consensus among the group that cross-training between ITS-Med and ITS should start as soon as possible. Learning each other’s functions and roles will be critical to both our social and functional integration.

UNANSWERED QUESTIONS General questions What is the future of the ITS-Med Systems and Network engineering (SANE) unit, with particular regard for the role SANE plays in email support? We hope that the Core services IPT will make recommendation on this area. We’re also hoping that the Core Services IPT report will help clarify the various roles in support of core services like e-mail, MeetingMaker, and network backup services. After all the relevant IPT reports are in we need to look at the various roles of functional ownership, direct user support, and technical responsibility for systems, and how those roles might be best organized for optimal efficiency and improved service to clients. Staff integration and clarification of roles and responsibilities How do we assign new roles to the ITS-Med staff who, largely because of the smaller organization, often perform a number of duties that cross over organization chart boundaries (DBA, systems administration, Data Center functions)? Will people be forced to chose a single, more specialized role for themselves, or can we work out ways (at least in the short run) to avoid forcing staff to immediately choose a single new role. We also need further clarification of staff roles, so that function that cross organization chart boundaries are well understood. For example, some DBAs perform roles that are really systems administration, and many ITS-Med staffers perform multiple roles. We believe the proposed structure for the new “Production Systems” group (see the enclosed org chart) offers the best current solution to the integration of the two major Data Center operations on campus. We also recognize the need for further clarification and evolution of job roles as the new organization develops.


Appendix A Data Center IPT Team Members Joseph Paolillo Responsible Steering Committee Member Director, Infrastructure Services, ITS 221 Whitney Avenue 432-6673 Members: Cheryl A. Boeher Mgr Database Admin ITS Data Center Services University Data Center, ITS 175 Whitney Avenue 432-9509 Sandra A. Bouton Mgr Sys Programming ITS Data Center Services University Data Center, ITS 175 Whitney Avenue 432-6619 Kathryn A. Dobbins Database Administrator ITS-Med Production Systems 100 Church Street South Suite 100 785-5921 Michael C. Cicarella Senior System Administrator ITS-Med Production Systems ITS-Med Production Systems 100 Church Street South Suite 100 737-2783

Susan Kelley Director ITS Data Center Services University Data Center, ITS 175 Whitney Avenue 432-6617 Patrick Lynch, IPT Leader Director ITS-Med MedMedia Group 100 Church Street South Suite 107 737-5033 494-3943, cell Cherrie M. MacIntyre Senior System Administrator ITS-Med Production Systems 100 Church Street South Suite 100 737-6043 Louis A. Passariello Associate Director ITS-Med Production Systems 100 Church Street South Suite 100 737-2695 Jeffrey R. Thorn Tech Coordinator Computer Operations ITS Data Center Services 155 Whitney Avenue 432-6582


Appendix B

Database Administration

History at Yale and a Proposed Future Directions In 1993 (or thereabouts) Administrative Systems began developing a system called Graduate Student Payment System. This involved 1 person from Technology and Planning and 3 people from Administrative Systems. The technology they chose to use was Oracle on Macintosh AUX; it’s development tools and it’s database. It was decided that with all of its complexities, and possibly the development of other applications utilizing similar software; a Database Administrator was required to perform these duties since it was deemed outside of the scope of support for developers. So why did the DBA role originally start out reporting to the Director of Administrative Systems? Because that was where the need was and the customer that they would support worked there. Soon more than one DBA was required to support more and more applications that were brought into or developed by the University; SCT/Banner, a Data Warehouse, OFA, and Oracle Financials. Why does it make sense to report to the same management as developers? Why does it make sense to report to the same management as systems programmers and operations? And why does it make sense that they report outside of both of those areas? The tasks that a Database Administrator performs can be split up into many areas depending on the needs of the environment and the stage that an application or series of applications is in. There is research and development, design, development, and production support. The only role listed that I have not seen DBAs be involved in at Yale is the Design Role. What do each do? Design: In large shops, or those that develop a lot of large scale databases/applications, there is usually a Designer or Database Architect. This person is usually a senior level person that has been to a lot of training that is less related to the RDBMS of choice and more on theory and customer relations: Case Method training; interviewing customers, defining process flows and turning them into a physical database layout (tables, views, indexes, etc) Development: In large shops there are DBAs that solely support development groups in developing large application projects. If a Database Architect is not available, this person works on table layouts/design with the developer. The idea here is to develop a ‘normalized’ database design and still be efficient from a performance perspective and logical from a programming perspective. These DBAs work a lot on techniques and exploring more efficient ways to utilize key database features, helping the developer to explore options – not producing “code”. Implied here is performance tuning of applications or certification before moving to production. Production Support: In all shops there are one or more DBAs responsible for supporting production applications. This includes code migration and planning, patch applications and upgrades, performance monitoring and tuning, and troubleshooting problems. These are the people who are usually on call 24x7, 365 days/yr. These DBAs have to be aware of other changes to their environment that may affect existing systems or systems being migrated to production. They should be in close contact with Development Staff, Systems staff and Operations Staff and be aware of any changes being made in the Security, Firewall, or Networking areas that may affect a healthy Production environment.


Research and Development: In most shops this is more of a responsibility than a role that is included in all three roles listed here. R&D is not just a nicety, but a requirement of Database Administration, especially using Oracle software due to its never ending changes. It explores changes in software and new technologies. A Designer would have to be aware of any structural changes that are being made in new versions of software, or new data types to migrate to. They would have to understand differences in the Optimizer to understand how structural changes in the backend of the applications will be affected by not changing the design at all…or trying to accommodate it. The Development DBA, especially if the Designer or Data Architect did not exist, needs to be aware of the same things, in addition to new functionality put into place in dictionary packages. They also need to understand behavioral differences in the development tools like; reports, forms, java, plsql, sql, etc. The Production Support DBA needs to explore all of the nuances listed above AND understand how they all work with existing applications and environments. In addition, explore changes in monitoring software and backup strategies. Yale University currently employs 7 Oracle Database Administrators (6 Central and 1 Medical School), 3 Administrative Personnel and 1 Database Manager that supports many administrative applications and some academic applications. Currently the charging strategy is allocations of full time employees to a paying department/customer. Finance and Administration by far is our largest paying customer. It has been voiced by Phil Long in the past, a desire to have Database Administration be more heavily utilized in the Academic arena – supporting more Student/Faculty relational databases. The largest issue with supporting more Academic applications (Oracle or SQL Server) is FTE cost. When asked how we could get around that problem, the only thing that was come up with (that was never pursued) was a change in the way that the University deals with Database Administration’s charging structure; from FTE rates to overhead. When Database Administration is aligned more closely with an application, there is a tendency to overlook abiding by standards and/or performance modifications that may take longer to implement. Why? Because the goal of the application group (Administrative or Academic) is driven by getting the project completed – that is as it should be. When a separate group, Database Administrators have points of focus (different primary support or customers) but also work as overseers of all database applications to ensure consistencies in the environments and focus on overall efficiencies in the database, on the machine and in the organization. There is also something to be said for being in the same proximity for when there are issues that come up that can be observed only because people are sitting near one another. Sharing of knowledge and looking for different perspectives is a reality. When a customer hires a resource in Database Administration currently, they get their primary DBA and all of the support and wisdom of the others in the group….thus providing our customers with the best possible support available. In trying to come with an organization chart that makes sense from a Database Administration’s perspective and taking into consideration all of the roles that we play, I believe that it does not belong in the Administrative Systems or Academic Systems organization solely because at Yale we do not separate our DBA roles to accommodate just the application needs. We also do not just support production systems, which I believe is why we were recently aligned to work under Data Center Services. In creating the Infrastructure group, I believe that the Director of ITS has aligned his resources into functions; the two development areas (Administrative and Academic) and the support groups that are required in order to successfully implement their applications (network services, security/firewall, operations, systems, and database administration). Therefore I believe that Database Administration should report to the Director of Infrastructure – where concerns and technical directions are hashed out on a common playing field where topics are discussed that affect the environments that all applications exist in.

DATA CENTERS INTEGRATION PLANNING TEAM

Technology

Transcript of DATA CENTERS INTEGRATION PLANNING TEAM