Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Alaska Dept of Fish and Game...

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | AKDFG BI Assessment

Alaska Dept of Fish and GamePlatform Performance and Business Intelligence Technical AssessmentFindings and Recommendations

Aaron Hinds – Assessment DirectorBarr Bauer – Master Principal Solution ArchitectChris Hughes – Business Intelligence Solution Architect Justin Kophs – Technology Account Manager Jim Hill – BI Account ManagerAugust 17, 2015

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | 2

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Health Check Findings and Recommendations

Findings

Recommendations

Next Steps

1

2

3


A structured method that produces recommendations to address your Business Objectives and Requirements.

Oracle STEPS Analysis Methodology


Mission Critical

What are the consequences of the system being down?

“there might be an impact on our ability to choose the appropriate closure

date and time and that, as some of our fleets could harvest up to 1 million

pounds of cod a day, the impact of a fishery being cut short or allowed to run

long could be significant. My number was off. I just read today that in 2014

one of our fleets hit a harvest rate in excess of 2 million pounds a day. To put

it another way, that’s a harvest rate of nearly $25,000 per hour.”


Key Findings Part 1 of 2

• Excellent staff!! • Created a robust Business Intelligence system that

is used for critical operations – Build it and they will come…..and they did– System is in use with a wide adoption/user pool it should be

properly maintained and operated as an enterprise resource– BI modeling expertise that you have is beyond most customers– Good BI planning/execution– Maintaining and improving BI user satisfaction


Key Findings Part 2 of 2• Ad hoc business arrangement around BI sysops (between BI and IT)• Unknown strategy for future state / future business requirements

– Is OceanAK the only driver for BI?– Adhoc/workgroup sysops approach might hamper organic growth of the platform

• Current System appears significantly underpowered and at risk of service outages• Available tools for system management and optimization not being used• Data Warehouse not being backed up enhancing risks of service outage• Current ETL process performance and reliability would be enhanced through redesign.• Not enough people/resources to write reports full time• Lots of data coming in from multiple sources, difficult to keep sync’d• High costs of licensing and maintenance


• R1: Develop & implement a sysops coordination plan

• R2: Define and prioritize existing and future business requirements and growth

• R3: Re-Architect the system (multiple recs)

• R4: Use Tuning and Diagnostics Tools

• R5: Implement a DW backup solution

• R6: Examine alternatives to storing images in the database

• R7: Manage system using OEM

• R8: Review latest tools and best practices for OBIEE

• R9: Implement OBIEE 12c

• R10: Integrate GIS project plan activities with existing BI priorities

• R11: Investigate Low Bandwidth / Public Facing OBI Options

• R12: Investigate Big Data Analytics

• R13: Investigate Platforming Options

• R14: Implement High Availability architecture

• R15: Address and improve ETL process

• R16: Review and develop future security architecture

Improve System Manageability and Increase User Satisfaction

Urgent or Critical Improvementsyear0

year 1


R1: Develop & implement a “sysops” coordination planProblem

– Infrastructure and system support responsibilities are not centralized and are spread across multiple groups.

– Adhoc posture to monitoring and troubleshooting the BI systems (DB, Network, Data Movement, Analysis)

– Reliability of the BI system is a critical need during the fishery season

Risks– Loss of staff productivity for troubleshooting across teams– Loss of revenue and/or damage to fishery ecology– Increased risk for longer outages and recovery times

Solution– Develop and define specific lines of responsibilities and

utilize guiding principles– Centralize monitoring capabilities and development

capabilities to ensure proper swim lanes between teams.

Year 0 Recommendation

Benefits Streamlined monitoring and troubleshooting activities Better BI Platform Stability and Manageability Ability to follow rational promotion model without major

impacts to the system More agile team to develop new capabilities

Oracle Products/Training/Reference Materials Enterprise Manager BI Management Pack Enterprise Manager ODI Management Pack Rittman Mead Article on BI Management Pack

See also R7: Manage system using OEM

http://www.oracle.com/technetwork/oem/pdf/511863.pdf

http://www.rittmanmead.com/2013/04/obiee-oem12cr2-and-the-bi-management-pack-part-1-introduction-to-oem12cr2/


Example Guiding Principles for Information ManagementR1 – Develop & implement an infrastructure coordination plan

# Principle Description

1 Data Availability Enterprise data will be made available in a consistent and structured format and frequency

2 Data Creation All enterprise data should be captured once at the point of its creation

3 Data as a Business Asset

Data will be treated as an asset that has value to the enterprise and will be managed accordingly. Accurate data is one of the cornerstones of the business and can be a key enabler of operational efficiencies, analytical & predictive insights, and organizational measurement against strategic goals. Data is created through authoritative source once and then organized and managed to retain its business context throughout its lifecycle

4 Enterprise Data Standardization

The meaning of strategic data elements should be standardized through clearly defined metadata (of which data dictionary is a subset) across enterprise applications and processes. Enterprise data (incl. reference data) would be standardized but made interoperable with other systems through appropriate integration infrastructure

5 Primary System of Record

All enterprise data will have an authoritative, official, primary data source that is the location for all Create, Update and Delete actions. All copies of enterprise data will be considered secondary and will not be updated as part of business transactions. Data should have a single master system of record for enterprise access.

6 Data Identifiers Key business objects in the enterprise will contain a globally unique identifier. That identifier will be in the form of the Universally Unique Identifier (UUID)

7 Data Integrity Authority to create and maintain the data will reside with those systems & people most knowledgeable about the data or those most able to control its accuracy

8 Data Cataloging Metadata should be managed and stored in a common and accessible metadata repository

9 Data Quality Data quality goals should be established, measured, reported, and monitored periodically for quality

10 Data Services Rationalization Redundant data stores should be eliminated and disparate data sourcing and data delivery approaches consolidated to use a single approach

11 Data Retention Data should only be retained in online storage in a manner that is consistent with business needs and data retention and archival (offline storage) needs to be reevaluated on a recurring basis


R2: Define and prioritize business requirements and plan for growthProblem

– Need a comprehensive list of user/constituent enhancements – Review and define internal business requirement

improvements and strategic initiatives (GIS) – Detailed development schedule and prioritization of those

business requirements

Risks– Slow reaction to business/constituent requirements can

decrease even greater adoption– Missed opportunities for system and business process

improvements

Solution– CMMI models provide guidance for developing/improving

processes that meet the business goals of an organization– Develop a process for growth and capacity planning


Benefits Better BI Platform Manageability More agile team to develop new capabilities

Oracle Products/Training/Reference Materials

Link to Information on Carnegie Mellon CMMI

http://www.sei.cmu.edu/library/assets/cmmi-overview071.pdf


R3: Re-Architect the System (Multiple Urgent/Critical Recommendations)Problem

– Current system capacity is frequently exceeded resulting in performance problems or service outages

– Production and Non-production environments are on the same server

– SQL Servers collocated on the same servers as the Oracle databases increasing load on the database server

Risks– Service outages impacting fishery business and attracting

attention and undesired publicity and attention

Solution– Develop a capacity plan for production and purchase new

larger servers to meet current and future capacity requirements.

– Separate non-production onto separate servers as a best practice by repurposing existing servers to support nonprod.


Solution cont.– Separate SQL Server databases similarly using this new

environment

Benefits More predictable performance Growth better managed for production environments More secure with production isolated from non production Production set up for future addition of HA and DR


CurrentState

Undersized


Non-Production Environments(Repurposed Existing Environments)

Production Environment(New, sized for Production, Managed by Capacity Plan)

Future State

Year 0Sized to Current Load

Year 0Sized to Current Load

Year 1Clustered for HA

Year 1RAC or DG for HA

Private Cloud or On Premise New Capability

Option to Move toPublic/Private Cloud




R4: Use Tuning and Diagnostic Tools to Improve Database PerformanceProblem

– Tuning and Diagnostic tools are not used when implementing and managing databases

– Database tuning issues may contribute to complex query problems

Risks– Poor query performance that impacts individual queries and

prevents system scaling

Solution– Training on use of the Oracle Tuning and Diagnostics Packs

for both DBAs and developers.– For Developers: Build tuning into new database

implementation process as a best practice. – For DBAs: Use the tuning and diagnostics packs for

troubleshooting performance problems as a best practice


Benefits Improved system performance Improved system scalability Consistent use of tools for best practices

Enablement Workshop to introduce AKDFG Staff to Oracle Enterprise

Manager, Diagnostics Pack, and Tuning Pack


R5: Implement a DW Backup Solution Problem

– DW is not backed up– DW growth is making conventional backup approaches

impractical

Risks– Data loss

– Service outages impacting fishery business and attracting attention and undesired publicity and attention

Solution– Use database partitioning to isolate static (historical) data from

active transactional data

– Use the transportable tablespace feature of Oracle Partitioning to make single backups of static data

– Back up active partitions on a frequent schedule using existing Oracle DB backup tools (rman)


Benefits Backup time is reduced considerably so that it could be done

nightly Static data is backed up once and not repeatedly backed up Restore DW from a catastrophic failure by:

• Copy back static data partition files to the restored system

• Restore active data partitions using rman

Oracle Products/Training/Reference Materials Oracle Partitioning http://www.oracle.com/technetwork/database/options/

partitioning/partitioning-wp-12c-1896137.pdf?ssSourceSiteId=ocomen


R6: Examine alternatives to storing content in the databaseProblem

– Binary images stored in the database are contributing to database growth

Risks– Increases size of database and database backups

– Inability to version content except through custom coding

Solution– Examine alternatives to managing binary content outside the

database (Store in file system and reference from database, for instance)

– Assess benefits of change including changes in database growth.

– Implement changes including new image storing process

– Migrate existing images to file system with compression


Benefits Reduce growth of the database Store images in the file system Take advantage of file-based compression tools. Position for future transition to a content management system

Enablement Oracle-led workshop to explore alternatives.


R7: Manage system using Oracle Enterprise ManagerProblem

– Need to manage more system components with fewer resources (production and non production, especially if Recommendations R1 and R2 are adopted)

– Not taking advantage of patch automation– Need a single management platform for database,

middleware (OBIEE) and hardware

Risks– Increased time to manage system and system complexity– Service outage and system inconsistencies as system grows

Solution– Training on using Oracle Enterprise Manager as a single pane

of glass management environment– Implement the following Management Packs: BI, Database

Lifecycle, Oracle Data Integrator


Benefits Single system management console allowing fewer

administrators to manage the entire system including production and non-production

Patch management and automation BI systems management from the central console Prepare for potential DR replication

Oracle Products/Training/Reference Materials Enterprise Manager BI Management Pack Enterprise Manager Database Lifecycle Management Pack Enterprise Manager ODI Management Pack Rittman Mead Article on BI Management Pack

http://www.oracle.com/technetwork/oem/pdf/511863.pdf

http://www.rittmanmead.com/2013/04/obiee-oem12cr2-and-the-bi-management-pack-part-1-introduction-to-oem12cr2/


R8: Review current and future tools & best practices for OBIEEProblem(s)

– Subject Areas are becoming unwieldy in terms of locating columns or cross joining SAs for analysis

– Performance and stability of mapping solutions in BI

– Rich BI content may consume bandwidth making app unusable (aka the Nome conundrum)

– Metadata management is difficult through the RPD and presentation layer – multiple naming conventions need to be rationalized

– General questions around charting capabilities

– Higher than desired learning curve (esp formulas)

Risks– Creating incorrect analyses (junk from the user formulas)

– Increased risk for system saturation and outages from high resource consumption queries

– Less user adoption of the curated BI information

– Inability to properly locate fisheries information geographically


Solution– Find opportunities for outreach and sharing with peers and other BI

users (like OOW, BIWA, web casts, seminars)

– Evaluate new OBI 11.1.1.9 capabilities like HTML5 charts

– Engage with a consultant (like RittmanMead) to address rpd and Subject Area Design

Benefits Greater insights and better user satisfaction Broader adoption of the system and greater dissemination of

information and understanding about fisheries

Oracle Products/Training/Reference Materials Monitoring OBIEE Performance for the End User with JMeter from

EM12c What’s New in OBIEE 11.1.1.9 for Systems Administrators and

Developers OBIEE Presentation Layer Docs BIWA Summit website

http://www.rittmanmead.com/2013/11/monitoring-obiee-performance-for-the-end-user-with-jmeter-from-em12c/




http://www.rittmanmead.com/2015/05/whats-new-in-obiee-11-1-1-9-for-systems-administrators-and-developers/

http://www.rittmanmead.com/2015/05/whats-new-in-obiee-11-1-1-9-for-systems-administrators-and-developers/

https://docs.oracle.com/middleware/11119/biee/BIEMG/presentationlayer.htm#BIEMG266

http://www.ioug.org/p/cm/ld/&fid=992


Example BIWA 2014 presentation on Heatmaps in OBIEER8: Review current and future tools & best practices for OBIEE


R9: Implement OBIEE 12cProblem

– User satisfaction issues around performance, usability and visualization options

– Performance of MapViewer in OBIEE is suspected not to support business use cases

Risks– Lack of insights being derived from OceanAK– Loss of staff productivity for troubleshooting user – Loss of revenue and/or damage to fishery ecology

Solution– Oracle BI 12c has a number of new tools such as:

• Visual analyzer for interactive HTML5 visual discovery

• BI ask (search based BI catalog)

• External Data sources: mashup local spreadsheets to the rpd data model. Analysts can join their local data to the OceanAK rpd


Solution cont.– The 12c version will have HTML5 Mapping component

(supports more map features like lots of individual GPS points)

Benefits increase time to deliver and time to value for your customers

(internal and external) New tools to manage promotion process and application

archiving (bar file) as well as promotion to BICS Increase user satisfaction Better mapping performance (drill into individual points)

Oracle Products/Training/Reference Materials BICS has previews of most of the 12c features:

BI Cloud Service (BICS) Home YouTube Channel for BICS

https://cloud.oracle.com/business_intelligence


https://www.youtube.com/playlist?list=PLOcpw36tp3yKLTdN5OLPRZodOV0eDMXQK


New option to OBIEE 12c: Visual AnalyzerR9: Implement OBIEE 12c


R10: GIS project plan activities merged with existing BI priorities Problem

– Performance of MapViewer in OBIEE is suspected not to support business use cases

– Desire to reuse ESRI content with OBIEE– Desire to better serve the public with fisheries information– Upcoming project to improve definition of Statistical Areas

Risks– Separate development threads may create data siloes

Solution– Coordinate with Oracle BI field staff (Conrad, Hughes, Lapp)– Consider a slowly changing dimensional model– Persist new SA definitions in Oracle Database– Follow best practices in design mapping services for OBIEE– Use 12c OBIEE or custom javascript for maps


Benefits Ensure that users can display and analyze harvest and related

data using geographic and location information Leverage work from the GIS project in OceanAK Show innovative map capabilities to the public and user

population

Oracle Products/Training/Reference Materials Fusion Middleware MapViewer Docs Oracle Spatial and Graph Developers guide

See also R9: implement OBIEE 12c

http://www.oracle.com/technetwork/middleware/mapviewer/documentation/index.html

http://www.oracle.com/technetwork/middleware/mapviewer/documentation/index.html

http://docs.oracle.com/database/121/SPATL/toc.htm


R11: Investigate Low Bandwidth / Public Facing Content OptionsProblem

– Network latency and bandwidth makes OceanAK unusable in places like Nome

– Rich content applications like OBIEE Answers and Dashboards arent suitable for public users

– Many OceanAK users will be using a device other than a desktop web browser (ie mobile device) to get content

Risks– Users will not get pertinent/timely information from

OceanAK to support their business– AKDFG will have to manipulate the BI systems in

adhoc/abnormal fashion to deliver content

Solution– Conduct a OBI Mobile workshop to evaluate options– Pending review, implement the BI Mobile Application

Development option in OBI Mobile


Solution cont.– Investigate Oracle BI Cloud (BICS) for low-bandwidth / public

facing options– Investigate ODI running against OBI ODBC connection to

deliver excel or SAS data sets to FTP endpoints

Benefits Low bandwidth locations can receive OceanAK content Will be able to solve 2 problems with one solution Includes JavaScript templating optimal for including ESRI

content or D3 graphs Includes templates for tables, phones, and desktop

Oracle Products/Training/Reference Materials Oracle BI Mobile Application Development (BIMAD) BIMAD Youtube Example BICS Youtube Channel

http://www.oracle.com/technetwork/middleware/bi-foundation/bi-mobile-app-designer-1983915.html

https://www.youtube.com/watch?v=uIRqqK2UUzE







R12: Investigate Big Data Analytics Problem

– OceanAK is planning on adding non-relational (big) data sources for analysis: documents, surveys, sensor data, remote sensing images, weather and climate data

Risks– Big data sources (sensors, documents, imagery) will require

extra ETL and modeling efforts in traditional BI– With traditional BI AKDFG may mis-estimate the value of the

big data to OceanAK users and may waste cycles on wild goose chases or miss the gem within the piles of data

Solution– Evaluate various big data approaches to consume, explore,

refine and transform data into meaning analytical content• Additional tools like Big Data Discovery provide an appropriate

platform for discovery and data evaluation

• OBIEE can source information Hive and Impala once the data is refined and modeled


Solution cont.– Conduct Big Data Discovery Workshop– Evaluate Oracle Spatial and Graph for Hadoop

Benefits Get clarity on the value of big data to OceanAK Learn how to combine Big Data sources to OceanAK Deploy Discovery focused applications that permit large

volumes of data to be accessed easily by endusers

Oracle Products/Training/Reference Materials Oracle Big Data Discovery video on Youtube Oracle Big Data Discovery introduction from RittmanMead

https://www.youtube.com/watch?v=tnAgdLeFCQY

https://www.youtube.com/watch?v=tnAgdLeFCQY

http://www.rittmanmead.com/2015/02/introducing-oracle-big-data-discovery-part-1-the-visual-face-of-hadoop/


Execution

Innovation

4th Generation Data Architecture for Big DataR12: Investigate Big Data Analytics

AKDFG BI Assessment

Enterprise Data

Other Data Sources

Data Streams

BusinessData

Social andLog Data

WarehouseData FactoryReservoir

Data Management Model FirstAnalytics

• Reporting-oriented• Often enterprise wide

in scope, cross LoB• “you know the

questions to ask”

Dashboards

• Telematics• Industry Services• Internet of Things• Sentiment

DataServicesData Streaming

Events Decisions

Analytics

1 2 3

Discovery Lab Data FirstAnalytics

• Data Exploration• Highly visual and/or

interactive• “you don’t know the

questions to ask”

Discovery

ModelingSearch

Analytics

BusinessIntelligence

Exploration

BusinessMonitoring


R13: Investigate Platforming Options Problem

– Real or perceived high costs in managing an Oracle infrastructure for BI

– Delivering BI to remote areas is difficult– Desire to better serve the public with fisheries information– Heavy burden on staff

Risks– Increased time to manage system and system complexity– Service outage and system inconsistencies as system grows– More public access to OceanAK will cause more operational

and system impacts

Solution– Evaluate BI Cloud Service (BICS) from Oracle Public Cloud– Investigate Database as a Service (DBaaS)– Investigate Managed Cloud offerings


Benefits Increase time to deliver and time to value for your customers

(internal and external) Patch management and automation managed by 3rd party BI Mobile license is included with BICS Scalability and performance is easier to plan and manage More agile team focused on BI and not infrastructure

Oracle Products/Training/Reference Materials BI Cloud Service (BICS) Home YouTube Channel for BICS Database as a Service Home



https://cloud.oracle.com/database


R14: Implement High Availability architecture Problem

– No hardware redundancy for mission critical system– Standard software maintenance (patching, upgrading,

promotion process) can take servers down for extended periods of time

Risks– Risk of extended system unavailability and impact on fishery

operations in the event of a major hardware failure

Solution– Database: implement Oracle RAC (hardware clustering), or

Oracle Data Guard (Mirroring).– Middle Tier: Implement WebLogic Clustering– Storage Tier: (if needed) implement available HW data

protections.


Benefits Builds in HW redundancy and service continuity in the event

of an major hardware failure. Depending on SLA to restore service, Data Guard may be an

option and has the additional benefit of having a copy of the data outside your current data center. Also, Data Guard is included in the Oracle Database Enterprise Edition license.

Standard software maintenance activities can occur on HA enabled systems without service interruptions

Oracle Products/Training/Reference Materials Real Application Clusters Data Guard Enterprise Deployment Guide for OBIEE

http://docs.oracle.com/cd/E28280_01/doc.1111/e15722.pdf


R15: Address and improve ETL process Problem

– Performance and latency problems with existing ETL processes.

– Recovery process from ETL failure is complex

Risks– Continued problems with ETL including data consistency– Effort to correct problems will increase as DW grows

Solution– Rearchitect the ETL process to have all remote sources

replicate/update data to a local Operational Data Store (ODS) then change the current ETL process to pull data only from the ODS.


Benefits Change will remove the latency and make the ETL process

run more quickly and reliably Data transmission problems can be corrected at the ODS

level and not be propagated into the DW. Transmission can be through file transfer or separate ODI

processes specific to each source.

Enablement Oracle-led workshop to explore approaches to rearchitecture


R16: Review and develop future security architecture Problem

– Access to system uses simple account management.

Risks– Exposes data to unauthorized access and data breaches

Solution– Review security requirements including how fishery access is

managed and access is audited.

– Determine if changes are needed including changes to how accounts are provisioned and deprovisioned.

– Assess changes to existing security software.

– Develop a roadmap for changes to the security environment


Benefits Security review and planning will identify any potential or actual

security problems. Security review may identify changes to managing users that can

simplify managing large number of fishery users by a small number of resources

Roadmap will help with budgeting for new systems.

Oracle Products/Training/Reference Materials Product recommendations will follow security assessment.


3. Next Steps

Calls To Action

Core Infrastructure Solutions and Learning

– Demos– Hands on workshop – DB Tools– DBaaS Options and Review

BI Solutions and Learning– Coordination call and webex for

issues and roadmap discussions with Field SC staff

– Engage a partner for BI Systems / best practices

– Education and outreach– BI Workshops– Present at BIWA or OOW

Next Steps as a Team– Refine Roadmap into executable plan

• Map timelines• Discuss Gaps• Phasing, Timing, and Swim Lanes

Correct?

AKDFG BI Assessment

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Alaska Dept of Fish and Game...

Documents

Transcript of Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Alaska Dept of Fish and Game...