How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for...
Transcript of How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for...
![Page 1: How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for E-Business Suite ... data access in reporting and data warehouse ... When a report](https://reader033.fdocuments.net/reader033/viewer/2022052712/5b492f087f8b9af5078d2e17/html5/thumbnails/1.jpg)
WHITE PAPER
How to Achieve Real-Time Analytics for E-Business Suite
Karthik MadaEmtec Inc.
Dean Alomerovic Humana Inc.
ABSTRACT
Oracle E-Business Suite mandatory reporting and business analytics increasingly require real-time transactional
data access in reporting and data warehouse environments. That presents a challenge in large multi-terabyte
systems. Oracle Active Data Guard, Golden Gate and Oracle Business Intelligence provide technology for
designing adequate solutions. Careful analysis is required to assess where each component performs to its full
strength and how they all fit together. Optimal solution is selected based on functionality, scalability, administration
and maintenance, and hardware resource requirements of the entire system.
CHALLENGES WITH E-BUSINESS SUITE REPORTING AND ANALYTICS
Oracle E-Business Suite plays a central role in many companies’ business landscape. Along with providing an
efficient OLTP environment, it is an indispensable resource for intelligent business decision making. To be efficient,
managerial and executive staff needs analysis and reports that convey the latest, real-time data. Even an hour of
delay may result in losing a client, revenue, or market expansion opportunity. Getting valuable information from
multi-terabyte E-Business Suite systems with minimum latency is a challenge.
As it is frequently common, there is no single solution that is optimal for all E-Business Suite implementations,
modules and dataset sizes. Therefore, we’ll explore multiple architectures and explain system constraints where
they fit well. In some cases, increasing capacity of the primary E-Business Suite environment may be all that is
needed. From return on investment (ROI) perspective, taking advantage of idle disaster recovery environments
through Oracle Active Data Guard is getting increasingly popular. In some other cases, nothing but reducing the
size of active datasets can help us achieve desired seconds-to-minutes reporting efficiency. Above all, Oracle
GoldenGate powerful real-time replication may be a key component there. It becomes obvious that overall
architecture efficiency is dependent upon full utilization and precise alignment of each component. Optimal solution
should be selected based on functionality, scalability, resource requirements and manageability of the entire
system.
©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com
![Page 2: How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for E-Business Suite ... data access in reporting and data warehouse ... When a report](https://reader033.fdocuments.net/reader033/viewer/2022052712/5b492f087f8b9af5078d2e17/html5/thumbnails/2.jpg)
OPERATIONAL VS ANALYTICS REPORTING
Oracle E-Business Suite with all of its 200 plus modules or applications is used for a huge variety of reporting
purposes. On high level, however, we can categorize these reports as either operational or analytics. Operational
reports are used for day-to-day business processes and decision making. They are typically based on an active
data set that contains the last quarter or two of transactions. The reports contain detailed information for the
active accounts and are used by managerial and business stuff to optimally steer daily operations. For operational
reporting, the data source can be either E-Business Suite directly or for larger implementations it may come from
an Operational Data Store.
On the other hand, analytics reports are based on larger data sets that usually include one to ten or more years of
transactional records. Executive staff uses them for strategic decision making and intelligence gathering. Data is
typically stored in a data warehouse and is aggregated in a way that enables efficient analysis. Functionally both
operational and analytics reports can coexist and be rendered by a single Oracle Business Intelligence Enterprise
Edition (OBIEE) instance, for example, and both typically offer drilldown capability to individual transactions.
WHERE IS TIME GONE AND WHY ARE REPORTS STALE?
When a report is executed against an Oracle database, it returns records consistent with the report start time.
Therefore, the report execution time is a measure of data staleness. In other words, if a report runs for two hours, it
will miss the last two hours of transactional data. For large volume data sets, more frequently than not transactional
records need to be transferred and transformed in some fashion to improve report runtime. Transformation time
adds to the total report lag time obviously. For example, if data is ETL-ed into an Operational Data Store (ODS) or
Data Warehouse (DW) environment and then a report is executed, the total lag time would be equal to the sum of
the ETL time and the report run time.
©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com
Figure 1. Components of report lag time
![Page 3: How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for E-Business Suite ... data access in reporting and data warehouse ... When a report](https://reader033.fdocuments.net/reader033/viewer/2022052712/5b492f087f8b9af5078d2e17/html5/thumbnails/3.jpg)
©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com
Transformation time and report run time are designated in the Figure 1 by tT and tR respectively. An ETL that runs
twice a day and has two-hour execution time would have tT of 14 hours. For Active Data Guard, tT would include
time to apply redo stream on the standby. For GoldenGate, tT comprises time to complete change data capture
(CDC) from the transaction logs, record CDCs in trail files, and deliver and apply them on the target. To reduce the
total report lag time, the goal is to minimize the sum of tT and tR.
SOLUTION 1: REINFORCE THE PRIMARY E-BUSINESS SUITE ENVIRONMENT
As a part of high availability architecture and performance requirements, it is typical for E-Business Suite
application tier to run on multiple nodes with web, forms, and concurrent processing distributed appropriately.
In a similar manner, in database tier we normally have a RAC where multiple DB instances may be individually
configured to handle specific OLTP or reporting workload. That is done by adjusting relevant DB parameters such
as parallelism and memory pools. In addition to E-Business Suite built-in reporting capability, companies typically
have Oracle Discoverer or similar tool that is multitoned for the same reasons. These powerful environments
are frequently capable of handling both transactional and reporting tasks. That is especially true for operational
reporting when data sets are not too large. If a need arises, the capacity can be increased by expanding nodes
pool in any tier. Advantage of this approach is the transformation time tT is equal to zero and there are no separate
environments to maintain. The solution is applicable when we have sufficient bandwidth in the database storage.
The solution is scalable and administration is not more difficult as a result.
SOLUTION 2: EMPLOY ACTIVE DATA GUARD STANDBY DATABASE FOR REPORTING
Disaster recovery requirements typically mandate companies to distribute E-Business Suite nodes across multiple
datacenters. In database tier, this is normally accomplished by having a dedicated standby database as shown in
the Figure 2. A standby is an exact replica of the primary database and remains idle until a disaster or a planned
switch-over occurs. The configuration is typically managed by Oracle Data Guard. For the last several years, there
has been increasing requirement to take advantage of this idle hardware from the return on investment perspective.
That is accomplished through Oracle Active Data Guard (ADG) that allows the standby to be open as a read-only
and thus be used for reporting needs while still continuously applying redo stream coming from the primary. ADG is
fully certified with E-Business Suite with caveat that the standby has to be a physical standby and not a logical one.
According to Doc ID 1070491.1, E-Business Suite is now certified to run many concurrent requests or reports
against a read-only standby. Certain requirements need to be met including that parallel concurrent processing
should be configured with some nodes and concurrent managers set up to handle ADG reports. It should be
noticed that these newly configured ADG concurrent managers still connect to the primary database; only the
reports run on the standby. Currently only Oracle Reports programs are supported to run on ADG. For reference,
full setup is described in detail in the document above. Additionally, the same Doc ID acknowledges that Oracle
Discoverer can officially be configured to take advantage of a read-only standby and provides the necessary steps.
![Page 4: How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for E-Business Suite ... data access in reporting and data warehouse ... When a report](https://reader033.fdocuments.net/reader033/viewer/2022052712/5b492f087f8b9af5078d2e17/html5/thumbnails/4.jpg)
©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com
Figure 2: Oracle Active Data Guard configuration
In addition to processing most of reporting tasks, ADG standby can substitute for the primary as the source of
numerous ETLs. Among them, standard ETLs distributed with both OBIEE 11g and 10g run against ADG with
no issues. In Oracle 12c, ADG has been enhanced to allow DML operations on global temporary tables and to
allow access to global or session sequences. These two enhancements have been crucial for enabling even more
reporting tools to run against the standby. On the top of that, ADG additionally can be used to run fast incremental
RMAN backups and helps prevent logical corruption on both primary and standby. Additional advantages of ADG
include a very low - a sub-second - latency in most environments with fast networks between primary and standby
thus resulting in transformation time tT being close to 0. The solution is applicable for operational reporting on
smaller and medium active datasets.
SOLUTION 3: USING GOLDENGATE AND ACTIVE DATA GUARD TOGETHER FOR REPORTING
For large E-Business Suite datasets, using technologies described so far, it may not be possible to achieve the
necessary response time for either operational or analytics reporting. These multi-terabyte datasets typically have
either transformation time or report time too high. In cases like this, Oracle GoldenGate can come to rescue as it
can help with both components, tT and tR. As Golden Gate has not been widely used with E-Business Suite, a
brief explanation of its functionality is in order.
GoldenGate is real-time, log based change data capture (CDC), transformation and replication software. It is
distinguished by very flexible, non-intrusive and low resource footprint architecture.
![Page 5: How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for E-Business Suite ... data access in reporting and data warehouse ... When a report](https://reader033.fdocuments.net/reader033/viewer/2022052712/5b492f087f8b9af5078d2e17/html5/thumbnails/5.jpg)
©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com
Three main groups of golden gate processes are extract, pump and replicat. Extracts are processes that capture
data changes from the source database transaction files. The Extract records these changes in a set of trail
files on the source database host. Pump processes transfer the trail files to the target environment from where
Replicat processes apply them to the targeted database. Manager process coordinates and administer the entire
environment.
So how do we take advantage of Golden Gate capabilities for operational and analytical reporting? For operational
reporting we typically create or instantiate an Operational Data Store (ODS) in a separate database. ODS includes
a subset of E-Business Suite tables that reporting is based on. To keep ODS lean, ODS tables are structured to
include only reporting columns form the corresponding source E-Business Suite tables. Furthermore, ODS does
not include all rows from the source tables but instead is populated only with the active data set that consists of
the last quarter or two of transactional data. A regular purging of aging records is scheduled on a weekly or monthly
basis.
Once ODS instantiation is complete, GoldenGate keeps replicating new changes from source to target tables
and thus keeps them in sync. The replication is very efficient with tT in seconds typically, but it may increase
significantly depending on the source transactions length. Operational reporting on the reduced dataset is quick
with tR ranging from a few seconds to a few minutes. Similarly, for analytical reporting GoldenGate can be
Figure 3. Oracle GoldenGate processes
![Page 6: How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for E-Business Suite ... data access in reporting and data warehouse ... When a report](https://reader033.fdocuments.net/reader033/viewer/2022052712/5b492f087f8b9af5078d2e17/html5/thumbnails/6.jpg)
©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com
configured to replicate changes from source to target in real-time manner. That way, large and long running ETLs
are completely avoided.
For GoldenGate replication, supplemental logging on the source database level is mandatory. It is employed for
tables without primary key or unique index. For those tables, supplemental logging causes additional entries to be
recorded in the redo stream so that the changed records can be identified on the target. Supplemental logging may
add up to 2% percent overhead on the database operation. One way to reduce that overhead is to make sure to
replicate from E-Business Suite only the schemas and tables of interest and nothing else. More frequently than not,
a set of lookup tables from applsys and apps schemas will need to be included in the replicated set as well.
It should be noticed that E-Business Suite as a whole still has some limitations regarding GoldenGate replication.
Reason? Datatypes ANYDATA and UROWID that E-Business Suite uses are not supported for direct GoldenGate
replication as additional logic is needed to get them fully functional in the target system. Furthermore, certain
E-Business Suite tables should be excluded from GoldenGate replication based on some twenty criteria identified
by E-Business Suite support team. A full list of exclusion criteria is given in Doc ID 1112325.1, but for illustration
purposes here are some: tables listed in DBA_QUEUE_TABLES, tables with row_size > 2MB, GL_INTERCO_%
tables, etc.
On the other hand, GoldenGate is fully functional with ADG and, as a result, it can be configured to replicate
E-Business Suite changes directly from the standby redo logs. Consequently, that helps eliminate any GoldenGate
impact on the primary environment.
ACTUAL IMPLEMENTATION OF GOLDENGATE AND ADG FOR REPORTING
Business requirement was to implement an efficient and scalable solution for near real-time E-Business Suite
financial reporting for operational and analytic purposes. The database was 6 TB in size and already had a disaster
recovery standby in place that was completely idle.
The first step of the solution was to configure the disaster recovery standby with ADG. Once that was completed,
the standby was used for all financial reports that execute fast enough to meet business requirements. Some
operational reports, however, absolutely needed a reduced dataset to meet a few-minute response time. As a
result, an ODS was established using GoldenGate that replicated directly from ADG. ODS was populated with
the last four month of GL transactions. GoldenGate also performed certain level of data transformation as a part
of replication to prepare data for faster reporting. For analytics reporting, an equivalent GoldenGate replication
implementation is under development.
On the application side, an existing OBIEE analytics instance was used to host the operational reporting
dashboards and reports as well. As a result of these changes, all major reporting was moved out of the primary
environment. Operational financial reporting from the ODS met the business requirements for real time data. The
reports included balance sheet, income statement and several others. This near real-time solution also helped us
retire an aging OBIEE 10g operational reporting that was based on standard ETLs executed four times a day. The
Figure 4 shows the complete solution architecture.
![Page 7: How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for E-Business Suite ... data access in reporting and data warehouse ... When a report](https://reader033.fdocuments.net/reader033/viewer/2022052712/5b492f087f8b9af5078d2e17/html5/thumbnails/7.jpg)
©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com
CHALLENGES AND LESSONS LEARNED FROM THE SOLUTION IMPLEMENTED
GoldenGate can be deployed to capture changes from ADG standby using either its online redo logs or archive
logs. To get replication underway as early as possible, the preference is to utilize online redo logs and GoldenGate
version 12.1.2.1.1 or higher should be used to accomplish that.
During its normal operation, GoldenGate needs access to archive logs for long running transaction. As a standard
practice, archive logs should stay in place for at least eight hours. However, if your E-Business Suite occasionally
has pretty long-running transactions, it is highly recommend considering adjusting that period accordingly. Also,
GoldenGate should be installed close the database server and should have local access to online redo and archive
logs. Otherwise, some replication architecture limitations may result.
A good GoldenGate health monitoring tools should be in place to help detect any problems as early as possible.
The tools should include status and statistics monitoring and alerting capability as a minimum. Additionally, a
versatile heartbeat mechanism should be in place designed to work with the most critical processes as well as
GoldenGate as a whole.
Remaining challenges are primarily related to long running E-Business Suite transactions that occasionally cause
extensive replication lags. Lag statistics shows us how well GoldenGate processes are keeping pace with the
Figure 4. Complete solution for E-Business Suite operational and analytics reporting
![Page 8: How to Achieve Real-Time Analytics for E-Business Suite · How to Achieve Real-Time Analytics for E-Business Suite ... data access in reporting and data warehouse ... When a report](https://reader033.fdocuments.net/reader033/viewer/2022052712/5b492f087f8b9af5078d2e17/html5/thumbnails/8.jpg)
©2016 Emtec Inc. ALL RIGHTS RESERVED www.emtecinc.com
amount of data that is being replicated or transformed. Specifically, some general ledger journal entries posting
jobs submit a million of records in a single transaction. The jobs typically complete within an hour on E-Business
Suite side and generate a huge amount of redo entries. As a result, the extract chokes on such a large transaction
and falls hours behind. To alleviate the problem, Oracle Support has recommended a few GoldenGate configuration
changes such as turning off bound recovery and breaking large transactions into small chunks that commit sooner.
The recommendations have helped to a large extent but the core problem still persists and emerges during the
busiest times such as period closing.
CONCLUSION
E-Business Suite frequently plays a central role in managing the company’s business activities. In addition to being
an efficient OLTP system, it is an indispensable source for efficient decision making. Both operational and analytics
reporting increasingly require access to real-time E-Business Suite transactional data. Intelligent solutions are
necessary to meet such requirements. Solutions vary among diverse E-Business Suite implementations, dataset
sizes, and hardware capabilities. Oracle GoldenGate, ADG and OBIEE provide powerful platforms for building such
solutions. Careful analysis and the technical expertise are needed in order to arrive at optimal solution in each
individual case.
The actual implemented solution followed the same principals and led to establishment of an efficient real-time GL
reporting ODS with an existing OBIEE analytics instance. Tremendous performance improvements were achieved
as a result. For example, Balance Sheet and Income Statement reports for all companies used to take more than
30 minutes, and now are completing in less than five minutes. In more typical scenarios when reports are run at an
individual company level, both reports are rendered in a few seconds and in real-time.
ABOUT EMTEC
Emtec is the right size provider of technology-empowered business solutions for world-class organizations. Our
local offices, highly-skilled associates, and global delivery capabilities ensure the accessibility and scale to align your
technology solutions with your business needs. Our collective focus is to continue to build clients for life: long-term
enterprise relationships that deliver rapid, meaningful, and lasting business value.
CONTACT
For more information or questions on How to Achieve Real-Time Analytics for E-Business Suite, please contact us