Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
-
Upload
azhar-mulla -
Category
Documents
-
view
229 -
download
0
Transcript of Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
-
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
1/26
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Home About Careers Clients Consulting Training Support Articles Blog
Performance and OBIEE part II Test DefineMarch 18th, 2013 by Robin M offatt
Take the first tentative s teps along the yellow brick road of investigating performance and OBIEE with this jum p into the n itty
gritty of how we s hould define a performance test.
An overview of defining a performance testThe first step in asses sing the performance of OBIEE is defining the testwe are going to us e. For troubleshooting a
performance problem in Production, this might be a s ingle report. For performance testing a new system it might be more
complex. The process to define it needs to be the sam e though.
Let us cons ider for a mom ent what it is we want to achieve through a performance test. It should be a validation or possibly
determination of som e or all of the following, which may or may not be formally documented in a project as Non-Functional
Requirements (NFRs):
Report response times
User concurrency levels
Hardware utilisation
It doesn t sound too difficult does it? We can run a report and time how long it takes, and we can run the sam e report multiple
times in parallel and s ee how long it takes then, and around all of that we can gather system OS stats.
This seems to be the level at which some people perceive the complexity of performancetesting; that is, not at all.
But, to put any weight in the above results would presume that the system under test had :
Only one report
No param eters or other interactivity on the report
Every user runs the report at the exact same time
Search theblog
Recent Posts
Analytics w ith Kibana andElasticsearch through Hadoop
part 3 Visualising the data inKibana
Analytics w ith Kibana and
Elasticsearch through Hadoop part 2 Getting data into
Elasticsearch
Analytics w ith Kibana and
Elasticsearch through Hadoop part 1 Introduction
UKOUG Partner of the YearAw ards
Oracle BI Cloud Service for SaaSApplication Reporting Part 1:Integrating BICS toSalesforce.com using REST APIs
Top PostsOBIEE 11g Security Week :
Managing Application Roles andPolicies, and Managing SecurityMigrations and Deployments
Upgrading OBIEE to 11.1.1.7
OBIEE 11gR1 : Architecture andUse of WebLogic Server
OBIEE 11g Security Week :Connecting to Active Directory,and Obtaining Group Membership
http://www.rittmanmead.com/2010/07/obiee-11gr1-architecture-and-use-of-weblogic-server/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-1-introduction/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-2-getting-data-into-elasticsearch/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-3-visualising-the-data-in-kibana/http://www.rittmanmead.com/2013/03/performance-and-obiee-test-define/http://www.rittmanmead.com/2013/03/performance-and-obiee-test-define/http://www.rittmanmead.com/http://www.rittmanmead.com/about/our-teamhttp://www.rittmanmead.com/about/careers/http://www.rittmanmead.com/clients/http://www.rittmanmead.com/consulting/http://www.rittmanmead.com/training/http://www.rittmanmead.com/support/http://www.rittmanmead.com/articles/http://www.rittmanmead.com/blog/http://www.rittmanmead.com/http://www.rittmanmead.com/2012/03/obiee-11g-security-week-managing-application-roles-and-policies-and-managing-security-migrations-and-deployments/http://www.rittmanmead.com/2014/11/oracle-bi-cloud-service-for-saas-application-reporting-part-1-integrating-bics-to-salesforce-using-rest-apis/http://www.rittmanmead.com/2014/11/oracle-bi-cloud-service-for-saas-application-reporting-part-1-integrating-bics-to-salesforce-using-rest-apis/http://www.rittmanmead.com/2014/11/ukoug-partner-of-the-year-awards/http://www.rittmanmead.com/2014/11/ukoug-partner-of-the-year-awards/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-1-introduction/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-1-introduction/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-2-getting-data-into-elasticsearch/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-2-getting-data-into-elasticsearch/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-2-getting-data-into-elasticsearch/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-2-getting-data-into-elasticsearch/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-3-visualising-the-data-in-kibana/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-3-visualising-the-data-in-kibana/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-3-visualising-the-data-in-kibana/http://www.rittmanmead.com/http://www.rittmanmead.com/http://www.rittmanmead.com/http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/http://www.rittmanmead.com/about/our-teamhttp://www.rittmanmead.com/about/careers/http://www.rittmanmead.com/clients/http://www.rittmanmead.com/consulting/http://www.rittmanmead.com/training/http://www.rittmanmead.com/support/http://www.rittmanmead.com/articles/http://www.rittmanmead.com/blog/http://www.rittmanmead.com/http://www.rittmanmead.com/2013/03/performance-and-obiee-test-define/http://www.rittmanmead.com/author/robin-moffatt/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-3-visualising-the-data-in-kibana/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-2-getting-data-into-elasticsearch/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-1-introduction/http://www.rittmanmead.com/2014/11/ukoug-partner-of-the-year-awards/http://www.rittmanmead.com/2014/11/oracle-bi-cloud-service-for-saas-application-reporting-part-1-integrating-bics-to-salesforce-using-rest-apis/http://www.rittmanmead.com/2012/03/obiee-11g-security-week-managing-application-roles-and-policies-and-managing-security-migrations-and-deployments/http://www.rittmanmead.com/2013/04/upgrading-obiee-to-11-1-1-7/http://www.rittmanmead.com/2010/07/obiee-11gr1-architecture-and-use-of-weblogic-server/http://www.rittmanmead.com/2012/03/obiee-11g-security-week-connecting-to-active-directory-and-obtaining-group-membership-from-database-tables/ -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
2/26
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
How many OBIEE implem entations do you know that match this des cription?
As soon as you accept this p remise, then you have to accept the need to des ign your performance tes t properly if you want
any kind of confidence in the res ults that you get.
Of course, there is the chance [sarcasm]vastly improbable and unheard of I know[/sarcasm] that Performance Testing is
a tick box on a project plan (normally about a week before Go Live), from which there is the expectation that it can deliver the
confidence of an accurate test with the s impli city and brevity of the one-report test I des cribe above. If that is the cas e then I
hope this article will explain why it is a fallacy.
Walk before you runThere are three key things to aim for in the design of a performance tes t:
1. Repeatability. You must be able to rerun the test, so that you can (a) prove bugs/performance problem s have beenfixed, and (b) re-use the test as a base line for future measurem ents
2. Accuracy. This can be difficult with a stack as complex as the OBIEE one, but you must do a ll you can to avoidconflation and mis interpretation of data.
3. Ease of Analysis. If the res ult of your test is unsucces sful, you will have to start analysing the is sue. If your test isunnecess arily big or complex then it will be very difficult and time consum ing to pare it back to find root cause.
Avoid the blunderbus approach. Its tempting (and fun!) to get carried away building a
huge test and run it with a big num ber of users but if it brings the system down then the
only immediately conclusion you can draw is that you can crash the system. To get to any
kind of root cause of the lim itation you will have to run more tests, and you will have to
sim plify the test that you did run to pinpoint the actual iss ue(s).
Build up small tes ts with single users and check things work at this level first. You can
then combine these together into larger tests for load testing. By working this way you get
the added benefit of a better unders tanding of the workings of your load test and system
being tested, as you will have gained this experience from the smal ler tests.
Think of it as as sem bling a cars engine for yourself versus buying a car out of the
showroom . If it breaks (or you want to tune it), youll be in a better place to start if you have
been under the bonnet [hood] already.
from Database Tables
Analytics w ith Kibana andElasticsearch through Hadoop -
part 3 - Visualising the data inKibana
Random Posts
Inside My Home OfficeDevelopment Lab (V MWare, OS X
Server)Photos and PresentationDownloads fr om the RittmanMead BI Forum 2014
Introducing the Updated Oracle /Rittman Mead InformationManagement Reference
Architecture Pt1. - Information
Architecture and the "DataFactory"
Preview ing TimesTen, Endecaand Oracle DW Sessions at the
Brighton BI Forum 2014Goodbye to Stew art...
Tags
11gBig Data Appliance
BIPBI Publisherdwem12cEndecaexalyticsextremebigit
goldengatehadoopHiveinit.dinstalllinuxMDS XMLmonitoring
new featuresnqcmdOBIA
obieeodiodi12copatchOracleOracle BI
Applications oracle data
http://www.rittmanmead.com/2012/03/obiee-11g-security-week-connecting-to-active-directory-and-obtaining-group-membership-from-database-tables/http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/2012/03/obiee-11g-security-week-connecting-to-active-directory-and-obtaining-group-membership-from-database-tables/http://www.rittmanmead.com/2014/11/analytics-with-kibana-and-elasticsearch-through-hadoop-part-3-visualising-the-data-in-kibana/http://www.rittmanmead.com/2013/08/inside-my-home-office-development-lab-vmware-os-x-server/http://www.rittmanmead.com/2014/05/photos-and-presentation-downloads-from-the-rittman-mead-bi-forum-2014/http://www.rittmanmead.com/2014/06/introducing-the-updated-oracle-rittman-mead-information-management-reference-architecture-pt1-information-architecture-and-the-data-factory/http://www.rittmanmead.com/2014/05/previewing-timesten-endeca-and-oracle-dw-sessions-at-the-brighton-bi-forum-2014/http://www.rittmanmead.com/2014/06/goodbye-to-stewart/http://www.rittmanmead.com/tag/11g/http://www.rittmanmead.com/tag/big-data-appliance/http://www.rittmanmead.com/tag/bip/http://www.rittmanmead.com/tag/bi-publisher/http://www.rittmanmead.com/tag/dw/http://www.rittmanmead.com/tag/em12c/http://www.rittmanmead.com/tag/endeca/http://www.rittmanmead.com/tag/exalytics/http://www.rittmanmead.com/tag/extremebi/http://www.rittmanmead.com/tag/git/http://www.rittmanmead.com/tag/goldengate/http://www.rittmanmead.com/tag/hadoop/http://www.rittmanmead.com/tag/hive/http://www.rittmanmead.com/tag/init-d/http://www.rittmanmead.com/tag/install/http://www.rittmanmead.com/tag/linux-2/http://www.rittmanmead.com/tag/mds-xml/http://www.rittmanmead.com/tag/monitoring/http://www.rittmanmead.com/tag/new-features/http://www.rittmanmead.com/tag/nqcmd/http://www.rittmanmead.com/tag/obia/http://www.rittmanmead.com/tag/obiee/http://www.rittmanmead.com/tag/odi/http://www.rittmanmead.com/tag/odi12c/http://www.rittmanmead.com/tag/opatch/http://www.rittmanmead.com/tag/oracle/http://www.rittmanmead.com/tag/oracle-bi-applications/http://www.rittmanmead.com/tag/oracle-data-integrator-2/http://www.rittmanmead.com/tag/oracle-endeca/http://www.rittmanmead.com/tag/oracle-data-integrator-2/ -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
3/26
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Define a UserThe first step in defining a good test is to define extremely precisely and clearlywhat it is we are testing. This sections
heading I take from Zed Shaws s uperb article Programmers need to learn statistics. For me it encapsulates in three
words the challenge that we face in des igning a good test. We know we want our system to support more than one us er, and
we probably have a us er concurrency figure (of which more be low) that the system must support. So, how do we define a
user? To quote from the aforementioned article, do we jus t click around a bit, you know, like a user? Maybe we run the
OBIEE-equivilent of SELECT 1 FROM DUALa thousand times sim ultaneously, and then declare our system can support a
thousand concurrent users?
The challenge is that good test or not, test results and system capacity will often be boiled down to a discuss ion of simp le
user numbers. Will the s ystem s upport 100 users? 200 users? And so on.
Modelling the workload of a transactional (OLTP) system is relativelystraightforward in that there is a finite number of
transactions, even if it is a large number. When we consider how to model an OBIEE user in a performance test, we have
additional challenges:
Navigation paths
In an OLTP system such as an online s tore once a user clicks on add to cart we can be pretty sure they will then
either (a) carry on shopping or (b) go to checkout. What about a BI user once theyve run their first report, which will
they run then? Maybe a different report, but quite likely it will be navigation within the existing report (eg drill down,
prompt changes , etc)
Ad hoc
Whilst some reporting will be fixed reports, much business value from BI comes in the analysis and exploration of
data in an ad hoc manner to discover trends etc to give competitive advantage. These ad hoc queries cannot, by
definition, be known in advance, so the performance test definition mus t find a way to address this.
Think about how many types of users you have on your system, and how you would characterise and group them. There will
be different sets of data they access , and varying degrees of interactivity with which they use the tool. Some will be using pre-
built reports with fixed filters, other may be doing extensive ad-hoc analyses , and som ewhere in the middle will be pre-built
reports but with a fair amount of interactivity through functionality such as prom pts and selection s teps.
Define a ReportSo for each of our User types that we define, we need to precisely define the workload for them. This wi ll be m ade up o f the
execution of one or more dashboards /reports. Defining the report that our model users are going to run is not as s imple as it
EndecaOracle EndecaInformation Discoveryow b
performanceReal TimeDecisionsreplicationReportService RTDrunReportsampleappscreenscripting
securitystartup testingtrainingXML
http://www.rittmanmead.com/tag/oracle-data-integrator-2/http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://zedshaw.com/essays/programmer_stats.html?__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186http://www.rittmanmead.com/tag/oracle-data-integrator-2/http://www.rittmanmead.com/tag/oracle-endeca/http://www.rittmanmead.com/tag/oracle-endeca-information-discovery/http://www.rittmanmead.com/tag/owb/http://www.rittmanmead.com/tag/performance-2/http://www.rittmanmead.com/tag/real-time-decisions-2/http://www.rittmanmead.com/tag/replication/http://www.rittmanmead.com/tag/reportservice/http://www.rittmanmead.com/tag/rtd/http://www.rittmanmead.com/tag/runreport/http://www.rittmanmead.com/tag/sampleapp/http://www.rittmanmead.com/tag/screen/http://www.rittmanmead.com/tag/scripting/http://www.rittmanmead.com/tag/security/http://www.rittmanmead.com/tag/startup/http://www.rittmanmead.com/tag/testing-2/http://www.rittmanmead.com/tag/training/http://www.rittmanmead.com/tag/xml/ -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
4/26
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
may appear. As well as the dis tinction to make between ind ividual reports (Analyses /Answers ) and Dashboards, we als o
need to consider:
Filters
How do we model a report which has six poss ible filters? We cant just call that report A, because each filter
permutation could cause different SQL to be generated by the BI reporting tool and cons equently vastly different
execution plan(s) on the database where the query runs.
Aggregate navigationOne of OBIEEs m any strengths is its aggregate navigation capability, enabling the us er to drill down into data whilst
in the background the tool switches between pre-defined aggregate tables on the database to pick the fastest one to
use. For performance testing we need to make sure that within a report each possib le aggregate is cons idered
separately.
As an example , let us examine a dashboard from SampleApp.
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.oracle.com/technetwork/middleware/bi-foundation/obiee-samples-167534.html?__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186 -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
5/26
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
At an ini tial glance, we could specify our Report as s imply being Das hboard 3.10. Butthe das hboard is m ade up of
three analyses(plus some s tatic text/image objects)
Product Line Analysis
Ragged Hierarchy
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/wp-content/uploads/2013/03/sampleapp_3.10_0.png -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
6/26
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Value Based (Parent Child) Hierarchy
We need to consider each of these reports individually. In addition, there are hierarchical columnswhich, like Prompts, must
be included in our test specification. Heres one of the reports from the dashboard in two different states of hierarchicalnavigation.
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
7/26
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
The first version, mostly collapsed, requires 3 queriesagains t the database. The second version requires 21 queries!
Clearly the performance of the two different versions of the report is going to differ.
Depending on the hierarchical column s tates, depending on dashboard prompt values, depending on aggregate drill pathsfollowed, the sam e individual report has the potential to behave quite differently.
Any report or das hboard thats using prom pts needs to have those prom pt values s tated in the test defin ition. The execution
of a query on the database can vary considerably depending on the predicates suppl ied, and it is in part from the
report/dashboard prompts that these predicates come.
Ad-hoc reports
Conspicuous by its absence from the above is how we deal with ad-hoc reporting with OBIEE in our performance tests. This
is the elephant in the roomwhen it comes to performance testing OBIEE, because we cannot go through the sam e
methodical process of cataloging the workload because it is ad hoc. This is discussed in detail later.
User workloadHaving precisely defined the reports which are going to be run, we can s equence them together to represent different types of
users . Clearly this is going to be a generalisation, an estimation, of how a particular group of users is go ing to tendto use
the reports. Unlike an OLTP transaction, where the steps are m uch more predefined and free from poss ible deviations,
users of OBIEE have the flexibility to run any number of reports, drill within existing reports, and s o on.
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://en.wikipedia.org/wiki/Elephant_in_the_room?__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186http://en.wikipedia.org/wiki/Ad_hoc?__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186#Ad_hoc_querying -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
8/26
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Taking the above SampleApp dashboard as an example, one us er (lets call them the Product Analyst) might have the
following test sequence:
1. View Dashboard 3.10in default state2. Drill into hierarchical columns of Product Line Analysis3. View 1.3 Sample Sales - Product Details
Some user types may just run one report and exit; they dont have to have a complex sequence.
You dont need to restrict yourself to one us er type per actual group of users . You could m odel s everal different instances of
each user group in o rder or increase the test coverage and reali sm of different reports running concurrently. Be aware that
the more us er types and tests you have, the longer it will take to build, and the more complex the troubleshooting could be.
Think Time
Users are not computers, which is a shame because it would make reasoning with them a lot easier. It also means that they
do not jus t one run report after another. They run a report, look at the numbers, get a cup of tea, change a prompt value, have
a chat with a colleague, and so on.
As part of your us er workload, plan in s ome think times. Don t spend too long thinking about it (geddit), becaus e remember
this user workload is an approximation of what a certain type of user is going to be do ing.
User concurrency
Once we have designed a s et of individual user workloads, we can bring these together into a
larger test to assess the performance of the system as a whole under load. The aim of load
testing is to assess what happens to the response time of reports, and hardware utilisation, as
the number of concurrent users on the system increases. There are three key things to define in
a load test:
the workload of each user
the different types of user
the number of concurrent users
The first two of these are dis cussed above. When it comes to us er concurrency, we need to be very careful how we derive our
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
9/26
df d mi b PRO i Are you a developer? Try out the HTML to PDF API
figure, because it has a big impact on the scale of the tests that we run and the discussion of the outcomes.
First, make s ure that you are clear in the definition of concurrency that you us e. These include:
Concurrent user sessions logged into OBIEE, viewing das hboards, formatting reports, and so on.
Concurrently executing queries at the BI Server level, which could be a single user running one report or one
dashboard.
Concurrently executing queries on the database, which could be a single user in OBIEE running a s ingle report or
dashboard requiring more than one SQL request to get its data.
Each of these have their own particular load on the system, but can eas ily differ by orders o f magnitude. Typically a user
concurrency figure will be derived from bums on seats , i.e. number of users . From this, a reckoning is made on how many
will be us ing the system at one time. So unles s its qualified otherwise, I would work with the first definition concurrent
user sessions logged into OBIEE. Butif this is your definition, remem ber it when we come to simu lating users later on
For some reason, user concurrency is often a contentious num ber to define. With the greatest of apologies to Monty Python,
I offer this paraphas ing of a familiar conversation:
yah, my system needs to support 100 concurrent users
100? I d take that as a QUIET DAY for m y system, bwahaha, I need to test at least 1,000 concurrent users!
1,000? You were LUCKY! My system would run a gazilli on concurrent users at three oclock in the morning, work twenty
hours down tpit and still run 5,000 concurrent users be fore having a handful of cold g ravel for its tea
There does s eem to be BSD s yndrome around user concurrency figures, with people always
pitching high num bers. This is counterproductive. If you run a performance test us ing a
completely simulated workloadwith an excess ively high num ber of users as the target then the
margin of error (how accurately your test represents reality) inherent in the s imula tion
approach will be magnified as the user concurrency increases. The net effect of this i s that you
cannot be confident that any problems that do happen will actually happen in real life. Conversly,
if you are taking real workloadand running it for many users, you can hones tly say if it doesnt
work in testing, it wont work in real life.
If you are worried about pitching your user estim ate too low and s o double it or stick a zero on
the end jus t in case, cons ider the following: If you define your test as being a fail if it does not support your high figure of
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttps://www.youtube.com/watch?v=13JK5kChbRw&__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186 -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
10/26
df di b PRO i Are you a developer? Try out the HTML to PDF API
concurrent users what then? You have to ask for money, either literally to buy bigger hardware, or in kind, to support the
time to optimis e the system you do have. Its not a great place to be in, and one which may be unnecess ary. A sens ible
approach is the following:
1. Load test to confirm whether the system can support the realistic number of usersin the s hort-term, e.g. go-live and afew months beyond
2. Deferpart of the load testing until after go-live, when you can run a test for medium /long-term user numbers using realuser workload for your tests. The benefit of this is that your second test will be far more accurate because yoursim ulation will not be based on how you thinkyou system will be us ed but how you knowis it being used.
If this approach doesnt suit, then your alternative should be to
1. take a realistic figure to start with for your firsttest2. run a secondperformance test to validate a future growth figure for your user bas e. If you dont have a future growth
figure, then you could consider the second test being to sim ply determine how many users your system cansupport byramping num bers up until things s tart to creak.
The point of doing it this way is that the firsttest as sum ing it gives a pos itive resu lt gives clarity and focus to your overall
results. If the secondtest fails, then at leas t you have the confidence from the first with which to proceed with a decis ion. If
you jus t run one big test then its an all or nothing approach, and if the test fails you have the option of either delaying launch
until a success ful test, or just cross ing your fingers and hoping.
User concurrency numbers for an existing system
If you have an existing s ystem then it is easy to determine your actual us er concurrency through either Usage Tracking(you
do have it enabled, don t you?) or the OBI DMS metrics.
If your user concurrency figure is based on users logged into OBIEE, not necess arily running reports, then the m etric
Oracle BI PS Sessions : Sessions Logged Inshould be appropriate.
For the number of queries (logical SQL) concurrently running on the BI Server, use Us age Tracking (see below), or DMS
metric t.b.c!
The number of concurrent queries on the databas e can either be obtained from your databases monitoring tool, or
through the Oracle BI DB Connection PoolDMS metrics.
To find out more about viewing and capturing OBI metrics over time, see here.
To estimate us er concurrency from Usage Tracking, you can us e the following query. It will analyse Usage Tracking to give
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/2013/03/performance-and-obiee-execute-and-measure#measure -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
11/26
df di b PRO i A d l ? T t th HTML t PDF API
an approximate number of concurrent users , but this is based on individual queries run on the BI Server, so it not the sam e
as users simply logged into OBIEE, nor the same as dashboard refreshes(since one dashboard could be many queries):
Estimating user concurrency numbers
If you dont have empirical numbers to use then 10% of your total user base is the rule of thumb often worked to.
User concurrency is multi dimensional
Just as there is no s uch thing as A User, to ensure an accurate performance test user concurrency figures should also be
specified in terms o f :
Typesof concurrent user, for example:
10 Concurrent Finance Power Users
2 Concurrent Finance Ad-Hoc Users
60 Concurrent HR Report Consumers
Periods of system usage, for example :
Monday morning peak
EOD peak
Average
You may only actually test the peak user concurrency model, but its a good idea to have a picture of the s ystem profile,
particularly when analysing the im pact and capacity for ad-hoc usage.
Summary : User and Report Definition
-- Usage Tracking Concurrency usage estimate-- Per minute granularitySELECT TO_CHAR(start_ts,'YYYY-MM-DD HH24:MI')ASstart_ts_min,
COUNT(DISTINCT user_name) ASdistinct_usersFROM s_nq_acct
GROUP BY TO_CHAR(start_ts,'YYYY-MM-DD HH24:MI')ORDER BY1DESC
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
12/26df di b PRO i A d l ? T t th HTML t PDF API
Remember:
Before you can measure something you really need to lay down a very concrete definition of what youre measuring.
Zed Shaw
Whatever kind of testing you are doing, aim to make it modu lar wherever possible. It will make both analysis and reuse of
test code easier.
Almos t all test defini tions s hould be made up of one or m ore of the following levels :
1. Specific Report or dashboardalong with allprom pt values, hierarchical column settings, etc. This is the base unitofyour performance test.
2. Then take these individual tests, and use them together to create an individual User Workloadmodel for each user.3. Finally, combine the User models to produce an overall Concurrent Userworkload simulation.
See below for a worked example of a sam ple test definition.
Points to remem ber in your test definition:
1. Whilst each dashboard does have its own nam e, a good idea is to use your own ID system to make it easy to refer to aspecific dashboard state. For example , when using prompts there might be two or more versions of the dashboard totest (for different prompt values).Its als o easier to refer to som ething like ID01 in conversation and no tes than it is it Finance Dashboard ProfitabilityEMEA blah blah blah.
2. Dont forget default values in your navigation. If you are testing s pecific dashboard p rompt values, youll need to alsotest the default values first because that is what will get executed when the user first clicks onto the dashboard.
3. As well as testing das hboard response times, factor in the usability of any prompts that the user will interact with. A fastdashboard is not quite so impress ive if the user had to wait a long time for the dashboard prompts to display theavailable values .
4. You need to be writing down nuts and bolts detail here. Resis t the temptation to skimp on the detail or assume youllremem ber you wont, and certainly your colleague who can reuse your testing wont appreciate having to fill in the
blanks to make us e of it.
What do you mean, not all of it has been tested?Before you run off to write down all your dashboards and types of user, consider
the following. Unless you have :
1. A very sm all system2. A lot of time, and/or an infinite number of monkeys
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://zedshaw.com/essays/programmer_stats.html?__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186 -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
13/26df di b PRO i
Are you a developer? Try out the HTML to PDF API
you are not going to be able to actually test all of your OBIEE system. Fact.
There are too many factors (prom pt values , aggregate paths, ad hoc us age) to be
able to model all plausible (let alone possible) permutations of these.
Designing the scope of a performance tes t needs to take into account the
acceptable riskof things going wrong vs time/budgetavailable vs functionality
required. By functionality, I mean factors such as the breadth of dashboards
delivered, or data made available.
In some s ystems the stability is param ount and therefore the relative riskmus t be minim ised. If this is the case and your
time/budgetis cons trained then therefore the functionalitywill have to be constrained.
If you need to deliver a lot of functionalityand ass uming that time/budgetis fixed, then you will need to accept an
increased riskto the stability of your system.
If you have limi tless time/budgetthen you can deliver lots of functionalityand do large and wide-ranging performance
tests to ensure minimal risk.
The point here is an important one because it is one that needs discus sing with the project owner, or more specifically,whoever will be accountable if things dont go right in Production. For a standard OLTP system it is much easier (relatively) to
design a performance test with a high degree of confidence that mos t [comm on] workloads have been simulated and tested.
However, for a reporting system such as OBIEE there are two reasons why it isnt so easily determined, as dis cussed above
and re-iterated here:
1. Permutations of fixed reporting each dashboard can have multiple prompt dimensions, hierarchical columnexpansions. Users can follow many different drill and navigation paths from any one dashboard.
2. Ad hoc reporting by definition, the report is not known until the user builds it, so we cannot test it beforehand
But, all is not lost. Even if we accept that to test every permutation of report that might run is impos sible, it is still perfectlyrealistic to perform a thoroughperform ance assessm entof a system prior to go-live. The mos t sens ible option is a hybrid
approach of:
Traditional performance testing of fixed reports as described above
Testing predefined ad-hoc reports (if that isnt a paradox)
Holistic and heuris tic evaluation of the system des ign and data model
Resource management to curtail the impact of bad queries
A ti i t i t id tif b d i d d l ith th ( i th T E A th d)
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/2013/03/performance-and-obiee-introduction#tea -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
14/26df di b PRO iAre you a developer? Try out the HTML to PDF API
Active moni toring to identify bad queries and deal with them (us ing the T.E.A. method)
Feedback loop : as system usage becomes clearer, refine original performance testing us ing a more accurately
informed m odel (us ing the T.E.A. method)
This may seem a bit of an excuse for a cop-out. If we cant performance test all our system and be confident in the results,
whats the point? Here, I defer to Cary Millsap, a great proponent of the rigorous, s cientific, and rational approach to
performance testing:
How can you possibly test a new application enough to b e confident that youre not going to wreck your production
implementation with performance problems?
You can m odel. And you can test. However, nothing you do will be perfect. It is extremely di fficult to create models and tests in
which youll foresee all your production prob lems in advance of actually encountering those problem s in production.
Some people allow the apparent futility of this ob servation to justify not testing at all. Dont get trapped in that mentality. The
following points are certain:
Youll catch a lot more prob lems if you try to catch them prior to production than if you dont even try.
Youll never catch all your prob lems in pre-production testing. Thats why you need a reliab le and efficient method for
solving the prob lems that leak through your pre-production testing processes.
Somewhere in the m iddle b etween no testing and complete production emulation is the right am ount of testing. The right
amount of testing for aircraft manufacturers is probab ly more than the right amount of testing for companies that sell baseb all
caps. But dont skip performance testing altogether. At the very least, your performance test plan will make you a more
competent diagnostician (and clearer thinker) when it comes time to fix the performance prob lems that will inevitab ly occur
during production operation.
An example OBIEE test definitionHeres an example test definition, put together from SampleAppv207.
What we are going to model is two groups of users . One likes pretty pictures and just runs two chart-based das hboards and
then logs out. The other is interested in numbers and does some filtering on an analysis dashboard.
First, well define the dashboards in use. For brevity, Ive omitted the cons tituent analysis details, but in norm al practice its a
good idea to include them (for future reference, ease of understanding, clarity, etc).
D hb d
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/2013/03/performance-and-obiee-introduction#teahttp://method-r.com/downloads/doc_details/44-thinking-clearly-about-performance?__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186http://www.oracle.com/technetwork/middleware/bi-foundation/obiee-samples-167534.html?__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186 -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
15/26df di b PRO iAre you a developer? Try out the HTML to PDF API
Dashboards
Dashboard : ID01
Dashboard: 3.20 Vanilla Visuals
Page: Standard Visuals
Path: /shared/3. Analysis and Dashboards/_portal/3.20 Vanilla Visuals
Dashboard Prompts: None
Analyses:
/shared/3. Analysis and Dashboards/Report Views/Gallery/Examples of Standard Visualizations
Dashboard ID02
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_3.20_1.png -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
16/26
Are you a developer? Try out the HTML to PDF API
Dashboard : ID02
Dashboard: 3.20 Vanilla Visuals
Page: Trellis-Product Sales
Path: /shared/3. Analysis and Dashboards/_portal/3.20 Vanilla Visuals Trellis-Product Sales
Dashboard Prompts: None
Drill state: Default (NB each chart can be clicked on to drill into detail)
Analyses:
/shared/3. Analysis and Das hboards/Report Views/Trellis/Revenue Breakup
/shared/3. Analysis and Dashboards/Report Views/Trellis/Product Cost and Uni t Price Trend
/shared/3 Analysis and Dashboards/Report Views/Trellis/Brand Cos t and Unit Price Trend
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_3.20_prod_trellis.png -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
17/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
/shared/3. Analysis and Dashboards/Report Views/Trellis/Brand Cos t and Unit Price Trend
Offices Region Hier: All collapsed, except AMERICASexpanded one level
Dashboard : ID03
Dashboard: 1.3 Sample Sales
Page: Brand Analysis
Path: /shared/1. Quick Demos/_portal/1.3 Sample Sales Brand Analysis
Dashboard Prompts:
T05 Per Name Year: Default (2008, 2009, 2010, 2011)
D4 Company: Default (none)
T31 C l W k D f lt (b t 0 d 53)
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_1.3_brand_analysis.png -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
18/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
T31 Cal Week: Default (between 0 and 53)
Products.Products Hierarchy: Default (none)
Analyses:
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Product Line Analysis
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Trending Analysis
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Gauges
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Top Customers Bookings
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Daily Revenue Timeline
Dashboard : ID04
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_1.3_brand_analysis.png -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
19/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Dashboard: 1.3 Sample Sales
Page: Brand Analysis
Path: /shared/1. Quick Demos/_portal/1.3 Sample Sales Brand Analysis
Dashboard Prompts:
T05 Per Name Year: 2010, 2011
D4 Company: Stockplus Inc.
T31 Cal Week: between 1 and 27
Products.Products Hierarchy: Default (none)
Analyses:
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_1.3_brand_analysis.png -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
20/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Product Line Analysis
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Trending Analysis
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Gauges
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Top Customers Bookings
/shared/1. Quick Demos/Simple Demo Dashboard/Brand Analysis/Daily Revenue Timeline
Tip:Use Catalog Manager to generate a report of dashboards and their constituent analyses to make the creation of the
above specification easier. :
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_catman0.pnghttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_catman1.pnghttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_catman1.pnghttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_catman2.png -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
21/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
User Workload
Having defined the dashboards (ID01 ID04), we define the user types and workloads:
User A
Description: Departm ent X, executive
Workload model:
http://www.rittmanmead.com/wp-content/uploads/2013/03/perf_catman1.pnghttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_catman1.pnghttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_catman1.pnghttp://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/wp-content/uploads/2013/03/perf_catman2.png -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
22/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
1. ID01
2. Think time: 60 seconds
3. ID02
4. Think time: 60 seconds
5. Logout
User BDescription: Department Y, analyst
Workload model:
1. ID03
2. Think time: 120 seconds
3. ID04
4. Think time: 90 seconds
5. Logout
Concurrency
Our concurrent user model looks like this. I am defining concurrent usersas active users, that is, executing dashboards or
simulated thinking.
Peak usage
User A : 5 concurrent users
User B : 2 concurrent users
Light usage
User A : 3 concurrent users
User B : 1 concurrent users
Types of testingBefore we can consider the types of test, we need to define what it iswe are tes ting, hence the long section above. Once
youve identified the types of user and s pecific reports they will run, you s hould also look at the type of test you want to run.
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
23/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
The terminology is frequently used interchangably, so heres my version of it:
Performance testing
With a single user, is the response time acceptable?
Load testing
With concurrent users, is the response time acceptable and s table?
Capacity testing
How does the system behave when concurrent users are sim ulated? do any areas s how signs of reaching capacity
of resource?
Soak testing
How does the system behave if the concurrent user workload is left to run for many hours? For example, are there any
mem ory leaks lurking, or connections which are not closed down properly and will eventually run out?
Stress testing
As we increas e the number of concurrent users beyond that required (and already tested success fully), how far willthe system go until something breaks or response times suffer?
In a high-availability deployment, what happens to response times if we simulate partial stack failures? Does the
system handle this gracefully under load?
These types of test do not have to be mututally exclusive in execution, but in defining your testing make sure you have covered
them all, or explained why they are not necess ary. An example would be performance tes ting a single report change on an
existing s ystem would probably not warrant a full scale s oak test.
In summaryThe output of this phas e of the method should typically be a set of test definitions, not one s ingle one. As already discuss ed,
keeping tests modular and simple is a much better approach than trying to kill all the birds with one boulder.
Footnote: The soft squidgy side of performance testing
Tangent: Enterprise change control processes
Taking the premise dis cussed above that it is impos sible to test every poss ible report execution raises an in teresting
question around change control and the stability of a Production environment. If your company is one where releases m ust
b t i l t t d d i d ff i th bl d f fi tb th i ht t t h thi ti th th
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
24/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
be triple tested and signed off in the blood of your firstborn the you might want to have this conversation sooner rather than
later. Why? Well if your system has any element of ad-hoc usage then there is a chance that a user will run a query that
disrupts other users on the system. There is always the potential for the perfect storm of it being a critical business period,
high us er traffic, and a rogue ad-hoc query. Without proper understanding of the nature of ad-hoc queries , an alm ighty witch
hunt can ensue, looking for who broke Production.
If this kind of risk is unacceptable then you will have to look into mitigating it with things like resource m anagement and
environment partitioning.
Estimating performance test for project planning
The challenge that the complexity and multiple facets of performance testing gives is finding a way to communicate it
accurately to project managers and business sponsors.
Unlike an es timate for code delivery, which will be days and after which the code will e ither function or not, performance
testing can pretty much take as long as you like, and the kicker is that the tests will only give varying degrees of confidence
that things shouldbe OK in production, but with no guarantees .
Doing the initial analysis on the system is a fixed time, as is building the test rig(s) with which to test it. Once you have donethis, then defining the number of dashboards to test can be done based on the time available. The more dashboards are
tested, the greater certainty can be had that any problems will be uncovered. Other fixed tasks s hould be a review of the
overall des ign, as well as time to properly document the tests undertaken and an analysis of the results obtained. It is a false
economy to skimp on the documentation, as to do so renders almos t useless the results ob tained for future work. Good
documentation of testing undertaken and results found can greatly shorten the time necess ary in the future when revisiting
the work either for a new phas e of development or to troubleshoot performance problems that have arisen.
What next?
Next up is designingthe test that weve just defined read on!
Other articles in this seriesThis article is part of a series on the subject of Performance and OBIEE:
1. Introduction2. Test Define3. Test Design
4. Test Build5 Execute
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.rittmanmead.com/2013/03/performance-and-obiee-test-designhttp://www.rittmanmead.com/2013/03/performance-and-obiee-test-designhttp://www.rittmanmead.com/2013/03/performance-and-obiee-introductionhttp://www.rittmanmead.com/2013/03/performance-and-obiee-test-definehttp://www.rittmanmead.com/2013/03/performance-and-obiee-test-designhttp://www.rittmanmead.com/2013/03/performance-and-obiee-test-buildhttp://www.rittmanmead.com/2013/03/performance-and-obiee-execute-and-measure -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
25/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Tweet 0 2Like
5. Execute6. Analyse7. Optimise8. Summary and FAQ
Comments?Id love yourfeedback. Do you agree wi th this method, or is it a waste of time? What have I overlooked or overemphas ised?
Am I flogging a dead horse?
Because there are several articles in this s eries, and Id like to keep the discus sion in one place, Ive enabled comments on
the summary and FAQpost here, and disabled comments on the others.
Related Posts:
Performance and OBIEE part I Introduction
Performance and OBIEE part VII Optimising OBIEE performance (Tuning)
Performance and OBIEE part VI Analysing results
Posted in Oracle BI Suite EE, Performance| Comments Off
Comments are closed.
Call us now to talk about your BI project:
+44 (0) 1273 911 268 (UK) or(888) 631-1410 (USA)
or+61 3 9596 7186 (Australia & New Zealand) or
+91 997 256 7970 (India)or+32 280 882 11 (Belgium)
Share 2
Rittman Mead Consulting ltd
http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttps://twitter.com/intent/tweet?original_referer=http%3A%2F%2Fwww.rittmanmead.com%2F2013%2F03%2Fperformance-and-obiee-test-define%2F&text=Performance%20and%20OBIEE%20%E2%80%93%20part%20II%20%E2%80%93%20Test%20%E2%80%93%20Define&tw_p=tweetbutton&url=http%3A%2F%2Fwww.rittmanmead.com%2F2013%2F03%2Fperformance-and-obiee-test-define%2F&via=rittmanmeadhttp://twitter.com/search?q=http%3A%2F%2Fwww.rittmanmead.com%2F2013%2F03%2Fperformance-and-obiee-test-define%2Fhttp://www.rittmanmead.com/2013/03/performance-and-obiee-execute-and-measurehttp://www.rittmanmead.com/2013/03/performance-and-obiee-analysing-resultshttp://www.rittmanmead.com/2013/03/optimising-obiee-performance-tuninghttp://www.rittmanmead.com/2013/03/performance-and-obiee-summary-and-faqhttp://www.rittmanmead.com/2013/03/performance-and-obiee-summary-and-faqhttp://www.rittmanmead.com/2013/03/performance-and-obiee-introduction/http://www.rittmanmead.com/2013/03/optimising-obiee-performance-tuning/http://www.rittmanmead.com/2013/03/performance-and-obiee-analysing-results/http://www.rittmanmead.com/category/oracle-bi-suite-ee/http://www.rittmanmead.com/category/performance/http://www.rittmanmead.com/ -
8/10/2019 Www Rittmanmead Com 2013 03 Performance and Obiee Test Defin
26/26
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Website Design & Build: tymedia.co.uk
Home
About Us> About us
> About our team
> Contact us
> Our clients
Services> Consulting
> Training
> Support
ConsultingServices> Projects
> Expert Services
> OBIEE 11g
> Sustainability
> On Discoverer?
> Oracle DW
Training> OBIEE
Bootcamp
> OBIEE End-User
> Exalytics
> ODI 11g
Bootcamp
> Oracle BI Apps
Resources> Articles
> Blog
> OBIEE 11g
Blog Authors> Mark Rittma n
> Venkat J
> Peter Scott
> Borkur S
> Mike Vic kers
> Robin Moffatt
> Jon Mead
Rittman Mead Consulting ltd.
Registered Office : Suite B,
First Floor Moore House,
13 Black Lion Street,
Brighton, East Sussex,
BN1 1ND, United Kingdom
Company No. : 6032852
VAT No. : 900 3839 48
Rittman Mead America, Inc.
Registered Office : 4550 North Point Parkway
390 Alpharetta, Georgia 30022, USA
Rittman Mead Oceania Pty Ltd.
Registered Office : 12 Moore Street,
Brighton East,
Victoria, 3187, Australia
Austral ian Company No. : 1 49 458 9 35
Rittman Mead Consulting Pvt Ltd.
Registered Office : Unit 105-106
Regent Prime
Whitefield Main Road
Whitefield
Bangalore
560066
Rittman Mead Belgium
Registered Office : Chausse de Louvain 426
1380 Lasne
Belgium
2010-2011 Rittm an Mead Cons ulting. | Privacy Policy | E: info@rittm anmead.com
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.rittmanmead.com%2f2013%2f03%2fperformance-and-obiee-test-define%2f&id=ma-141108035341-df455196http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.tymedia.co.uk/?__hstc=150481449.35d69ee28ad764e69b542078dfc1c354.1415436819737.1415436819737.1415436819737.1&__hssc=150481449.1.1415436819737&__hsfp=1770896186http://www.rittmanmead.com/http://www.rittmanmead.com/about/http://www.rittmanmead.com/about/our-team/http://www.rittmanmead.com/about/contact-us/http://www.rittmanmead.com/clients/http://www.rittmanmead.com/consultinghttp://www.rittmanmead.com/traininghttp://www.rittmanmead.com/supporthttp://www.rittmanmead.com/consulting/expert-services/http://www.rittmanmead.com/consulting/expert-services/http://www.rittmanmead.com/consulting/obiee-11g-services/http://www.rittmanmead.com/consulting/prepare-build-and-sustain/http://www.rittmanmead.com/consulting/migrating-from-discoverer-to-oracle-bi-suite-enterprise-edition/http://www.rittmanmead.com/consulting/data-warehousing/http://www.rittmanmead.com/training/trn202-oracle-bi-ee-11g-bootcamp/http://www.rittmanmead.com/training/trn-205-oracle-bi-ee-11g-create-reports-dashboards-alerts-and-scorecards/http://www.rittmanmead.com/training/trn-701-exalytics-for-systems-administrators/http://www.rittmanmead.com/training/trn-403-odi-11g-bootcamp/http://www.rittmanmead.com/training/trn-40-oracle-bi-apps-bootcamp/http://www.rittmanmead.com/articleshttp://www.rittmanmead.com/bloghttp://www.rittmanmead.com/obiee11ghttp://www.rittmanmead.com/author/mark-rittman/http://www.rittmanmead.com/author/venkatakrishnan-j/http://www.rittmanmead.com/author/peter-scott/http://www.rittmanmead.com/author/borkur-steingrimsson/http://www.rittmanmead.com/author/mike-vickers/http://www.rittmanmead.com/author/robin-moffatt/http://www.rittmanmead.com/author/jon-mead/http://www.rittmanmead.com/privacy-policy/mailto:[email protected]