1

Qualogy 2014-08-29 M. Schildmeijer

Whitepaper

Oracle Exalogic FMW Optimization

2

Inhoudsopgave

1. Preface ................................................................................................................................. 3

2. WebLogic Domain Level ...................................................................................................... 4

2.1 Domain Enhancements .................................................................................................. 4

2.2 JDBC SDP enhancement .............................................................................................. 4

2.3 Using AdminServer VIPS for floating IP ......................................................................... 5

2.4 WebLogic Admin channel over InfiniBand private net .................................................... 5

2.4.1 HTTP Client channel for AdminServer ................................................................ 5

2.4.2 T3 Client channels .............................................................................................. 6

2.5 Using Embedded Coherence (Several FMW products, SOA suite, IDM) ....................... 7

2.5.1 Configuring Coherence Caches for Dehydrations .................................................... 7

2.5.2 Enabling the property ............................................................................................... 7

2.5.3 Update SOA JVM settings ....................................................................................... 8

2.5.4 Overview of tuning settings ...................................................................................... 8

2.6 Cluster replications ....................................................................................................... 10

2.7. Configure JMS for Exalogic ......................................................................................... 12

2.7.1 JDBC Store ........................................................................................................... 12

2.7.2 JMS and Exalogic .................................................................................................. 12

2.8 Other Considerations .................................................................................................... 13

2.8.1 Hugepages and Transparent HugePages .............................................................. 13

3. Oracle Traffic Director enhancements ................................................................................ 14

3

1. Preface

For each WebLogic / FMW environment running on Exalogic, some extra optimization work

needs to be done in order to get the biggest benefit in terms of resources and performance

you can get when combining an Exalogic with an Oracle FMW. This whitepaper describes

the different configurations to optimize.

This whitepaper focuses on:

Java JVM/WebLogic/Exalogic optimizations

Domain Level enhancements such as: Cluster replication channels, SDP

enhancements

JDBC SDP enhancements

Domain specific enhancements

JMS & TLogs in database (after optimization)

SOA Suite specific optimizations

Cluster channel session replications

Coherence WKA over InfiniBand

Separate SOA traffic from Admin traffic

HTTP

Use of Oracle Traffic Director or use MNs Apache HTTP Proxy

SSL termination and offload

4

2. WebLogic Domain Level

2.1 Domain Enhancements

On a domain level, the next options have to be set:

-Dweblogic.ScatteredReadsEnabled=true Better I/O in high network volume

environments ( OSB, Webcenter Portal)

-Dweblogic.GatheredWritesEnabled=true

-Dweblogic.replication.enableLazyDeserialization=true

To be set in setDomainEnv

# Enable Java Exalogic optimizations

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}

-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true

-Dweblogic.ScatteredReadsEnabled=true

-Dweblogic.GatheredWritesEnabled=true

-Dweblogic.replication.enableLazyDeserialization=true"

export EXTRA_JAVA_PROPERTIES

2.2 JDBC SDP enhancement

Besides configuring JDBC and GridLink for RAC to use InfiniBand, DB connections must use

the SDP protocol to improve performance:

Complete the following steps:

1. Ensure that you have created the Grid Link Data Sources for the JDBC connectivity

on both Exadata nodes

2. The console automatically generates the complete JDBC URL, as shown in the

following example: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))(C

ONNECT_DATA=(SERVICE_NAME=)))

3. In the JDBC URL, replace TCP protocol with SDP protocol. For example: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=sdp)(HOST=)(PORT=1521))(C

ONNECT_DATA=(SERVICE_NAME=)))

4. Manually add the system property -Djava.net.preferIPv4Stack=true to the

startWebLogic.sh script. Locate the following line in the startWebLogic.sh script:

a. . ${DOMAIN_HOME}/bin/setDomainEnv.sh $*

b. Add the following property immediately after the above entry: JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true

-Doracle.net.SDP=true"

By having all these enhanced features, storing JMS artifacts in a JDBC store is more valid

than ever.

5

2.3 Using AdminServer VIPS for floating IP

The way of setting up an AdminServer VIP for HA and failover will be described in another

document. We assume that it is available for use.

2.4 WebLogic Admin channel over InfiniBand private net

The private InfiniBand fabric including WebLogic Clusters and Coherence clusters requires a

set of IP addresses for all WebLogic Managed Server-to-Administration Server traffic and for

cluster communication. These IP addresses are associated with the BOND0 interface.

Within Exalogic it is possible to use a private vnet that can be reserved for administrative

purposes, such as accessing the AdminServer and its consoles, internal cluster

communication and so on.

This can be easily achieved by setting all listen addresses to the IP that is bound to

InfiniBand

We do the same for all managed servers and their listen addresses

Trouble is that we cant access the AdminServer url over IPOIB. For this we create separate

channels.

Also for the managed servers, which have client access, we create channels to connect to

the outside world.

There are several channels to identify:

There are several channels for EoIB interfaces, such as:

AdminServer Network Channel

HTTP and T3 Network Channels

These are needed for internal resources, like AdminServers or to let managed servers

communicate with the outside world.

You must create two network channels for the Administration Server. The network channels

are necessary for routing HTTP traffic and T3 traffic (TCP-based protocol used by WebLogic

Server) coming in from an external data center via Ethernet over InfiniBand (EoIB). You must

create the following network channels:

2.4.1 HTTP Client channel for AdminServer

1. Log in to the WLS console and click Lock & Edit in the Change Center.

2. In the left pane of the Console, expand Environment, and then Servers.

3. In the Servers tab, click AdminServer (admin).

4. Select Protocols Channels. Click New.

6

5. Enter AdminHTTPClient as the name of the new network channel, select http as the

protocol, then click next.

6. Enter the following information in the Network Channel Addressing page:

Listen address: < the EOIB IP address >. This address is the floating IP

assigned to the Administration Server using the BOND interface.

Listen port: 7001

7. Click next, and select Enabled on the Network Channel Properties page and click

Finish.

8. Activate these changes.

2.4.2 T3 Client channels

Repeat the same steps as above but choose T3 as a protocol.

Repeat these steps for all managed servers as well.

Example channels configured in a domain

AdminServer

Managed Servers

When started, you see the several interfaces which are listened to:

Sep 5, 2014 4:49:15 PM CEST>

And you still can use the normal AdminServer address to logon, either hostname or Admin

VIP configured.

When you configure Oracle Traffic Director, this should be configured using the IPoIB, in

every other setup you choose the EoIB, but then you miss all IB enhancements.

7

2.5 Using Embedded Coherence (Several FMW products, SOA

suite, IDM)

With optimizations in SOA Suite the process state can now reside in Oracle Coherence -

which is an in-memory data grid. Coherence uses direct memory access for replicating data.

We configure coherence using Unicast; therefore we should disable all multicast entries,

located in the setDomainEnv.

Furthermore, we should set all WKA addresses to use the IPoIB interface, by setting the

virtual hostname that is set for the IP address. This is typically the VIP address used for SOA

internal traffic (attached to IB interface)

-Dtangosol.coherence.wka1=SOAHOST1-PRIV-V1

-Dtangosol.coherence.wka2=SOAHOST2-PRIV-V1

-Dtangosol.coherence.localhost=SOAHOST1-PRIV-V1

SOAHOST1-PRIV-V1 is the virtual host name that maps to the virtual IP where WLS_SOA1

is listening (in SOAHOST1), and so on.

2.5.1 Configuring Coherence Caches for Dehydrations

Dehydrations take place in the SOA repository database, but you can use Coherence to

offload this kind of work. If you configure the BPEL engine with the CacheEnabled property,

the engine runs much fewer reads against database. Because many reads require locks and

version checks, eliminating them improves BPEL engine performance. Configuring Oracle

Coherence for dehydration requires the following steps:

2.5.2 Enabling the property

To enable the cache enabled property in SOA, follow these steps:

1. Log in to Oracle Enterprise Manager Fusion Middleware Control

2. Expand Farm_domain_name, SOA, and then right click soa-infra (server_name)

(either one of the servers).

3. Click SOA Administration, and then BPEL Properties.

4. Select More BPEL Configuration properties.

5. Enter CacheEnabled for the property QualityOfService property.

6. Select Apply.

Next, set the required server properties for using in-process coherence cache for

dehydration.

To enable the cache enabled property in SOA:

1. Log into the Administration Console

2. In the Domain Structure window, expand the Environment node.

8

3. Click Servers.

4. Click the name of the server (WLS_SOA1 or WLS_SOA2, which are represented as

hyperlinks) in the Name column of the table.

5. The settings page for the selected server appears.

6. Click Lock & Edit.

7. Click the Server Start tab.

8. In the Arguments field, add the following for all SOA managed servers

-Dbpel.cache.localStorage=true

-Dbpel.cache.threadCount=20

-Dbpel.cache.cubeInstance.sizeLimit=4g

-Dbpel.cache.invokeMessage.sizeLimit=2g

-Dbpel.cache.deliveryMessage.sizeLimit=2g

-Dbpel.cache.deliverySubscription.sizeLimit=2g

2.5.3 Update SOA JVM settings

To optimize behavior with local storage caches and improve performance, update the SOA

servers' start parameters to include the following flags:

-Djava.net.preferIPv4Stack=true

-Xlargepages:exitOnFailure=true

-Doracle.xdkjava.exalogic.optimization=true

-Xmsg -Xmxg ( heapsize used)

2.5.4 Overview of tuning settings

Maybe an open door, but every setting you optimize must have a reason, so do not use all

these setting just like that. Do it one by one with a group of parameters, and test it. The

outcome of your analysis will give you a guideline of what is applicable and what is not. To

be set in the setDomainEnv, or better use a separate parameter file and reference it to the

setDomainEnv.

Java Optimizations: EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}

-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true

-Dweblogic.ScatteredReadsEnabled=true

-Dweblogic.GatheredWritesEnabled=true

-Dweblogic.replication.enableLazyDeserialization=true"

export EXTRA_JAVA_PROPERTIES

Coherence Optimizations (after Java Optimizations are enabled) # Used with InfiniBand

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}

-Dsoa.archives.dir=${SOA_ORACLE_HOME}/soa

-Dsoa.oracle.home=${SOA_ORACLE_HOME}

-Dsoa.instance.home=${DOMAIN_HOME}

-Dtangosol.coherence.log=jdk

-Dtangosol.coherence.transport.reliable=imb

9

InfiniBand enabling! IMB driver must be enabled for that. Only for Linux x86- 64 bit platforms.

-Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl

-Dweblogic.transaction.blocking.commit=true

-Dweblogic.transaction.blocking.rollback=true

-Djavax.net.ssl.trustStore="

export EXTRA_JAVA_PROPERTIES

BPEL Optimizations. The main parameter is the bpel.cache.localStorage; set it to true

and then tune the other parameters to your needs, like cache sizes and worker

threads. EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}

-Dbpel.cache.localStorage=true

-Dbpel.cache.threadCount=20

-Dbpel.cache.cubeInstance.sizeLimit=4g

-Dbpel.cache.invokeMessage.sizeLimit=2g

-Dbpel.cache.deliveryMessage.sizeLimit=2g

-Dbpel.cache.deliverySubscription.sizeLimit=2g"

export EXTRA_JAVA_PROPERTIES

Note that all the values are examples and can differ from your situation. Always measure,

analyze, test and decide!

10

2.6 Cluster replications

This applies only for stateful applications such as B2B Console, or SOA/BPM Composer.

To enable session replication enhancements for the SOA_Cluster:

1. Shut down the servers in the SOA_Cluster.

2. To set replication ports for a managed server, such as WLS_SOA1:

a. Under Domain Structure, click Environment and Servers. Click on the SOA

managed server, Configuration tab, Cluster tab.

b. In the Replication Ports field, enter a range of ports for configuring multiple

replication channels. For example, replication channels for managed servers

in SOA_Cluster can listen on ports starting from 8006 to 8009. To specify this

range of ports, enter 8006-8009.

c. Repeat these steps for all managed servers

3. Create a custom network channel for each managed server in the cluster:

Log on to the Admin Console in the Servers table; click the SOA managed server

instance.

a. Select Protocols, and then Channels, and click New.

b. Listen Address: SOAHOST1-PRIV-V1 ( (IPoIB Address )

4. Listen port: 8006 Remove the default external listen port.

5. Click next, and in the Network Channel Properties page, select Enabled and

Outbound Enabled, then click Finish.

6. Under the Network Channels, select ReplicationChannel, the network channel you

created for the WLS_SOA1 managed server.

11

7. Expand Advanced, and select Enable SDP Protocol and click Save.

You must repeat these steps to create a network channel, each for the remaining managed

servers in the SOA_Cluster cluster.

Next you should specify the replication channel to you cluster

1. Click the Replication tab.

2. In the Replication Channel field, ensure that ReplicationChannel is set as the name of

the channel to be used for replication traffic.

3. In the Advanced section, select the Enable One Way RMI for Replication option, and

click save.

4. Activate the changes and restart the managed servers.

5. Add the system property -Djava.net.preferIPv4Stack=true to the start parameters for

the WebLogic servers.

a. In the Domain Structure window, expand the Environment node and click

servers.

b. Click the name of the server.

c. Enter ReplicationChannel as the name of the new network channel and select

t3 as the protocol, then click next.

d. Click the Server Start tab. and add the following for the managed servers into

the Arguments field:

-Djava.net.preferIPv4Stack=true

Note that the domain should be restarted.

See logs, to see if it succeeded.

12

2.7. Configure JMS for Exalogic

Although not a specific Exalogic enhancement, using all InfiniBand and SDP features storing

JMS Artifacts in a database is the preferred way to go.

2.7.1 JDBC Store

From here use the common methods to point JMS into JDBC store. However, keep the

following in mind:

Ensure that the JDBC data source used for the database work exists on the same

server instance as the JMS destinationthe transaction will still be two-phase, but it

will be handled with less network overhead.

Configure multiple services to share the same store if they will commonly be invoked

within the same transaction.

Do no use the XA driver for the JDBC Store, because the JMS infrastructure of

WebLogic itself is fully equipped for XA

JMS often executes direct database operations to invoke its store services within the

same transaction, so using a JDBC data source with Logging Last Resource (LLR)

might be a good option. It will do two-phase commit in a single local transaction DB

operation, this improves the overall transaction performance.

2.7.2 JMS and Exalogic

There are some enhancements specifically for Exalogic in doing message compression,

either persistency is file based or JDBC bases. The options you have to set are:

StoreMessageCompressionEnabled: Enabling Message Store Compression. For

Exalogic set this to true to use with the belonging options.

MessageCompressionOptions: Along with the option above, choose one of the

compression options in it. Note that compression is done by gzip, so you can choose

the best compression:

GZIP_DEFAULT_COMPRESSION

GZIP_BEST_COMPRESSION

GZIP_BEST_SPEED

LZF

PagingMessageCompressionEnabled setting. It's the JMS paging store to perform

message body compression on persistent and non-persistent messages.

JMS also makes use of the domain wide Exalogic optimizations by using concurrent queues

for pending messages. A buffer queue is used for incoming requests which increases

throughput by scheduling these requests and prevent locks.

13

2.8 Other Considerations

Some other considerations you could think of, either by determining if its applicable to your

situation, it may come out of some load or stress test scenario and analysis.

2.8.1 Hugepages and Transparent HugePages

Hugepages allocation can be good for applications which have to allocate large memory

blocks.

In your Exalogic VM infrastructure design, decide which VM will be used for such

applications like WebCenter Portal, and if they need it. In some situation, it can be necessary

to reserve a compute node for these environments instead of doing dynamic allocation, and

you might consider using Transparent HugePages for these.

For more info, see this fantastic article: http://www.qualogy.com/hugepages-for-weblogic/

http://www.qualogy.com/hugepages-for-weblogic/

14

3. Oracle Traffic Director enhancements

In almost every Enterprise Deployment Guide from a Fusion Middleware Product, Oracle

uses the Oracle WebTier as a controlling layer for you HTTP calls.

In Exalogic, a software loadbalancer is added, called Oracle Traffic Director. Depending on

your approach of Cloud infrastructure with one of these components

Network topology

Segregations of duties and tasks rolled out over several departments, or done by one

department overall? Who takes care of what? (Cloud Infrastructure team, Middleware

Team, all by one DevOps team)

Other solutions than those provided by Oracle (3rd party WebTiers, hardware

loadbalancers)

Besides the common steps to take in order to use Oracle Traffic Director, there are some

specific topics to address. These will be covered in a separate whitepaper.