A DAPT IST-2001-37126 Clustering Implementation in JBoss Jakša Vučković Università di Bologna.

AADAPT DAPT IST-2001-37126IST-2001-37126

Clustering Implementation in JBossClustering Implementation in JBoss

Jakša VučkovićJakša Vučković

Università di BolognaUniversità di Bologna

Clustering Implementation in JBoss – Jakša Vučković, Università di Bologna2 ADAPT

Clustering J2EEClustering J2EE

• Replicating ResourcesReplicating Resources• CommunicationCommunication• Concurrency ControlConcurrency Control• Failure HandlingFailure Handling• Controlling Access to External ResourcesControlling Access to External Resources


Replicating ResourcesReplicating Resources

• Stateful Session Bean ReplicationStateful Session Bean Replication• Entity Bean Replication Entity Bean Replication • HTTP Session ReplicationHTTP Session Replication• JNDI Tree ReplicationJNDI Tree Replication


Sticky Sessions and Homogeneous DeploymentSticky Sessions and Homogeneous Deployment

• JBoss’ staff recommends the use of sticky sessions and JBoss’ staff recommends the use of sticky sessions and homogeneous deploymenthomogeneous deployment– Using sticky sessions means that all requests of a client Using sticky sessions means that all requests of a client

arrive to the same node during a sessionarrive to the same node during a session

– Homogeneous deployment means that each application Homogeneous deployment means that each application component is replicated an all the nodes, so requests don’t component is replicated an all the nodes, so requests don’t need to span on different nodesneed to span on different nodes

• Sticky sessions + homogeneous deployment imply that Sticky sessions + homogeneous deployment imply that each request will be entirely processed on one nodeeach request will be entirely processed on one node


Stateful Session BeansStateful Session Beans

• Each instance of a SFSB is associated to a clientEach instance of a SFSB is associated to a client• The lifetime of a SFSB is determined by a timeout The lifetime of a SFSB is determined by a timeout

from the last client accessfrom the last client access• EJBs cannot generate threadsEJBs cannot generate threads• All access to a SFSB instance should be serializedAll access to a SFSB instance should be serialized• SFSBs are non-reentrantSFSBs are non-reentrant


SFSB ReplicationSFSB Replication

• After each invocation of a SFSB, if the bean changed, After each invocation of a SFSB, if the bean changed, a message is broadcast to the clustera message is broadcast to the cluster

• The message contains the serialized state of the session The message contains the serialized state of the session beanbean

• Before broadcasting the message a distributed lock is Before broadcasting the message a distributed lock is acquiredacquired

• The broadcast operation is synchronous (blocking) The broadcast operation is synchronous (blocking) with acknowledgements from all the nodes of the with acknowledgements from all the nodes of the clustercluster


RMI ClientsRMI Clients

• The clients downloads an “intelligent” stub from the The clients downloads an “intelligent” stub from the server and uses it to access the beanserver and uses it to access the bean

• The stub encapsulates the load balancing policy and The stub encapsulates the load balancing policy and the failover mechanismthe failover mechanism

• The stubs maintains a list of server addresses which is The stubs maintains a list of server addresses which is updated on each requestupdated on each request


RMI Clients - FailoverRMI Clients - Failover

• If a call to a node fails the client stub fails over to If a call to a node fails the client stub fails over to another server or raises an exception to the client another server or raises an exception to the client applicationapplication

• The stub fails over to another server when:The stub fails over to another server when:– there is a communication exceptionthere is a communication exception– The server does not responds because it crashed or a network The server does not responds because it crashed or a network

partition occurredpartition occurred

• The stub reports an exception when:The stub reports an exception when:– There are no more servers to fail over toThere are no more servers to fail over to– The server reported a GenericClusteringException with the The server reported a GenericClusteringException with the

flag COMPLETED_YES or COMPLETED_MAYBEflag COMPLETED_YES or COMPLETED_MAYBE


RMI Clients - FailoverRMI Clients - Failover

• If a client makes a request to a node and does not If a client makes a request to a node and does not receive the response four scenarios are possible:receive the response four scenarios are possible:

1.1. The node crashed before receiving the request – failover The node crashed before receiving the request – failover safesafe

2.2. The node received the request but crashed before The node received the request but crashed before broadcasting the state – failover safebroadcasting the state – failover safe

3.3. The node received and processed the request, broadcast the The node received and processed the request, broadcast the state and then crashed before sending the reply – the state and then crashed before sending the reply – the failover node must recognize a duplicate requestfailover node must recognize a duplicate request

• The client indicates in the request message if it is a The client indicates in the request message if it is a failover or the original requestfailover or the original request


Partial Replication ProblemPartial Replication Problem

• We have two beans S1 and S2 on two nodes A and BWe have two beans S1 and S2 on two nodes A and B

Client

S1 S2S2

node A

node B

S1 S2S2S1



• The client invokes S1 on node A The client invokes S1 on node A

Client

S1 S2S2

node A

node B

S1 S2S2S1



• S1 invokes S2S1 invokes S2

Client

S1 S2S2

node A

node B

S1 S2S2S1



• Invocation of S2 is completed and its state is replicated Invocation of S2 is completed and its state is replicated on node B on node B

Client

S1 S2S2

node A

node B

S1 S2S2S1



• S1 tries to make another call to S2 but node A crashesS1 tries to make another call to S2 but node A crashes

Client

S1 S2S2

node A

node B

S1 S2S2S1



• Now we have an inconsistent state in node BNow we have an inconsistent state in node B

Client

S1 S2

node B



• There are two possible solutions to this problemThere are two possible solutions to this problem::– A replication aware Transaction Manager would need to roll A replication aware Transaction Manager would need to roll

back the state of S2 in B alsoback the state of S2 in B also

– The updated state is broadcast to other nodes only when The updated state is broadcast to other nodes only when committing the whole transactioncommitting the whole transaction


Network PartitionsNetwork Partitions

• A network failure could divide the cluster in two or A network failure could divide the cluster in two or more partitions that cannot communicatemore partitions that cannot communicate

Node A

Node C

Node B

Node D

No shared data (Entity Beans, DB) is accessed!!!



• The client stub maintains a list of available serversThe client stub maintains a list of available servers• If a client is not able to contact the server instance it is If a client is not able to contact the server instance it is

bound to, it tries to contact the next from the list until it bound to, it tries to contact the next from the list until it reaches an instance that is its own partitionreaches an instance that is its own partition

Node A

Node C

Node B

Node Dclient






Node A

Node C

Node B

Node Dclient




• Each SFSB is associated to a client, so there will be no Each SFSB is associated to a client, so there will be no other clients accessing the instance on another partitionother clients accessing the instance on another partition

Client2

Client1

s2s3

s1

s2s3

s1

s2s3

s1

s2s3

s1A B

C D




• Each clients list of available servers will be updated to Each clients list of available servers will be updated to contain only the nodes on its partition contain only the nodes on its partition

Client2

Client1

s2s3

s1

s2s3

s1

s2s3

s1

s2s3

s1A B

C D



Network Partitions MergingNetwork Partitions Merging

• When the cluster merges the clients will still have the When the cluster merges the clients will still have the old list of nodesold list of nodes

Client2

Client1

s2s3

s1

s2s3

s1

s2s3

s1

s2s3

s1A B

C D




• Only when the client makes the next request, the bean Only when the client makes the next request, the bean gets replicated on the other nodes and the client’s list is gets replicated on the other nodes and the client’s list is updatedupdated

Client2

Client1

s2s3

s1

s2s3

s1

s2s3

s1

s2s3

s1A B

C D




• S3 is not updated so the client must maintain a separate S3 is not updated so the client must maintain a separate list of nodes for each SFSBlist of nodes for each SFSB

Client2

Client1

s2s3

s1

s2s3

s1

s2s3

s1

s2s3

s1A B

C D



Network Partitions Merging ProblemNetwork Partitions Merging Problem

• Suppose node C crashes nowSuppose node C crashes now

Client2

Client1

s2s3

s1

s2s3

s1

s2s3

s1A B

D




• Client 2 wants to invoke s2 and fails over to node AClient 2 wants to invoke s2 and fails over to node A

Client2

Client1

s2s3

s1

s2s3

s1

s2s3

s1A B

D




• If s2 invokes s3 we find an “out of date” state on this If s2 invokes s3 we find an “out of date” state on this nodenode

Client2

Client1

s2s3

s1

s2s3

s1

s2s3

s1A B

D



Entity BeansEntity Beans

• Entity Beans are persistentEntity Beans are persistent• Unlike Session Beans, Entity Beans are not associated Unlike Session Beans, Entity Beans are not associated

to clientsto clients• Can be accessed by multiple clients, but not Can be accessed by multiple clients, but not

concurrentlyconcurrently• If an Entity Bean is declared as reentrant it can be If an Entity Bean is declared as reentrant it can be

accessed concurrently but only from the same accessed concurrently but only from the same transactiontransaction



• JBoss does not replicate Entity Beans, replication is JBoss does not replicate Entity Beans, replication is left to the database tierleft to the database tier

• JBoss controls concurrent access to the databaseJBoss controls concurrent access to the database• All JBoss nodes share the same databaseAll JBoss nodes share the same database• It is transparent to JBoss if the database is clustered or It is transparent to JBoss if the database is clustered or

notnot



• Entity Bean state is loaded from the DB before each Entity Bean state is loaded from the DB before each business method invocation and stored after the business method invocation and stored after the invocation (Commit option B and C)invocation (Commit option B and C)

• To each J2EE transaction is associated a DB To each J2EE transaction is associated a DB transactiontransaction

• There are to policies to deal with concurrency:There are to policies to deal with concurrency:– OptimisticOptimistic

– PessimisticPessimistic


Optimistic PolicyOptimistic Policy

• Transactions are executed concurrentlyTransactions are executed concurrently• Each Transaction has it’s own instance of the Entity Each Transaction has it’s own instance of the Entity

BeanBean• Shared row locking is used in the databaseShared row locking is used in the database• Before committing the container verifies if the Bean Before committing the container verifies if the Bean

changed in the databasechanged in the database– If yes the transaction is rolled back and re-executedIf yes the transaction is rolled back and re-executed

– If not the new value is stored in the DBIf not the new value is stored in the DB


Pessimistic PolicyPessimistic Policy

• Transactions are not executed concurrentlyTransactions are not executed concurrently• Exclusive Row locking at the DB is usedExclusive Row locking at the DB is used• A transaction blocks if it tries to access a locked row A transaction blocks if it tries to access a locked row

until the row is releaseduntil the row is released• A Distributed Deadlock Detection algorithm is used to A Distributed Deadlock Detection algorithm is used to

detect Deadlocksdetect Deadlocks


Entity Bean Replication FailuresEntity Bean Replication Failures

• If the node owner of a transaction crashes the DB rolls If the node owner of a transaction crashes the DB rolls back the transactionback the transaction

• Network partitions must be handled by the DB Network partitions must be handled by the DB clustering mechanismclustering mechanism

• With a non replicated database, the approach of a With a non replicated database, the approach of a primary partition is used where the primary partition is primary partition is used where the primary partition is defined to be the one containing the databasedefined to be the one containing the database


Entity Bean ReplicationEntity Bean Replication

• Relies intensively on DBRelies intensively on DB– Maximizes DB I/OMaximizes DB I/O

• If we don’t want a single point of failure, we must use If we don’t want a single point of failure, we must use a replicated databasea replicated database

• Entity Beans are not used as cacheEntity Beans are not used as cache• A distributed cache with distributed locks is under A distributed cache with distributed locks is under

developmentdevelopment


HTTP Session ReplicationHTTP Session Replication

• An HTTP Session is implemented as a serializable objectAn HTTP Session is implemented as a serializable object

• The HTTP Session state is NOT broadcast to the other nodes The HTTP Session state is NOT broadcast to the other nodes but replication is implemented through an Entity Beanbut replication is implemented through an Entity Bean

• The HTTP Session is a cmp-field of that beanThe HTTP Session is a cmp-field of that bean

• The CMP engine serializes the HTTP Session object along with The CMP engine serializes the HTTP Session object along with its contentsits contents

• The HTTP Session contains also references (remote objects) of The HTTP Session contains also references (remote objects) of SFSBsSFSBs

• SFSBs do not get serialized with the HTTP SessionSFSBs do not get serialized with the HTTP Session


HTTP Session ReplicationHTTP Session Replication

• Failures are handled by the EJB Replication Failures are handled by the EJB Replication mechanism mechanism

• Client fail over behavior depends on the HTTP Client fail over behavior depends on the HTTP dispatcherdispatcher

• A JBoss HTTP Dispatcher is under developmentA JBoss HTTP Dispatcher is under development


JNDI Tree ReplicationJNDI Tree Replication

• Each node has a local JNDI TreeEach node has a local JNDI Tree• All nodes share a replicated HA-JNDI TreeAll nodes share a replicated HA-JNDI Tree• When a client looks up an object on node NWhen a client looks up an object on node N

– The replicated HA-JNDI service is invoked firstThe replicated HA-JNDI service is invoked first

– If it cannot find it there it looks in the local JNDI TreeIf it cannot find it there it looks in the local JNDI Tree

– If it’s not there it asks all other nodes if they have it in their If it’s not there it asks all other nodes if they have it in their local JNDI Treelocal JNDI Tree

– If it hasn’t been found a NameNotFoundException is thrownIf it hasn’t been found a NameNotFoundException is thrown


JNDI Tree ReplicationJNDI Tree Replication

• Unlike clients, EJBs access only the local JNDI TreeUnlike clients, EJBs access only the local JNDI Tree• This solution was chosen because ofThis solution was chosen because of

– Compatibility with existing applicationsCompatibility with existing applications

– Distinction between local and replicated objectsDistinction between local and replicated objects

– Low network traffic in homogeneous clustersLow network traffic in homogeneous clusters


Distributed CacheDistributed Cache

• Deadline for June (JavaONE)Deadline for June (JavaONE)• Based on JavaGroupsBased on JavaGroups• Will be a Jboss ServiceWill be a Jboss Service• Highly ConfigurableHighly Configurable• Will be applied to Entity Bean Clustering and Session Will be applied to Entity Bean Clustering and Session

Bean ClusteringBean Clustering• Will enable replication of the DatabaseWill enable replication of the Database


Distributed Cache SemanticsDistributed Cache Semantics

• Three semantics:Three semantics:– AsynchronousAsynchronous

• broadcast update and return immediatelybroadcast update and return immediately

– SynchronousSynchronous• broadcast update and wait for acknowledgementsbroadcast update and wait for acknowledgements

– Serialized SynchronousSerialized Synchronous• acquires locks before updating a beanacquires locks before updating a bean

• ensures ordering but not consistencyensures ordering but not consistency

• consistency is obtained at another levelconsistency is obtained at another level


Cache ContentsCache Contents

• The cache will implement the XAResource interfaceThe cache will implement the XAResource interface• This will enable the use of the cache in a Two Phase This will enable the use of the cache in a Two Phase

Commit protocolCommit protocol• The mapping of the XAResource semantics to the The mapping of the XAResource semantics to the

cluster is not a trivial problemcluster is not a trivial problem• This could enable to do the DB update, cluster This could enable to do the DB update, cluster

replication and sending the response to the client an replication and sending the response to the client an atomic operationatomic operation


ConclusionsConclusions

• JBoss clustering algorithm does not handle all failure patterns JBoss clustering algorithm does not handle all failure patterns correctlycorrectly

• Simply changing the GC semantics will not solve the problemsSimply changing the GC semantics will not solve the problems

• To make the existing mechanism work correctlyTo make the existing mechanism work correctly– A replication aware Transaction Manager needs to be implementedA replication aware Transaction Manager needs to be implemented

– During state merging the “out-of-date” beans must be markedDuring state merging the “out-of-date” beans must be marked

• An alternative would be to re-implement the clustering from An alternative would be to re-implement the clustering from scratchscratch

• The development of the Distributed Cache by JBoss authors is a The development of the Distributed Cache by JBoss authors is a good opportunity to redefine the clustering algorithmgood opportunity to redefine the clustering algorithm

A DAPT IST-2001-37126 Clustering Implementation in JBoss Jakša Vučković Università di Bologna.

Documents

Transcript of A DAPT IST-2001-37126 Clustering Implementation in JBoss Jakša Vučković Università di Bologna.