DepSync for Deployment Artifact

Synchronization Between the Cluster

Nodes

Pradeep Fernando

Senior Software Engineer

pradeep@wso2.com

Kasun Gajasinghe

Software Engineer

kasung@wso2.com

Agenda

• The need for deployment synchronization

• Brief introduction to WSO2 product clusters

• Demo - Complete Application server cluster with DepSync +

fronted by WSO2 ELB

Why Deployment Synchronization?

• Artifact distribution and deployment should be transparent to

the end-users

• Manual artifact copying is acceptable for some extent in

standalone product clusters.

• Will that work in an elastically scaling cloud environment ??

Typical WSO2 Product Cluster

• Manager and Worker nodes.

• Manager is for administration purposes - artifact upload,

applying security,etc

• Worker nodes responsible for serving requests.

• The setup is only valid for products that support artifact

deployment:

• Application Server

• Enterprise Service Bus

• etc

Management and Worker Node Separation

Why Management & Worker Node Separation

Crash Course• Proper separation of concerns - management nodes

specialize in management of the setup while worker nodes specialize in serving requests to deployment artifacts

• Only management nodes are authorized to add new artifacts into the system or make configuration changes

• Worker nodes can only deploy artifacts & read configuration

• Lower memory foot in the worker nodes because the management console related OSGi bundles are not loaded

• Improved security - management nodes can be behind the internal firewall & be exposed to clients running within the organization only, while worker nodes can be exposed to external clients.

Deployment Synchronization

• Distributing deployment artifacts & related metadata to all nodes in a homogeneous cluster is a typical requirement for a clustered deployment of any middleware platform

• Provides a consistent, and reliable cluster

• Automated synchronization without need of any user interaction

• Artifact Metadata contains service policies, and other important service configuration details

Deployment Synchronizer

• WSO2 products provides this through DeploymentSynchronizer (DepSync) mechanism

• A new feature addition to WSO2 products

• Synchronization done using a central repository, usually, a SVN server

• We call it SVN-based Deployment Synchronizer

• Extensible - may be a Git based one in the future.

The Deployment Synchronization Mechanism

What it synchronizes?

• Web Applications• JAXWS / JAXRS Applications• Proxy Services• Data Services• BPEL• Basically all the deployable artifacts in axis2 repo of

Carbon products (ex. repository/deployment/server)• Service Metadata – (details on policies, transports etc.)

Enabling DepSync : Carbon.xml

DepSync Configuration

<DeploymentSynchronizer>

<Enabled>true</Enabled>

<AutoCommit>true|false</AutoCommit>

<AutoCheckout>true</AutoCheckout>

<RepositoryType>svn</RepositoryType>

<SvnUrl>https://10.100.3.115/svn/repos/as</SvnUrl>

<SvnUser>wso2</SvnUser>

<SvnPassword>wso2123</SvnPassword>

<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>

</DeploymentSynchronizer>

ex. local-file system based SvnUrl - file:///C:/demo/svnrepo

Multicast vs. WKA

Multicast• Cluster is going to be set up in a network where

multicasting is allowed

Well-Known Address• Cloud based deployment• Members are distributed across datacenters & regions• Multicasting blocked

DepSync and Multi-Tenancy

• DepSync synchronizes the artifacts among the product cluster in a tenant aware manner.

• Ghost deployment is supported in the depsync aware manner in order reduce the initial deployment and request serving latencies.

DEMONSTRATION

Configuring a minimal WSO2 cluster setup w/ Deployment Synchronization

Deployment diagram

Process

1. Set up the Environment for Deployment Synchronizer

2. Configure Load Balancer

3. Configure Management Node - Application Server

4. Configure Worker Node - Application Server

Set up the Environment for

Deployment Synchronizer

PART 1

Set up the Environment for Deployment

Synchronizer

• We will setup the SVN-based DepSync• For this, you need the following

o svnkit osgi bundleo TortoiseSVN / Silk SVN (Windows), Subversion

command line package(Unix) -optional

Set up the Environment for Deployment

Synchronizer

• Download svnkit bundle from here - http://goo.gl/CVR2F• If you will be using a remote svn location for depsync, you may

skip the creation of local svn repository.• To create a local svn repository,

o if you have svn command line client, enter following command.

svnadmin create C:\demo\svnrepo

o if you are using TortoiseSVN, then first create a folder at any place you like. Say - C://demo/svnrepo . Now go in to that folder, right-click -> TortoiseSVN -> "Create Repsitory here"

o Now, the SVN URL for this location is -file:///C:/demo/svnrepo

Set up the Environment for Deployment

Synchronizer

• We will use this svn url when configuring the manager and worker nodes. So, remember the location where you created this. DepSync config will also be configured there.

Enabling DepSync : Carbon.xml

DepSync Configuration

<DeploymentSynchronizer>

<Enabled>true</Enabled>

<AutoCommit>true|false</AutoCommit>

<AutoCheckout>true</AutoCheckout>

<RepositoryType>svn</RepositoryType>

<SvnUrl>https://10.100.3.115/svn/repos/as</SvnUrl>

<SvnUser>wso2</SvnUser>

<SvnPassword>wso2123</SvnPassword>

<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>

</DeploymentSynchronizer>

ex. local-file system based SvnUrl - file:///C:/demo/svnrepo

Configure Load Balancer

PART 2

Configure Load Balancer -

Loadbalancer.conf

• Download and extract wso2elb-2.0.0.zipo Let the extracted directory be WSO2_LB_HOME

• Open loadbalancer.conf file in the WSO2_LB_HOME/repository/conf

Configure Load Balancer

• Configuring the WSO2 Elastic Load Balancer is now complete

• To start the servero Go to the WSO2_LB_HOME/bin folder using command

lineo Enter the command `wso2server.bat`o Notice the logs printed by TribesClusteringAgent

Configure Management Node

- Application Server

PART 3

Configure Management Node

• Download and extract wso2as-5.0.0.zip

o Rename extracted directory to wso2as-5.0.0-manager

o Let the extracted directory be WSO2_AS_MGR_HOME

Configure Management Node -

axis2.xml

• Enable clustering at axis2 level.<clustering

class="org.apache.axis2.clustering.tribes.TribesClusteringAgent"

enable="true">

• Change the membershipScheme of clustering to wka (well-known address)

<parameter name="membershipScheme">wka</parameter>

• Set the cluster domain. This must be the same which we defined in loadbalancer.conf

<parameter name="domain">wso2.as.domain</parameter>

Configure Management Node -

axis2.xml

• Set localMemberHost<parameter

name="localMemberHost">mgt.appserver.wso2.com</parameter>

• Set localMemberPort<parameter name="localMemberPort">4250</parameter>

• Add a new property named "subDomain"and set it to 'mgt' inside the parameter "properties"

<parameter name="properties"><property name="backendServerURL"

value="https://${hostName}:${httpsPort}/services/"/>

<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>

<property name="subDomain" value="mgt"/>

</parameter>

• Add load balancer as a well-known member. o Use the IP/hostname and port defined in LB's

axis2.xml<members>

<member>

<hostName>127.0.0.1</hostName>

<port>4000</port>

</member>

</members>

Configure Management Node –

catalina-server.xml

• Since the mgt node is fronted by an LB, we need to configure proxy ports associated with HTTP and HTTPS connecters.o Defaults for http and https are 8280 and 8243

respectively.<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"

port="9763"

proxyPort="8280“

---

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"

port="9443“

proxyPort="8243“

---

Configure Management Node –

carbon.xml

• Set port offset to avoid port conflicts with other nodes (LB).

<Offset>1</Offset>

• Update mgtHostName and HostName elements in carbon.xml as shown below

<HostName>appserver.wso2.com</HostName><MgtHostName>mgt.appserver

.wso2.com</MgtHostName>

• Copy svnkit bundle to repository/components/dropins

Configure Management Node -

Deployment Synchronizer

• The deployment synchronizer config is in carbon.xml.• Uncomment the DeploymentSynchronizer xml segment.• Set AutoCommit to true.• Set the SVNUrl

• No need to set username/password if you didn't set it for localsvn repo

<DeploymentSynchronizer>

<Enabled>true</Enabled>

<AutoCommit>true</AutoCommit>

<AutoCheckout>true</AutoCheckout>

<RepositoryType>svn</RepositoryType>

<SvnUrl>file:///C:/webinar-setup/demo/final/svnrepo</SvnUrl>

<SvnUser>username</SvnUser>

<SvnPassword>password</SvnPassword>

<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>

</DeploymentSynchronizer>

Configure Management Node

• Configuring the WSO2 Application Server is now complete• To start the server

o Go to the WSO2_AS_MGR_HOME/bin folder using command line

o Enter the command `wso2server.bat`

o Notice the logs printed by TribesClusteringAgent and corresponding cluster joining logs in load balancer.

• You can now login to management console via https://mgt.appserver.wso2.com:8243/carbon/

• Notice that requests we sent to services (including WSDL) will be directed to worker nodes.

Configure Worker Node -

Application Server

PART 4

Configure Worker Node

• We can use a copy of a management node, and convert it to a worker node easily since it will have most of the settings

• So, instead of using a fresh appserver copy, make a copy of the management node we just created

• Rename the new folder as wso2as-5.0.0-worker• Let this directory be WSO2_AS_WORKER_HOME

Configure Worker Node -

axis2.xml

• In addition to the settings we already have, do the following.

• Set the localMemberPort to 4251<parameter name="localMemberPort">4251</parameter>

• This node belongs to the "worker"subDomain, so, change the property "subDomain" to worker (instead of mgt)<parameter name="properties">

<property name="backendServerURL"

value="https://${hostName}:${httpsPort}/services/"/>

<property name="mgtConsoleURL"

value="https://${hostName}:${httpsPort}/"/>

<property name="subDomain" value="worker"/>

</parameter>

Configure Worker Node –

carbon.xml

• First set a new port offset <Offset>2</Offset>

• Comment out/remove the mgtHostName configuration• Set the AutoCommit property inside Deployment Synchronizer

to false. This MUST be done because the worker nodes of a cluster should NOT commit (write) artifacts

<DeploymentSynchronizer>

<Enabled>true</Enabled>

<AutoCommit>false</AutoCommit>

<AutoCheckout>true</AutoCheckout>

<RepositoryType>svn</RepositoryType>

<SvnUrl>file:///C:/webinar-setup/demo/final/svnrepo</SvnUrl>

<SvnUser>username</SvnUser>

<SvnPassword>password</SvnPassword>

<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>

</DeploymentSynchronizer>

Configure Worker Node –

carbon.xml

•

• Now, it's time to start the server.

o NOTE: The workerNode system property must be set when starting the worker nodes every time.

wso2server.bat -DworkerNode=true

Testing the cluster

There are many ways to test our cluster deployment. Let's follow a simpler path.

• Log in to the management console of Application Server management node

• Deploy a new Axis2 Web service (Go to Manage --> Axis2 Services --> Add --> AAR service)

• Once the service is deployed in the management node go to the services list and click on Tryit

• Invoke the service

QUESTIONS?

THANK YOU!!