Microservices, Containers, Databases and Persistence Models · Microservices interact by...

Copyright©2017, Oracleand/oritsaffiliates.Allrightsreserved.|

Microservices,Containers,DatabasesandPersistenceModelsTIP4071

Kuassi MensahDirectorProductManagement,Oracle

PaulParkinsonConsultantMemberTechnicalStaff,Oracle

October23,2018

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|

SafeHarborStatement

Thefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,timing, andpricingofanyfeaturesorfunctionalitydescribedforOracle’sproductsmaychangeandremainsatthesolediscretionofOracleCorporation.


Agenda• Concepts:Microservices,Containers,Kubernetes• TransactionModels• MicroservicesPersistenceandDatabases• Demo

3


Concepts:Microservices,Containers,Kubernetes

4


MicroServicesBuildfromScratchorBreakMonolithicappsintoModules• Reducecomplexity• Modulespecificdatamodel• Scaleatmodulelevel• FasterDevelopment– API-First,Events-First,DomainDrivenDesign

• Blue/greendeployments• Idempotent• FaultIsolation


Containers/Docker• OSvirtualization• Modularity• Reducedsize• Greaterscaling• Isolatedenvironmentforrunningapps,webserver,persistencestore,etc• Consistencyinruntimeenvironments(includingtestingandproduction)• Operationalsimplicity• Productivity• Enableportabilityfromon-premisestotheCloud

Source:Docker


ContainersOrchestration/KubernetesContainersmustbemanaged(resource),scheduled,loadbalancedandconnectedtoexternalworld

Kubernetes:Greekwordforaship'scaptain• Goal:robustplatformforrunningthousandsofcontainersinproduction.

• Systemforautomatingdeploymentandscalingofcontainerizedapps• Velocity– constantupdates,pause/resumedeployments,versioncontrol

• Immutability– newimageforchanges(maintainshistory,lineage)

• DeclarativeAPI&config– describedesiredstateofsystem,abstracts“how”

• Selfhealing– continuouslymaintainsdesiredstateovertime.

• Autoscaling


Kubernetes/MicroServicesIdealformicroservices…• Pods,arunnableunitofwork(containerimagesdevelopedbydifferentteamsintoasingledeployableunit);usuallyholds1or2containers• Loadbalancing,naminganddiscoveryisolateonemicroservicefromanother• Namespacesprovideisolationandaccesscontrolsothateachmicroservicecancontrolthedegreetowhichotherservicesinteractwithit.• PlatformservicestellstherestoftheKubernetesenvironmentwhatservices yourapplicationprovides.• Api-registry:tracksofallavailableservicesandtheresourcestheyexpose.


KubernetesPlatformServicesonOKE

9

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.| 10

PainPointsAddressed


IstioRoutingRules• IstioRouteRulescanbeusedtochangewhichservicereceivesrequests• Thisallowsinvokertopassrequeststodifferentservicewithnocodechangeorredeployment• Routerulescanshifttrafficallatonce,orroutepercentagestonewservices

RouteRule.yaml

kind: VirtualServicemetadata:

name: processnamespace: demo

spec:hosts:- processhttp:- route:

- destination:host: processsubset: v2

11

kind: DestinationRulemetadata:

name: processnamespace: demo

spec:host: processsubsets:- name: v1

labels:version: v1

- name: v2labels:version: v2


EventBroker• Addsrules,callbacks,andstructuretovanillamessagebus• Rulesaresimpleandstraightforward• Rulescanbeupdatedtoalterrequestflow• EBcanleverageanymessagebusimplementationunderneath

EventRule.yaml

apiVersion: auraevents.oracledx.com/v1alpha1kind: Rulemetadata:name: rule-process

labels:auraevents.oracledx.com/eventType:

processspec:

action:url: http://process.demo:2222

callbackURL: http://storefront.sf:8888/callback

12


EventBroker• CloudEvents standard ExampleCloudEvent

{"cloudEventsVersion":"0.1","eventType":"com.example.someevent","source":"/mycontext","eventID":"A234-1234-1234","eventTime":"2018-04-05T17:31:00Z","comExampleExtension1":"value","comExampleExtension2":{"otherValue": 5

},"contentType":"text/xml","data":"<muchwow=\"xml\"/>"

}

13


SauronIntegration• EasyintegrationofSauronManagedService• Providesvisualrepresentationofapplicationandsystemmetrics• ProvidesalertingfeaturesviaeMail,Slack,etc.

Prometheusconfig- job_name: 'events-broker'

scrape_interval: 5s

scrape_timeout: 5s

static_configs:

- targets: ['129.213.46.56:30350']

• Businessmetricsbasedonsmartfilteringofeventmetricsindashboard

14


MarketplaceProvisioningofATPDBaaS• ServiceBrokeraddsconnectioninfointoclusterviaKubernetesSecrets• ServiceiswrittentoconsumeSecret• Thisallowsservicetobedeployedindifferentenvironmentswithoutanycodechanges,andpickuptheappropriateDBconnectionautomatically

Deployment.yamlenv: - name:DB_ADMIN_USER valueFrom:

secretKeyRef:name:atp-demo-binding

key:user_name - name:DB_ADMIN_PWD valueFrom:

secretKeyRef:name:atp-user-cred

key:password - name:WALLET_PWD valueFrom:

secretKeyRef:name:atp-user-cred

key:wallet_password volumeMounts: - name:creds mountPath:/db-demo/creds

15


TransactionsModels

16


MicroServices Integration• AnapplicationbecomeanassemblyofMicroServices thatcommunicateamongthemselves• MSintegrationchoices:filesystem,database,RPC,Event• Event-basedIntegration:bestchoice– Asynchronous,loosecoupling– Pub/Sub,Notification– Statetransfer– Immutable– SupportforEventSourcingandCommandQueryResponsibilitySegratation (CQRS)


EventSourcingMicroservicesinteractby”sourcingevents”in/outfromtheEventstoreviatheEventBroker• Eventstore(Kafka,AQinthefutureorvanillaOracleDB)isthesinglesourceoftruth:• Producers:logeventsintheEventstorethenpublishanotification• Consumers:notifiedwhenEventsarepublishedthen“read”theEventstore

WebServer Shipping ServiceOrdersService

PublishEvent

ReceiveEvent


CommandQueryResponsibilitySegregation- CQRSSeparationofresponsibilitieswithinaboundedcontext(Designdomain)• Commands:makechangestostate/data-- donotquerystate

• Queries:viewstate/data(scalewithMaterializedViews)-- donotchangestate

• OftencombinedwithEventSourcing

WebServer Shipping ServiceOrdersServiceCommand

Query

MaterializedViews


BusinessTransactions– RequirementsDistributedTransactionsthatspanmultiplemicroservices– E.g.,Bookingatripinclude:flight,hotel,car,shows

• MicroserviceAcallsBthatcallsC.– TheworkfloworbusinesstransactioncommitsonlywhenCcompletes

• HowdoyouundoA&BshouldCfail?– Traditionaltwo-phaseCOMMITTXisananti-patternwithMicroServices– Eachservicecommitsindividually(locally):howtoensureconsistency?• SAGAPattern• HighconcurrencywithEscrow/Promises


BusinessTransactionsConsistencywithSAGAPattern“Asagaisasequenceoflocaltransactions.Eachlocaltransactionupdatesthestate(database)andpublishesamessageoreventtotriggerthenextlocaltransaction inthesaga.Ifalocaltransaction failsbecauseitviolatesabusinessrulethenthesagaexecutesaseriesofcompensating transactions thatundothechanges thatweremadebytheprecedinglocaltransactions.”https://microservices.io/patterns/data/saga.html

Twoapproachesforcompensatingfailedmember(s)ofadistributedTx• Choreography– Theservicesinteractwitheachothertocoordinatewellknown/definedactivity(thedemohasanexample)

• Orchestration– Anorchestratorisusedtocoordinatetheactivity.– Allmessagesarepassedthrough(interceptedandissuedby)thisorchestrator.

• CompensationcansimplybeTxABORT (thiscanbeautomated)


BusinessTransactionsConsistencywithEscrow/PromisesEscrow– longrunningbusinesstransactions• Counters(anycolumnwith“commutative”properties)usedforinventoriesmanagement– Stockonhand,money/share/creditofaccount,availableseats,...– manipulatedwithplus/minus-- commutative– Onerequestissufficientamount := amount+/-change

• Queriesreturntwodistinctstates:#availableinstock,#inflow• Benefits:parallelupdate,automaticcompensation(theinversefunction)

Promises(notJava/NodeJSpromises)

• Apromiseisagrantedescrowrequestwithtimelimits(i.e.,On-holdreservation)– Expirationneedscarefulhandlingofraceconditions


MicroservicesPersistenceandDatabasesDataModels,Domains,Databases

23


PolyglotorMulti-ModelDatabase?Microservicesarchitecturegoals:flexibilityandagility• Polyglotsystem– aspecializeddatabaseforeachmicroservice– mayleadtoanexpensiveanddifficultplatformtooperate

• Multi- modeldatabase(i.e.,AutonomousTransactionProcessing)– EachservicegetsorshareanATPserviceinstance– highavailabilityanddurabilityoptions

Spatial

Graph

Relational

XML UnstructuredData


OracleAutonomousDatabase

2525

Self-Driving• Automatesallmonitoring,managementandtuning

• Createsmissioncriticalscale-outdatabasewithDR

• Fullcompatibilityenablessimpledatabasemigration

Self-Repairing• Recoversautomaticallyfromanyfailure

• 99.995%uptimeincludingmaintenance,guaranteed

• Elasticallyscalescomputeorstorageasneeded

Self-Securing• Automaticallyappliessecurityupdatesonline

• Secureconfigurationwithfulldatabaseencryption

• SensitivedatahiddenfromOracleorcustomeradmins


PDBSharding forMicroservices

• PDBSharding inDB19c– EachPDBcanbesharded acrossmultipleCDBs

• Providesfaultisolationandgeo-distributionformicroservices– LossofanentireCDBmakesonlypartofaPDBunavailable

• Alsoallowseachmicroservices toscaleitsPDBindividually– Moreefficientuseofresourcescomparedtoscalingamonolithicapplication(CDB).

26

Scalability,faultisolationandgeo-distribution Shard-1 Shard-1Shard-1

Shard-2 Shard-2

Shard-3

Recomm

ProductCatalog

ProductCatalog

ProductCatalog

CheckOut

CheckOut


OnlineShoppingKafkaasEventBrokerandStore

Catalog Cart Order Inventory Shipping

PlaceOrder

BrowseCatalog Product

InCart

KubernetesEventBrokerandEventRules

Kafka/EventStore/EventLog

PublishCloudEvent

ConsumeEvent


AQSupportforMicroservices• AQ/ATPasEventstore• QueueoperationsandDMLDataandmessageinthesamelocaltransaction• Sharded Queuesfurnishhighscalability• Transactedsessionguaranteesnolossofmessageandexactlyoncedelivery

• Clientinitiatednotification• Handleslargemessagebacklogs

• Queuelevelaccessprivileges(enqueue,dequeue)supportsCQRS• AQsupportforSAGA-patterndistributedtransactions• HighAvailabilityandDisasterRecovery


OnlineShoppingKafkaEventBroker&Store+AQMessageproviderservice


PlaceOrder(insertdb+AQmessage)


InCart

K8EventBrokerandEventRules


AQEventProviderService

CloudEvent


AQEnhancementstoSupportMicroServices• RetentionsupportinSharded Queues• CloudEventssupport• Logcompaction• AutonomousQueuesinDBaaS• ChangeQueryNotificationsupportinATPusingAQsecurenotification• Lightweight,seekable andretroactivesubscribers• Topicauto-creation• KafkaAPI(Java)


OnlineShoppingAQasEventBrokerandStore




InCart

AQBroker,Storeand EventRules




AsynchronousDatabaseAccessforMicroServicesMicroservicesarchitecturegoals:flexibilityandagility• SynchronousDBAccess(JDBC)– Themicroservicewaitstilldataupdate(DML)iscompleted– Publishestheevent

• AsynchronousDbAccess(ADBA)– ThemicroservicesubmitsaDMLoperation+AQeventpublisgingthenmoveson.a) Create a transaction object b) create/submit a RowCountOperation to insert an orderc) get a CompletionStaged) create another RowCountOperation to publish an AQ event e) session.commitMaybeRollback(trans) f) submit the whole transaction and return

– Hands-onlabandSession@https://bit.ly/2DzV3uT


RefactoringMonolithic

• CapturesaSQLTuningSet(STS)whileaSQLworkloadisrunning• ProducesaMapofSQLvsTables• GeneratesaGraph


RefactoringMonolithic– BoundedContexts

• Identifynon-conflictingdatamodel• DefineserviceswithinBoundedContextsboundaries


Demo

35


OnlineShopping

Catalog Cart Order Inventory Supplier



InCart

K8EventBrokerandEventRules



CloudEvent


OnlineShopping- Workflow1. Catalogservice:Graphdatamodel– notimplemented

2. Orderservice:entersanorderintheDBandpublishesanAQmessageinthesamelocalTx

3. Dbeventprovider:picksuptheAQmessagepublishesittotheCloudEventBroker

4. Supplierservice:suppliestheInventoryserviceandpublishesanAQmessageinthesamelocalTx.

5. Inventoryservice:receivesanOrderplaceeventthenpublishesa“lackofinventoryfororderitem”event

6. Orderservice:picksupthateventandconstructsitsstatefromtheeventsourcingmechanismw/oqueryingthedatabase


Reportoffailedordertoclient(theresultofmissinginventory)…


OrderinsertedintableandAQmessagesentinsamelocaltransaction(src code)…


OrderservicelogsshowinginsertoforderandAQsend

OrderinsertedintableandAQmessagesentinsamelocaltransaction(logs)…


AQEventproviderserviceMediatingAQmessagereceivedfromitintoaCloudEventthatispublishedtoeventbrokerforprocessingbasedonrules.Therecanbemultiplerulesforasingleeventtype(eg ordercompletiontriggersemail,delivery,etc.notification)


InventoryservicereceivingeventasaresultofruleprocessingoforderplacedCloudeventandthenpublishingeventindicating(lackof)inventoryfororderitem.


Orderservicereceivinginventoryeventandcancellingorderasaresult(settingstatustofailedandreturningittotheclient).

Choreographysaga:servicescoordinatedirectlywitheachothertomaintaindataintegrityforabusinessactivity.E.g.,the lackofinventory(intheinventoryservice/database)inthiscaseresultsinacancelled/failedorder(intheorderservice/database).


Supplierserviceinterfacetoinventoryserviceaddsinventory…


Reportoffailedordertoclient(theresultofinventorynowexisting)…


Samemessage/workflowandorderisnowsuccessfulasinventoryexists.


CQRS(aggregate)queryprovidingdatagleanedfromeventsourcing.Dataissourcedfromeventsreceivedandeachservicebuildsandaccessesthespecificstate/representationitrequireslocallyallowingforCommandQueryResponsibilitySegregation.Theorderserviceorderdetail,eg,ineffecthasinformationfromotherservices’databases(inventoryandsuggesteditems).


QandA


SeeourNextSessions@https://bit.ly/2DzV3uT

60

Microservices, Containers, Databases and Persistence Models · Microservices interact by...

Documents

Transcript of Microservices, Containers, Databases and Persistence Models · Microservices interact by...