itlchn 20 - Kien truc he thong chung khoan - Phan 1

PowerPoint Presentation

Kin trc H thng Core chng khon & mt s gii php v non-functional

Trnh by:

Nguyn Nht Quang

V Vit Anh

H Ni, 05/2015

Hi tho

1

Slides c son tho da trn cc t liu ca ng nghip, cc nh cung cp phn mm core chng khon v cc nh cung cp phn mm khc.

Cc ti liu khi cng b c chnh sa khc vi nguyn bn trnh tit l b mt thng tin, nhng qua ngi trnh by vn truyn ti c cc tng c bn.

Mc ch cng b ti liu l giao lu, hc hi cng ng ngi lm ngh CNTT, thu nhn nhng kin phn bin khi quay li h thng c th lm cho n tt ln.

Ti liu v trnh by cng l dp cng ng CNTT c c khi nim v hiu hn v h thng chng khon, giao dch trn th trng chng khon, t c c s tng tc tt hn i vi anh em ang lm trong lnh vc ny.

Li ng

2

Gii thiu

3

V Vit Anh

Tt nghip i hc Bch khoa Paris, i hc vin thng quc gia Php

Thc s i hc Columbia (New York city)

Kinh nghim:

Lm vic ti Tervela Inc (NYC). Tervela l mt trong nhng nh tin phong v middleware ti Wallstreet, l cng ty u tin a gii php messaging da trn hardware

Sng lp OCTech

Stock Exchange Gateway, Order Routing System C, C++, ZeroMQ, Oracle, PostgresDB

ang dn dt OCTech l 1 cng ty chuyn pht trin cc PM trong ngnh ti chnh, CK vi cht lng cao

Mt trong nhng SP c gii CK bit n l GW kt ni vi 2 s GDCK

OCTech c gn 10 nm hot ng trong lnh vc ny.

4

Nguyn Nht Quang

Bch khoa H Ni, in t Vin thng + i hc Quc gia HN, CNTT

Cc cng ty lm vic: CDiT, VTN, Hanoi Telecom, CBOSS, SHS, FPT

Kinh nghim:

H thng Telecom Billing (VNPT, VTN) C, C++, STL, Lua, Delphi, Oracle

H thng Mobile Tracking System (BCA) C, C++, Oracle

H thng Core chng khon (iBroker, eSMS, Miroku) C#, C++, Java, Oracle, MSSQL, MySQL, ProC, WildFly, Spring, ActiveMQ, Infinispan, Nginx, Crystal Report, iReport.

Stock Exchange Gateway (C++, C#, MSSQL, MSMQ)

iu ng ni nht l Q c khong 10 nm lm trong lnh vc phn mm CK, cho c 2 pha bn lm sn phm v bn tiu th SP

5

AGENDA

Overview v kin trc

Overview v TTCK

Lch s pht trin Core CK ti VN & cc vn

Kin trc h thng Core chng khon

Cc gii php k thut & case study

6

KIN TRC L G?

AGENDA

Overview v kin trc

Overview v TTCK




Khng c kin trc

Nu khng c KT, bt k mt cng trnh, mt sn phm no s c xy dng v hon thin mt cch v t chc, khng c quy c, ln xn v thng khng p ng c cng nng s dng

8

Kin trc cng trnh ln, c ch, sng ch

Ti sao cn kin trc?

n gin hng dn bin ci c / hoc khng c g thnh ci mi

10

Sng to v ti s dng

DO NOT REINVENT THE WHEELS

Khng nn pht minh li bnh xe ci m chng c g pht minh thm c - trn tra, n gin nhng hon m, kiu g cng phi trn, nu khng n s khng phi l ci bnh xe na !

11

Kt lun phn 1

Khi nim v kin trc

Vai tr thit yu ca n trong vic xy dng bt c 1 th g bao gm phn mm, c bit l cc phn mm ln, phc tp, nhiu yu cu kht khe

12

OVERVIEW V TH TRNG CHNG KHON

AGENDA

Overview v kin trc

Overview v TTCK




Th trng Chng khon

S GDCK Stock Exchange

10 Stock Exchanges ln trn th gii:

1. Frankfurt Stock Exchange, c

2.Indonesia Stock Exchange

3. New YorkStock Exchange, M

4. Madrid Stock Exchange, Ty Ban Nha

5.Hong Kong Stock Exchange

6. BahrainStock Exchange

7. London Stock Exchange, Vng quc Anh

8. Chicago Board of Trade, M

9. Dubai Financial Market, UAE

10. Buenos Aires Stock Exchange, Argentina

15

S GDCK Ni trao i mua bn CK

16

Bng gi / bng in chng khon

Khp lnh thng qua S GDCK

Nguyn tc: u tin mua cao, bn thp

Tch ly bn bn: t gi thp n cao

Tch ly bn mua: t gi cao n thp

2 bn chm nhau ti mc gi m KL cc bn mua-bn t c l ln nht l mc gi khp lnh

18

Core chng khon trong cc mi tng quan

Khch hng

CTCK (Broker Firms, Securities Firms - SSI, HSC, VNDS)

Mi gii

u mi tip nhn v chuyn lnh

Thnh vin lu k, thanh ton b tr

Nghip v khc: t vn, cho vay k qu

Chi nhnh, phng GD

S GDCK (HOSE, HNX)

Trung tm lu k CSD (VSD)

Ngn hng thanh ton (BIDV)

Ngn hng

Ngn hng lu k (HSBC)

Trung tm thanh khon (CCP)

i tc t lnh (Reuters, Bloomberg)

Datafeed vendors (Reuters, Bloomberg, StoxPro, StockBiz)

SMS Vendors

19

Ni dung trao i d liu

Cng ty Chng khon

Knh khc

S GDCK

Ngn hng Thanh ton (BIDV)

Trung tm Lu k

Ngn hng

Web , Mobile

KH

Lnh

Lnh

Lnh

Lnh

Xc nhn s d v giao dch CK

Xc nhn thanh ton

KQ Lnh

Chuyn tin

Chuyn tin

Lnh thanh ton

KQ Lnh

So khp, i sot

M TK, thay i thng tin, yu cu GD

i qua nhanh

20

Thi gian trao i d liu

Cng ty Chng khon

Knh khc

S GDCK

Ngn hng Thanh ton (BIDV)

Trung tm Lu k CK

Ngn hng

9:00-14:45

Web , Mobile

Mi lc

9:00-14:45

9:00-14:45

Ngy 1 ln

Ngy 1 ln

15:30

Ngy 1 ln

Ngy 1 ln

9:00-14:45

Ngy 1 ln

Ngy 1 ln

Ngy 1 ln

KH

i qua nhanh

21

LCH S PHT TRIN CORE CK TI VN

& CC VN

AGENDA

Overview v kin trc

Overview v TTCK




Cc NCC Core chng khon ti VN

CORE NI

FPT.BOSC (FPT) chim lnh th trng trong giai on u

FPT.Index

FPT.UTS: t 2009, thay th cho Index

@Direct (FSS)

VSSD.SBS 15 CTCK, t 2007, trin khai cho SHS,BSC,

Thng Nht

MegaStock (RPSoft) VTS, Click & Phone, WSS

Stock24 (TLS/MBS)

KLS

Navisoft eBroker, eTrading (ABS, Tn Vit)

GoLine

HT2D SMS, iTrading, Rabbit Trade

FSS - Flex

i qua nhanh

23

Core chng khon ti VN

CORE NGOI

FreeWill

Lotte HPT (Tong Yang HPT)

AFE (Hongkong)

TTL (Hongkong)

Syscom (Taiwan): PNS

CMS-Nova/TAW (Australia + India)

3i-Infotech (India)

Okasan & Goline V-GAIA (hp tc Nht Vit): MHBS

Formis BASS EBOS/EBETS (Malysia): ABS

Miroku (NRI FPT, phi sinh)

i qua nhanh

24

Th trng Chng khon Vit Nam

CC PHN MM khc CORE

Gateway:

OCTech (Gateway, ORS)

Stockbiz

K ton:

Bravo

Oracle

Web, datafeed, dch v d liu

Vietstock (web, datafeed)

Stox/StoxPlus

CafeF, Cophieu68, iStock.vn...

BANK Connection

CC H THNG PHN MM CA CC C QUAN QUN L

H thng ca S HNX

H thng ca S HOSE

H thng VSD

FIS, FSS, Navisoft

Kin trc ng dng 1 phn mm Core hin ti

Mt s giao din Trading

AFE mn hnh t lnh cho mi gii

27


SH-PRO, LotteHPT, mn hnh t lnh

28


SH-Mobile, LotteHPT, mn hnh t lnh + gi hin ti + t lnh nhanh vi 3 gi tt nht

29

Th trng chng khon VN

PHNG THC T LNH v QUN L DANH MC

Thi gian u: Qun l lnh = Excel, i din sn, t lnh bng gi in...

Thng sn, kt ni CTCK S bng phn mm, b i din sn. T ng cp nht kt qu GD.

Nhn lnh t ng, giao dch online qua Internet, mobile

SN PHM, DCH V:

Lnh thng thng, chuyn tin th cng

Giao dch k qu, chuyn tin online, thanh ton T+3

Giao dch phi sinh

Bn khng, giao dch trong ngy, thanh ton T+2, T+0

30

PHNG THC T LNH v QUN L DANH MC

Thi gian u: Qun l lnh = Excel, i din sn, t lnh bng gi in...

Thng sn, kt ni CTCK S bng phn mm, b i din sn. T ng cp nht kt qu GD.

Nhn lnh t ng, giao dch online qua Internet, mobile

31

Thc trng & Cc vn

TTCK Vit Nam hot ng c hn 15 nm vi kh nhiu thng trm, vi s gp mt ca nhiu NCC trong v ngoi nc. S lng v v cht lng cc gii php core kh a dng.

32

Kin trc chp v, d dng, hoc khng c KT

1. Nng cp t h thng c l khng kh thi, tng t nh xy 1 nh cao tng trn nn mng ca nh cp 4

2. Nu vic nng cp kh thi, khng c kinh ph, kinh nghim, con ngi, kin thc thc hin nng cp

3. Cng vic kinh doanh vn phi chy, khch hng khng th dng giao dch phi chp v vic d dng l khng th trnh khi

33

Thc trng

S lng cc CTCK: gn 100 CTCK, gi rt xung cn 71 CTCK ang hot ng (theo danh sch thnh vin hnx.vn ngy 3/5/2016) qu ng, qu nguy him

Th trng cn tn ti nhiu c im c: T+3, nhiu khu giao dch cha t ng, giy t th tc rm r, cha c nhiu sn phm ti chnh tt, KHNG MINH BCH, b thao tng, lm gi (i li, tay to).

Cht lng hng ha cha cao, vn ha th trng thp, thanh khon thp (ngy 3/5/2016, ngun cafef.vn)

Cc vn hin ti

Nng lc thp (Capacity): s lng khch hng cng lc cao nht l vi nghn, thc t < 1000.

Thi gian t 1 lnh l chm (tc thp): chc lnh / s

Li: p lnh, sai lnh, bn khng...

Nu th trng pht trin, volume tng, dch v mi trin khai khng p ng c.

Performance h thng khng p ng c nhu cu hin ti v giao dch 1 phn do gn y TTCK kh khn, cc CTCK khng u t nhiu cho HT nh trc, c bit bo tr, nng cp...1 phn do cc vendor ngoi kh khn, rt vn v ngun lc khi VN, cc vendor ni khng c chin lc di hn v sc khe ti chnh tt nn b ht hi.

Quy nh v giao dch, c bit giao dch k qu, room, t l an ton ti chnh...check cc iu kin trc khi t lnh lm chm tc t lnh h thng.

Core ngoi c u t pht trin bi bn hn, mc d cng ngh kh c nhng kin trc kh tt nn vn tr c n ngy nay. Core ni vn gp nhiu vn v kin trc, khng c chin lc u t pht trin di hn sinh ra cc vn v hiu nng, tc thm ch li h thng nu chy trong thi gian di.

35

Mt s trng hp s c phn mm Core

(2012) HSC (AFE) b li h thng GD do nng cp HT, thay i cc tham s hot ng ca HT.

(2014) SHS (LotteHPT) nghn c chai khi giao dch, h thng t lnh , khch hng phn nn, bc xc. B nhiu ln, trong thi gian di, m khng khc phc c trit .

(2014) VNDirect (FSS) b li y nhm lnh ln S HNX, hn 20k lnh trong 1 pht. Phi khc phc hu qu rt nng n, thit hi hng chc t ng.

KIN TRC H THNG CORE CHNG KHON

AGENDA

Overview v kin trc

Overview v TTCK




H thng Core chng khon CMS, ly 2 t u ca tn 2 h thng ang pht trin + system

37

Kin trc tng th Core chng khon

5 hng tip cn:

BA Business

PA Performance or Non-Function requirements

DA - Data

AA - Application

TA - Technology

Xy dng kin trc h thng da trn phng php lun vi 5 hng tip cn

C g hi tng ng vi TOGAF

38


SA

Non-functional requirements

Phn tch th trng CK

Cu trc nghip v

Yu cu nghip v

Kin trc, nh ngha d liu

Cu trc chc nng, ng dng

Cng ngh, Software Stack,

Lc Software, Network & Hardware

PA

Kin trc nng lc

BA

Kin trc nghip v

DA

Kin trc d liu

AA

Kin trc ng dng

TA

Kin trc cng ngh

39

Kin trc nghip v

Application (window form)

Web (thick/thin)

Stock info Mng.

Securities Deposit & Withdraw

Rights basic Info Mng.

Rights execution/Correction

Inquiries/Reports

Market Data

Stock Price Data

Tick Data

Other data

Buying/Selling Order

Order Correction/Cancel

Odd lot Order

PT Order

Conditional Order

Deals Correction

Order & Match List Inquiry

Inquiries/Reports

Buying/Selling Settlement

Put-thru/Bond Settlement Date Mng.

Settlement Cash Transfer to Bank

Commission Grade /ratio Mng.

Inquiries/Reports

Open/Close Account

Customer/Account info.

Signature/Password Mng.

Fault Mng.

Variable List Check

Account History

Account Balance

Cash deposit/withdrawal

Bank information Mng.

Transfer

Interest on deposit

Bank Connection

Stock basket, limit policy

Stock evaluation

Interest/Fee Mng.

Pay in advance/Repayment

Loan/Repayment

Margin/Loan info Mng.

Inquiries/Reports

Basic Information Mng.

Common policy

Authentication

Authorisation

Inquiries/Reports

Person in Charge Registration

Account activities/status

Broker performance report

Branch performance report

Agency performance report

FRONT

Account

Cash

Data

Order

Settlement

Finance Service

Securities and Custody

Remiser

Common

BACK

Gateway to Exchanges

SMS Gateway

Call center

Bank

External Institutions

(Securities Depository)

VSD Gateway

Accounting

Mobile (web /app)

Qun l ti khon, tin, chng khon, t lnh, thc hin quyn, qun l mi gii, dch v ti chnh...

Cc nghip v lin quan n vic kt ni vi S GDCK, Trung tm lu k, ngn hng

....

40

Front v Back office

Front Office

Back Office

Thi gian thc

S d ti khon Tin, Giao dch Tin

D liu th trng, gi CK

Thi gian thc

Thng tin KH, hp ng, y quyn

S d CK, GDCK

S d Tin, GD Tin (Thanh ton, bin nhn, np, rt, chuyn khon, c tc...)

Chnh sch dch v (K qu, Hn mc, Giao dch)

BOD (Ngy 1 ln, u ngy)

TT Khch hng, ti khon, y quyn

Thng tin CTCK, mi gii

S d ti khon CK ca KH

S d ti khon Tin ca KH

Chnh sch dch v (K qu, Hn mc, Giao dch)

EOD (Ngy 1 ln, cui ngy)

Lnh, KQGD

D liu th trng, gi CK

FO giao tip vi KH, thng qua knh Internet

BO giao tip vi user l NV cng ty, trong mng LAN

FO, BO ni vi VSD, Bank ty

FO phi ni vi S GDCK

41


SA


Phn tch th trng CK

Cu trc nghip v

Yu cu nghip v





PA

Kin trc nng lc

BA

Kin trc nghip v

DA

Kin trc d liu

AA

Kin trc ng dng

TA

Kin trc cng ngh

Performance Architecture Market Analysis

43
Year4GDP ($bil)% Market Cap/GDPMarket Cap ($bil)2014171.2232%54.792022468.04 (based on GRavg)70% (governments report)327.6 Market Cap increases 5.97 times (327.6/ 54.79) ~ 6 times
Gi s mc tng GPD trung bnh = GRavg t c 14.07% (rt cao)

tm ra nng lc phc v ca h thng, cn phn tch so snh cc thng s (TT, vn ha, s lnh...) hin ti vi th trng trong tng lai

Tng lai my nm ph thuc vo chin lc KD ca CTCK, vo vng quay khu hao mong mun ca CTCK..

Hin ti 1 ngy gi tr GD khong 200tr USD, 2022 mong mun t khong 1 t USD

43

Performance Architecture - Account Number Analysis

44
Account number analysisCurrent 1.5 mil accounts in VN.10 top BRs take 65% market share --> let assume take 65% account number ~ 1milEach BR in top10 take 100k accounts on averageAverage (top 10 market share) - both retail & institutional customersOrder number per day2-10k, peak may be 30kNo of transactions per day100k 500k altogetherNo of order transactions per day10-50k, peak may be 100k

1,5 triu ti khon

Mi CTCK khong 100k ti khon

S lnh 1 ngy khong 2-10k, nh l 30k (VND)

S transaction 1 ngy khong 10-50k tnh c t lnh, truy vn s d, chuyn tin..., nh c th t 100k

Kh nng x l song song, ng thi cao, i hi chnh xc tuyt i, an ton kht khe.

Hi khc so vi 1 h thng TMT, bo (mc d vi cc trang ny tng s lng request c th ln hn)

44

Nu th trng phi sinh hot ng mnh

45

T l giao dch phi sinh chim 1 t trng cao trong tng khi lng giao dch

T l thanh khon ca derivatives / cash ty thuc th trng CK mi nc, c nhng nc tnh bng ln

c bit ti TTCK M, khi lng giao dch phi sinh chim ~ 95%, khi lng giao dch thc chim ~5%

Thc t trn 1 s th trng khc VD Vng, FX, t l trading ca phi sinh rt cao

Cn dn 1 s ngun. TTCK n c ngun nhng s liu thng k kh di dng phc tp.

45

Performance Architecture - System targets

46
Loi hnhMcHin ti (1CTCK)Mc tiu 2022 (1CTCK, 6 ln)Reliability / MTBF45,000 hours (5 years)100,000 hours (12 years)AvailabilityOperating ratio99.999% (five nine)ServiceabilityMTTR~ 1 hour (100,000/99,999)RTON/ANone (auto fail-over & fallback)RPON/ANone (auto fail-over & fallback)IntegrityData CollisionYesNo
Market Cap tng ~ 6 ln n 2022, nu tnh c th trng phi sinh c th s nhiu hn (~ vi chc ln). S tho lun thm trong phn gii php k thut.

Performance Architecture - System targets Additional Explanation

47

47

Performance Architecture - System targets

48
Phn loiMcCh tiuHin ti (1CTCK)Mc tiu (1CTCK) 2022 (6 times)Mc tiu (tnh c Phi sinh, 1CTCK) 2022 (60 times)Hiu nngKhi lng giao dchCh giao dch Trading10,00060,000 order requests600,000 order requestsTt c giao dch (trading + truy vn + np rt tin...)1 triu6 triu60 triuS giao dch ng thiCh giao dch trading1,0006,00060,000Tt c giao dch (trading + truy vn + np rt tin...)10,00060,000600,000D liu th trngS lng M chng khon6601,0001,000S lng ti khon100,000600,0006,000,000 (scaled) trOrder response time (tnh c tr ng truyn)100ms-1s50-100 millisecond50-100 millisecondOrder response time (Ch Gateway)30ms-500ms10-20 millisecond10-20 millisecondData distribution time100ms-1s10-100 millisecond10-100 millisecondBng thngS lnh / giy1006006,000 (scaled)S giao dch / giy1,0006,00060,000 (scaled)

y c 1 mu thun, l yu cu va ch c nhiu, va chy nhanh

So snh xe bus vi xe ua (ferrari), boing 747 v F117

Dung ha: my bay Concord

48


PA

Kin trc nng lc

BA

Kin trc nghip v

DA

Kin trc d liu

AA

Kin trc ng dng

TA

Kin trc cng ngh

SA


Phn tch th trng CK

Cu trc nghip v

Yu cu nghip v





DA: Kin trc d liu

Qun l giao dch v s d tin

-S d tin

-Giao dch tin

Qun l Khch hng

Knh giao dch

- Ti sn

- Web

- Mobile

- Desktop

- Call center, SMS

Qun l giao dch t lnh

- Lnh thng

- Lnh margin

- Qun l DM

- Lnh nng cao

Tham s chung

-Chnh sch margin

-Chnh sch hn mc

-Chnh sch GD

.......

Core Back

- Thanh ton b tr

- Thc hin quyn

- QL CK

- QL Tin

- Bo co

Kt ni HT khc

-Gateway

-Interface

Qun l TT Th trng

- Th trng

- Gi CK

Common

Market DB

Cust DB

CRM

CRM

Sales force mngt

Bo co

-Bo co

-H tr ra quyt nh

Qun l CTCK v Mi gii

-CTCK

-Mi gii

Cash DB

BF DB

Trade DB

K ton

- GL

- Management

Qun l giao dch v s d CK

-S d CK

-GD CK

Securities DB

Back DB

Phn hoch d liu theo nghip v, ch no cn b tr DB ch no khng

Gip cho vic lu tr, truy vn, x l d liu nhanh chng, sun s

Gip cho vic bo tr bo dng d liu tn km v thun tin

Gip cho vic pht trin cc phn mm v tinh nh CRM, BPM...

50


PA

Kin trc nng lc

BA

Kin trc nghip v

DA

Kin trc d liu

AA

Kin trc ng dng

TA

Kin trc cng ngh

SA


Phn tch th trng CK

Cu trc nghip v

Yu cu nghip v





Kin trc ng dng m hnh logic

y khng phi l v d tt v AA, y l m hnh tch hp, c bng dng ca Techical

AA diagram phi l cc khi ng dng (logic) v chc nng ca n, quan h gia cc khi ng dng

52


PA

Kin trc nng lc

BA

Kin trc nghip v

DA

Kin trc d liu

AA

Kin trc ng dng

TA

Kin trc cng ngh

SA


Phn tch th trng CK

Cu trc nghip v

Yu cu nghip v





Phn TA l phn tn nhiu cng sc, thi gian nht

53

TA M hnh trin khai ci t

Khi c kin trc ng dng, anh em TA v Dev s ngi li la chn cc cng ngh ph hp cho mi my ch (logic)

54

TA S my ch thc t

Sau a ra s trin khai vi thng tin ca cc my ch cn thit

55

TA Load balancing

Mt trong nhng vn rt quan trng i vi cc h thng hiu nng cao, c lng truy cp ln l phi chia ti

56

TA Load balancing

S dng 2 loi request t client: RESTful, Websocket

HTTPS RESTful ly truy vn thng tin ti khon, gi chng khon ban u, t lnh...

Websocket update gi chng khon, update trng thi lnh trong phin giao dch...

HTTP/S l giao thc stateless nhng Websocket l statefull, do vy phi chia vic qun l connection ra lm 2 i vi load balancing.

Cng 80 cho websocket WS

Cng 443 cho HTTPS

Load Balancer phn phi request n cc application server da vo tn min.

57

TA Load balancing

Nginx: Load balancing + fault tolerance cho web server

Tc v hiu nng

Nh

n nh, tin cy

D dng cu hnh

Cng ng s dng

LVS Linux Virtual Server

KeepAlived

Cu hnh

Qun l connection theo domain

m bo performance cho websocket bng vic nn d liu

Multi-tenancy: nhiu cng ty, nhiu chi nhnh. Dch v theo tn min. SSL accelerator.
#Tn cng tyChc nngTn minCng LBCng ng dng1AAARESTfulAAA.tenmien.com 44380802BBBRESTfulBBB.tenmien.com44380813CommonWebSocketWebsocket.tenmien.com809080

TA M hnh Load Balancing

TA Load Balancing kt ni

TA Database

TA Database

MySQL l s la chn tt v free, mnh

Oracle t hn, l s la chn ti u hn v tc , hiu nng, dnh cho cc CTCK c tin

Scale-out

HA

Clustering

Replication

http://dev.mysql.com/doc/refman/5.0/en/ha-overview.html

62

TA M hnh Database

http://imysql.com/mysql-refman/5.6/replication.html tham kho kh tt v y v replication ca MySQL

H thng la chn m hnh Master Slave tc ghi tt hn

63

TA MySQL Master-Slave Replication

http://imysql.com/mysql-refman/5.6/replication.html tham kho kh tt v y v replication ca MySQL

64

TA Database ng b d liu

MySQL replicate d liu t master server sang slave server. C 2 phng thc:

PT bn ng b chm hn nhng m bo c d liu c nht qun

Vic copy d liu c MySQL thc hin bng cch copy binary log file t master sang slave server.
#Phng thcM t1Asynchronous Bt ng bKhi application update d liu vo master server, Mysql gi phn hi v Application m khng cn i d liu c replicate sang slave server.2Semi-synchronous Bn ng bKhi application update d liu vo master server, Mysql gi phn hi v Application sau khi i d liu c replicate sang slave server.

Phi cn bng gia cc non-funtional target vi phng thc ng b database

S tho lun k hn v option s dng PT no phn Tho lun Gii php K thut

65

TA M hnh Database bn ng b

App write (thong qua Connection Thread)

Update binary log

Update master DB

Send binary log tu Master Server qua Slave Server; Slave server tu dong Update Slave DB

Send ACK cho Connection Thread

Connection Thread send OK cho App

66

TA Database HA

S dng LVS, KeepAlived, VIP

Cc server tht s s dng chung mt IP o, sau dng KeepAlived qun l

Ti mt thi im KeepAlived s kim tra xem l ci IP o ang tr ti my vt l no. Nu connection OK th gi kt ni , nu khng th release v tr sang con khc.

VIP = Virtual IP; LVS = Linux Virtual Server

67

TA M hnh Database HA

68

TA M hnh Database MHA

TA M hnh Database MHA

TA Caching

TA Caching

Cc cng ngh khc:

Couchbase, MongoDB, Redis, Memcached, Ehcached...

La chn Redhat Infinispan, cc chc nng

Transactions Persistence Map/Reduce Querying

Tiu ch chn: support Java, lu c object v text, tng thch vi App Server, nh pht trin uy tn, OSS, hiu nng tt, HA, cn bng c ghi, max value...

Tham kho

http://infinispan.org/docs/7.0.x/user_guide/user_guide.html

http://db-engines.com/en/system/Couchbase%3BMemcached%3BRedis

http://vschart.com/compare/jboss-infinispan/vs/redis-database/vs/couchbase

72

Infinispan Core Architecture

73

TA Cache Server Model

Architecture: Clustered Lib

Use as library

More feature

Richer APIs

Programmatic/declarative configuration

Extendable/embeddable

Faster (API call)

App doesnt know its on cluster

Library Mode

Cluster

Infinispan

JVM

App

Infinispan

JVM

App

Infinispan

JVM

App

Run mode:

Library mode: App chy cng JVM vi Infinispan. Rich API, c th dng Listener API, gi API nhanh hn. D pht trin app hn.

Client-Server mode: App chy khc JVM vi Infinispan, process ca ng dng l c lp. C th dng cc giao thc HotRod, REST, memcached... giao tip App vi Cache.

Architecture: Server

Use as server

Remote

Memcached, REST, Hot Rod, WebSocket

Data tier shared by multiple apps

App doesnt affect cluster

Non-java clients

Java, C++,.NET, Ruby, Python

Server Mode

Cluster

Infinispan

JVM

Infinispan

JVM

Infinispan

JVM

App

App

App

TA Cache Listener / Notifications

Listener API: event notification

77

TA Cache - Mode c, ghi d liu

Replicated mode: tt c d liu c replicated qua tt c cc server. c d liu nhanh hn, ghi lu hn Distributed mode.

Distributed mode: d liu c khng replicated qua tt c server. c d liu lu hn, ghi nhanh hn.

TA Cache Clustering

Tt c cc server lin quan n vic trao i d liu c a vo trong 1 cluster. S dng multicast network ng b d liu gia cc server trong cluster.

Cc server gm c:

Application servers

OMS servers

Cache servers

Datafeed Servers

Internal GW servers

External GW servers

Cache HA: Cluster a node

1 node cht vn OK

80

TA Cache Configuration

TA Cache Dataflow

ng i ca d liu c vn ??!

Qu long ngong, khng nhanh, khng tin cy

Sai

82

TA Cache Dataflow

Trong lc ny, ng i ca d liu ngn, nhanh hn, tin cy hn

TA Message

TA Message

JMS, Message Queue

Truyn thng bt ng b (Async Comm)

i tng tham gia: GW ca back v front, GW kt ni S GD

TA MQ HA

Cc gii php k thut

& Case Study

AGENDA

Overview v kin trc

Overview v TTCK




Tng hp cc vn k thut khi xy dng Core chng khon

HA:

Fail-over, fall back

Load balancing

Scalability

Performance & throughput (response time, speed, concurrent trans, payload...), cn bng gia tc v an ton an ninh thng tin

Trao i message real-time

Nhiu n v kt ni, Multi-tenancy

Truy cp DB s dng ORM hay khng

Java: EJB hay Spring?

Entity ? Conversion? DTO ?

Multi-cast?

Which memory techniques are used?

Cache?

Message queue / JMS?

Kt lun

m t v TTCK, cc i tng tham gia, vai tr ca Core i vi 1 trong nhng i tng quan trng nht CTCK

Kin trc s b ca 1 Core system. Phng php tip cn khi xy dng kin trc . Cc cng vic cn thc hin khi thc hin kin trc la chn cng ngh, design (overview, data, app, hardware...)

Cc vn ca TTCK VN v cc core system hin ti. Cc vn k thut t ra khi thc hin 1 kin trc. (kin trc khc c th gp cc vn khc)

Biz quyt nh tt c

Khng c kin trc tt mi, ch c KT ph hp vi chin lc kinh doanh ca DN trong 1 giai on nht nh m thi.

Cc phng din khc cn xem xt: Budget, Non-functional target, Stakeholders, Market/Marco Economy, Customers

Slides trnh by Case Study + Tho lun (bi Mr V Vit Anh)

Thm kho thm : http://bit.ly/itlchn-stockcore-event

Front OfficeBack OfficeKhch hngNgn hngNgn hng Thanh tonTrung tm Lu k CKS GDCKS GDCK

Data Feed GatewayExchange GatewayBack Office SystemView (HTML5)Controller (JS framework)Front End Services Layer (JS framework)Restful + Websocket Resource LayerBusiness LayerMainDatabasesBrowser/Mobile BrowserRest Services callJSON compressedDataJDBCExchange SystemTrading Module Trading Gateway Receiving ModuleBOD DataReal-time DataEOD DataClientPresentationBusinessData AccessDatabaseData AccessReal-time Data Gateway (TCP/IP, FIX, REST, SOAP)Data Feed SystemBloombergsBroadcast gateway (UDP)UDPData SyncJPACommon Services ( Configuration, Logging, Security, Audit, Email, SMS, Scheduling)Broadcast Module ReutersDomestic Data VendorsPeriodical Data Gateway (REST, SOAP)File Gateway (Secured FTP)JPACacheCache for Biz LayerCache for DataFeedCache Client protocolBO GatewayMQ gatewaySFTP/SCP gatewayCache Replicate protocolCache Client protocolData FeedDatabasesFile ServerFor Data FeedSCP / SFTPGateway & IFExternal SystemsJDBCFile ServerFor BOSCP/SFTPCache Replicate protocolCache Client protocolVM or TCPMQOMS/EMS (Active/Standby)ExecutionReceive MQSending MQTCPOrder ProcessingTCPVMVMTrading Gateway Sending ModuleTCP/IPTCP/IP2.12.22.32.42.52.63.13.23.3b3.3a3.4a3.4b3.5b1.11.21.31.41.51.61.7MQVM or TCP4.14.24.34.5SCP/SFTPSCP/SFTPFile ServerFO-BO IF5.15.25.45.55.6Variable Protocols (TCP/IP, FIX, REST, SOAP or SFTP)

Application ServerRed Hat Enterprise Linux 7.1VM or Physical MachineOracle JDK 8Hibernate 4EJB 3.2Angular JSWildfly Application Server 9.0.1.FinalTwitter bootstrap, CSS 3JQueryDRBDInternal Load Balancing ServerRed Hat Enterprise Linux 7.1VM or Physical Machinenginx 1.9.3CacheRed Hat Enterprise Linux 7.1VM or Physical MachineInfinispanDRBDOracle JDK 8log4JDatabase ServerRed Hat Enterprise Linux 7.1VM or Physical MachineMySQL 5.7.7DRBDInternal GatewayRed Hat Enterprise Linux 7.1VM or Physical MachineActiveMQ/ZeroMQDRBDOracle JDK 8log4JDatafeedRed Hat Enterprise Linux 7.1VM or Physical MachineActiveMQ/ZeroMQDRBDOracle JDK 8log4JHibernate 4EJB 3.2InfinispanSFTP (OpenSSH)Exchange GatewayRed Hat Enterprise Linux 7.1VM or Physical MachineActiveMQ/ZeroMQDRBDOracle JDK 8log4JFIXEngineRESTeasyWebSocket (JavaEE)

InvestorsHTTPSHTTPVRRPHTTPBack GWReplicationExternalGWstandbyExternalGWactiveSynchronizer StandbySynchronizer ActiveBackExchangesData VendorsDatafeed standbyDatafeed activejGroupTCPjGroupTCPSFTPTCP(MQ)Auto-tFIXHOSEHNXCORE SYSTEMHT BN NGOIOMS activeDB masterLoad Balancer activeCache activeLoad Balancer standbyCache activeOMS standbyApp server activeDB slaveApp server activeData Vendors

KeepAlivedKeepAlivedNGINXNGINXVirtual IPHeart BeetJBOSSJBOSSRequestRequestStatus CheckStatus CheckLB1 ActiveLB2 StandbyApp Server 1 ActiveApp Server 2 ActiveWebsocketRESTful

App Server 2Websocket:1234Web: 8888App Server 1Websocket:1234Web: 8888LoadBalancerApp - Mobile - WebHTTPS:443ctck.tenmien.com:443To App Server 1 :8888websocket.tenmien.com:80To App Server 2 :8888To Application Server#1 :1234To Application Server#2 :1234WS:80

MasterWriteVIPReadLVSApplicationSlaveSlave

Old MasterWriteReplicateVIPReadLVSApplicationNew MasterSlaveVIP

Remote App 1 (C++)Remote App 2 (Java)Remote App 3 (.NET)Network (TCP)Node (JVM)MemCached, HotRod, REST, WebSocket (*)Embedded App (Java)Transport (JGroups)NotificationTransactions / XAQueryMap / ReduceMonitoringStorage Engine(RAM + Overflow)Node (JVM)MemCached, HotRod, REST, WebSocket (*)Embedded App (Java)Transport (JGroups)NotificationTransactions / XAQueryMap / ReduceMonitoringStorage Engine(RAM + Overflow)TCP/UDP

Remote App 1 (C++)Remote App 2 (Java)Remote App 3 (.NET)Network (TCP)Node (JVM)MemCached, HotRod, REST, WebSocket (*)Embedded App (Java)Transport (JGroups)NotificationTransactions / XAQueryMap / ReduceMonitoringStorage Engine(RAM + Overflow)Node (JVM)MemCached, HotRod, REST, WebSocket (*)Embedded App (Java)Transport (JGroups)NotificationTransactions / XAQueryMap / ReduceMonitoringStorage Engine(RAM + Overflow)TCP/UDP

M hnh phn tnWrite (Key, Value)Node 2Node 1Node 4Read (Key, Value)Node 3M hnh nhn bnWrite (Key, Value)Node 2Node 1Node 4Read (Key, Value)Node 3ValueData ItemKeyValueData ItemKeyValueData ItemKeyValueData ItemKeyValueData ItemKeyValueData ItemKeyValueData ItemKeyValueData ItemKeyValueData ItemKeyValueData ItemKey

CLUSTERK1K1K2K2K3K3K4K4K5K5

CLUSTERK1K1K2K2K3K3K4K4K5K5

MulticastReplicationDatafeedOMS CacheApp serversGatewaysSynchronizerCache server with only cache function

jGroupjGroupjGroupjGroupjGroupjGroupjGroupjGroupNet Area 1Net Area 2Net Cluster (jGroup protocol)MulticastRoutingDatafeedOMS CacheApp serversGatewaysSynchronizer

OMSGatewayApplicationCacheApp ServerDatabaseDesktopMobileWebOrder1.Write cacheApplicationCache2.ReplicationCache Listener3.NotificationOrder Handler4.Write CacheCache5.ReplicationApplicationCache ListenerMessage Handler6.Notification7.MQOrder APICache Listener

OMSGatewayApplicationCacheApp ServerDatabaseDesktopMobileWebOrder1.Write cacheApplicationCache2.ReplicationCache Listener3.NotificationOrder Handler4.Write CacheApplicationMessage Handler5.MQOrder APICache Listener

Failover HandlerMaster GWSlave GWFailover Master-IP -> Slave-IPMessage QueueApplicationCacheMessage QueueApplicationCacheBack GWMessageActive OMSActive OMSApplicationCacheApplicationCacheReplicateReplicate

Failover HandlerMaster Front GWSlave Slave GWFailover Master-IP -> Slave-IPMessage AdapterBack GWMessagePersistent&Exclusive lockreplicationJDBCJDBCMessage AdapterMessageMaster DBSlave DB

itlchn 20 - Kien truc he thong chung khoan - Phan 1

Technology

Transcript of itlchn 20 - Kien truc he thong chung khoan - Phan 1