Data on Steroids: Building a Fast Financial Applica7on Using Hybrid Cloud & SOA

Alex Legault - Associate Director, Products

2

Leading resource for public and private market analysis

Keep these numbers as up-to-date as possible. Reference pitchbook.com for latest counts. Round numbers to increments of 5. Update the date if you change the numbers as well. Access Sharepoint. Delete this text box when you’re finished.

910K companies

220K investors

785K deals

40K funds

1.5M professionals

What we’re tracking…

200,000 valuations

786,000 investments

26,000 LPs

3

HybridCloud+SOAMigra3on

4

Why?

1

2

3

4

5

Geographically spread teams

Different missions/priori7es

Varied tech stacks

Emphasis delivering features faster

Up7me resiliency – reduce firefigh7ng

5

•  Projects built with new mindset; Execu7ve buy-in easier with new features.

•  Rehost non-cri7cal apps, experiment with backups, image hos7ng, etc

•  Iden7fy tech and arch choices that will hinder migra7on. Ex: SQL Server cost...

•  Next, more isolated, natural SOA candidates. Learn from “quick wins”

•  Tackle complex, main parts of the app, once you’ve learned something

•  Only to introduce new services as needed

ThePlan

6

TeamStructureisCri3cal!

Arch teams to move fast I.  Break up team by func7on II.  Strong team leads needed III.  Formalize Devops

Delegate decision-making I.  Tech choices, Cloud choice II.  Code III.  QA IV.  Long-term deployment

Agile

I.  Deploy code quickly II.  Don't expect other teams to respond

as quickly

7

WhyAWS?

A.  ManagedServices–DataScienceteamshouldfocusonmodels,notmaintainingdatabases,configuringservers,etc

B.  Servicesthatplaywelltogether

C.  HighpaceofinnovaIon

D.  GreatnetworkofotherengineersworkingonAWSifyouneedsupport

8

LessonsLearned

Publisher 18 hrs 6hrs

Timing buffered disk reads

119.14 MB/sec 654.91 MB/sec

839 MB copied 96.3 MBs 1.4 GB/s

Cost $2500/mo $1200/mo

Not as simple as re-hos7ng

9

RootsofDataSciatPitchBook

DataScienIst(SEA)

SendRorpyfile

OffshoreTeam

RewriteinJavaorusewrapper

RedployenIremonolith

10

RootsofDataSciatPitchBook

DataScienIst(SEA)

SendRorpyfile

OffshoreTeam

RewriteinJavaorusewrapper

RedployenIremonolith

Painful itera7on process!

11

NewandImprovedDataScience

DataScienIst(SEA)

ScalableNameTaggerModel

12

MiniServicesvsMicroservices

Monolith

Profile

Signals

Gravity

Dashboard

Search

SR

News

Financials

VS. VS.

13

LessonLearned

Only split services when necessary

Search Result Service

Search Engine Service

Hkps Ask yourself,

Why?

14

AddingNewDatasets

Researcher

SendCSV

OffshoreTeam

SQLImportRunDatabase

Switch

Using old structure

Build/runcrawlerfromlocal

15

AddingNewDatasets

API Gateway Pipeline DB

Data Store

Researcher UI

Orchestrator

Gatekeeper Update Queue

Crawl Service

Feature Extractors

Using hybrid structure

16

What’s next

S7ll working on deployment tooling…

Teams on AWS have full

CI/CD using Jenkins

Rackspace teams have CI,

not CD

Peace of mind that everything is fully

portable between

clouds and SO

Troubleshoo7ng is a pain…

Need beker monitoring

tools

…Reliant on devops as superheroes

1 2 3 4

A N I M A T E D

Q&A