TECHNOLOGYRADARMay 2015 — Our thoughts on the technology and trends that are shaping the future

1

2

3

TECHNOLOGY ADVISORY BOARD

4

5

6

THEMES FOR THIS ISSUE

7

TECHNIQUES8

TECHNIQUES8

9

ADOPT 1. Consumer-driven contract testing NEW

2. Focus on mean time to recovery 3. Generated infrastructure diagrams NEW 4. Structured logging

TRIAL 5. Canary builds 6. Datensparsamkeit 7. Local storage sync 8. NoPSD 9. Offline-first web applications NEW 10. Products over projects NEW 11. Threat Modelling NEW

ASSESS 12. Append-only data store 13. Blockchain beyond Bitcoin 14. Enterprise Data Lake 15. Flux NEW 16. “git-based CMS” NEW 17. Phoenix environments NEW 18. Reactive architectures NEW

HOLD 19. Long lived branches with Gitflow 20. Microservice envy 21. Programming in your CI/CD tool 22. SAFe™ 23. Security sandwich 24. Separate DevOps team

TECHNIQUES

10

TECHNIQUESArchitectures for the digital world (theme: innovations in architecture)

10

TECHNIQUES18

15

14

12

REACTIVE ARCHITECTURES

FLUX

ENTERPRISE DATA LAKE

APPEND-ONLY DATA STORE

Architectures for the digital world (theme: innovations in architecture)

Attrition

Acquisition

Retention

Activation

Referral

Ads

ARCHITECTURES FOR THE DIGITAL JOURNEY

11

Emails

Google

Legacy Systems

RIGHT DATA, RIGHT PLACE, RIGHT TIME

12

Legacy SystemsLegacy Systems Web Analytics Operational Metrics Insights

13

Business Logic

(functions)

Legacy Systems

Archivers

File Store (S3)

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

Microservices

Append-only

Database

Flux-based Web Application

User Actions

View Rendering

(react.js)

Transactions, Web Analytics,

Operational Logs

Subscribed Events Event Queue

(Time Series Database, Apache Kafka, AWS Kinesis,

Eventstore, …)

13

Business Logic

(functions)

Legacy Systems

Archivers

File Store (S3)

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

Microservices

Append-only

Database

Flux-based Web Application

User Actions

View Rendering

(react.js)

Transactions, Web Analytics,

Operational Logs

Subscribed Events

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)

14

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)

Transactions, Web Analytics,

Operational Logs

Subscribed Events

Business Logic

(functions)

Legacy Systems

Archivers

File Store (S3)Microservices

Append-only

Database

Flux-based Web Application

User Actions

View Rendering

(react.js)

14

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

Transactions, Web Analytics,

Operational Logs

Subscribed Events

Business Logic

(functions)

Legacy Systems

Archivers

File Store (S3)Microservices

Append-only

Database

Flux-based Web Application

User Actions

View Rendering

(react.js)

15

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

Append-only

Database

Flux-based Web Application

User Actions

View Rendering

(react.js)

Transactions, Web Analytics,

Operational Logs

Subscribed Events

Business Logic

(functions)

Legacy Systems

Archivers

File Store (S3)Microservices

15

Transactions, Web Analytics,

Operational Logs

Subscribed Events

Business Logic

(functions)

Legacy Systems

Archivers

File Store (S3)Microservices

16

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

File Store (S3)

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)

Microservices

Append-only

Database

Flux-based Web Application

User Actions

View Rendering

(react.js)

Transactions, Web Analytics,

Operational Logs

Subscribed Events

Business Logic

(functions)

Legacy Systems

Archivers

16

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

File Store (S3)

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)

16

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

File Store (S3)

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)

ALL DATA IN MOTION IS IMMUTABLE

16

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

File Store (S3)

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)

ALL DATA IN MOTION IS IMMUTABLE

FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER

16

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

File Store (S3)

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)

ALL DATA IN MOTION IS IMMUTABLE

FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER

MANAGING AND PUBLISHING EVENTS BRINGS COMPLEXITY

16

Insights Analytics

Reports, Model

Parameters

Spark, Hadoop

File Store (S3)

Event Queue (Time Series Database,

Apache Kafka, AWS Kinesis,

Eventstore, …)PUSHES RESPONSIBILITY FOR DATA QUALITY BACK ON THE SOURCE SYSTEMS

ALL DATA IN MOTION IS IMMUTABLE

FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER

MANAGING AND PUBLISHING EVENTS BRINGS COMPLEXITY

17

ADOPT 1. Consumer-driven contract testing NEW

2. Focus on mean time to recovery 3. Generated infrastructure diagrams NEW 4. Structured logging

TRIAL 5. Canary builds 6. Datensparsamkeit 7. Local storage sync 8. NoPSD 9. Offline-first web applications NEW 10. Products over projects NEW 11. Threat Modelling NEW

ASSESS 12. Append-only data store 13. Blockchain beyond Bitcoin 14. Enterprise Data Lake 15. Flux NEW 16. “git-based CMS” NEW 17. Phoenix environments NEW 18. Reactive architectures NEW

HOLD 19. Long lived branches with Gitflow 20. Microservice envy 21. Programming in your CI/CD tool 22. SAFe™ 23. Security sandwich 24. Separate DevOps team

TECHNIQUES

TOOLS18

TOOLS18

19

TOOLSADOPT 48. Composer 49. Go CD 50. Mountebank 51. Postman

TRIAL 52. Boot2docker 53. Brighter NEW 54. Consul

55. Cursive 56. Gitlab 57. Hamms NEW

58. IndexedDB 59. POLLY NEW 60. Rest-assured NEW 61. Swagger 62. Xamarin 63. ZAP NEW

ASSESS 64. Apache Kafka NEW 65. Blackbox 66. Bokeh/Vega NEW 67. Gor NEW 68. NaCL NEW 69. Origami NEW 70. Packet beat 71. pdfmake NEW 72. PlantUML NEW 73. Prometheus NEW 74. Quick NEW 75. Security Monkey NEW

HOLD 76. Citrix for development

20

TOOLS

20

TOOLS

57 HAMMS

WE BUILD DISTRIBUTED SYSTEMS BY DEFAULT

21

22

22

FAILURE IS THE NORM…but testing it is tough

HAMMSSimulating badly behaved servers

https://github.com/kevinburke/hamms

STARTING

24

STARTING

24

IF AT FIRST YOU DON’T SUCCEED…

25

http://127.0.0.1:5512/?key=demo&tries=6

IF AT FIRST YOU DON’T SUCCEED…

25

http://127.0.0.1:5512/?key=demo&tries=6

NOBODY IS 100% PERFECT

26

http://127.0.0.1:5513/?failrate=0.25

NOBODY IS 100% PERFECT

26

http://127.0.0.1:5513/?failrate=0.25

OTHER FAILURE MODES

27

Invalid content lengths and types Non-http responses

Slow responses Partial responses

WHEN WOULD YOU USE HAMMS?

28

Automated testing Usability testing

BENEFIT OF COMMON PROTOCOLS

29

HTTP, TCP/IP

Well-known failure cases

Build on other people’s hard work!

GOING BEYOND UNIT TESTING…

30

ZED ATTACK PROXY

CONSUMER-DRIVEN CONTRACT TESTING

REST-ASSURED

GORHAMMSTEST.GENERATIVE

QUICKCHECK

SIMULANT

RESOURCES

31

https://github.com/kevinburke/hamms

32

TOOLSADOPT 48. Composer 49. Go CD 50. Mountebank 51. Postman

TRIAL 52. Boot2docker 53. Brighter NEW 54. Consul

55. Cursive 56. Gitlab 57. HAMMS NEW

58. IndexedDB 59. POLLY NEW 60. Rest-assured NEW 61. Swagger 62. Xamarin 63. ZAP NEW

ASSESS 64. Apache Kafka NEW 65. Blackbox 66. Bokeh/Vega NEW 67. Gor NEW 68. NaCL NEW 69. Origami NEW 70. Packet beat 71. pdfmake NEW 72. PlantUML NEW 73. Prometheus NEW 74. Quick NEW 75. Security Monkey NEW

HOLD 76. Citrix for development

LANGUAGES & FRAMEWORKS

33

LANGUAGES & FRAMEWORKS

33

34

LANGUAGES & FRAMEWORKS

ADOPT 77. Nancy

TRIAL 78. Dashing 79. Django Rest 80. Ionic Framework 81. Nashorn 82. Om 83. React.js 84. Retrofit 85. Spring Boot

ASSESS 86. Ember.js NEW 87. Flight.js 88. Haskell Hadoop library 89. Lotus 90. Reagent 91. Swift

HOLD 92. JSF

35

LANGUAGES & FRAMEWORKS

35

LANGUAGES & FRAMEWORKS

85 SPRING BOOT

NANCY77

A TALE OF TWO WEB FRAMEWORKS

36

Java/Spring C#/.NET

Lightweight ✓ ✓

Low-ceremony ✓ ✓

Self-hosted ✓ ✓

Opinionated ✓ ✓

boot

ON THE SURFACE, VERY SIMILAR

37

ON THE SURFACE, VERY SIMILAR

37

BUT WHAT’S UNDER THE COVERS?

38

Spring Boot’s pom.xml

1847 lines in total!

BUT WHAT’S UNDER THE COVERS?

38

Nancy’s Nuget page

BUT WHAT’S UNDER THE COVERS?

38

Nancy’s Nuget page

FRAMEWORKS VS. COMPOSITION

39

Spring Framework

Your Spring Boot App

JettyYour App

Code

Owin

Nancy.Owin

Nancy

Composes

Calls higher-order functions

40

LANGUAGES & FRAMEWORKS

ADOPT 77. Nancy

TRIAL 78. Dashing 79. Django Rest 80. Ionic Framework 81. Nashorn 82. Om 83. React.js 84. Retrofit 85. Spring Boot

ASSESS 86. Ember.js NEW 87. Flight.js 88. Haskell Hadoop library 89. Lotus 90. Reagent 91. Swift

HOLD 92. JSF

PLATFORMS41

PLATFORMS41

42

PLATFORMSADOPT TRIAL 25. Apache Spark NEW 26. Cloudera Impala NEW 27. DigitalOcean 28. TOTP Two-Factor Authentication

HOLD 45. Application Servers NEW 46. OSGi 47. SPDY NEW

ASSESS 29. Apache Kylin NEW 30. Apache Mesos 31. CoreCLR and CoreFX NEW 32. CoreOS 33. Deis NEW 34. H2O NEW 35. Jackrabbit Oak 36. Linux security modules 37. MariaDB 38. Netflix OSS Full stack 39. OpenAM 40. SDN 41. Spark.io 42. Text it as a service / Rapidpro.io 43. Time-series Databases NEW 44. U2F

43

PLATFORMS

43

PLATFORMS26CLOUDERA IMPALA

44

IMPALA Elegant, nimble querying of

big, lumbering data

45

HADOOP Fast processing

Scalable Commodity hardware

Elegant failure handling HDFS

MAPREDUCE

46

• Powerful … but complicated • Not always the right model

Input

Splitting Mapping

SortingReducing

SQL

47

SELECT …

FROM …

WHERE …

GROUP BY …

Declarative

Simple

Well known

RELATIONAL DATABASES

48

Not always good choice for adhoc analysis:

RELATIONAL DATABASES

48

Not always good choice for adhoc analysis:

• Data needs to be optimised for consumption

ETL

?

?

RELATIONAL DATABASES

48

Not always good choice for adhoc analysis:

• Data needs to be optimised for consumption

• Scaling is tough

?

?

??

ETL

?

?

IMPALA

49

An SQL engine that runs on Hadoop

IMPALA IN ACTION

50

External tables

create EXTERNAL TABLE books( id BIGINT, isbn STRING, category STRING, publish_date TIMESTAMP, publisher STRING, price FLOAT )

ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION '/data/books/';

IMPALA IN ACTION

51

SELECT category,

count(*)

FROM books

GROUP BY category

Real-time querying of large datasets

IMPALA IN ACTION

51

SELECT category,

count(*)

FROM books

GROUP BY category

+---------------------------+----------+| category | count(*) |+---------------------------+----------+| COMPUTERS | 312604 || SPORTS-RECREATION | 312766 || PETS | 313039 || CRAFTS-HOBBIES | 312438 || FAMILY-RELATIONSHIPS | 312435 || HOUSE-HOME | 313365 || EDUCATION | 313014 |

Runtime : 5 seconds(~15 million rows)

Real-time querying of large datasets

IMPALA IN ACTION

52

Arbitrary joins

SELECT category, sum(quantity) as total_quantity

FROM books

JOIN [SHUFFLE] transactions ON books.id = transactions.book_id

GROUP BY category;

IMPALA IN ACTION

52

Arbitrary joins

SELECT category, sum(quantity) as total_quantity

FROM books

JOIN [SHUFFLE] transactions ON books.id = transactions.book_id

GROUP BY category;

+---------------------------+----------------+| category | total_quantity |+---------------------------+----------------+| STUDY-AIDS | 10171749 || SELF-HELP | 10238182 || TECHNOLOGY-ENGINEERING | 10207003 || SOCIAL-SCIENCE | 10177075 || HISTORY | 10238243 || LAW | 10229825 || PERFORMING-ARTS | 10231744 || POLITICAL-SCIENCE | 10193267 |

Runtime : 19 seconds

WHEN WOULD YOU USE IMPALA?

53

Adhoc/exploratory data analysis

Existing Business Intelligence tools via

ODBC/JDBC

DIVERSIFY CONSUMERS OF YOUR DATA

54

?

?

?

?

Our Data

Curious Person

Financial Analyst

Customer Experience

Guru

Dashboard Whizz

DIVERSIFY CONSUMERS OF YOUR DATA

54

?

BI Tools

?

?

?

?

Our Data

Curious Person

Financial Analyst

Customer Experience

Guru

Dashboard Whizz

DIVERSIFY CONSUMERS OF YOUR DATA

54

?

BI Tools

Data Scientists

Graph Fans

Map Reducers

?

?

?

?

Our Data

Curious Person

Financial Analyst

Customer Experience

Guru

Dashboard Whizz

MADE POSSIBLE BY V2.0 DESIGN CHOICES

55

Diagram taken from: http://hortonworks.com/blog/apache-hadoop-2-is-ga/

FURTHER RESOURCES

56

http://impala.io/

Tutorials from Cloudera: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_tutorial.html

Impala and Amazon EMR: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-impala.html

Hadoop: http://hadoop.apache.org/

Yarn + hadoop v2: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html

57

PLATFORMSADOPT TRIAL 25. Apache Spark NEW 26. Cloudera Impala NEW 27. DigitalOcean 28. TOTP Two-Factor Authentication

HOLD 45. Application Servers NEW 46. OSGi 47. SPDY NEW

ASSESS 29. Apache Kylin NEW 30. Apache Mesos 31. CoreCLR and CoreFX NEW 32. CoreOS 33. Deis NEW 34. H2O NEW 35. Jackrabbit Oak 36. Linux security modules 37. MariaDB 38. Netflix OSS Full stack 39. OpenAM 40. SDN 41. Spark.io 42. Text it as a service / Rapidpro.io 43. Time-series Databases NEW 44. U2F

58

Scott Shaw

@scottwshaw

Jen Smith

@JenniferSmithCo

thoughtworks.com/radar