TECHNOLOGYRADARMay 2015 — Our thoughts on the technology and trends that are shaping the future
1
2
3
TECHNOLOGY ADVISORY BOARD
4
5
6
THEMES FOR THIS ISSUE
7
TECHNIQUES8
TECHNIQUES8
9
ADOPT 1. Consumer-driven contract testing NEW
2. Focus on mean time to recovery 3. Generated infrastructure diagrams NEW 4. Structured logging
TRIAL 5. Canary builds 6. Datensparsamkeit 7. Local storage sync 8. NoPSD 9. Offline-first web applications NEW 10. Products over projects NEW 11. Threat Modelling NEW
ASSESS 12. Append-only data store 13. Blockchain beyond Bitcoin 14. Enterprise Data Lake 15. Flux NEW 16. “git-based CMS” NEW 17. Phoenix environments NEW 18. Reactive architectures NEW
HOLD 19. Long lived branches with Gitflow 20. Microservice envy 21. Programming in your CI/CD tool 22. SAFe™ 23. Security sandwich 24. Separate DevOps team
TECHNIQUES
10
TECHNIQUESArchitectures for the digital world (theme: innovations in architecture)
10
TECHNIQUES18
15
14
12
REACTIVE ARCHITECTURES
FLUX
ENTERPRISE DATA LAKE
APPEND-ONLY DATA STORE
Architectures for the digital world (theme: innovations in architecture)
Attrition
Acquisition
Retention
Activation
Referral
Ads
ARCHITECTURES FOR THE DIGITAL JOURNEY
11
Emails
Legacy Systems
RIGHT DATA, RIGHT PLACE, RIGHT TIME
12
Legacy SystemsLegacy Systems Web Analytics Operational Metrics Insights
13
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
Transactions, Web Analytics,
Operational Logs
Subscribed Events Event Queue
(Time Series Database, Apache Kafka, AWS Kinesis,
Eventstore, …)
13
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
14
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
14
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
15
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)Microservices
15
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)Microservices
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER
MANAGING AND PUBLISHING EVENTS BRINGS COMPLEXITY
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)PUSHES RESPONSIBILITY FOR DATA QUALITY BACK ON THE SOURCE SYSTEMS
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER
MANAGING AND PUBLISHING EVENTS BRINGS COMPLEXITY
17
ADOPT 1. Consumer-driven contract testing NEW
2. Focus on mean time to recovery 3. Generated infrastructure diagrams NEW 4. Structured logging
TRIAL 5. Canary builds 6. Datensparsamkeit 7. Local storage sync 8. NoPSD 9. Offline-first web applications NEW 10. Products over projects NEW 11. Threat Modelling NEW
ASSESS 12. Append-only data store 13. Blockchain beyond Bitcoin 14. Enterprise Data Lake 15. Flux NEW 16. “git-based CMS” NEW 17. Phoenix environments NEW 18. Reactive architectures NEW
HOLD 19. Long lived branches with Gitflow 20. Microservice envy 21. Programming in your CI/CD tool 22. SAFe™ 23. Security sandwich 24. Separate DevOps team
TECHNIQUES
TOOLS18
TOOLS18
19
TOOLSADOPT 48. Composer 49. Go CD 50. Mountebank 51. Postman
TRIAL 52. Boot2docker 53. Brighter NEW 54. Consul
55. Cursive 56. Gitlab 57. Hamms NEW
58. IndexedDB 59. POLLY NEW 60. Rest-assured NEW 61. Swagger 62. Xamarin 63. ZAP NEW
ASSESS 64. Apache Kafka NEW 65. Blackbox 66. Bokeh/Vega NEW 67. Gor NEW 68. NaCL NEW 69. Origami NEW 70. Packet beat 71. pdfmake NEW 72. PlantUML NEW 73. Prometheus NEW 74. Quick NEW 75. Security Monkey NEW
HOLD 76. Citrix for development
20
TOOLS
20
TOOLS
57 HAMMS
WE BUILD DISTRIBUTED SYSTEMS BY DEFAULT
21
22
22
FAILURE IS THE NORM…but testing it is tough
HAMMSSimulating badly behaved servers
https://github.com/kevinburke/hamms
STARTING
24
STARTING
24
IF AT FIRST YOU DON’T SUCCEED…
25
http://127.0.0.1:5512/?key=demo&tries=6
IF AT FIRST YOU DON’T SUCCEED…
25
http://127.0.0.1:5512/?key=demo&tries=6
NOBODY IS 100% PERFECT
26
http://127.0.0.1:5513/?failrate=0.25
NOBODY IS 100% PERFECT
26
http://127.0.0.1:5513/?failrate=0.25
OTHER FAILURE MODES
27
Invalid content lengths and types Non-http responses
Slow responses Partial responses
WHEN WOULD YOU USE HAMMS?
28
Automated testing Usability testing
BENEFIT OF COMMON PROTOCOLS
29
HTTP, TCP/IP
Well-known failure cases
Build on other people’s hard work!
GOING BEYOND UNIT TESTING…
30
ZED ATTACK PROXY
CONSUMER-DRIVEN CONTRACT TESTING
REST-ASSURED
GORHAMMSTEST.GENERATIVE
QUICKCHECK
SIMULANT
32
TOOLSADOPT 48. Composer 49. Go CD 50. Mountebank 51. Postman
TRIAL 52. Boot2docker 53. Brighter NEW 54. Consul
55. Cursive 56. Gitlab 57. HAMMS NEW
58. IndexedDB 59. POLLY NEW 60. Rest-assured NEW 61. Swagger 62. Xamarin 63. ZAP NEW
ASSESS 64. Apache Kafka NEW 65. Blackbox 66. Bokeh/Vega NEW 67. Gor NEW 68. NaCL NEW 69. Origami NEW 70. Packet beat 71. pdfmake NEW 72. PlantUML NEW 73. Prometheus NEW 74. Quick NEW 75. Security Monkey NEW
HOLD 76. Citrix for development
LANGUAGES & FRAMEWORKS
33
LANGUAGES & FRAMEWORKS
33
34
LANGUAGES & FRAMEWORKS
ADOPT 77. Nancy
TRIAL 78. Dashing 79. Django Rest 80. Ionic Framework 81. Nashorn 82. Om 83. React.js 84. Retrofit 85. Spring Boot
ASSESS 86. Ember.js NEW 87. Flight.js 88. Haskell Hadoop library 89. Lotus 90. Reagent 91. Swift
HOLD 92. JSF
35
LANGUAGES & FRAMEWORKS
35
LANGUAGES & FRAMEWORKS
85 SPRING BOOT
NANCY77
A TALE OF TWO WEB FRAMEWORKS
36
Java/Spring C#/.NET
Lightweight ✓ ✓
Low-ceremony ✓ ✓
Self-hosted ✓ ✓
Opinionated ✓ ✓
boot
ON THE SURFACE, VERY SIMILAR
37
ON THE SURFACE, VERY SIMILAR
37
BUT WHAT’S UNDER THE COVERS?
38
Spring Boot’s pom.xml
1847 lines in total!
BUT WHAT’S UNDER THE COVERS?
38
Nancy’s Nuget page
BUT WHAT’S UNDER THE COVERS?
38
Nancy’s Nuget page
FRAMEWORKS VS. COMPOSITION
39
Spring Framework
Your Spring Boot App
JettyYour App
Code
Owin
Nancy.Owin
Nancy
Composes
Calls higher-order functions
40
LANGUAGES & FRAMEWORKS
ADOPT 77. Nancy
TRIAL 78. Dashing 79. Django Rest 80. Ionic Framework 81. Nashorn 82. Om 83. React.js 84. Retrofit 85. Spring Boot
ASSESS 86. Ember.js NEW 87. Flight.js 88. Haskell Hadoop library 89. Lotus 90. Reagent 91. Swift
HOLD 92. JSF
PLATFORMS41
PLATFORMS41
42
PLATFORMSADOPT TRIAL 25. Apache Spark NEW 26. Cloudera Impala NEW 27. DigitalOcean 28. TOTP Two-Factor Authentication
HOLD 45. Application Servers NEW 46. OSGi 47. SPDY NEW
ASSESS 29. Apache Kylin NEW 30. Apache Mesos 31. CoreCLR and CoreFX NEW 32. CoreOS 33. Deis NEW 34. H2O NEW 35. Jackrabbit Oak 36. Linux security modules 37. MariaDB 38. Netflix OSS Full stack 39. OpenAM 40. SDN 41. Spark.io 42. Text it as a service / Rapidpro.io 43. Time-series Databases NEW 44. U2F
43
PLATFORMS
43
PLATFORMS26CLOUDERA IMPALA
44
IMPALA Elegant, nimble querying of
big, lumbering data
45
HADOOP Fast processing
Scalable Commodity hardware
Elegant failure handling HDFS
MAPREDUCE
46
• Powerful … but complicated • Not always the right model
Input
Splitting Mapping
SortingReducing
SQL
47
SELECT …
FROM …
WHERE …
GROUP BY …
Declarative
Simple
Well known
RELATIONAL DATABASES
48
Not always good choice for adhoc analysis:
RELATIONAL DATABASES
48
Not always good choice for adhoc analysis:
• Data needs to be optimised for consumption
ETL
?
?
RELATIONAL DATABASES
48
Not always good choice for adhoc analysis:
• Data needs to be optimised for consumption
• Scaling is tough
?
?
??
ETL
?
?
IMPALA
49
An SQL engine that runs on Hadoop
IMPALA IN ACTION
50
External tables
create EXTERNAL TABLE books( id BIGINT, isbn STRING, category STRING, publish_date TIMESTAMP, publisher STRING, price FLOAT )
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION '/data/books/';
IMPALA IN ACTION
51
SELECT category,
count(*)
FROM books
GROUP BY category
Real-time querying of large datasets
IMPALA IN ACTION
51
SELECT category,
count(*)
FROM books
GROUP BY category
+---------------------------+----------+| category | count(*) |+---------------------------+----------+| COMPUTERS | 312604 || SPORTS-RECREATION | 312766 || PETS | 313039 || CRAFTS-HOBBIES | 312438 || FAMILY-RELATIONSHIPS | 312435 || HOUSE-HOME | 313365 || EDUCATION | 313014 |
Runtime : 5 seconds(~15 million rows)
Real-time querying of large datasets
IMPALA IN ACTION
52
Arbitrary joins
SELECT category, sum(quantity) as total_quantity
FROM books
JOIN [SHUFFLE] transactions ON books.id = transactions.book_id
GROUP BY category;
IMPALA IN ACTION
52
Arbitrary joins
SELECT category, sum(quantity) as total_quantity
FROM books
JOIN [SHUFFLE] transactions ON books.id = transactions.book_id
GROUP BY category;
+---------------------------+----------------+| category | total_quantity |+---------------------------+----------------+| STUDY-AIDS | 10171749 || SELF-HELP | 10238182 || TECHNOLOGY-ENGINEERING | 10207003 || SOCIAL-SCIENCE | 10177075 || HISTORY | 10238243 || LAW | 10229825 || PERFORMING-ARTS | 10231744 || POLITICAL-SCIENCE | 10193267 |
Runtime : 19 seconds
WHEN WOULD YOU USE IMPALA?
53
Adhoc/exploratory data analysis
Existing Business Intelligence tools via
ODBC/JDBC
DIVERSIFY CONSUMERS OF YOUR DATA
54
?
?
?
?
Our Data
Curious Person
Financial Analyst
Customer Experience
Guru
Dashboard Whizz
DIVERSIFY CONSUMERS OF YOUR DATA
54
?
BI Tools
?
?
?
?
Our Data
Curious Person
Financial Analyst
Customer Experience
Guru
Dashboard Whizz
DIVERSIFY CONSUMERS OF YOUR DATA
54
?
BI Tools
Data Scientists
Graph Fans
Map Reducers
?
?
?
?
Our Data
Curious Person
Financial Analyst
Customer Experience
Guru
Dashboard Whizz
MADE POSSIBLE BY V2.0 DESIGN CHOICES
55
Diagram taken from: http://hortonworks.com/blog/apache-hadoop-2-is-ga/
FURTHER RESOURCES
56
http://impala.io/
Tutorials from Cloudera: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_tutorial.html
Impala and Amazon EMR: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-impala.html
Hadoop: http://hadoop.apache.org/
Yarn + hadoop v2: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
57
PLATFORMSADOPT TRIAL 25. Apache Spark NEW 26. Cloudera Impala NEW 27. DigitalOcean 28. TOTP Two-Factor Authentication
HOLD 45. Application Servers NEW 46. OSGi 47. SPDY NEW
ASSESS 29. Apache Kylin NEW 30. Apache Mesos 31. CoreCLR and CoreFX NEW 32. CoreOS 33. Deis NEW 34. H2O NEW 35. Jackrabbit Oak 36. Linux security modules 37. MariaDB 38. Netflix OSS Full stack 39. OpenAM 40. SDN 41. Spark.io 42. Text it as a service / Rapidpro.io 43. Time-series Databases NEW 44. U2F
58
Scott Shaw
@scottwshaw
Jen Smith
@JenniferSmithCo
thoughtworks.com/radar
Top Related