Microservices architecture pitfalls

WJUG meeting ◦ march 2015

Mateusz GajewskiSolutions Architect @ Allegro

Twitter: @wendigo

About me

given: I started working in Allegro in 2009 (5 mln AO, 50 devs)

when: Allegro reached 40 mln AO, 400 devs

then: I am Solutions Architect

2

Agenda

• Microservices, microservices, microservices! ;)

• Some challenges & pitfalls:

• Architectural,

• Operational,

• Organisational

3

Let’s go back in time to year 2012

4

5

Back then we wanted

• agile development,

• scalability,

• resilience,

• lower costs,

• hybrid cloud.

6

Basically SOA + JVM was an answer!

7

But our system was too BIG & too complex to do it with

existing enterprise solutions

8

s/Enterprise/OSS/g Solutions ;)

9

we’ve started to do *buzzword*

10

And now, literally everyone is doing microservices!!??

11

Microservices by Fowler

12

Lots of *buzzwords*

http://martinfowler.com/articles/microservices.html

SOA ≈

microservices?

13

microservices architecture ≈

fine-grained SOA − enterprise (commercial) sh*t

≈ highly scalable, distributed system

14

Distributed systems

• concurrency of components,

• independent failure of components,

• lack of a global clock.

15

The Eight Fallacies of Distributed Computing

16

by Peter Deutsch

1991

#1: Network is reliable

17

#2: Latency is zero

18

#3: Bandwidth is infinite

19

#4: Network is secure

20

#5: Topology doesn’t change

21

#6: There is one administrator

22

#7: Transport cost is zero

23

#8: Network is homogeneous

24

distributed systems are hard →

microservices are much harder ;)

25

What have we learnt?

26

Act I: architectural constraints

27

CAP is not just theorem it’s reality against us

28

bye, bye ACID semantics

29

Long live BASE guarantees!

Basically

Available,

Soft state,

Eventually consistent

30

distributed transactions add complexity

31

it’s far cheaper to do compensation

32

33

http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

you need idempotent APIs and events sinks

34

35

choreography > orchestration

So we’ve built Hermes a.k.a circulatory system

36

network can be congested!

37

REST+JSON on top of HTTP/1.1is fine

38

REST+JSON on top of HTTP/2.0with TLS is finer

39

we don’t rely on network anymore

net splits in public clouds happens everytime!

40

we adopted antifragile organization

41

42

powerful tandem

43

+

Reactive programming Circuit breaker pattern

you need to support non-native old services, clients

and systems

44

45

conclusion: constant architecture

improvement

46

47

Act II: operational troubles

creating new service should be instant!

48

49

automationwith gradle & axions

50

51

so now we’ve got over 1800 repositories grouped under

250 projects

52

all with CI, code quality checks,

security checks, integrated with sonar & artefact repository

53

but what withservices upgrades?

54

we’ve initially built our own service stack

… and it was ok - for a while

55

now we are extending spring-boot

with so called andamio project

56

rapid deployments integrated with CI/CD environment and canary tests are must-have

57

war files ▾

scp + puppet ▾

golden images ▾

docker (immutable images) ▾

58

frequency of changes →

automatedmonitoring, logging

& operational insights

59

graphite statsd cabot

tessera kibana

logstash zabbix

newrelic selena

pingdom …60

Monitoring As A Service +

SLA Monitoring +

61

we need to build real-time anomaly detection soon

62

63

Act III: organizational shift

strategic DDD is good for splitting up monolith

64

but leave tactical DDD up to teams

65

huge polyglot hangover

66

acquiring distributed skills

67

you build it - you run it

68

coupling avoidance

69

please don’t audit me

70

distributed (micro) data curation

71

So after two years…

72

73

Final thoughts

74

75

76

77

Thanks!

Any questions?

Visit our blog: allegrotech.io

Follow us on twitter: @allegrotechblog

Check our OSS projects: github.com/allegro

And meetup group: meetup.com/allegrotech

78