We broke up with the monolith, and started dating #eventSourcing - #symfonyCat
-
Upload
javier-ferrer-gonzalez -
Category
Technology
-
view
741 -
download
1
Transcript of We broke up with the monolith, and started dating #eventSourcing - #symfonyCat
We broke up with the Monolith, and started dating Event SourcingMigrations, Event Sourcing, AWS, and other Demeter things
@JavierCane#symfonyCat - Symfony Catalunya 2016
@JavierCane
Saliendo del monolito, al Rico EventitoMigraciones, Event Sourcing, y otras cosas del meter
#symfonyCat - Symfony Catalunya 2016
Contents
1. Starting point 2. Needs 3. Migration action plan 4. Sync flow example 5. Error handling 6. Issues found 7. Conclusions
Starting point Mobile first app
! In-house REST API
◕ symfony/symfony
◕ friendsofsymfony/rest-bundle
◕ jms/serializer
◕ nelmio/api-doc-bundle
Starting point Where do we come
! Startup with less than 2 years
◕ Externalize services (Parse, Kahuna) ! Funding: $200M
◕ Ads on TV in USA and Turkey
Needs New chat system!
! From:
◕ REST API
◕ “Chat” (messaging system, polling) ! To:
◕ WebSockets
◕ Chat (real time, push)
◕ Service isolated from the main API
◕ Backwards Compatible (mobile!)
◕ Scale! ! QA tested, Coming to a theater near you!
Needs New chat system!
! To:
◕ WebSockets
◕ Chat (real time, push)
◕ Service isolated from the main API
◕ Backwards Compatible (mobile!)
◗ Not loosing anyone
◗ Enables a “soft migration”
◕ Scale! ! QA tested, Coming to a theater near you!
Migration action plan Maintaining BC
! Develop new chat with Scala and Akka, and WS ! Maintain BC taking advantage of Event Sourcing
! Why not share DB?
◕ Limit scheme modifications for the new chat
◕ Chaos
◕ Coupling
Migration action plan Maintaining BC
! SQS: Amazon Simple Queue Service
◕ Managed message queuing service
◕ Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available
! Why not sync through SQS?
◕ Coupling
◗ 2 subscribers => publish to 2 SQS
Migration action plan Maintaining BC
! SNS: Amazon Simple Notification Service
◕ Send push messages to subscribers (no polling)
◕ Scale as your needs grow ! Why sync through SNS?
◕ Decoupled
◗ N subscribers => publish to 1 SNS
◗ Open-Closed Principle compliant at a services level
Migration action plan Maintaining BC
! 1 SNS per microservice
◕ SNS per event: Too much detail. New event => New SNS
◕ Unique SNS: Not enough detail. Too much noise on the subscribed SQS
Migration action plan Maintaining BC
Migration action plan Maintaining BC
! Delete all messages coming from the legacy chat on the SQS until day X
! Migrate all conversations and messages from day 0 to X using a script
! Start consuming the SQS
Sync flow example From the new chat to the legacy one
{ "data": { "id": uuid, "type": "conversation_archived", "attributes": { "conversation_id": uuid, "talker_id": uuid, "archived_at": timestamp, }, "meta" : { "created_at": timestamp, "from_legacy": bool }
First level: “protocol” data
Sync flow example From the new chat to the legacy one
"data": { "id": uuid, "type": "conversation_archived", "attributes": { "conversation_id": uuid, "talker_id": uuid, "archived_at": timestamp, }, "meta" : { "created_at": timestamp, "from_legacy": bool } }}
Domain aggregate data
Sync flow example From the new chat to the legacy one
"id": uuid, "type": "conversation_archived", "attributes": { "conversation_id": uuid, "talker_id": uuid, "archived_at": timestamp, }, "meta" : { "created_at": timestamp, "from_legacy": bool } }}
Event metadata
Conv. archivedConv. archived
Get items
Conv. archived
Archive conv.
Error handling Order not guaranteed
Conv. createdConv. created
Conv. created
Conv. archivedConv. archived
Get items
Conv. archived
Archive conv.
Error handling Order not guaranteed
Conv. createdConv. created
Conv. createdLets queue it with ‘++$attempts;’
Conv. archivedConv. archived
Get items
Conv. archived
Archive conv.
Error handling Order not guaranteed
Conv. createdConv. created
Conv. created
Create conv.Conv. archived’
Get items
Conv. archived’ Archive conv.
! If we find an error processing an event =>
◕ Increment its processing attempts and queue it again ! If the number of processing attempts is greater than 5 =>
◕ Queue it into a dead letter SQS ! Symfony command in order to reprocess the dead letter
Error handling Order not guaranteed
Error handling Identity generation
Conv. created
Message sentConv. created
Message sentGet items
Conv. created
Message sent
Send message
Error handling Identity generation
Conv. created
Message sentConv. created
Message sentGet items
Conv. created
Message sent
Send message
Conversation does not exists! Lets create it with ID ‘b’!
Error handling Identity generation
Conv. created
Message sentConv. created
Message sentGet items
Conv. created
Message sent
Create conv.
Send message
Conversation does not exists! Lets create it with ID ‘b’!
Lets create the conversation ‘a’!
Issues found SQS -> SNS
! Order not guaranteed & events duplication ! Alternatives:
◕ RabbitMQ
◗ Why not: go fast & don’t worry about clusters
◕ Kafka
◗ Why not: Same as RabbitMQ
◗ BTW: Order guaranteed, no duplication, partitions
◕ Kinesis
◗ Why not: More expensive
◗ BTW: Order guaranteed, no duplication, partitions
◕ Furthermore: Do not add more uncertainty
Issues found Initial data migration
! Source DB node: Slave outside the production LB ! Destination DB node: Increase AWS provisioned IOPS
Issues found Not following SRP
! REST API
◕ Send message endpoint => create conversation if not exists. #basuritas
◕ GET conversation messages => mark messages as read
◗ Check if a conversation exists, new endpoint: ”HEAD /conversation/{id}”
Conclusions
! Event Sourcing ~~ DIP at a services level
◕ Dependency Inversion Principle (CodelyTV video)
◕ Allow OCP: One SNS publishes to many SQS
◕ Encourage SRP
◕ Decoupleeeeee
Conclusions
! Event Sourcing ~~ DIP at a services level
◕ Dependency Inversion Principle (CodelyTV video)
◕ Allow OCP: One SNS publishes to many SQS
◕ Encourage SRP
◕ Decoupleeeeee
Conclusions
! Isolated services
◕ Different technologies
◕ Different contexts
◕ Change context -> Mental challenge
! From PHP to Scala: 🙃 🙂 🙃 💃 🙃 😀 😐 😳 🤔 🙂 🙃 🙂 …
! Know more at #SCBCN16! (October 1st & 2nd in BCN) ! A lot of work, if you want to deal with that, we're hiring!
References / To take away
! A Series of Fortunate Events ! The anatomy of Domain Event ! 6 Code Smells with your CQRS Events and How to Avoid Them ! json:api standard ! Sending Amazon SNS Messages to Amazon SQS Queues ! Subscribing a Queue to an Amazon SNS Topic
Credits
! Presentation base template by SlidesCarnival ! Graphics generated using draw.io ! Icon set Small-n-flat ! A Series of Fortunate Events