Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017
-
Upload
big-data-spain -
Category
Technology
-
view
471 -
download
0
Transcript of Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017
![Page 1: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/1.jpg)
![Page 2: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/2.jpg)
Scaling a backend for big dataand blockchain environment
![Page 3: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/3.jpg)
1. Introduction (The companies, project and me)
2. Backend challenge
3. Big Data Integration
4. Blockchain Integration
·············· P. 3
··································································· P. 7
································································· P. 17
··························································· P. 21
![Page 4: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/4.jpg)
@ganchix
Head of Blockchain / Backend
Me
ganchix
![Page 5: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/5.jpg)
The companies
![Page 6: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/6.jpg)
![Page 7: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/7.jpg)
Fecha
The project
• Financial platform• Marketplace
• Tokenization• Operate with cryptocurrency
• Liquidity predictive models• Credit scoring• Product recommendation
![Page 8: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/8.jpg)
Backend challenge - The evolution
Dapp
Smart Contract
Ethereum
![Page 9: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/9.jpg)
Backend challenge - The evolution
![Page 10: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/10.jpg)
Backend challenge - The evolution
Dapps problem:
• Noncryptocurrency users• Problem with some integrations• Legal problem
![Page 11: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/11.jpg)
Backend challenge - The evolution
![Page 12: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/12.jpg)
Backend challenge - The evolution
![Page 13: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/13.jpg)
Fecha
Backend challenge - Why microservices?
• Migration of Dapp easier
• Easy to scale
• Polyglot Database and Languages
![Page 14: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/14.jpg)
Why don't use exclusively blockchainwith a database?
![Page 15: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/15.jpg)
1. Spring CloudNetflix and Kubernetes• Easy to learn.• Nice integrations• Spring 5 reactive
2. Docker• Most adopted vendor technology for containers• Well supported
3. Kubernetes• Multi-cloud provider and on-premises data centers• Self-repair and health check capabilities• Auto-scale
Backend challenge - Microservice Architecture Stack
![Page 16: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/16.jpg)
Backend challenge - Microservice Architecture Stack
![Page 17: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/17.jpg)
Backend challenge - Deployment
![Page 18: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/18.jpg)
PFM values generation from user data. Apache Spark + Cassandra
Forecast prediction and regeneration of this modelsApache Spark + Cassandra
Product recommendations based on the economic profile of the user and his real needs. Apache Spark + Cassandra + Neo4j
Credit scoring calculationApache Spark + Cassandra
Big Data Integration - Tasks
![Page 19: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/19.jpg)
• Tasks are hard, needs:• Time• Resources
• Not Real Time is needed.
• Event Driven Architecture.
Big Data Integration - Events
![Page 20: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/20.jpg)
Big Data Integration - How?
![Page 21: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/21.jpg)
Big Data Integration - RabbitMQ vs Kafka
KafkaRabbitMq• RabbitMQ is designed as a general
purpose message broker
• Support existing protocols like AMQP, STOMP, MQTT.
• Finer-grained consistency control/guarantees on a peer-message.
• Complex routing.
• Apache Kafka is designed for high volume publish-subscribe messages and streams, meant to be durable, fast, and scalable.
• Event Sourcing
• Your application needs access to stream history.
• No complex routing.
https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka
![Page 22: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/22.jpg)
• Deployed in Kubernetes.
• Only accessible by NodeJS API.
• All keys are stored in secrets vaults.
• Used for:• Tokenization• Transactions of users
Blockchain Integration - Private Ethereum
![Page 23: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/23.jpg)
Blockchain Integration - Private Ethereum
![Page 24: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/24.jpg)
Blockchain Integration - Ethereum Main Net
![Page 25: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/25.jpg)
Blockchain Integration - Ethereum Main Net
![Page 26: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/26.jpg)
Blockchain Integration - Ethereum Main Net
• We are the owner of the wallets
• We use Infura to connect blockchain
• Used for:• Payment• Transfers
![Page 27: Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017](https://reader036.fdocuments.net/reader036/viewer/2022062600/5aad0a507f8b9a003b8b4855/html5/thumbnails/27.jpg)
@ganchix
@2getherbank
@bigeeksoftwa
re
Thanks!
ICO is coming!