NoSQL and Cloud Services - Philip Balinow, Comfo
26
-
Upload
beitconference -
Category
Presentations & Public Speaking
-
view
75 -
download
0
Transcript of NoSQL and Cloud Services - Philip Balinow, Comfo
SOME DEFINITIONS
def. CLOUD COMPUTING
def. BIG DATA
SOME DEFINITIONS
def. CLOUD COMPUTING - distributed computing over a network- the ability to run a program or application on many connected computers at the same time.
def. BIG DATA- data sets so large and complex that it becomes difficult to process using traditional data processing applications
SOME DEFINITIONS, CONTD.
Q: No, seriously. What is the Cloud?
A: Well, there are three types
Infrastructure-as-a-Service
Platform-as-a-Service
Software-as-a-Service
THE CLOUD – PROS AND CONS
WHY CLOUD COMPUTING IS SO GREAT
Better hardware utilization
Economy of scale
Usage-based pricing
In-built resilience (here be monsters)
No front-up costs
No long-term contracts
Retail
Finance
E-Commerce
Telecommunication
B2B
Publishing/Media
Government & NGO
Automotive
Travel
KOMFO'S CLIENTS
KOMFO PLATFORM WORKFLOW OVERVIEW
EXTERNAL PROVIDERS
OUR TECHNOLOGY
PHP, Python, C
MySQL
MongoDB, Elasticsearch
Javascript (node.js)
Ruby
Freedom to use any tool fit for the job
KOMFO MAIN CHALLENGES
Continuously changing workload
Fast feature changes
BigData
(Predictive) Analytics
Security
Availability
HOW TO USE THE CLOUD
Automation
Horizontal scaling
Break tasks into many small sub-tasks
Synchronize all the workers
Write for eventual consistency
HOW TO USE THE CLOUD
Beware! There are traps*
*Minimally shared with fair weighting. Allows burst when idle resources are available. In rare cases, resources may be throttled back under heavy host contention.
–**Disk I/O is shared across the host.
–***A vCPU corresponds to a physical CPU thread.
–
HOW TO USE THE CLOUD
There are more traps*
Coordinate tasks – good messaging
system (AMQP, DB, MemCached)
Asynchronous task execution (see
above, also API Callback hooks)
Implement transactions in software
CLOUD ARCHITECTURE
Messaging
Ensure communication between dynamic
number of nodes
Message-oriented middleware
Exactly-once delivery
At-least-once delivery
Transaction-based delivery
Timeout-based delivery
MIX & MATCH
Crunch numbers in the cloud
Application servers
Slow running tasks
Temporary services
Test servers
Automation – automatic deployment of multi-
tiered environments
MIX & MATCH, CONTD.
Traditional servers for:
Incompatible apps (single-threaded, memory,
disk intensive, specialized hardware) do not
work well in cloud environments
Database servers are best kept on dedicated
machines
OK, so we have an (endlessly) scalable cloud app now.
What are we forgetting?
DATABASES, NOSQL
def. NoSQL a mechanism for storage and retrieval of data
that is modeled in means other than the tabular relations used in relational databases.
DATABASES
Postgres, Hadoop, MongoDB, Cassandra, Riak
In-memory dataset for faster operation
No predefined structure
Integrated sharding, load-balancing and failover
Versatility - can be used for anything from data
storage to real-time messaging to search indexes
DATABASES
Use the best tool for the job depending on the task
NoSQL Advantages
Some sources generate a lot of data
Complex interconnections, cyclical
dependencies
Aggregations must be performed on both new
and old data
Structure of foreign sources may change on
short notice
DATABASES
Use the best tool for the job depending on the task
NoSQL Disadvantages (Classic SQL advantages)
Not ACID compliant
No transactions
No relations between data
Lack of structure means aggregations are slow
DATABASES, LONG TERM STRATEGY
Data quickly becomes irrelevant
Archive it, but keep it accessible
Online Data Warehouse solutions
Amazon Redshift
Keep Everything
Terabytes for pennies
Summary
The cloud rocks, mmkay?
Questions?
