Optimized your sql server operation using big data echo system
-
Upload
tzahi-hakikat -
Category
Technology
-
view
128 -
download
0
Transcript of Optimized your sql server operation using big data echo system
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
1
Optimized your sql server operation using big data echo system
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
2
Agenda
• Introduction
• Big data– The bad the good & the ugly
• Open source projects
• Puppet • Grafana• Logstash, ElasticSearch and Kibana
• Summary
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
3
Agoda.com
• Agoda.com is one of the world`s leading online hotel booking platforms.
• We work with hundreds of thousands of hotels worldwide to offer discounted hotel rates to customers.
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
4
Agoda Production environment
• 600 instances
• Support 7 data engines • Sql server • Cassandra• Verica• Cubes• Elastic search• Spark • Hadoop
• 4 data centers
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
5
Agoda data team
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
6
Data Operation Roles & Responsibilities
• Installations & upgrade • Maintenance
• Monitoring - active , proactive
• Performance Tuning
• Deploy database changes
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
7
Stairway (pyramid) to heaven
Hardware
Operating system
Data engine
Application
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
8
The bad the good & the ugly
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
9
Puppet
• Puppet is an open source data center automation and configuration management framework.
• Scalable to thousands of nodes
• Web-based dashboard for reporting
• Hundreds of community developed modules
• puppetlabs/sqlserver
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
10
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
11
Puppet – use cases
• Install
• Upgrade
• Migration –Storage , server
• Deployment lifecycle – safe & fast
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
12
No upload, No cry
Establish a baseline
Identify bottlenecks
OptimizeMeasure performance
Repeat (if desired)
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
13
Grafana
• Grafana provides a powerful and elegant way to create, explore, and share dashboards,
• Open source
• Real time
• Feature Highlights • Annotations• Template Variables• Rich Graphing
• play.grafana.org
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
14
Grafana architecture
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
15
Collect Source
• Performance monitor• Query • XE • Error logs • JVM
MethodUnix – collectDWindows - Graphite-PowerShell
Architecture Policy based Central collector vs local collector
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
16
Cluster management
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
17
Scale using replication
Master server
DC 1 DC 2 DC 3 DC 4
20xfront end
2xRPL 2xRPL 2xRPL
2xRPL
20x front end
12x front end
12x front end
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
18
Monitor Replication
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
19
Replication – performance monitor counters Agent Performance object Counter Description
Log Reader Agent SQL Server: Replication Logreader
Logreader: Delivered Cmds/sec
The number of commands per second delivered to the Distributor.
Log Reader Agent SQL Server: Replication Logreader
Logreader: Delivered Trans/sec
The number of transactions per second delivered to the Distributor.
Log Reader Agent SQL Server: Replication Logreader
Logreader: Delivery Latency The current amount of time, in milliseconds, elapsed from when transactions are applied at the Publisher to when they are delivered to the Distributor.
Distribution Agent SQL Server: Replication Dist. Dist: Delivered Cmds/sec The number of commands per second delivered to the Subscriber.
Distribution Agent SQL Server: Replication Dist. Dist: Delivered Trans/sec The number of transactions per second delivered to the Subscriber.
Distribution Agent SQL Server: Replication Dist. Dist: Delivery Latency The current amount of time, in milliseconds, elapsed from when transactions are delivered to the Distributor to when they are applied at the Subscriber.
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
20
Logstash, ElasticSearch & KibanaLogstash
• Centralize data processing of all types• Normalize varying schema and formats• Quickly extend to custom log formats
• ElasticSearch
• Search server based on Lucene.• Scaleable
• Kibana
• Flexible analytics and visualization platform• Real-time summary and charting of streaming data
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
21
Logstash, ElasticSearch & Kibana
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
22
Elastic Search – Use cases
• Application logs
• Data engine logs
• OS logs
• Hardware logs - network , storage
• Free text search
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
23
Kapunkap
All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved.AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd.Agoda is part of priceline.com (NASDAQ:PCLN)
24
We are looking for new talents!
• Are you ready to work in a fast paced environment, with opportunity for
career progression?
• Do you want to be part of a multicultural team, with colleagues from all
around the world?
• Do you want to utilize and improve your foreign language skills further?
• Are you ready to take up challenges and give prompt solutions for
customers with their requests?