Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data...

47
1 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware Lenovo Big Data Validated Design for Cloudera Enterprise and VMware Dan Kangas (Lenovo) Weixu Yang (Lenovo) Ajay Dholakia (Lenovo) Dwai Lahiri (Cloudera) Last update: 20 June 2017 Version 1.2 Configuration Reference Number BDACLDRXX63 Deployment considerations for high-performance, cost- effective and scalable solutions Contains detailed bill of material for different servers and associated networking Describes the reference architecture for Cloudera Enterprise, powered by Apache Hadoop and Apache Spark Solution based on the powerful, versatile Lenovo System x3650 M5 server, bare- metal and virtualized

Transcript of Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data...

Page 1: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

1 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Dan Kangas (Lenovo)

Weixu Yang (Lenovo)

Ajay Dholakia (Lenovo)

Dwai Lahiri (Cloudera)

Last update: 20 June 2017 Version 1.2 Configuration Reference Number BDACLDRXX63

Deployment considerations for high-performance, cost-effective and scalable solutions

Contains detailed bill of material for different servers and associated networking

Describes the reference architecture for Cloudera Enterprise, powered by Apache Hadoop and Apache Spark

Solution based on the powerful, versatile Lenovo System x3650 M5 server, bare-metal and virtualized

Page 2: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

2 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Table of Contents

1 Introduction ............................................................................................... 4

2 Business problem and business value ................................................... 5

2.1 Business problem .................................................................................................... 5

2.2 Business value ......................................................................................................... 5

3 Requirements ............................................................................................ 6

3.1 Functional requirements........................................................................................... 6

3.2 Non-functional requirements .................................................................................... 6

4 Architectural overview ............................................................................. 7

4.1 Cloudera Enterprise ................................................................................................. 7

4.2 VMware vSphere ...................................................................................................... 7

5 Component model .................................................................................... 9

5.1 Apache Spark in CDH 5.8 ...................................................................................... 12

6 Operational model .................................................................................. 14

6.1 Hardware description ............................................................................................. 14 1.1.1 Lenovo System x3650 M5 Server ............................................................................................. 14 1.1.2 Lenovo System x3550 M5 Server ............................................................................................. 15 1.1.3 Lenovo RackSwitch G8052 ....................................................................................................... 15 1.1.4 Lenovo RackSwitch G8272 ....................................................................................................... 16

6.2 Cluster nodes ......................................................................................................... 17 1.1.5 Data nodes ................................................................................................................................ 17 6.2.1 Master nodes ............................................................................................................................. 19

6.3 Systems management ........................................................................................... 22

6.4 Networking ............................................................................................................. 23 6.4.1 Data network .............................................................................................................................. 23 6.4.2 Hardware management network ............................................................................................... 24 6.4.3 Multi-rack network...................................................................................................................... 24

6.5 Predefined cluster configurations ........................................................................... 25

7 Deployment considerations ................................................................... 29

Page 3: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

3 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

7.1 Increasing cluster performance .............................................................................. 29

7.2 Designing for lower cost ......................................................................................... 29

7.3 Designing for high ingest rates ............................................................................... 29

7.4 Designing for in-memory processing with Apache Spark ....................................... 30

7.5 Designing for Hadoop in a Virtualized Environment ............................................... 31 7.5.1 VMware vSphere Design ........................................................................................................... 31 7.5.2 Cloudera Software Stack Configuration ..................................................................................... 32 7.5.3 Virtualized Configuration Summary ........................................................................................... 34

7.6 Estimating disk space ............................................................................................ 36

7.7 Scaling considerations ........................................................................................... 36

7.8 High availability considerations .............................................................................. 37 7.8.1 Networking considerations ........................................................................................................ 37 7.8.2 Hardware availability considerations ......................................................................................... 37 7.8.3 Storage availability ..................................................................................................................... 38 7.8.4 Software availability considerations ........................................................................................... 38

7.9 Migration considerations ........................................................................................ 38

8 Appendix: Bill of Materials ..................................................................... 39

8.1 Master node ........................................................................................................... 39

8.2 Data node .............................................................................................................. 40

8.3 Management/Administration network switch .......................................................... 41

8.4 Data network switch ............................................................................................... 41

8.5 Rack ....................................................................................................................... 41

8.6 Cables .................................................................................................................... 42

9 Acknowledgements ................................................................................ 43

Resources ..................................................................................................... 44

Document history ......................................................................................... 46

Page 4: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

4 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

1 Introduction This document describes the reference architecture for Cloudera Enterprise on bare-metal and on VMware vSphere with locally attached storage. It provides a predefined and optimized hardware infrastructure for the Cloudera Enterprise, a distribution of Apache Hadoop and Apache Spark with enterprise-ready capabilities from Cloudera. This reference architecture provides the planning, design considerations, and best practices for implementing Cloudera Enterprise with Lenovo products.

Lenovo and Cloudera worked together on this document, and the reference architecture that is described herein was validated by Lenovo and Cloudera.

Cloudera brings the power of Hadoop to the enterprise. Hadoop is an open source software framework that is used to reliably manage large volumes of structured and unstructured data. Cloudera expands and enhances this technology to withstand the demands of your enterprise, adding management, security, governance, and analytics features. The result is that you get a more enterprise ready solution for complex, large-scale analytics.

VMware vSphere brings virtualization to Hadoop with many benefits that cannot be obtained on physical infrastructure or in the cloud. Virtualization simplifies the management of your big data infrastructure, enables faster time to results and makes it more cost effective. It is a proven software technology that makes it possible to run multiple operating systems and applications on the same server at the same time. Virtualization can increase IT agility, flexibility, and scalability while creating significant cost savings. Workloads get deployed faster, performance and availability increases and operations become automated, resulting in IT that's simpler to manage and less costly to own and operate.

This reference architecture's intended audience is IT professionals, technical architects, sales engineers, and consultants to assist in planning, designing, and implementing the big data solution with Lenovo hardware. It is assumed that you are familiar with Hadoop components and capabilities. For more information about Hadoop, see “Resources” on page 44.

Page 5: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

5 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

2 Business problem and business value This section describes the business problem that is associated with big data environments and the value that is offered by the Cloudera solution that uses Lenovo hardware.

2.1 Business problem By 2012, the world generated 2.5 million TB of data, daily, a level that is expected to increase to 40 million TB by 2020. In all, 90% of the data in the world today was created in the last two years alone. This data comes from everywhere, including sensors that are used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone global positioning system (GPS) signals. This data is big data.

Big data spans the following dimensions: ● Volume: Big data comes in one size: large – in size, quantity and/or scale. Enterprises are awash with

data, easily amassing terabytes and even petabytes of information. ● Velocity: Often time-sensitive, big data must be used as it is streaming into the enterprise to maximize

its value to the business. ● Variety: Big data extends beyond structured data, including unstructured data of all varieties, such as

text, audio, video, click streams, and log files.

Big data is more than a challenge; it is an opportunity to find insight into new and emerging types of data to make your business more agile. Big data also is an opportunity to answer questions that, in the past, were beyond reach. Until now, there was no effective way to harvest this opportunity. Today, Cloudera uses the latest big data technologies such as the in-memory processing capabilities of Spark in addition to the standard MapReduce scale-out capabilities of Hadoop, to open the door to a world of possibilities.

2.2 Business value Hadoop is an open source software framework that is used to reliably manage and analyze large volumes of structured and unstructured data. Cloudera enhances this technology to withstand the demands of your enterprise, adding management, security, governance, and analytics features. The result is that you get an enterprise-ready solution for complex, large-scale analytics.

How can businesses process tremendous amounts of raw data in an efficient and timely manner to gain actionable insights? Cloudera allows organizations to run large-scale, distributed analytics jobs on clusters of cost-effective server hardware. This infrastructure can be used to tackle large data sets by breaking up the data into “chunks” and coordinating data processing across a massively parallel environment. After the raw data is stored across the nodes of a distributed cluster, queries and analysis of the data can be handled efficiently, with dynamic interpretation of the data formatted at read time. The bottom line: Businesses can finally get their arms around massive amounts of untapped data and mine that data for valuable insights in a more efficient, optimized, and scalable way.

Cloudera that is deployed on Lenovo System x servers with Lenovo networking components provides superior performance, reliability, and scalability. The reference architecture supports entry through high-end configurations and the ability to easily scale as the use of big data grows. A choice of infrastructure components provides flexibility in meeting varying big data analytics requirements.

Page 6: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

6 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

3 Requirements The functional and non-functional requirements for this reference architecture are desribed in this section.

3.1 Functional requirements A big data solution supports the following key functional requirements:

● Ability to handle various workloads, including batch and real-time analytics ● Industry-standard interfaces so that applications can work with Cloudera ● Ability to handle large volumes of data of various data types ● Various client interfaces

3.2 Non-functional requirements Customers require their big data solution to be easy, dependable, and fast. The following non-functional requirements are key:

● Easy:

o Ease of development o Easy management at scale o Advanced job management o Multi-tenancy o Easy to access data by various user types

● Dependable:

o Data protection with snapshot and mirroring o Automated self-healing o Insight into software/hardware health and issues o High availability (HA) and business continuity

● Fast:

o Superior performance o Scalability

● Secure and governed:

o Strong authentication and authorization o Kerberos support o Data confidentiality and integrity

Page 7: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

7 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

4 Architectural overview 4.1 Cloudera Enterprise Figure 1 shows the main features of the Cloudera reference architecture that uses Lenovo hardware. Users can log into the Cloudera client from outside the firewall by using Secure Shell (SSH) on port 22 to access the Cloudera solution from the corporate network. Cloudera provides several interfaces that allow administrators and users to perform administration and data functions, depending on their roles and access level. Hadoop application programming interfaces (APIs) can be used to access data. Cloudera APIs can be used for cluster management and monitoring. Cloudera data services, management services, and other services run on the nodes in cluster. Storage is a component of each data node in the cluster. Data can be incorporated into Cloudera Enterprise storage through the Hadoop APIs or network file system (NFS), depending on the needs of the customer.

A database is required to store the data for Cloudera manager, hive metastore, and other services. Cloudera provides an embedded database for test or proof of concept (POC) environments and an external database is required for a supportable production environment.

Figure 1. Cloudera architecture overview

4.2 VMware vSphere When Hadoop is virtualized, all of the components of Hadoop, including the NameNode, ResourceManager, DataNode, and NodeManager, are running within purpose-built Virtual Machines (VMs) rather than on the native OS of the physical machine. However, the Hadoop services or roles of the Cloudera software stack are installed with Cloudera Manager exactly the same way as with the physical machines. With a virtualization infrastructure, two or more VMs can be run on the same physical host server to improve cluster usage efficiency and flexibility.

Page 8: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

8 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

The VMware-based infrastructure with direct attached storage for HDFS is used to maintain the storage-to-CPU locality on a physical node. VMs are configured for one-to-one mapping of a physical disk to a vSphere VMFS virtual disk - see Figure 2 below.

Figure 2. One-to-one mapping of local storage

Page 9: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

9 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

5 Component model Cloudera Enterprise provides features and capabilities that meet the functional and nonfunctional requirements of customers. It supports mission-critical and real-time big data analytics across different industries, such as financial services, retail, media, healthcare, manufacturing, telecommunications, government organizations, and leading Fortune 100 and Web 2.0 companies.

Cloudera Enterprise is the world’s most complete, tested, and popular distribution of Apache Hadoop and related projects. All of the packaging and integration work is done for you, and the entire solution is thoroughly tested and fully documented. By taking the guesswork out of building out your Hadoop deployment, Cloudera Enterprise gives you a streamlined path to success in solving real business problems with big data.

The Cloudera platform for big data can be used for various use cases from batch applications that use MapReduce or Spark with data sources, such as click streams, to real-time applications that use sensor data.

Figure 3 shows the Cloudera Enterprise key capabilities that meet the functional requirements of customers.

Figure 3. Cloudera Enterprise key capabilities

Page 10: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

10 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Cloudera Enterprise solution contains the following components:

● Analytic SQL: Apache Impala (incubating)

Impala is the industry’s leading massively parallel processing (MPP) SQL query engine that runs natively in Hadoop. Apache-licensed, open source Impala project combines modern, scalable parallel database technology with the power of Hadoop, enabling users to directly query data stored in HDFS and Apache HBase without requiring data movement or transformation. Impala is designed from the ground up as part of the Hadoop system and shares the same flexible file and data formats, metadata, security, and resource management frameworks that are used by MapReduce, Apache Hive, Apache Pig, and other components of the Hadoop stack.

● Search Engine: Cloudera Search

Cloudera Search is Apache Solr that is integrated with Cloudera Enterprise, including Apache Lucene, Apache SolrCloud, Apache Flume, Apache Tika, and Hadoop. Cloudera Search also includes valuable integrations that make searching more scalable, easy to use, and optimized for near-real-time and batch-oriented indexing. These integrations include Cloudera Morphlines, which is a customizable transformation chain that simplifies loading any type of data into Cloudera Search.

• NoSQL - HBase

A scalable, distributed column-oriented datastore. HBase provides real-time read/write random access to very large datasets hosted on HDFS.

• Stream Processing: Apache Spark

Apache Spark is an open source, parallel data processing framework that complements Hadoop to make it easy to develop fast, unified big data applications that combine batch, streaming, and interactive analytics on all your data. Cloudera offers commercial support for Spark with Cloudera Enterprise. Spark is 10 – 100 times faster than MapReduce which delivers faster time to insight, allows inclusion of more data, and results in better business decisions and user outcomes.

● Machine Learning: Spark MLlib

MLlib is the API that implements common machine learning algorithms. MLlib is usable in Java, Scala, Python and R. Leveraging Spark’s excellence in iterative computation, MLlib runs very fast, high-quality algorithms.

● Cloudera Manager

Cloudera Manager is the industry’s first and most sophisticated management application for Hadoop and the enterprise data hub. Cloudera Manager sets the standard for enterprise deployment by delivering granular visibility into and control over every part of the data hub, which empowers operators to improve performance, enhance quality of service, increase compliance, and reduce administrative costs. Cloudera Manager makes administration of your enterprise data hub simple and straightforward, at any scale. With Cloudera Manager, you can easily deploy and centrally operate the complete big data stack.

Cloudera Manager automates the installation process, which reduces deployment time from weeks to minutes; gives you a cluster-wide, real-time view of nodes and services running; provides a single,

Page 11: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

11 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

central console to enact configuration changes across your cluster; and incorporates a full range of reporting and diagnostic tools to help you optimize performance and utilization.

• Cloudera Manager Metrics

Cloudera Manager monitors a number of performance metrics for services and role instances that are running on your clusters. These metrics are monitored against configurable thresholds and can be used to indicate whether a host is functioning as expected. You can view these metrics in the Cloudera Manager Admin Console, which displays metrics about your jobs (such as the number of currently running jobs and their CPU or memory usage), Hadoop services (such as the average HDFS I/O latency and number of concurrent jobs), your clusters (such as average CPU load across all your hosts) and so on.

• Cloudera Manager Backup And Disaster Recovery (BDR)

Cloudera Manager provides an integrated, easy-to-use management solution for enabling data protection in the Hadoop platform. Cloudera Manager provides rich functionality that is aimed towards replicating data that is stored in HDFS and accessed through Hive across data centers for disaster recovery scenarios. When critical data is stored on HDFS, Cloudera Manager provides the necessary capabilities to ensure that the data is available at all times, even in the face of the complete shutdown of a data center. Cloudera Manager also provides the ability to schedule, save, and (if needed) restore snapshots of HDFS directories and HBase tables.

• Cloudera Manager API

The Cloudera Manager API provides configuration and service lifecycle management, service health information and metrics, and allows you to configure Cloudera Manager. The API is served on the same host and port as the Cloudera Manager Admin Console, and does not require an extra process or extra configuration. The API supports HTTP Basic Authentication, accepting the same users and credentials as the Cloudera Manager Admin Console.

• Cloudera Navigator

A fully integrated data management and security tool for the Hadoop platform. Cloudera Navigator provides three categories of functionality:

o Auditing data access and verifying access privileges. Cloudera Navigator allows administrators to configure, collect, and view audit events, and generate reports that list the HDFS access permissions granted to groups. Cloudera Navigator tracks access permissions and actual accesses to all entities in HDFS, Hive, HBase, Hue, Impala, Sentry, and Solr.

o Searching metadata and visualizing lineage. Metadata management features allow DBAs, data modelers, business analysts, and data scientists to search for, amend the properties of, and tag data entities. Cloudera Navigator supports tracking the lineage of HDFS files, datasets, and directories, Hive tables and columns, MapReduce and YARN jobs, Hive queries, Impala queries, Pig scripts, Oozie workflows, Spark jobs, and Sqoop jobs.

o Securing data and simplifying storage and management of encryption keys. Data encryption and key management provide protection against potential threats by malicious actors on the network or

Page 12: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

12 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

in the datacenter. It is also a requirement for meeting key compliance initiatives and ensuring the integrity of enterprise data.

• Cloudera Kafka

Cloudera Distribution of Apache Kafka is a distributed commit log service. Kafka functions much like a publish/subscribe messaging system, but with better throughput, built-in partitioning, replication, and fault tolerance. Kafka is a good solution for large scale message processing applications. It is often used in tandem with Apache Hadoop, Apache Storm, and Spark Streaming.

For more information, see this website: cloudera.com/content/cloudera/en/products-and-services/product-comparison.html

The Cloudera solution is operating system independent. Cloudera supports many Linux® operating systems, including Red Hat Linux and SUSE Linux. For more information about the versions of supported operating systems, see this website:

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_cm_requirements.html.

5.1 Apache Spark in CDH 5.8 Spark has recently become very popular and is being adopted as a preferred framework for a variety of big data use-cases ranging from batch applications that use MapReduce or Spark with data sources such as click streams, to real-time applications that use sensor data.

The Spark stack is shown in Figure 4. As depicted, the foundational component is the Spark Core. Spark is written in the Scala programming language and offers simple APIs in Python, Java, Scala and SQL.

Figure 4. The Spark stack

In additional to the Spark Core, the framework allows extensions in the form of libraries. Most common extensions are Spark MLlib for machine learning, Spark SQL for queries on structured data, Spark Streaming for real-time stream-processing, and Spark GraphX for handling graph databases. Other extensions are also available. Cloudera doesn’t currently support GraphX or SparkR. There are also caveats for Spark SQL support - please refer to Cloudera's Spark documentation.

Page 13: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

13 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

The Spark architecture shown in Figure 4 enables a single framework to be used for multiple projects. Typical big data usage scenarios to date have deployed the Hadoop stack for batch processing separately from another framework for stream processing, and yet another one for advanced analytics such as machine learning. Apache Spark combines these frameworks in a common architecture, thereby allowing easier management of the big data code stack and also enabling reuse of a common data repository.

The Spark stack shown in Figure 4 can run in a variety of environments. It can run alongside the Hadoop stack, leveraging Hadoop YARN for cluster management. Spark applications can run in a distributed mode on a cluster using a master/slave architecture that uses a central coordinator called “driver” and potentially large number of “worker” processes that execute individual tasks in a Spark job. The Spark executor processes also provide reliable in-memory storage of data distributed across the various nodes in a cluster. The components of a distributed Spark application are shown in Figure 5.

Figure 5. Distributed Spark application component model

A key distinguishing feature of Spark is the data model, based on RDDs (Resilient Distributed Datasets). This model enables a compact and reusable organization of data-set that can reside in the main memory and can be accessed by multiple tasks. Iterative processing algorithms can benefit from this feature by not having to store and retrieve data-sets from disks between iterations of computation. These capabilities are what deliver the significant performance gains compared to MapReduce.

RDDs support two types of operations: Transformations and Actions. Transformations are operations that return a new RDD, while Actions return a result to the driver program. Spark groups operations together to reduce the number of passes taken over the data. This so-called lazy evaluation technique enables faster data processing. Spark also allows caching data in memory for persistence to enable multiple uses of the same data. This is another technique contributing to faster data processing.

Page 14: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

14 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

6 Operational model This section describes the operational model for the Cloudera reference architecture. To show the operational model for different sized customer environments, four different models are provided for supporting different amounts of data. Throughout the document, these models are referred to as starter rack, half rack, full rack, and multi-rack configuration sizes. The multi-rack is three times larger than the full rack.

A Cloudera deployment consists of cluster nodes, networking equipment, power distribution units, and racks. The predefined configurations can be implemented as is or modified based on specific customer requirements, such as lower cost, improved performance, and increased reliability. Key workload requirements, such as the data growth rate, sizes of datasets, and data ingest patterns help in determining the proper configuration for a specific deployment. A best practice when a Cloudera cluster infrastructure is designed is to conduct the proof of concept testing by using representative data and workloads to ensure that the proposed design works.

6.1 Hardware description This reference architecture uses Lenovo System x3650 M5 and x3550 M5 servers and Lenovo RackSwitch G8052 and G8272 top of rack switches.

1.1.1 Lenovo System x3650 M5 Server

The Lenovo System x3650 M5 server (as shown in Figure 6) is an enterprise class 2U two-socket versatile server that incorporates outstanding reliability, availability, and serviceability (RAS), security, and high-efficiency for business-critical applications and cloud deployments. It offers a flexible, scalable design and simple upgrade path to 26 2.5-inch hard disk drives (HDDs) or solid-state drives (SSDs), or 14 3.5-inch HDDs with doubled data transfer rate through 12 Gbps serial-attached SCSI (SAS) internal storage connectivity and up to 1.5 TB of TruDDR4 Memory. On-board it provides four standard embedded Gigabit Ethernet ports and two optional embedded 10 Gigabit Ethernet ports without occupying PCIe slots.

Figure 6. Lenovo System x3650 M5

Combined with the Intel® Xeon® processor E5-2600 v4 product family, the Lenovo x3650 M5 server offers an even higher density of workloads and performance that lowers the total cost of ownership (TCO). Its pay-as-you-grow flexible design and great expansion capabilities solidify dependability for any kind of workload with minimal downtime.

The x3650 M5 server provides internal storage density of up to 128 TB in a 2U form factor with its impressive array of workload-optimized storage configurations. It also offers easy management and saves floor space and power consumption for most demanding use cases by consolidating storage and server into one system.

Page 15: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

15 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

The reference architecture recommends the storage-rich System x3650 M5 model for the following reasons:

• Storage capacity: The nodes are storage-rich. Each of the 14 configured 3.5-inch drives has raw capacity up to 8 TB and each, providing for 112 TB of raw storage per node and over 1000 TB per rack.

• Performance: This hardware supports the latest Intel Xeon processors and TruDDR4 Memory. • Flexibility: Server hardware uses embedded storage, which results in simple scalability (by adding

nodes). • More PCIe slots: Up to 8 PCIe slots are available if rear disks are not used, and up to 2 PCIe slots if

both Rear 3.5-inch HDD Kit and Rear 2.5-inch HDD Kit are used. They can be used for network adapter redundancy and increased network throughput.

• Better power efficiency: Innovative power and thermal management provides energy savings. • Reliability: Lenovo is first in the industry in reliability and has exceptional uptime with reduced costs.

For more information, see the Lenovo System x3650 M5 Product Guide:

https://lenovopress.com/lp0068-lenovo-system-x3650-m5-e5-2600-v4

1.1.2 Lenovo System x3550 M5 Server

The Lenovo System x3550 M5 server (as shown in Figure 7) is a cost- and density-balanced 1U two-socket rack server. The x3550M5 features a new, innovative, energy-smart design with up to two Intel Xeon processors of the high-performance E5-2600 v4 product family processors a large capacity of faster, energy-efficient TruDDR4 Memory, up to twelve 12Gb/s SAS drives, and up to three PCI Express (PCIe) 3.0 I/O expansion slots in an impressive selection of sizes and types. The server’s improved feature set and exceptional performance is ideal for scalable cloud environments.

Figure 7: Lenovo System x3550 M5

For more information, see the Lenovo System x3550 M5 Product Guide: https://lenovopress.com/lp0067-lenovo-system-x3550-m5-e5-2600-v4

1.1.3 Lenovo RackSwitch G8052

The Lenovo System Networking RackSwitch G8052 (as shown in Figure 8) is an Ethernet switch that is designed for the data center and provides a simple network solution. The Lenovo RackSwitch G8052 offers up to 48 1 GbE ports and up to 4 10 GbE ports in a 1U footprint. The G8052 switch is always available for business-critical traffic by using redundant power supplies, fans, and numerous high-availability features.

Page 16: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

16 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Figure 8. Lenovo RackSwitch G8052

Lenovo RackSwitch G8052 has the following characteristics:

A total of 48 1 GbE RJ45 ports Four standard 10 GbE SFP+ ports Low 130W power rating and variable speed fans to reduce power consumption

For more information, see the Lenovo RackSwitch G8052 Product Guide: https://lenovopress.com/tips1270-lenovo-rackswitch-g8052

1.1.4 Lenovo RackSwitch G8272

Designed with top performance in mind, Lenovo RackSwitch G8272 is ideal for today’s big data, cloud, and optimized workloads. The G8272 switch offers up to 72 10 Gb SFP+ ports in a 1U form factor and is expandable with six 40 Gb QSFP+ ports. It is an enterprise-class and full-featured data center switch that delivers line-rate, high-bandwidth switching, filtering, and traffic queuing without delaying data. Large data center grade buffers keep traffic moving. Redundant power and fans and numerous HA features equip the switches for business-sensitive traffic.

The G8272 switch (as shown in Figure 9) is ideal for latency-sensitive applications. It supports Lenovo Virtual Fabric to help clients reduce the number of I/O adapters to a single dual-port 10 Gb adapter, which helps reduce cost and complexity. The G8272 switch supports the newest protocols, including Data Center Bridging/Converged Enhanced Ethernet (DCB/CEE) for support of FCoE and iSCSI and NAS.

Figure 9: Lenovo RackSwitch G8272

The enterprise-level Lenovo RackSwitch G8272 has the following characteristics:

48 x SFP+ 10GbE ports plus 6 x QSFP+ 40GbE ports Support up to 72 x 10Gb connections using break-out cables 1.44 Tbps non-blocking throughput with low latency (~ 600 ns) Up to 72 1Gb/10Gb SFP+ ports OpenFlow enabled allows for easily created user-controlled virtual networks

For more information, see the Lenovo RackSwitch G8272 Product Guide:

https://lenovopress.com/tips1267-lenovo-rackswitch-g8272

Page 17: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

17 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

6.2 Cluster nodes The Cloudera reference architecture is implemented on a set of nodes that make up a cluster. A Cloudera cluster consists of two types of nodes, data nodes and Master nodes. Data nodes use System x3650 M5 servers with locally attached storage and Master nodes use System x3550 M5 servers.

Data nodes run data (worker) services for storing and processing data.

Master nodes run the following types of services:

Management (control) services for coordinating and managing the cluster Miscellaneous services (optional) for file and web serving

1.1.5 Data nodes

Table 1 lists the recommended system components for data nodes.

Table 1. Data node configuration

Component Data node configuration

System System x3650 M5

Processor 2 x Intel Xeon processor E5-2680 v4 2.4GHz 14-core

Memory - base 512 GB: 16 x 32GB 2400MHz RDIMM

Disk (OS) 2 x 2.5” HDD or SSD

Disk (data) 4 TB drives: 14 x 4TB NL SATA 3.5 inch (56 TB Total) 6TB drives; 14 x 6TB NL SATA 3.5 inch (84 TB total) 8 TB drives: 12 x 8TB NL SATA 3.5 inch (96 TB Total)

HDD controller OS: ServeRAID M1215 SAS/SATA Controller

HDFS: N2215 SAS/SATA HBA

Hardware storage protection OS: RAID1

HDFS: None (JBOD). By default, Cloudera maintains a total of three copies of data stored within the cluster. The copies are distributed across data servers and racks for fault recovery.

Hardware management network adapter

Integrated 1GBaseT IMM interface)

Data network adapter Broadcom NetXtreme Dual Port 10GbE SFP+ Adapter

Page 18: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

18 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Figure 10: Data node disk assignment

The Intel® Xeon® processor E5-2680 v4 is recommended to provide sufficient performance. A minimum of 256 GB of memory is recommended for most MapReduce workloads with 512 GB or more recommended for HBase, Spark, and memory-intensive MapReduce workloads. Two sets of disks are used: one set of disks is for the operating system and the other set of disks is for data. For the operating system disks, RAID 1 mirroring should be used.

Each data node in the reference architecture has internal directly attached storage. External storage is not used in this reference architecture. Available data space assumes the use of Hadoop replication with three copies of the data, and 25% capacity reserved for efficient file system operation and to allow time to increase capacity, if needed.

In situations where higher storage capacity is required, the main design approach is to increase the amount of data disk space per node. Using 8 TB drives instead of 4 TB drives increases the total per node data disk capacity from 56 TB to 112 TB, a 100% increase.

When increasing data disk capacity, there is a balance between performance and throughput. For some workloads, increasing the amount of user data that is stored per node can decrease disk parallelism and negatively affect performance. Increasing drive sizing also affects rebuilding and repopulating the replicas if there is a disk or node failure. Higher density disks or nodes results in longer rebuild times. Drives that are larger than 4 TB are not recommended based on the balance of capacity and performance. In this case, higher capacity can be achieved by increasing the number of nodes in the cluster.

For higher IO throughout, the data node can be configured with 24 2.5-inch SAS drives, which have less storage capacity but much higher IO throughout. In such cases, it is recommended to use 3 host bus adapters in the configuration.

For the HDD controller, just a bunch of disks (JBOD) is the best choice for a Cloudera cluster. It provides excellent performance and, when combined with the Hadoop default of 3x data replication, also provides significant protection against data loss. The use of RAID with data disks is discouraged because it reduces performance and the amount data that can be stored. Data nodes can be customized according to client needs.

Page 19: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

19 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

A minimum of three data nodes are required as Hadoop has three copies of data by default. Three data nodes should be used for test or POC environments only. A minimum of five data nodes are required for production environment to reduce risk of losing more than one node at a time

6.2.1 Master nodes The Master node is the nucleus of the Hadoop Distributed File System (HDFS) and supports several other key functions that are needed on a Cloudera cluster. The Master node hosts the following functions:

YARN ResourceManager: Manages and arbitrates resources among all the applications in the system. Hadoop NameNode: Controls the HDFS file system. The NameNode maintains the HDFS metadata, manages the directory tree of all files in the file system, and tracks the location of the file data within the cluster. The NameNode does not store the data of these files. ZooKeeper: Provides a distributed configuration service, a synchronization service, and a name registry for distributed systems. JournalNode: Collects, maintains, and synchronize updates from NameNode. HA ResourceManager: Standby ResourceManager that can be used to provide automated failover. HA NameNode: Standby NameNode that can be used to provide automated failover. Other hadoop component management services: HBase master, HiveServer2, and Spark History Server. Cloudera Manager: Optional management platform for Cloudera Enterprise.

Table 2 lists the recommended components for a Master node. Master nodes can be customized according to client needs.

Table 2. Master node configuration

Component Master node configuration

System System x3550 M5

Processor 2 x Intel Xeon processor E5-2650 v4 2.2 GHz 12-core

Memory - base 128 GB – 8 x 16 GB 2133 MHz RDIMM (minimum)

Disk (OS / local storage)

OS: 2x 2.5” HDD or SSD Data: 8 x 2TB 2.5” HDD

HDD controller ServeRAID M5210 SAS/SATA Controller

Hardware management network adapter

Integrated 1GBaseT IMM Interface

Data network adapter Broadcom NetXtreme Dual Port 10GbE SFP+ Adapter

The Intel® Xeon® processor E5-2650 v4 is recommended to provide sufficient performance. A minimum of 128 GB of memory is recommended. A choice of 240GB and 480GB SSD drives is suggested for the operating system and local storage.

The Master node uses 10 HDDs for the following storage pools:

Two drives are configured with RAID 1 for operating system Two drives are configured with RAID 1 for NameNode metastore

Page 20: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

20 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Four drives are configured with RAID 10 for database One drive is configured with RAID 0 for ZooKeeper One drive is configured with RAID 0 for Quorum Journal Node store

This design separates the data stores for different services and provides best performance. SSD and PCIe flash storage can be used to provide improved I/O performance for the database.

Figure 11: Cloudera Master node disk assignment

Because the Master node is responsible for many memory-intensive tasks, multiple Master nodes might be needed to split functions. For most implementations, the size of the Cloudera cluster is a good indicator of how many Master nodes are needed. Table 3 provides a high-level guideline for a cluster that provides HA NameNode and ResourceManager failover when configured with multiple Master nodes. For a medium size cluster with 20 - 100 data nodes, consider the use of a better configuration for the Master nodes, such as more memory and CPU core.

Table 3. Number of Master nodes

Number of Data Nodes

Number of Master nodes

Breakout of function

< 100 3

● Cloudera Manager, Journal Node, ZooKeeper

● ResourceManager, HA Hadoop NameNode, JournalNode, ZooKeeper

● HA ResourceManager, Hadoop NameNode, JournalNode, ZooKeeper

> 100 5

● Cloudera Manager, Journal Node, ZooKeeper

● ResourceManager, HA Hadoop NameNode, JournalNode, ZooKeeper

● HA ResourceManager, Hadoop NameNode, JournalNode, ZooKeeper

● JournalNode, ZooKeeper, other roles

Page 21: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

21 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

● JournalNode, ZooKeeper, other roles

Note: If there are multiple racks, Master nodes should be separated across racks.

If you plan to scale up the cluster significantly, it might be best to separate out each of these functions from the beginning, even if the starting configuration is smaller and requires fewer Master nodes. The number of Master nodes can be customized based on specific needs.

Table 4. Service Layout Matrix Node Master Node Master

Node Master Node Data

Nodes Service/ Roles

ZooKeeper ZooKeeper ZooKeeper ZooKeeper HDFS NN,QJN NN,QJN QJN Data Node YARN RM RM History Server Node

Manager Hive MetaStore,

WebHCat, HiveServer2

Management Oozie, CM, Management

Services

Cloudera Agent

Navigator Navigator, KMS

HUE HUE Search Solr Spark Runs on

YARN Impala statestore,

catalog impalad

HBASE HMaster HMaster HMaster Region Servers

Installing and managing the Cloudera Stack

The Hadoop ecosystem is complex and constantly changing. Cloudera makes it simple so enterprises can

focus on results. Cloudera Manager is the easiest way to administer Hadoop in any environment, with

advanced features like intelligent defaults and customizable automation. Combined with predictive

maintenance included in Cloudera’s Support Data Hub, Cloudera Enterprise keeps the business up and

running.

Reference Cloudera's latest Installation documentation for detailed instructions on Installation

Page 22: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

22 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

6.3 Systems management Systems management includes cluster system management and hardware management.

Cluster systems management uses Cloudera Manager and is adapted from the standard Hadoop distribution, which places the management services on separate servers than the data servers. Because the Master node hosts important and high-memory functions, it is important that it is a powerful and fast server. The recommended Master nodes can be customized according to client needs.

Hardware management uses the Lenovo XClarity™ Administrator, which is a centralized resource management solution that reduces complexity, speeds up response, and enhances the availability of Lenovo® server systems and solutions.

The Lenovo XClarity Administrator provides agent-free hardware management for Lenovo’s System x® rack servers and Flex System™ compute nodes and components, including the Chassis Management Module (CMM) and Flex System I/O modules.

Figure 12 shows the Lenovo XClarity Administrator interface in which Flex System components and rack servers are managed and are seen on the dashboard. Lenovo XClarity Administrator is a virtual appliance that is quickly imported into a virtualized environment server configuration.

Figure 12: XClarity Administrator interface

Page 23: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

23 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

In addition, xCAT provides a scalable distributed computing management and provisioning tool that provides a unified interface for hardware control, discovery, and operating system deployment. It can be used to facilitate or automate the management of cluster nodes. For more information about xCAT, see “Resources” on page 44.

6.4 Networking Regarding networking, the reference architecture specifies two networks: a data network and an administrative or management network. Figure 13 shows the networking configuration for Cloudera.

Figure 13: Cloudera network configuration

6.4.1 Data network The data network is a private cluster data interconnect among nodes that is used for data access, moving data across nodes within a cluster, and importing data into the Cloudera cluster. The Cloudera cluster typically connects to the customer’s corporate data network.

Two top of rack switches are required; one for out-of-band management and one for the data network that is used by Cloudera. At least one 1GbE switch is required for out-of-band management of the nodes. The data switch should be 10GbE, depending on workload requirements.

Page 24: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

24 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

The recommended 1 GbE switch is the Lenovo RackSwitch G8052. The 10 Gb Ethernet switch can provide extra I/O bandwidth for better performance. The recommended 10 GbE switch is the Lenovo System Networking RackSwitch™ G8272.

The two Broadcom 10 GbE ports of each node are link aggregated to the recommended G8272 rack switch for better performance and improved HA. The data network is configured to use a virtual local area network (VLAN).

NOTE: Cloudera does not support multi-homing on either worker or master nodes.

6.4.2 Hardware management network The hardware management network is a 1 GbE network that is used for out-of-band hardware management. Through the integrated management modules II (IMM2) within the System x3650 M5 server, out-of-band management enables the hardware-level management of cluster nodes, such as node deployment, basic input/output system (BIOS) configuration, status and power states.

Hadoop has no dependency on the IMM2. Based on customer requirements, the administration links and management links can be segregated onto separate VLANs or subnets. The administrative or management network is typically connected directly to the customer’s administrative network.

The reference architecture requires one 1 Gb Ethernet top-of-rack switch for the hardware management network. Administrators also can access all of the nodes in the cluster through the customer admin network, as shown in Figure 13. On the nodes, the management link connects to the dedicated IMM2 port on the integrated 1 GBaseT adapter.

6.4.3 Multi-rack network The data network in the predefined reference architecture configuration consists of a single network topology. Appropriate other Lenovo RackSwitch G8316 core switches per cluster is required for cross racks communication. In this case, the second Broadcom10 GbE port can be connected to the second Lenovo RackSwitch G8272. The over-subscription ratio for G8272 is 1:2.

Figure 14 shows how the network is configured when the Cloudera cluster is installed across more than one rack. The data network is connected across racks by two aggregated 40 GbE uplinks from each rack’s G8272 switch to a core G8316 switch.

Page 25: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

25 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Figure 14: Cloudera cross rack network configuration1

A 40GbE switch is recommended for interconnecting the data network across multiple racks. Lenovo System Networking RackSwitch G8316 is the recommended switch. A best practice is to have redundant core switches for each rack to avoid a single point of failure. Within each rack, the G8052 switch can optionally be configured to have two uplinks to the G8272 switch to allow propagation of the administrative or management VLAN across cluster racks through the G8316 core switch. For large clusters, the Lenovo System Networking RackSwitch G8332 is recommended because it provides a better cost value per 40 Gb port than the G8316. Many other cross rack network configurations are possible and might be required to meet the needs of specific deployments or to address clusters larger than three racks.

If the solution is initially implemented as a multi-rack solution, or if the system grows by adding racks, the nodes that provide management services are distributed across racks to maximize fault tolerance.

6.5 Predefined cluster configurations The intent of the predefined configurations is to ease initial sizing for customers and to show example starting points for four different-sized workloads. The starter rack configuration consists of three nodes and a pair of rack switches. The half rack configuration consists of nine nodes and a pair of rack switches. The full rack configuration (a rack fully populated) consists of 18 nodes and a pair of rack switches. The multi-rack contains a total of 57 nodes; in each rack there are 19 nodes, a Master node and a pair of switches.

1 4. To simplify the diagram, only one G8272 is drawn in the diagram, and in recommended configuration, two G8272 are configured for HA.

Page 26: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

26 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Table 5 lists four predefined configurations for the Cloudera reference architecture. The table also lists the amount of space for data and the number of nodes that each predefined configuration provides. Storage space is described in two ways: the total amount of raw storage space when 4 TB or 8 TB drives (raw storage) are used and the amount of space for the data that the customer has (available data space). Available data space assumes the use of Hadoop replication with three copies of the data and 25% capacity that is reserved for intermediate data (scratch storage). The estimates that are listed in Table 4 do not include extra space that is freed up by using compression because compression rates can vary widely based on file contents.

Table 5. Pre-defined configurations

Starter rack Half rack Full rack Multi-rack *

Raw storage (4TB) 168 TB 504 TB 1008 TB 3192 TB

Available data space (4TB) 42 TB 126 TB 252 TB 798 TB

Raw storage (8 TB) 336 TB 1008 TB 2016 TB 6384 TB

Available data space (8 TB) 84 TB 252 TB 504 TB 1592 TB

Number of Data Nodes 3 9 18 57

Number of Master nodes 3 3 3 3

Number of Racks 1 1 1 3

Number of 10 GbE cables 12 24 42 120

Number of 1 GbE cables 8 14 23 66

* Multi-rack < 100 nodes

Figure 15 shows an overview of the architecture in three different one-rack sized clusters without network redundancy: a starter rack, a half rack, and a full rack. Figure 16 shows a multi-rack-sized cluster without network redundancy.

Page 27: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

27 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Figure 15: Starter rack, half rack, and full rack Cloudera predefined configurations

Page 28: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

28 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Figure 16: Multi-rack Cloudera predefined configurations

Page 29: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

29 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

7 Deployment considerations This section describes other considerations for deploying the Cloudera solution.

7.1 Increasing cluster performance There are two approaches that can be used to increase cluster performance: increasing node memory and the use of a high-performance job scheduler and MapReduce framework. Often, improving performance comes at increased cost and you must consider the cost-to-benefit trade-offs of designing for higher performance.

In the Cloudera predefined configuration, node memory can be increased to 1025 GB by using 16 x 64GB RDIMMs. An even larger memory configuration can provide greater performance, depending on the workload.

7.2 Designing for lower cost There are two key modifications that can be made to lower the cost of a Cloudera reference architecture solution. When lower-cost options are considered, it is important to ensure that customers understand the potential lower performance implications of a lower-cost design. A lower-cost version of the Cloudera reference architecture can be achieved by using lower-cost node processors and lower-cost cluster data network infrastructure.

The node processors can be substituted with the Intel Xeon E5-2630 v4 2.2 GHz 10-core processor. This processor supports 1600 MHz and 1866 MHz RDIMMs, which can also lower the per-node cost of the solution.

The use of a lower-cost network infrastructure can significantly lower the cost of the solution, but can also have a substantial negative effect on intra-cluster data throughput and cluster ingest rates. To use a lower-cost network infrastructure, use the following substitutions to the predefined configuration: Within each cluster, substitute the Lenovo RackSwitch G8316 core switch with the Lenovo RackSwitch G8272.

7.3 Designing for high ingest rates Designing for high ingest rates is difficult. It is important to have a full characterization of the ingest patterns and volumes. The following questions provide guidance to key factors that affect the rates:

● On what days and at what times are the source systems available or not available for ingest? ● When a source system is available for ingest, what is the duration for which the system remains

available? ● Do other factors affect the day, time, and duration ingest constraints? ● When ingests occur, what is the average and maximum size of ingest that must be completed? ● What factors affect ingest size? ● What is the format of the source data (structured, semi-structured, or unstructured)? Are there any

data transformation or cleansing requirements that must be achieved during ingest?

To increase the data ingest rates, consider the following points:

● Ingest data with MapReduce job, which helps to distribute the I/O load to different nodes across the cluster.

● Ingest when cluster load is not high, if possible. ● Compressing data is a good option in many cases, which reduces the I/O load to disk and network. ● Filter and reduce data in earlier stage saves more costs.

Page 30: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

30 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

7.4 Designing for in-memory processing with Apache Spark Methods from the Lenovo Big Data Reference Architecture for Cloudera Enterprise apply for general Spark considerations as well; however, there are additional considerations. Conceptually, Spark is similar in nature to high performance computing.

It is important that memory capacity be carefully considered, as both the execution and storage of Spark should be able to reside fully in memory, to achieve maximum performance, however there continue to be performance benefits even when an application doesn’t fully fit within memory Disk access, for storage or caching, is very costly to Spark processing. The memory capacity considerations are highly dependent on the application. To get an estimate, load an RDD of a desired dataset, into cache, and evaluate the consumption. Generally, for workloads with high execution and storage requirements, capacity is primary consideration.

Additional considerations for memory configuration include the bandwidth and latency requirements. Applications with high transactional memory usage should focus on DIMM configurations that result in four DIMMs per channel using dual rank DIMMs. The following table provides ideal data node memory configurations for bandwidth/latency sensitive workloads.

Table 6. Memory configurations for data nodes Capacity DIMM Description Feature Quantity 128GB 16GB TruDDR4 Memory (2Rx4, 1.2V) PC4-19200 CL17 2400MHz LP

RDIMM ATCA 8

256GB 32GB TruDDR4 Memory (2Rx4, 1.2V) PC4-19200 CL17 2400MHz LP RDIMM

ATCB 8

384GB 32GB TruDDR4 Memory (2Rx4, 1.2V) PC4-19200 CL17 2400MHz LP RDIMM

ATCB 12

512GB 32GB TruDDR4 Memory (2Rx4, 1.2V) PC4-19200 CL17 2400MHz LP RDIMM

ATCB 16

768GB 64GB TruDDR4 Memory (4Rx4, 1.2V) PC4-19200 PC4 2400MHz LP LRDIMM

ATGG 12

1024GB 64GB TruDDR4 Memory (4Rx4, 1.2V) PC4-19200 PC4 2400MHz LP LRDIMM

ATGG 24

Similarly, processor selection may vary based on the level of desired level of parallelism for the workloads. For example, Apache recommends 2-3 tasks per CPU core. Large working sets of data can drive memory constraints, which can be alleviated through further increasing parallelism, resulting in smaller input sets per task. In this case, higher core counts can be beneficial. Naturally, the nature of the operations is considered, as they may be simple evaluations or complex algorithms.

Page 31: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

31 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

7.5 Designing for Hadoop in a Virtualized Environment This section of the reference architecture shows the important configuration items for the vSphere VMs; the Hadoop Virtualized Extensions (HVE) which maintain the storage, memory and CPU locality on each physical node; and the Cloudera software stack.

7.5.1 VMware vSphere Design

Virtual Network Switch

Standard vswitches may be employed, which need to be configured for each ESXi host in the cluster. A key configuration parameter to verify is the MTU size to ensure that the same MTU size being set at the physical switches, guest OS, ESXi VMNIC and the vswitch layers and that it's set to Jumbo Frames for highest performance - the is recommended for Hadoop environments.

Storage Group

Each provisioned disk is either mapped to one vSphere datastore (which in turn contains one VMDK or virtual disk) or mapped to one raw device mapping (RDM). Configure virtual disks in “independent persistent” mode for optimal performance. Eager Zeroed Thick virtual disks provide the best performance. And make sure to disable SIOC, and disable storage DRS.

vSphere Tuning Best Practices

Power Policy is an ESXi parameter. The balanced mode may be the best option. Evaluate your environment and choose accordingly. In some cases, performance might be more important than power optimization. Avoid memory and CPU over-commitment, and ensure Transparent Huge Pages setting is Disabled for the EXSi Hypervisor and Guest OS.

VMXNET3 Virtual Network Driver

This driver is supported in RHEL and CentOS with the installation of VMware tools. Verify MTU size for jumbo frames at the guest level as well as ESXi and switch level. Only VMXNET3 drivers at the Guest layer can leverage this. Similarly, other offload features can be leveraged only when using the VMXNET3 driver. Use regular platform tuning parameters, such as ring buffer size. However, RSS and RPS tuning must be specific to the VMXNET3 driver.

HBA Driver Type

Use the VMware PVSCSI storage adapter. This provides the best performance characteristics (reduced CPU utilization and increased throughput), and is optimal for I/O-intensive guests (as with Hadoop). Tune queue depth in the guest OS SCSI driver, as needed.

I/O Scheduler

The I/O scheduler used for the OS disks might need to be different if using VMDKS. Instead of using CFQ, use deadline or noop elevators (deadline should be the default with rhel OS). This varies and must be tested. Any performance gains must be quantified appropriately; for example, 1-2% improvement vs. 10-20% improvement.

Page 32: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

32 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Memory Tuning

Disable or minimize anonymous paging by setting vm.swappiness=0 or 1. When tuning memory from the default settings to a best performance condition, configure VMs so that one or more VMs fit within a NUMA node size associated with one CPU socket, as far as their collective memory goes. The goal is to not have a VM access memory across a NUMA boundary. Figure 17 below shows the virtualized cluster layout with 2 VMs per physical node and locally attached disks for each VM.

Figure 17. Example Virtualized Cluster Topology

7.5.2 Cloudera Software Stack Configuration Guidelines for installing the Cloudera stack on a virtualized platform are nearly identical to those for bare-metal, except for enabling the Hadoop Virtualized Extensions (HVE) functionality.

Enabling Hadoop Virtualization Extensions (HVE)

HVE is part of the Apache Project and adds physical node awareness to Hadoop and Spark for a virtualized environment. This enables HDFS to maintain all of the data block replicas across physical nodes as it does in the bare-metal environment. Refer to the Apache Project HVE descriptions and user guide at this link: (https://issues.apache.org/jira/browse/HADOOP-8468). Following are considerations for HVE:

1. Enable HVE when there is more than one Hadoop VM per physical node in virtualized environments.

2. Use the Node Group definition to group VMs that reside on the same physical node to enable HDFS to distribute block replication across physical nodes.

Page 33: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

33 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

3. HVE extensions can be used to create node groups to further specify locality and awareness for spreading HDFS block replicas across physical servers of a common model type, with a certain level of power supply redundancy or nodes from certain hardware purchase cycles, for example.

The following diagram illustrates the addition of a new level of abstraction (in red) called NodeGroups. The NodeGroups represent the physical hypervisor on which the nodes (VMs) reside.

Rx = Server Rack

NGx = Node Group

Nx = Physical Server Node

Figure 18. HVE Node groups All VMs under the same node group run on the same physical host. With awareness of the node group layer, HVE refines the following policies for Hadoop on virtualization: Replica Placement Policy • No duplicated replicas are on the same node or nodes under the same node group. • First replica is on the local node or local node group of the writer. • Second replica is on a remote rack of the first replica. • Third replica is on the same rack as the second replica. • The remaining replicas are located randomly across rack and node group for minimum restriction.

Page 34: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

34 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Replica Choosing Policy • The HDFS client obtains a list of replicas for a specific block sorted by distance, from nearest to farthest: local node, local node group, local rack, off rack. Balancer Policy • At the node level, the target and source for balancing follows this sequence: local node group, local rack, off rack. • At the block level, a replica block is not a good candidate for balancing between source and target node if another replica is on the target node or on the same node group of the target node.

HVE typically supports failure and locality topologies defined from the perspective of virtualization. However, you can use the new extensions to support other failure and locality changes, such as those relating to power supplies, arbitrary sets of physical servers, or collections of servers from the same hardware purchase cycle.

7.5.3 Virtualized Configuration Summary In this reference architecture the following configuration was used to create the Cloudera software stack running on the ESXi hypervisor. As a starting point and to maintain best locality for HDD, CPU and memory, 2 VMs per physical node is demonstrated in this reference architecture. Additional configurations such as 4VMs or more per node are possible to meet specific customer requirements.

Cluster Hardware Design

Figure 19. Cluster Design for Virtualized Cloudera on VMware ESXi

Component Configuration

System System x3650 M5

Processor 2 x Intel Xeon processor E5-2680 v4 2.4GHz 14-core

Memory - base 512 GB: 16 x 32GB 2400MHz RDIMM

Disk (OS) 2 x 2.5” SSD

Disk (data) 14 x 4TB NL SATA 3.5 inch, 4TB each (56 TB Total)

HDD controller OS: ServeRAID M1215 SAS/SATA Controller

HDFS: N2215 SAS/SATA HBA

Hardware storage protection OS: RAID1

HDFS: None (JBOD). By default, Cloudera maintains a total of three copies of data stored within the cluster. The copies are distributed across data servers and racks for fault recovery.

Network 10Gb Ethernet, 2x bonded interfaces

Hardware management network adapter

Integrated 1GBaseT IMM interface)

Page 35: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

35 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Component Configuration

Data network adapter Broadcom NetXtreme Dual Port 10GbE SFP+ Adapter

Cluster Software Stack

Figure 20. Virtualized Cloudera Software Stack:

Component Version

vSphere (ESXi + vCenter Server) 6.50

Guest Operating System Red Hat rhel7.2

Cloudera Hadoop Distribution CDH 5.11

Cloudera Manager CDH 5.11

Java Oracle 1.8.0

ESXi Hypervisor and Guest OS Configuration:

Below are the key configuration parameters used in this reference architecture. Many valid configurations exist and additional information can be obtained from the Cloudera and VMware whitepapers shown in the References section on page 44:

ESXi

• 2 VMs per physical node

• 8 physical nodes * 2 = 16 total VMs

Memory

• CPU to Memory locality: 1 VM per CPU • 6% of node physical Memory allocated for ESXi; remainder Memory allocated to VMs • Anonymous paging: vm.swappiness=0 or 1

Disks

• 8 Physical Disks per VM • 8 Disks per DataNode • VMware PVSCSI storage adapter used (all 4 virtual SCSI controllers used) for best I/O performance • Queue depth in guest OS SCSI driver: 4294967295 (default value) • Eager-zeroed thick VMDKs (on EXT4 filesystem in guest OS)

Network

• VMXNET3 network driver used with MTU=90000 for jumbo frames on guest OS and virtual switch • Enabled TCP segmentation offload (TSO) at the ESXi level (should be enabled by default). Only

VMXNET3 drivers at the Guest layer can leverage this.

Page 36: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

36 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

7.6 Estimating disk space When you are estimating disk space within a Cloudera Enterprise cluster, consider the following points:

For improved fault tolerance and performance, Cloudera Enterprise replicates data blocks across multiple cluster data nodes. By default, the file system maintains three replicas. Compression ratio is an important consideration in estimating disk space and can vary greatly based on file contents. If the customer’s data compression ratio is unavailable, assume a compression ratio of 2.5:1. To ensure efficient file system operation and to allow time to add more storage capacity to the cluster if necessary, reserve 25% of the total capacity of the cluster.

Assuming the default three replicas maintained by Cloudera Enterprise, the raw data disk space, and the required number of nodes can be estimated by using the following equations:

Total raw data disk space = (User data, uncompressed) * (4 / compression ratio)

Total required data nodes = (Total raw data disk space) / (Raw data disk per node)

You should also consider future growth requirements when estimating disk space.

Based on these sizing principals, Table 7 shows an example for a cluster that must store 500 TB of uncompressed user data. The example shows that the Cloudera cluster needs 800 TB of raw disk to support 500 TB of uncompressed data. The 800 TB is for data storage and does not include operating system disk space. A total of 15 nodes are required to support a deployment of this size.

Table 7. Example of storage sizing with 4TB drives

Description Value

Size of uncompressed user data 500 TB

Compression ratio 2.5:1

Size of compressed data 200 TB

Storage multiplication factor 4

Raw data disk space needed for Cloudera cluster 800 TB

Storage needed for Cloudera Hadoop 3x replication 600 TB

Reserved storage for headroom 200 TB

Raw data disk per node (with 4TB drives) 56 TB

Minimum number of nodes required (800/56) 15

7.7 Scaling considerations The Hadoop architecture is linearly scalable but it is important to note that some workloads might not scale completely linearly.

Page 37: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

37 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

When the capacity of the infrastructure is reached, the cluster can be scaled out by adding nodes. Typically, identically configured nodes are best to maintain the same ratio of storage and compute capabilities. A Cloudera cluster is scalable by adding System x3650 M5 data nodes, Master nodes, and network switches. As the capacity of racks is reached, new racks can be added to the cluster.

When a Cloudera reference architecture implementation is designed, future scale out should be a key consideration in the initial design. There are two key aspects to consider: networking and management. These aspects are critical to cluster operation and become more complex as the cluster infrastructure grows.

The cross rack networking configuration that is shown in Figure 14 provides robust network interconnection of racks within the cluster. As racks are added, the predefined networking topology remains balanced and symmetrical. If there are plans to scale the cluster beyond one rack, a best practice is to initially design the cluster with multiple racks (even if the initial number of nodes fit within one rack). Starting with multiple racks can enforce proper network topology and prevent future re-configuration and hardware changes. As racks are added over time, multiple G8316 switches might be required for greater scalability and balanced performance.

Also, as the number of nodes within the cluster increases, so do many of the tasks of managing the cluster, such as updating node firmware or operating systems. Building a cluster management framework as part of the initial design and proactively considering the challenges of managing a large cluster pays off significantly in the long run.

Proactive planning for future scale out and the development of cluster management framework as a part of initial cluster design provides a foundation for future growth that can minimize hardware reconfigurations and cluster management issues as the cluster grows.

7.8 High availability considerations When a Cloudera cluster on Lenovo servers, is implemented, consider availability requirements as part of the final hardware and software configuration. Typically, Hadoop is considered a highly reliable solution. Hadoop and Cloudera best practices provide significant protection against data loss. Generally, failures can be managed without causing an outage. There is redundancy that can be added to make a cluster even more reliable. Some consideration must be given to hardware and software redundancy.

7.8.1 Networking considerations Optionally, a second redundant switch can be added to ensure HA of the hardware management network. The hardware management network does not affect the availability of the Cloudera Hadoop file system functionality, but it might affect the management of the cluster; therefore, availability requirements must be considered.

To support HA in the network, link aggregation is used between the 10Gb ports of a server network adapter and the top-of-rack switch. Virtual Aggregation Groups (vLAG) can be used between redundant switches.

7.8.2 Hardware availability considerations The redundancy of each individual data node is not necessary with Hadoop. HDFS default 3x replication provides built-in redundancy and makes loss of data unlikely. If Hadoop best practices are used, an outage from a data node loss is extremely unlikely as the workload can be dynamically re-allocated. The loss of a data node will not cause a job to fail; workload is automatically re-allocated to another data note.

Page 38: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

38 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Multiple Master nodes are recommended so that if there is a failure, function can be moved to an operational Master node. Having multiple Master nodes does not automatically resolve the issue of the NameNode being a single point of failure. For more information, see “Software availability considerations”.

Within racks, switches and nodes must have redundant power feeds with each power feed connected from a separate PDU.

7.8.3 Storage availability HDFS 3x replication provides more than sufficient protection. Higher levels of replication can be considered if needed.

Cloudera also provides manual or scheduled snapshots of volumes to protect against human error and programming defects. Snapshots are useful for rollback to a known data set.

7.8.4 Software availability considerations Operating system availability is provided by using mirrored drives for the operating system.

NameNode HA is recommended and can be achieved by using three master nodes. Active and standby nodes communicate with a group of separate daemons called JournalNodes to keep their state synchronized. When any namespace modification is performed by the active NameNode, it durably logs a record of the modification to most of these JournalNodes. The standby NameNode can read the edits from the JournalNodes and is constantly watching them for changes to the edit log. As the standby Node sees the edits, it applies them to its own namespace.

An external database is required for Cloudera Manager, Hive metastore and so on, and HA configuration of external database is recommended to avoid single point of failure. Embedded databases should only be used for test or POC environment.

7.9 Migration considerations If migrating data or applications to Cloudera is required, you must consider the type and amount of data to be migrated. Most data types can be migrated, but you must understand migration requirements to verify viability. Cloudera Enterprise provides tools to move data between external SQL databases and Hadoop.

Other considerations should be given to whether applications must be modified to use Hadoop functionality. Significant effort might be required in some cases.

Page 39: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

39 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

8 Appendix: Bill of Materials This appendix includes the Bill of Materials (BOMs) for different configurations of hardware for the Big Data Solution from Cloudera deployments. There are sections for Master nodes, data nodes, and networking.

The BOM includes the part numbers, component descriptions, and quantities. Table 5 lists how many core components are required for each of the predefined configuration sizes.

The BOM lists in this appendix are not meant to be exhaustive and must always be verified with the configuration tools. Any discussion of pricing, support, and maintenance options is outside the scope of this document.

This BOM information is for the United States; part numbers and descriptions can vary in other countries. Other sample configurations are available from your Lenovo sales team. Components are subject to change without notice.

8.1 Master node Table 8 lists the BOM for the Master node.

Table 8. Master node 8869AC1 Lenovo System x3550 M5 1 A5FW System x Gen-II Universal Slides Kit 1 A5AF System x3550 M5 PCIe Riser 2, 1-2 CPU (LP x16 CPU1 + LP x16 CPU0) 1 A5AG System x3550 M5 PCIe Riser 1 (1x LP x16 CPU0) 1 A4Z6 Broadcom NetXtreme Dual Port 10GbE SFP+ Adapter 1 ATKQ x3550 M5 Base Chassis 8x2.5 1 A3YZ ServeRAID M5210 SAS/SATA Controller 1 ATCA 16GB TruDDR4 Memory (2Rx4, 1.2V) PC4-19200 CL17 2400MHz LP RDIMM 8 ATLY Intel Xeon Processor E5-2650 v4 12C 2.2GHz 30MB Cache 2400MHz 105W 1 ATMN Addl Intel Xeon Processor E5-2650 v4 12C 2.2GHz 30MB 2400MHz 105W 1 A5B0 System x 900W High Efficiency Platinum AC Power Supply 1 A5B0 System x 900W High Efficiency Platinum AC Power Supply 1 ATLK System x3550 M5 front IO cage Advanced 1 A59W System x3550 M5 4x 2.5" HS HDD Kit 1 A59X System x3550 M5 4x 2.5" HS HDD Kit PLUS 1 A1ML Integrated Management Module Advanced Upgrade 1 ATL3 System Documentation and Software-US English 1 ATYV System x Advanced LCD Light Path Kit 1 ATKT x3550 M5 MLK Planar 1 5978 Select Storage devices - configured RAID 1 AT81 2TB 7.2K 6Gbps NL SATA 2.5 G3HS 512e HDD 6 A2K7 Primary Array - RAID 1 1 A578 240GB SATA 2.5" MLC G3HS Enterprise Value SSD 2 6400 2.8m, 13A/125-10A/250V, C13 to IEC 320-C14 Rack Power Cable 2 A5AL System x Enterprise 1U Cable Management Arm (CMA) 1

Page 40: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

40 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

ATKX x3550 M5 Label GBM 1 ATKW x3550 M5 Fan Filler 1 ATKU -SB- x3550 Packaging 1 ATRW System x M5 Rear USB Port Cover 1 2302 RAID Configuration 1 A52A 2U Bracket for Broadcom NetXtreme Dual Port 10GbE SFP+ Adapter 1 A597 LCD OP Cable 1 A595 ODD Filler 1

8.2 Data node Table 9 lists the BOM for the Data node.

Table 9. Data node 8871AC1 Lenovo System x3650 M5 1 ATG2 System Documentation and Software-US English 1 A5GE x3650 M5 12x 3.5" HS HDD Assembly Kit 1 A5GL System x3650 M5 Rear 2x 3.5" HDD Kit (Cascaded) 1 A3YY N2215 SAS/SATA HBA 1 A5GH System x3650 M5 Rear 2x 2.5" HDD Kit (Independent RAID) 1 ATE4 System x3650 M5 Planar BDW 1 ATET Intel Xeon Processor E5-2680 v4 14C 2.4GHz 35MB Cache 2400MHz 120W 1 ATFJ Addl Intel Xeon Processor E5-2680 v4 14C 2.4GHz 35MB 2400MHz 120W 1 A5FV System x Enterprise Slides Kit 1 ATE0 System x3650 M5 12x 3.5" Base without Power Supply BDW 1 A5EW System x 900W High Efficiency Platinum AC Power Supply 1 A45W ServeRAID M1215 SAS/SATA Controller 1 A5EW System x 900W High Efficiency Platinum AC Power Supply 1 ATE9 System x3650 M5 EIA L - VGA 1 A4Z6 Broadcom NetXtreme Dual Port 10GbE SFP+ Adapter 1 A483 Populate and Boot From Rear Drives 1 5977 Select Storage devices - no configured RAID required 1 A5VM 6TB 7.2K 6Gbps NL SATA 3.5" G2HS 512e HDD 14 A577 120GB SATA 2.5" MLC G3HS Enterprise Value SSD 2 ATCB 32GB TruDDR4 Memory (2Rx4, 1.2V) PC4-19200 CL17 2400MHz LP RDIMM 16 6400 2.8m, 13A/125-10A/250V, C13 to IEC 320-C14 Rack Power Cable 2 ATGF -SB- System x3650 M5 WW Packaging 1 ATE2 System x3650 M5 System Agency Label 1 ATE3 System x3650 M5 System Level Code 1 ATRG system x M5 rear USB port cover 1 A52A 2U Bracket for Broadcom NetXtreme Dual Port 10GbE SFP+ Adapter 1

Page 41: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

41 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

A5FT System x3650 M5 Power Paddle Card 1 A5G1 System x3650 M5 EIA Plate 1 A5V5 System x3650 M5 Right EIA for Storage Dense Model 1 ASQA System x3650 M5 Rear 2x 2.5" HDD Label (Independent RAID-Riser1) 1

8.3 Management/Administration network switch Table 10 lists the BOM for the Management/Administration network switch.

Table 10. Management/Administration network switch

Code Description Quantity

7159G52 Lenovo RackSwitch G8052 (Rear to Front) 1

6201 1.5m, 10A/100-250V, C13 to IEC 320-C14 Rack Power Cable 2

A3KP IBM System Networking Adjustable 19" 4 Post Rail Kit 1

2305 Rack Installation of 1U Component 1

8.4 Data network switch Table 11 lists the BOM for the data network switch.

Table 11. Data network switch

Code Description Quantity

7159CRW Lenovo RackSwitch G8272 (Rear to Front) 1

6311 2.8m, 10A/100-250V, C13 to IEC 320-C14 Rack Power Cable 2

A3KP IBM System Networking Adjustable 19" 4 Post Rail Kit 1

2305 Rack Installation of 1U Component 1

8.5 Rack Table 12 lists the BOM for the rack.

Table 12. Rack

Code Description Quantity

1410HPB Intelligent Cluster 42U 1100mm Enterprise V2 Dynamic Rack 1

6012 DPI Single-phase 30A/208V C13 Enterprise PDU (US) 4

2202 Cluster 1350 Ship Group 1

2304 Rack Assembly - 42U Rack 1

2310 Cluster Hardware & Fabric Verification - 1st Rack 1

Different cluster sizing leaves different unused rack space; therefore, consider the use of blank plastic filter panels for the rack to better direct cool air flow.

The number of PDUs in the rack depends on the server numbers in the rack. Four PDU should be used for the

Page 42: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

42 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

half rack configuration and six PDUs for a full rack.

8.6 Cables Table 13 lists the BOM for the cables, for each node.

Table 13. Cables

Code Description Quantity

A1PJ 3m IBM Passive DAC SFP+ Cable 2

A4RA CAT5E IntraRack Cable Service 1

Page 43: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

43 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

9 Acknowledgements This reference architecture document has benefited very much from the detailed and careful review comments provided by colleagues at Lenovo and Cloudera.

Lenovo business review

• Prasad Venkatachar – Sr. Solutions Product Manager

Cloudera technical review

• Alex Moundalexis – Sr. Solutions Architect, Partner Enablement

• Rick Hallihan – Solutions Architect, Partner Enablement

Cloudera business review

• Thomas Pinckney – Sr. Director, Business Development/ Global Partner Sales

• Sandy Lii – Director, Partner Marketing

VMware technical review

• Technical staff members at VMware

Page 44: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

44 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Resources For more information, see the following resources:

Lenovo System x3650 M5 (Cloudera Data Node):

• Product page: shop.lenovo.com/us/en/systems/servers/racks/systemx/x3650-m5/ • Lenovo Press product guide: lenovopress.com/tips1193

Lenovo System x3550 M5 (Cloudera Master node):

• Product page: shop.lenov o.com/us/en/systems/servers/racks/systemx/x3550-m5/ • Lenovo Press product guide: lenovopress.com/tips1194

Lenovo RackSwitch G8052 (1GbE Switch):

• Product page: shop.lenovo.com/us/en/systems/browsebuy/%20rackswitch-g8052.html • Lenovo Press product guide: lenovopress.com/tips0813

Lenovo RackSwitch G8272 (10GbE Switch):

• Product page: shop.lenovo.com/us/en/systems/browsebuy/lenovo-rackswitch-g8272.html • Lenovo Press product guide: lenovopress.com/tips1267

Lenovo XClarity Administrator:

• Product page: shop.lenovo.com/us/en/servers/thinkserver/system-management/xclarity • Lenovo Press product guide: lenovopress.com/tips1200

Cloudera:

• Cloudera Distribution for Hadoop (CDH): cloudera.com/content/cloudera/en/products-and-services/cdh.html

• Cloudera products and services: cloudera.com/content/cloudera/en/products-and-services.html

• Cloudera solutions: cloudera.com/content/cloudera/en/solutions.html • Cloudera resources: cloudera.com/content/cloudera/en/resources.html • Cloudera RA with VMware and local attached storage:

cloudera.com/documentation/other/reference-architecture/PDF/cloudera_ref_arch_vmware_local_storage.pdf

VMware:

• VMware Hadoop Deployment Guide • Big Data Performance on vSphere • Virtaulized Hadoop Performance with VMware vSphere 6.0 on High-Performance Servers

Open source software:

• Hadoop: hadoop.apache.org • Spark: spark.apache.org • Flume: flume.apache.org

Page 45: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

45 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

• HBase: hbase.apache.org • Hive: hive.apache.org • Hue: gethue.com • Impala: rideimpala.com • Oozie: oozie.apache.org • Mahout: mahout.apache.org • Pig: pig.apache.org • Sentry: entry.incubator.apache.org • Sqoop: sqoop.apache.org • Whirr: whirr.apache.org • ZooKeeper: zookeeper.apache.org • Parquet: parquet.apache.org • Hadoop Virtualization Extensions (HVE):

https://issues.apache.org/jira/browse/HADOOP-8468 ● xCat: xcat.org

Page 46: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

46 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Document history Version 1.0 19 August 2015 • First version

Version 1.1 3 October 2016 • Updated to use CDH 5.8

• Updated to use servers with Intel E5-2600 v4 processor family

Version 1.2 20 June 2017 • Added sections on virtualized Hadoop with CDH 5.11 on VMware vSphere 6.5

Page 47: Lenovo Big Data Validated Design for Cloudera Enterprise ... · PDF file1 Lenovo Big Data Validated Design for Cloudera Enterprise and ... 4 Architectural overview ... Lenovo Big Data

47 Lenovo Big Data Validated Design for Cloudera Enterprise and VMware

Trademarks and special notices © Copyright Lenovo 2017.

References in this document to Lenovo products or services do not imply that Lenovo intends to make them available in every country.

Lenovo, the Lenovo logo, ThinkCenter, ThinkVision, ThinkVantage, ThinkPlus and Rescue and Recovery are trademarks of Lenovo.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used Lenovo products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-Lenovo products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by Lenovo. Sources for non-Lenovo list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. Lenovo has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-Lenovo products. Questions on the capability of non-Lenovo products should be addressed to the supplier of those products.

All statements regarding Lenovo future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local Lenovo office or Lenovo authorized reseller for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in Lenovo product announcements. The information is presented here to communicate Lenovo’s current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard Lenovo benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-Lenovo websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this Lenovo product and use of those websites is at your own risk.