Apache Ambari - What's New in 2.4

of 88 /88
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari 2.4.0 What’s New August 2016

Transcript of Apache Ambari - What's New in 2.4

Page 1: Apache Ambari - What's New in 2.4

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari 2.4.0What’s NewAugust 2016

Page 2: Apache Ambari - What's New in 2.4

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

What is Apache Ambari?

A completely open source management platform for provisioning, managing, monitoring and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop.

Page 3: Apache Ambari - What's New in 2.4

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

What Ambari Does

Simplified Installation, Configuration and Management

Centralized Security Setup

Full Visibility into Cluster Health

Highly Extensible and Customizable

• Wizard-driven and automated cluster provisioning• Smart Configurations and Cluster Recommendations• Automated Rolling and Express cluster upgrades

• Reduce complexity to administer security across the platform• Automate setup Kerberos• Simplify the configuration of Apache Ranger

• Predefined alerts based on operational best practices• Advanced metrics visualization with Grafana

• Seamlessly fit into your enterprise environment• Bring custom Services under management via Ambari Stacks• Customize the UI with Ambari Views

Page 4: Apache Ambari - What's New in 2.4

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

What’s New in Ambari 2.4

Alerts: Customizable SCRIPT Parameters (AMBARI-14898)

Alerts: Retry Check Counts (AMBARI-15686)

Alerts: New HDFS Alerts (AMBARI-14800)

New Host Page Filtering (AMBARI-15210)

Remove Service (AMBARI-14759)

Support for SLES 12 Technical Preview (AMBARI-16007)

Stability: Database Consistency Checking (AMBARI-16258)

Customizable Ambari Log + PID Dirs (AMBARI-15300)

New Version Registration Experience (AMBARI-15724)

Log Search Technical Preview (AMBARI-14927)

Operational Audit Logging (AMBARI-15241)

Role-Based Access Control (AMBARI-13977)

Automated Setup of Ambari Kerberos (AMBARI-15561)

Automated Setup of Ambari Proxy User (AMBARI-15561)

Customizable Host Reg. SSH Port (AMBARI-13450)

Core Features Security Features

View URLs (AMBARI-15821), View Refresh (AMBARI-15682)

Inherit Cluster Permissions (AMBARI-16177)

Remote Cluster Registration (AMBARI-16274)

Views Framework Features

Page 5: Apache Ambari - What's New in 2.4

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Alert Retry Check Counts

Page 6: Apache Ambari - What's New in 2.4

6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Alert Check Counts

Customize the number of times an alert is checked before dispatching a notification Avoid dispatching an alert notification (email, snmp) in case of transient issues

Page 7: Apache Ambari - What's New in 2.4

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Configuring the Check Count

Set globally for all alerts, or override for a specific alert

Global Setting

Alert Override

Page 8: Apache Ambari - What's New in 2.4

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

State Change Types

SOFT state changes do not perform a dispatch HARD state changes (to non-OK) perform dispatch Regardless of change:

– The Ambari Web UI will show the current state (OK/WARN/CRIT)– The state change is written to ambari-alerts.log

2016-05-31 13:20:52,294 [CRITICAL] [SOFT] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)2016-05-31 13:22:52,290 [CRITICAL] [HARD] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)

Note: check counts are not configurable for AGGREGATE alert types. All state changes are considered HARD.

Page 9: Apache Ambari - What's New in 2.4

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Example: Check Count = 3

Check 1/3State: OK

Change: n/a

Check 1/3State: OK

Change: n/a

Check 1/3State: CRIT

Change: SOFT

Check 2/3State: CRIT

Change: n/a

Check 3/3State: CRIT

Change: HARD

Check 1/3State: OK

Change: HARD

DISPATCH

Check Interval Check Interval Check Interval Check Interval Check Interval

no state changestate changes to CRIT

performing multiple checks

back to OK

Page 10: Apache Ambari - What's New in 2.4

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Alert Customizable Params

Page 11: Apache Ambari - What's New in 2.4

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Alert Types and Thresholds

Ability to customize Thresholds for SCRIPT and SERVER alerts Ability to customize Connection Timeout for METRIC alerts

Alert Type Description Thresholds (units)

WEB Connects to a Web URL. Alert status is based on the HTTP response code. Response Code (n/a)Connection Timeout (seconds)

PORT Connects to a port. Alert status is based on response time. Response (seconds)

METRIC Checks the value of a service metric. Units vary, based on the metric being checked.

Metric Value (units vary)Connection Timeout (seconds)

AGGREGATE Aggregates the status for another alert. % Affected (percentage)

SCRIPT Executes a script to handle the alert check. Varies

SERVER Executes a server-side runnable class to handle the alert check. Varies

NEW!

Page 12: Apache Ambari - What's New in 2.4

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Alerts: Customizable METRIC Connection Timeout

Ability to set Connection Timeout threshold via Ambari Web UI

NEW!

Page 13: Apache Ambari - What's New in 2.4

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Alerts: Customizable SCRIPT Thresholds

Ability to set various thresholds via Ambari Web UI

Page 14: Apache Ambari - What's New in 2.4

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Alerts: NEW!!! Ambari Server Performance Alert

Measures the Ambari Server REST API and Backend Database response

Page 15: Apache Ambari - What's New in 2.4

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: New HDFS Alerts

Page 16: Apache Ambari - What's New in 2.4

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

New HDFS Alerts Watch Trends

NameNode Client RPC Queue Latency (Hourly/Daily) NameNode Client RPC Processing Latency (Hourly/Daily) NameNode Service RPC Queue Latency (Hourly/Daily) NameNode Service RPC Processing Latency (Hourly/Daily) NameNode Heap Usage (Daily/Weekly) HDFS Storage Capacity Usage (Daily/Weekly)

Page 17: Apache Ambari - What's New in 2.4

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: New Host Filtering

Page 18: Apache Ambari - What's New in 2.4

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

New Host Filtering Control in Ambari Web

Ability to perform complex host filtering from Ambari Web Make it easier to find hosts

NEW!

Page 19: Apache Ambari - What's New in 2.4

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Search by Host Attribute, Service or Component

Page 20: Apache Ambari - What's New in 2.4

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Host Attribute Filtering

Host Name IP Host Status Cores RAM Stack Version + Version State Rack

Page 21: Apache Ambari - What's New in 2.4

21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Service Filtering

Page 22: Apache Ambari - What's New in 2.4

22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Component Filtering

Page 23: Apache Ambari - What's New in 2.4

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Host Filter: Examples

Page 24: Apache Ambari - What's New in 2.4

24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Remove Service

Page 25: Apache Ambari - What's New in 2.4

25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Remove Service

Ability to perform Remove Service from Ambari Web Eliminates need to use Ambari REST API Checks for Service dependencies Service must be stopped All configuration information and history is also removed This operation is not reversible

NEW!

Page 26: Apache Ambari - What's New in 2.4

26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Other Items

Page 27: Apache Ambari - What's New in 2.4

27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Customizable Ambari Log + PID Dirs (AMBARI-15300)

Ambari Server and Agents write log activity output to log files and use a PID-file that contains the process identification number (PID) for their running process.

Log Location PID Location

Ambari Server /var/log/ambari-server/ambari-server.log /var/run/ambari-server/ambari-server.pid

Ambari Agent /var/log/ambari-agent/ambari-agent.log /var/run/ambari-agent/ambari-agent.pid

Page 28: Apache Ambari - What's New in 2.4

28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Customize Ambari Server Log + PID

vi /etc/ambari-server/conf/ambari.properties

pid.dir=/var/run/ambari-server

vi /etc/ambari-server/conf/log4j.properties

ambari.log.dir=${ambari.root.dir}/var/log/ambari-server

Ambari Server PID Ambari Server Log

1. Stop Ambari Server prior to modifying log or pid directories.2. You must manually create the new directories and be sure to set the directory

ownership + permissions to allow the Ambari Server process access.

Page 29: Apache Ambari - What's New in 2.4

29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Customize Ambari Agent Log + PID

vi /etc/ambari-agent/conf/ambari-agent.ini

[agent]

logdir=/var/log/ambari-agent

piddir=/var/run/ambari-agent

1. Stop Ambari Agent prior to modifying log or pid directories.2. You must manually create the new directories and be sure to set the directory

ownership + permissions to allow the Ambari Agent process access.

Page 30: Apache Ambari - What's New in 2.4

30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Customizable Host Registration SSH Port

Customize SSH Port when performing Host Registration automatically

NEW!

Page 31: Apache Ambari - What's New in 2.4

31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Stability: Database Consistency Checking

On Ambari Server start, Ambari runs a database consistency check looking for issues. If any issues are found, Ambari Server start will abort and a message will be printed to

console “DB configs consistency check failed.” Check Ambari Server log file for more details:/var/log/ambari-server/ambari-server-check-database.log

Ability to “skip” check and force Ambari Server startambari-server start --skip-database-check

Important: if you “skip” the check to force Ambari Server start, do not make any changes to your cluster topology or perform a cluster upgrade until you correct

the database consistency issues.

Page 32: Apache Ambari - What's New in 2.4

32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: View Framework Enhancements

Page 33: Apache Ambari - What's New in 2.4

33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

View URLs (AMBARI-15821)

Ability to create a “short URL” or “vanity URL” for view instances Provide users with a non-version or instance specific URL to a view

/#/main/views/{viewName}/{viewVersion}/{viewInstanceName}/#/main/view/{viewName}/{shortURL}

Page 34: Apache Ambari - What's New in 2.4

34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

View Refresh (AMBARI-15682)

Automatically deploy new views into Ambari Server w/o a restart

1. Copy view archive to: /var/lib/ambari-server/resources/views/

2. Ambari Server detects the new view, automatically extracts + deploys

3. View is available for creating instances

4. Click “Refresh” in Views UI

Page 35: Apache Ambari - What's New in 2.4

35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Remote Cluster ConfigurationAMBARI-16274

Page 36: Apache Ambari - What's New in 2.4

36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: View <-> Cluster Communication

Deployed Views “talk” with cluster using REST APIs (as applicable)

CLUSTER

ATS

RM

Ambari Server

Tez UIView

Tez UI View talks with cluster using

REST APIs toATS and ResourceManager

Ambari DB

LDAPAuthN

Page 37: Apache Ambari - What's New in 2.4

37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Operational vs. Standalone Ambari Server

Ambari Agent

Host

Ambari Agent

Host

Ambari Agent

Host

Standalone Ambari ServerOne or More Ambari Server Instances

No Agents, no requirement to operate the cluster

Operational AmbariOne Ambari Server Instance

Talking with Agents, Managing the cluster

Ambari Server

Ambari DB

LDAPAuthN

Ambari Server

Ambari DB

LDAPAuthN

Page 38: Apache Ambari - What's New in 2.4

38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Local Cluster vs. Non-Local

Ambari Server

Ambari DB

LDAPAuthN

Ambari Server

Ambari DB

LDAPAuthN

Standalone Ambari ServerOne or More Ambari Server Instances

No Agents, no requirement to operate the cluster

Operational AmbariOne Ambari Server Instance

Talking with Agents, Managing the cluster

LOCALCLUSTER

NON-LOCAL

CLUSTER

Page 39: Apache Ambari - What's New in 2.4

39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Introducing Remote Cluster Configuration (AMBARI-16274)Option DescriptionLocal Cluster When you select this Local Cluster option, Ambari will automatically determine the

cluster configuration properties needed for the view instance.Criteria:• Ambari Server running the views is also managing the cluster

Remote Cluster When you select Remote Cluster option, Ambari will automatically determine the cluster configuration properties needed for the view instance.Criteria:• The cluster is not local to the Ambari Server running the views (i.e. Standalone)• Cluster is being managed by Ambari

Custom When you select Custom option, you must enter all configuration information, and are responsible for updating if the cluster configuration changes.Criteria:• The cluster running the view is not local to the Ambari Server• The cluster is not being managed by Ambari

NEW!

Page 40: Apache Ambari - What's New in 2.4

40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Local vs Remote View Configuration

Ambari Server

Views

Cluster

Ambari Server

Views

ClusterAmbari Server

LOCAL CLUSTER

REMOTE CLUSTER

Operational Ambari

Manages cluster

Standalone Ambari

Manages cluster

Talks to cluster

Obtains view config

Obtains view config

Talks to cluster

Operational Ambari

Page 41: Apache Ambari - What's New in 2.4

41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

View Configuration: Minimizing Need for Custom

Cluster Config Ambari Server Cluster Mgmt Ambari 2.2 or Earlier Ambari 2.4

No HA, No Kerberos Operational Ambari Local Local

HA or Kerberos Operational Ambari Custom Local

No HA, No Kerberos Standalone Ambari Custom Remote

HA or Kerberos Standalone Ambari Custom Remote

No HA, No Kerberos Standalone Non-Ambari Custom Custom

HA or Kerberos Standalone Non-Ambari Custom Custom

Page 42: Apache Ambari - What's New in 2.4

42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Inherit Cluster PermissionsAMBARI-16177

Page 43: Apache Ambari - What's New in 2.4

43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Inherit Cluster Permissions (AMBARI-16177)

Ability to automatically grant View “Use” permission based on Cluster role Note: Option is only available when using a Local Cluster Configuration

Explicitly grant users and groups Use

permission

Automatically grant users and groups Use permission based on

Cluster roles

Page 44: Apache Ambari - What's New in 2.4

44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Log Search

TECH PREVIEW

Page 45: Apache Ambari - What's New in 2.4

45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Log Search

Solr

A M B A R ILog

Search

Search Cluster Component Logs from within Ambari

Goal: When issues arise, be able to quickly find issues across all components

⬢ Capabilities– Rapid Search of all cluster component logs– Search across time ranges, log levels, and for keywords

⬢ Core Technologies: – Apache Ambari– Apache Solr– Apache Ambari Log Search

Tech Preview

Page 46: Apache Ambari - What's New in 2.4

46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Log Search Architecture

A M B A R I

L O G F E E D E R

L O G F E E D E R

L O GF E E D E R

L O GF E E D E R

L O G F E E D E R

L O G F E E D E R

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

Solr

L O G S E A R C H

U I

Tech Preview

Page 47: Apache Ambari - What's New in 2.4

47 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Log Search Details

W O R K E RN O D E

L O G F E E D E R

Solr

L O G S E A R C H

U I

Solr

Solr

A M B A R I

Java ProcessMulti-output SupportGrok

Solr CloudLocal Disk StorageTTL

Tech Preview

Page 48: Apache Ambari - What's New in 2.4

48 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Considerations

Log Feeders are CPU intensive, consider 1 dedicated core Solr instances should use dedicated hardware with at least 32GB of RAM dedicated to

the Solr instance By default, logs will age out after 7 days

Tech Preview

Page 49: Apache Ambari - What's New in 2.4

49 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: RBAC

Page 50: Apache Ambari - What's New in 2.4

50 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

New Role Based Access Control

Introducing new “roles” for more granular division of control for cluster operations

Old Permission New Role Notable Permissions

Operator Cluster Administrator Full operational control, including upgrades. Ambari Admins are implicitly granted this Role.

Cluster Operator Adding and removing hosts.

Service Administrator Manage configurations, move components.

Service Operator Service stop and start and service-specific operations such as HDFS Rebalance.

Read-Only Cluster User View cluster service and host information.

Note: Users flagged as “Ambari Administrators / Ambari Admins” are implicitly granted Cluster Administrator permission.

Page 51: Apache Ambari - What's New in 2.4

51 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Managing Cluster RolesAssign roles to users or groups

Manage roles in Block or List View

layouts

Page 52: Apache Ambari - What's New in 2.4

52 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Managing Cluster Roles View users or groups

Change current role assignment

Page 53: Apache Ambari - What's New in 2.4

53 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Security Enhancements

Page 54: Apache Ambari - What's New in 2.4

54 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Summary of Security Enhancements

Automatic Setup of Ambari Server as a Proxyuser Automatic Setup of Ambari Server for Kerberos

Page 55: Apache Ambari - What's New in 2.4

55 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Automatic Setup of Ambari Server as a Proxyuser

Page 56: Apache Ambari - What's New in 2.4

56 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Proxyusers

HDFS and WebHCat (as part of Hive) support the concept of a Proxyuser Proxyuser allows UserA to access the service on behalf of UserB (i.e. the proxyuser is

allowed to impersonate other users) Proxyuser is a commonly used capability of Hadoop

HDFS

“UserA” is setup as a proxyuser

UserA can access HDFS as “UserA” on behalf of “UserB”

HDFS ops performed are as “UserB”

Page 57: Apache Ambari - What's New in 2.4

57 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: HDFS Proxyuser Setup

A proxyuser needs to be configured in core-site.xml configuration:hadoop.proxyuser.{proxyuser-name}.hosts

hadoop.proxyuser.{proxyuser-name}.groups

If these settings are not present, impersonation will not be allowed and connection to the service via proxyuser will fail

Page 58: Apache Ambari - What's New in 2.4

58 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Ambari + Proxyuser

Ambari Views use proxyuser to access the cluster (such as Hive View and Pig View) Ambari Server needs to access a service on behalf of an authenticated user

Ambari Server HDFS

(running as user “ambari”)

“joe” authenticates to Ambari

(setup for proxyuser “ambari”)hadoop.proxyuser.ambari.hosts=*hadoop.proxyuser.ambari.groups=*

Ambari Server can talk to HDFS as “ambari” proxyuser on behalf of “joe”

Configuration of proxyuser is commonly “missed” when setting up Ambari Views

Page 59: Apache Ambari - What's New in 2.4

59 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

New: Automatic Ambari Server Proxyuser Setup

Proxyuser configurations are automatically added for HDFS and WebHCat For example: Ambari Server as running as “ambari”, the following configurations are

added during HDFS service installhadoop.proxyuser.ambari.hosts

hadoop.proxyuser.ambari.groups

Page 60: Apache Ambari - What's New in 2.4

60 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Automatic Setup of Ambari Server for Kerberos

Page 61: Apache Ambari - What's New in 2.4

61 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Hadoop + Kerberos

Strongly authenticating and establishing a user’s identity is the basis for secure access in Hadoop. Users need to be able to reliably “identify” themselves and then have that identity propagated throughout the Hadoop cluster.

Once this is done, those users can access resources (such as files or directories) or interact with the cluster (like running MapReduce jobs).

Besides users, Hadoop cluster resources themselves (such as Hosts and Services) need to authenticate with each other to avoid potential malicious systems or daemon’s “posing as” trusted components of the cluster to gain access to data.

Page 62: Apache Ambari - What's New in 2.4

62 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Hadoop + Kerberos

Service Component

A

Service Component

B

Hadoop Cluster

KDC

keytabkeytab

Service Component

C

keytab

Service Component

D

keytab

Service Component

X

Service Component

X

keytabkeytab

Service Component

X

keytab

Service Component

X

keytab

Kerberos is used to secure the

Components in the cluster. Kerberos

identities are managed via

“keytabs” on the Component hosts.

Principals for the

cluster are managed in

the KDC.

Page 63: Apache Ambari - What's New in 2.4

63 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Automated Kerberos Setup with Ambari

Page 64: Apache Ambari - What's New in 2.4

64 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Principal and Keytab Generation & Distribution

1. User provides KDC Admin Account credentials to Ambari

2. Ambari connects to KDC, creates principals (Service and Ambari) needed for cluster

3. Ambari generates keytabs for the principals

4. Ambari distributes keytabs to Ambari Server and cluster hosts

5. Ambari discards the KDC Admin Account credentials (optional)

AmbariServer KDC

1 2

4

3

5

Cluster

Page 65: Apache Ambari - What's New in 2.4

65 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Ambari + Hadoop + Kerberos

Ambari Server communicates with the cluster to retrieve information (such as metrics) Especially important for Ambari Views (e.g. Files, Hive, Pig) Therefore: Ambari Server ALSO needs to be “setup for Kerberos”

Ambari Server Cluster

Kerberos enabled

Page 66: Apache Ambari - What's New in 2.4

66 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Manual Setup of Ambari Server for Kerberos

Manual setup of Ambari Server for Kerberos (outside of “Enable Kerberos” wizard):

1. Create principal for Ambari Server

2. Generate keytab for Ambari Server

3. Place keytab on Ambari Server host

4. Run “ambari-server setup-security” on Ambari Server

5. Restart Ambari Server

Configuration of Ambari Server for Kerberos is commonly “missed” when setting up Ambari Views

Page 67: Apache Ambari - What's New in 2.4

67 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

New: Automatic Setup of Ambari Server for Kerberos

When enabling Kerberos and choosing an automated option (MIT or AD), Ambari Server will be setup for Kerberos automatically:

1. A principal will be created for Ambari Server

2. A keytab will be generated and placed on Ambari Server

3. Ambari Server is setup for Kerberos

Note: you will still need to perform the Ambari Server restart for the Kerberos identity to get picked-up by Ambari.

Page 68: Apache Ambari - What's New in 2.4

68 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

What about Proxyuser + Kerberos?

Page 69: Apache Ambari - What's New in 2.4

69 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

New: Automatic Proxyuser Setup with Kerberos

When a cluster has Kerberos enabled, the proxyuser needs to be configured based on the primary part of the Kerberos principal name

hadoop.proxyuser.{principal-name-primary}.hosts

hadoop.proxyuser.{principal-name-primary}.groups

Ambari will adjust proxyuser configurations during Kerberos setup

Ambari Server HDFS

(running as user “ambari”)(setup with principal “[email protected]

“joe” authenticates to Ambari

(setup for proxyuser “ambari-server”)hadoop.proxyuser.ambari-server.hosts=*hadoop.proxyuser.ambari-server.groups=*

Ambari Server can talk to HDFS as “ambari-server” proxyuser on behalf

of “joe”

Page 70: Apache Ambari - What's New in 2.4

70 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Ops Audit Logging

Page 71: Apache Ambari - What's New in 2.4

71 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Operational Audit Logging

Ambari will create entries in an audit log as Ambari + Cluster operations are performed Using the audit log, you can determine who performed the operation and when the

operation was performed as well as other operation-specific information The Ambari Audit log can be found at: /var/log/ambari-server/ambari-audit.log

Page 72: Apache Ambari - What's New in 2.4

72 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

List of Operations

Stop/Start Service Stop all Services Add Service Move Component Turn On/Off Maintenance Mode Download Client Configurations Blueprint Export Update Configuration **

Login (success/failed) / Logout Create User, Group Delete User, Group Change Group Membership Change User Status, Admin Change User Password Grant/Revoke User, Group Cluster Roles

Service Operations User Operations

** Note: When a Service Configuration change is made, an entry is also written to a specific log file ambari-config-changes.log for configuration changes that provides even more detail on the change.

Page 73: Apache Ambari - What's New in 2.4

73 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

List of Operations (continued)

Add/Remove Host Enable/Disable/Edit Alert Add/Update/Delete Alert Group Add/Upgrade/Delete Notification Enable/Disable Kerberos Regenerate Kerberos Keytabs Rename Cluster Add/Remove Remote Clusters

Register/Deregister Version Cluster Upgrade

Cluster Operations Upgrade Operations

Create/Delete View Instance Edit View Instance Grant/Revoke View Permissions Create/Delete View URLs

View Operations

Page 74: Apache Ambari - What's New in 2.4

74 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Example: Change Group Membership

Add/Remove group members creates a “Membership change” audit entry

2016-06-02T23:12:09.930Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT), url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser)2016-06-02T23:12:34.700Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT), url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser, mike)

Page 75: Apache Ambari - What's New in 2.4

75 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Example: Stop ZooKeeper

A single operation (like “Stop ZooKeeper”) might generate multiple audit entries Relate entries via RequestId()

2016-06-02T23:14:35.206Z, User(admin), RemoteIp(192.168.64.1), Operation(INSTALLED: ZOOKEEPER_SERVER/ZOOKEEPER on c6401.ambari.apache.org (MyCluster)), Host name(c6401.ambari.apache.org), RequestId(7), Status(Successfully queued)2016-06-02T00:31:56.016Z, User(admin), Operation(Stop ZooKeeper Server), Status(IN_PROGRESS), RequestId(7)2016-06-02T00:31:56.025Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(QUEUED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)2016-06-02T00:31:57.370Z, User(admin), Operation(Stop ZooKeeper Server), Status(COMPLETED), RequestId(7)2016-06-02T00:31:57.370Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(COMPLETED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)

Page 76: Apache Ambari - What's New in 2.4

76 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Version Registration Experience

Page 77: Apache Ambari - What's New in 2.4

77 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Introducing the Version Definition File (VDF)

This is a meta file describing which Services are included and at which version

Page 78: Apache Ambari - What's New in 2.4

78 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari will “discover” Available Versions

Tabs for list of available Stacks

List of discovered Versions

List of Services w/version #

Page 79: Apache Ambari - What's New in 2.4

79 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

“Default Version Definition” for Backwards Compat

Ambari provides a “default” Version

Definition.

Page 80: Apache Ambari - What's New in 2.4

80 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Add New Version via File Upload or URL

Page 81: Apache Ambari - What's New in 2.4

81 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Changes in Install / Version Registration Flow

Scenario Ambari 2.4 ChangeInternet Access / Public Repositories No change.

No Internet Access / Local repositories - Upload a VDF for the Local Repository you created- Set the Local Repository URLs

OR

- Choose the Default Version Definition- Set the Local Repository URLs

Page 82: Apache Ambari - What's New in 2.4

82 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Other UX Changes: Local vs. Public Repository Radio

Explicit Choice

Page 83: Apache Ambari - What's New in 2.4

83 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Other UX Changes: Local vs. Public Repository Radio

Choose Local

Must enter Base URLs

Page 84: Apache Ambari - What's New in 2.4

84 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Other UX Changes: OS Add/Remove

Page 85: Apache Ambari - What's New in 2.4

85 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Other UX Changes: RedHat Satellite/Spacewalk

Explicit Choice

- Ambari will not write the .repo files

- User must register the repositories channels via Satellite

Page 86: Apache Ambari - What's New in 2.4

86 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Other UX Changes: Viewing, Install and Upgrade

Page 87: Apache Ambari - What's New in 2.4

87 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Other UX Changes: Managing Versions

Page 88: Apache Ambari - What's New in 2.4

88 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Thank You