HDP2.5 Updates

1 ©HortonworksInc.2011–2016.AllRightsReserved

HortonworksDataPla.ormUpdatesYutaImai,Hortonworks

HortonworksDataPla.orm

HortonworksDataPla.orm:ReleaseStrategy

MorefrequentreleasesofSpark,Hive,AmbariandotherApacheDataAccessprojects

ExtendedServices

LongerreleasearcsforcoreApacheHadoopcomponents:HDFS,YARNandMapReduce

HadoopCore2016 2017

2016 2017

HORTONWORKSDATAPLATFORM

HDP2.3isApacheHadoop;not“basedon”Hadoop

Falcon

4.10.2

DATAMGMT DATAACCESS GOVERNANCE&INTEGRATION OPERATIONS SECURITY

HDP2.2Dec2014

HDP2.1April2014

HDP2.0Oct2013

HDP2.2Dec2014

HDP2.1April2014

HDP2.0Oct2013

0.12.0 0.12.0

0.12.1 0.13.0 0.4.0

1.4.4 1.4.4 3.3.23.4.5

0.4.00.5.0

0.14.0 0.14.0 3.4.6 0.5.0 0.4.00.9.30.5.2

4.0.04.7.2

1.2.1 0.60.0 0.98.4 4.2.0 1.6.1 0.6.0 1.5.21.4.5 4.1.02.0.0

1.4.0 1.5.1 4.0.0

1.5.1 1.4.4 3.4.5

2.7.1 1.4.6 1.0.0 0.6.0 0.5.02.1.00.8.2 3.4.61.5.25.2.1 0.80.0 0.5.01.7.04.4.0 0.10.0 0.6.10.7.01.2.10.15.0HDP2.3Oct2015 4.2.0

0.96.1

0.98.0 0.9.1

1.4.1 1.1.2

2.7.3 1.4.6 1.3.0 0.9.0 0.6.02.4.00.10.0 3.4.61.5.25.5.1 0.91.0 0.7.01.7.04.7.0 1.0.1 0.10.00.7.01.2.1+2.1***0.16.0HDP2.5*

2H20164.2.01.6.2+

2.0** 1.1.2

2.7.1 1.4.6 1.2.0 0.6.0 0.5.02.2.10.9.0 3.4.61.5.25.2.1 0.80.0 0.5.01.7.04.4.0 0.10.0 0.6.10.7.01.2.10.15.0HDP2.4Mar2016 4.2.01.6.0 1.1.2

OngoingInnovadoninApache

*HDP2.5–ShowscurrentApachebranchesbeingused.FinalcomponentversionsubjecttochangebasedonApachereleaseprocess.

**Spark1.6.2+Spark2.0–HDP2.5supportinstallaEonofbothSpark1.6.2andSpark2.0.Spark2.0isTechnicalPreviewwithinHDP2.5.

***Hive2.1isTechnicalPreviewwithinHDP2.5.

HortonworksDataPla.orm2.5KeyHighlights

•  InteracYveQueryinSeconds:HivewithLLAP(TechnicalPreview)•  EnterpriseSparkatScale:ApacheZeppelinNotebookforSpark•  Real-TimeApplicaYons:StormandHBase/Phoenix•  StreamlinedOperaYons:ApacheAmbari•  DynamicSecurity:ApacheAtlas+RangerIntegraYon•  HortonworksDataCloud(TechnicalPreview)•  HortonworksHDB(ApacheHAWQ)

InteracdveQueryinSecondsHivewithLLAPTechnicalPreview

7 ©HortonworksInc.2011–2016.AllRightsReserved7 ©HortonworksInc.2011–2016.AllRightsReserved

Hive2withLLAPEnableInteracdveQueryInSeconds

DeveloperProducYvity:InteracYvequeryinseconds

EaseofUseandAdopYon:100%compaYblewithHiveSQL

EnterpriseReadiness:LinearscalingatTerabytesvolumeofdata

StreamlinedOperaYons:LLAPintegraYonwithAmbariwithautomateddashboards

Why LLAP? •  PeoplelikeHive•  Disk->Memisgehngfurtheraway

–  CloudStorageisn’tco-located–  DisksareconnectedtotheCPUvianetwork

•  Securitylandscapeischanging–  Cells&Columnsarethenewsecurityboundary,notfiles–  Safelymaskingcolumnsneedsaprocessboundary

•  Concurrency,Performance&Scaleareatconflict–  Concurrencyat100kqueries/hour–  Latenciesat2-5seconds/query–  Petabytescalewarehouses(withterabytesof“hot”data)

LLAP Process

Query Fragment

What is LLAP? •  Hybrid model combining daemons and containers

for fast, concurrent execution of analytical workloads (e.g. Hive SQL queries)

•  ConcurrentquerieswithoutspecializedYARNqueuesetup•  MulY-threadedexecuYonofvectorizedoperatorpipelines

•  Asynchronous IO and efficient in-memory caching •  Relational view of the data available thru the API •  Highperformancescans,execuYoncodepushdown•  Centralizeddatasecurity

LLAP Process

Query Fragment

Hive2withLLAP:ArchitectureOverview

Storage

YARNCluster

LLAPDaemon

QueryExecutors

LLAPDaemon

QueryExecutors

LLAPDaemon

QueryExecutors

LLAPDaemon

QueryExecutors

QueryCoordinators

Coord-inator

HiveServer2(Query

Endpoint)

ODBC/JDBC SQL

Queries In-MemoryCache(SharedAcrossAllUsers)

HDFSandCompaYble S3 WASB Isilon

MR vs Tez vs Tez+LLAP

HDFS In-Memorycolumnarcache

Map – Reduce Intermediate results in HDFS

Tez Optimized Pipeline

Tez with LLAP Resident process on Nodes

MaptasksreadHDFS

Tez Tez with LLAP (auto)

Tez with LLAP (all)

allauto

Hive2withLLAP:PreliminaryNumbers

q3 q7 q12 q13 q19 q21 q26 q27 q42 q43 q45 q52 q55 q60 q73 q84 q89 q91 q98

Hive2.0andLLAP:TPC-DSat10TBScale,18Nodes

Hive2.0-Tez

Minquerydme:Query55:2.38s

KeyFeatures:EDWOffload

Ã  ACIDGAforStreamingandSQL:–  50+stabilizaYonfixes.–  TestedatmulY-terabytescalewithsimultaneousingest,deleteandquery.

Ã  BererBIToolCompaYbilitythroughExpandedOLAPCapabiliYes:–  MulYparYYon-by,mulYorder-by.–  OrderbyUDF/UDAF.–  NullorderspecificaYon(nullsfirstornullslast).

Ã  FasterETLwithMoreScalableParYYonLoads:–  2xfasterdynamicparYYonloads.

Ã  ProceduralExtensions(TechPreview):–  Proceduralstructures:loops,if/else.–  Determinemin/maxparYYonvalues.–  CopydatafromexternalsourceslikeFTP.–  SimplifiesETL/dataloadprocesses.

HCatalog Stream Mutation API

ORCORC

Bucket

SQL Compliance

DataTypes SQLFeatures FileFormats FuturesNumeric CoreSQLFeatures Columnar ProceduralExtensions(PL/SQL)

FLOAT/DOUBLE Date,TimeandArithmeYcalFuncYons ORCFile PrimaryKey/ForeignKeyDECIMAL INNER,OUTER,CROSSandSEMIJoins Parquet Non-EquijoinINT/TINYINT/SMALLINT/BIGINT DerivedTableSubqueries Text ScalableCrossProductBOOLEAN Correlated+UncorrelatedSubqueries CSV EnhancedOLAP

String UNIONALL LogfileCHAR/VARCHAR UDFs,UDAFs,UDTFs Nested/Complex ACIDMERGESTRING CommonTableExpressions Avro MulYSubqueryBINARY UNIONDISTINCT JSON Comparisontosub-select

Date,Time AdvancedAnalydcs XML INTERSECTandEXCEPTDATE OLAPandWindowingFuncYons CustomFormatsTIMESTAMP CUBEandGroupingSets OtherFeaturesIntervalTypes NestedDataAnalydcs XPathAnalyYcs

ComplexTypes NestedDataTraversalARRAY LateralViewsMAP ACIDTransacdonsSTRUCT INSERT/UPDATE/DELETEUNION

ApacheHive:JourneytoSQL:2011Analydcs

LegendExisYng

Projected:HDP3.0

Projected:HDP2.5

TrackHiveSQLComplete:HIVE-13554

EnterpriseSparkatScale

ApacheZeppelinGA:TheDataScienceNotebook

Web-baseddatasciencenotebook

InteracYvedataingesYonanddataexploraYon

EasysharingandcollaboraYon

Securewithsinglesign-onandencrypYon

ApacheSpark2.0(TechnicalPreview)

StructuringSpark:DataFrames,DatasetsandStreaming

InteracYvedataingesYonanddataexploraYon

EasysharingandcollaboraYon

Securewithsinglesign-onandencrypYon

DynamicSecurityPoliciesApacheAtlasandRangerIntegradon

ApacheAtlas+Ranger-PowerfulTogether

DynamicMaskingandRowLevelFiltering

Dept SSN CCNo Name DOB MRN PolicyID01 232323233 4539067047629850 JohnDoe 9/12/1969 8233054331 nj23j424

02 333287465 5391304868205600 JaneDoe 9/13/1969 3736885376 cadsd984

RangerPolicyEnforcement

Dept SSN CCNo MRN Name

01 xxxxx3233 4539xxxxxxxxxxxx null JohnDoe

02 xxxxx7465 5391xxxxxxxxxxxx null JaneDoe

Dept SSN Name MRN01 232323233 JohnDoe 8233054331

MarkeYnggroupsseesCCandSSNasmaskedvaluesandMRNisnullified

Deptemployeeonlyseesdataspecifictothatdepartment

TeradataConnector

ApacheKaRa

Expanded Native Connector: Dataset Lineage

CustomAcdvityReporter

MetadataRepository

ApacheAtlasEnablesBusinessCatalogforEaseofUse

Ã  Organizedataassetsalongbusinessterms–  AuthoritaYve:HierarchicalbusinessTaxonomyCreaYon–  Agilemodeling:ModelConceptual,Logical,Physicalassets–  DefiniYonandassignmentoftagslikePII(Personally

IdenYfiableInformaYon)

Ã  Comprehensivefeaturesforcompliance–  MulYpleuserprofilesincludingDataStewardandBusiness

Analysts–  ObjectaudiYngtotrack“Whodidit”–  MetadataVersioningtotrack”whatdidtheydo”

KeyBenefits:EasywaytocreatebusinessTaxonomyUsefulformulYpleusertypesincludingDataStewardandBusinessAnalystsComprehensivefeaturesforcompliance

BusinessCatalog ModelandexploremetadataviathenewBusinessCataloginApacheAtlas

DataSteward

RealTimeApplicadonspoweredbyStormandHBase/Phoenix

What’sNewinStorm

DeveloperProducYvity:Slidingandtumblingwindowingsupport

DeveloperProducYvity:NewconnectorsforsearchandNoSQLDatabase

EnterpriseReadiness:AutomaYcbackpressure

StreamlinedOperaYons:ResourceawareschedulingandStormviewforAmbari

What’sNewinHBaseandPhoenix

DeveloperProducYvity:PhoenixandHiveIntegraYontorunHBASEqueriesinHIVE

EnterpriseReadiness:IncrementalBackup/Restore

EnterpriseReadiness:Performanceboostforhigh-scaleloads

DeveloperProducYvity:AdHocAnalyYcswithconnectortoanyODBCBItool

StreamlinedOperadonsApacheAmbari

StreamlinedOperadonsPhase1:AdvancedMetricsVisualizaYon&Dashboarding

AmbariMetricsSystem

AMBAR I Grafana

Goal:Quicklyunderstandclusterhealthmetricsandkeyperformanceindicators⬢  Capabilides

–  CentralizedDashboardingfocusingoncomponentHealth&Performance

–  Ad-HocGraphCreaYon

⬢  Pre-BuiltDashboards–  HDFS–  YARN–  HBase

⬢  CoreTechnologies–  AmbariMetricsSystem–  Grafana

Ambarinowincludespre-builtdashboardsforvisualizingthemostimportantclusterhealth.

StreamlinedOperadonsPhase2:ConsolidatedClusterAcYvityReporYng

Goal:Quicklyvisualizeandreportonhowbusinessusersandtenantsareusingthecluster,top10queue’s,users,mostBmeconsumingjobs⬢  Capabilides

–  TopKAcYvityReporYng–  Chargeback

⬢  ServicesCovered–  YARN–  MapReduce–  Hive/Tez–  Spark–  HDFS

⬢  CoreTechnologies–  HortonworksSmartSense–  ApacheZeppelin

SmartSenseAMBAR I

AmbariMetricsSystem

Zeppelin

AcdvityExplorer:ClusterUdlizadonRepordng

Preview:StreamlinedOperadonsInvestments

AMBAR I Log

Search

Phase3:Centralized&ContextualLogSearch

Goal:Whenissuesarise,beabletoquicklyfindissuesacrossallHDPcomponents⬢  Capabilides

–  RapidSearchofallHDPcomponentlogs–  SearchacrossYmeranges,loglevels,andforkeywords

⬢  CoreTechnologies:–  ApacheAmbari–  ApacheSolr–  ApacheAmbariLogSearch

HortonworksDataCloud

Abstract: Governance and Security in Cloud Today’stransportaYonmarketplaceiscompeYYveandquicklyevolving.Ouen,unexpectedregulaYonscanposeaseriousrisktooperaYonsandtheboromline.WithHortonworksDataCloud(HDC),we’llshowhowtogainagilityinadapYngtonewchallengesthatcanturnproblemsintoopportuniYes.•  QuicklyprovisionanewanalyYccloudenviroment•  ClassifyandTagassetstofindandunderstandyourdata•  SecurityandAuditsservicetomeetcompliancerequirements

Learn More

http://hortonworks.github.io/hdp-aws/index.html

HortonworksHDBPoweredbyApacheHAWQ

WhatisHDB/ApacheHAWQ?

Hadoop-native SQL query engine and advanced analytics MPP database

that offers high-performance interactive ANSI SQL query execution

and machine learning for Data Analysts & Data Scientists who want to find insights from large/complex

datasets.

HORTONWORKS

HDB powered by Apache HAWQ

HortonworksHDBPoweredByApacheHAWQ

1.  Interactive query performance •  Queryperformanceinseconds •  Compatible with any ANSI SQL compliant BI Tool •  Largernumberofconcurrentusers

2.  MADlib big data Machine Learning in SQL for data scientists and data analysts •  Classification e.g. predict loan default •  Regression e.g. predict value of a sale •  Clustering e.g. marketing campaign segmentation, …

3.  Data federation using HAWQ Extension Framework •  SQL queries against otherdatasources

BI Tool X

BI Tool Y

BI Tool Z

HDP HORTONWORKS DATA PLATFORM

HORTONWORKS

SQL-89 SQL-92 SQL-2003

Advanced Analytics Performance

Exceptional MPP performance, low latency, high scalability, ACID reliability,

fault tolerance

Most Complete Language Compliance

Higher degree of SQL compatibility, SQL-92, 99, 2003, OLAP, leverage

existing SQL skills

Best-in-class Query Optimizer

Maximize performance and do advanced queries with confidence

Elastic Architecture for Scalability

Scale-up/down or scale-in/out, expand/shrink clusters on the fly

Tightly integrated w/MADlib Machine

Learning Advanced MPP analytics, data science at

scale, directly on Hadoop data

HDB/HAWQAdvantages

NewinHDF2.0

NewFeaturesofHDF2.0Ã EnterpriseproducYvityviastreamlinedoperaYons

– AmbariIntegraYonofApacheNiFi,Kava,Storm– ApacheRangerauthorizaYon– Modernized,moreintuiYveUI– MulY-tenancyofdataflows

Ã 170+processors,30%morethaninApacheNiFi1.0

Ã EdgeintelligencewithApacheMiNiFiÃ  IncreasedsecurityopYonswithApacheKava0.10

Ã 10XstreaminganalyYcsperformance,windowingandproducYvitytoolswithApacheStorm1.0

AmbariIntegradon

ComprehensiveStorm-AmbariViews

Muld-tenantAuthorizadon

ReadPermission

Muld-tenantAuthorizadon

NOReadPermission(talkaboutlevels,whereyoucanassignpermissions)

HDF2.0has170+Processors,30%IncreasefromHDF1.2

Extract

Duplicate

GeoEnrich

Replace

ConvertSplit

Translate

RouteContent

RouteContext

RouteText

ControlRate

DistributeLoad

GenerateTableFetch

JoltTransformJSON

PrioridzedDelivery

Encrypt

Evaluate

Execute

Syslog

AllApacheprojectlogosaretrademarksoftheASFandtherespecYveprojects.

EdgeIntelligencewithApacheMiNiFi

Ã  GuaranteeddeliveryÃ  Databuffering

‒  Backpressure‒  Pressurerelease

Ã  PrioriYzedqueuingÃ  FlowspecificQoS

‒  Latencyvs.throughput‒  Losstolerance

Ã  Dataprovenance

Ã  Recovery/recordingarollinglogoffine-grainedhistory

Ã  DesignedforextensionrÃ  SmallFootprint(~40MB)r

KeyFeatures

NewStreamProcessingFeaturesHDF2.0

Ã  NewStormConnectors

Ã  Storm-KavaSpoutusingnewclientAPIs

Ã  StormDistributedLogSearch

Ã  StormDynamicWorkerProfiling

Ã  KavaGrafanaIntegraYon

Ã  StormGrafanaIntegraYon

Ã  ImprovedNimbusHA

Ã  StormAutomaYcBackPressure

Ã  StormDistributedcache

Ã  StormWindowingandStateManagement

Ã  StormPerformanceimprovements

Ã  ImprovedKavaSASL

Ã  StormTopologyEventinspector

Ã  StormResourceAwareScheduling

Ã  StormDynamicLogLevels

Ã  PacemakerStormDaemon

Ã  KavaRackAwareness

DeveloperProducdvity EnterpriseReadiness OperadonalSimplicity

ThankYou

HDP2.5 Updates

Technology

Transcript of HDP2.5 Updates

General Awareness Updates (December 2014)media.careerlauncher.com.s3.amazonaws.com/acads_download_section/BANk... · General Awareness Updates (December 2014) GA Updates (December

For Updates, visit our Product Updates Page at www

TTR updates

UPDATES ON: BIR SSS PHILHEALTH DOLE SEC Outsourcing …...BIR updates SSS updates PHILHEALTH updates DOLE updates SEC updates ALLOCATION AND NOTIFICATION REQUIREMENT The qualified

Search updates

Geïmplementeerde ICF Updates 2011 2018 - whofic.nl Updates 2011... · 1 Geïmplementeerde ICF Updates 2011 – 2018 Hierna volgende updates m.b.t. de periode 2011 – 2018 die voor

Software Updates

GTKAD9.2 Updates

Targeted Updates

Federated Learningcs294-163/fa19/slides/... · 2019-10-02 · Updates of M(i) Updates of M(i) Updates of M(i) i) Gradient updates for M(i) Gradient updates for M(i) Gradient updates

BU CROCCS UPDATES Issue 5, 2016 BU CROCCS Updates

AG2017 PROGRAM UPDATES AG2017 PROGRAM ... - … · 2017 PROGRAM UPDATES AWA AG CNFNC AWA AG CNFNC 2017 PROGRAM UPDATES AG2017 PROGRAM UPDATES AG2017 PROGRAM UPDATES. Farm to Table

COSUG updates

Licensing & updates

Financial Updates Quarterly Highlights Construction Updates

PCIT Training UPDATES Center UPDATES · 2017-06-30 · 9/4/2014 1 UPDATES UPDATES UPDATES PCIT Training Center Susan Timmer, Ph.D. UC Davis PCIT Training Center • First 5 LA- Conference

Nevada Garrett Lee Smith Local Evaluation Updates Local Evaluation Updates Nevada Garrett Lee Smith Local Evaluation Updates Local Evaluation Updates June.

Agile updates

Albion updates

Microsoft SQL Updates Oracle Java JRE Updates