ID2223 Lecture 2: Distributed ML and Linear …...[Distributed Machine Learning with Apache Spark,...

ID2223 Lecture 2: Distributed ML and Linear Regression

Terminology

•Observations. Entities used for learning/evaluation

•Features. Attributes (typically numeric) used to represent an observation

•Labels. Values/categories assigned to observations

•Model. Parameters/weights that are adjusted by training to predict label(s) given observations.

•Training, Validation, and Test Data. Observations for training and evaluating a learning algorithm

- Training data is given to the algorithm for training

- Validation data is withheld from training and is used to measure the performance of training

- Test data is withheld at train time

2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 2/77

Supervised Learning

2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 3

DataPrediction

SupervisedLearning

Learn from labeled observations.

Labels ‘teach’ the algorithm to learn a mapping from observations to labels

Labelled Observations

Supervised Learning

•Classification. Assign a category to each item

- Categories are discrete

[Image from Sze et al, 2017]

Supervised Learning

•Regression. Predict a real value for each item

- Labels are continuous

- Can define ‘closeness’ when comparing prediction with label

Unsupervised Learning

2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling 6

List of

Data Points

List of

Cluster LabelsUnsupervised

Learning

UnsupervisedLearning

Clustering. Partition observations into homogeneous regionsDimensionality Reduction. Transform an initial feature representation into

a more concise representation

Non-Distributed Representations

Not Compositional

- Nearest Neighbour

- Decision Trees (DTs)

• Random Forests

• Gradient Boosted DTs

- Clustering

2017-11-02 7/58

Regions defined by

learned Prototypes

[Bengio, BayArea DL School, 16]

Parametric Learning

“A learning model that summarizes data with a set of

parameters of fixed size (independent of the number of

training examples) is called a parametric model. No

matter how much data you throw at a parametric model, it

won’t change its mind about how many parameters it

needs.”— Artificial Intelligence: A Modern Approach, page 737

Parametric vs Non Parametric Learning

•Parametric = bounded number of parameters

•Non-parametric = unbounded number of parameters

•Parametric or non-parametric models affect

- the way the learning is stored

- the method for learning

•Possible Combinations:

Supervised Parametric Supervised Non-Parametric

Unsupervised Parametric Unsupervised Non-Parametric

2017-11-02 ID2223, Large Scale Machine Learning and Deep Learning, Jim Dowling

Supervised Parametric Learning

•Imagine a machine where you can input some data, turn some dials and observe its performance

- Output data is a prediction

2017-11-02

[Grokking Deep Learning, Manning ’16]`

Supervised Parametric Learning Algorithm

1. Predict

2. Compare to actual result (true pattern)

3. Adjust the dials (parameters)to improve predictions

2017-11-02

[Grokking Deep Learning, Manning ’16]

Machine Learning Pipeline

Machine Learning Hierarchy of Needs

(Distributed

Deep Learning)

Deep Learning,

RL, Automated ML

Labeled Data, ML Experimentation

B.I. Analytics, Metrics, Aggregates,

Features, Training/Test Data

Reliable Data Pipelines, ETL, Unstructured and

Structured Data Storage, Real-Time Data Ingestion

Analytics

Prediction

Importing Data

•Import the raw data (observations) from some source

- Real-time data (Kafka)

- Data-at-rest (HDFS)

•Different data formats likely

•Data may have duplicate, missing columns, invalid values

- Clean and wrangle the data

•Store the data in a format that is efficient for querying (partitioned)

Clean/Partition Data

Acquire Raw Data

Feature Extraction

•Extract features to represent the observations

- Exploit domain knowledge

•Nearly always want numeric features

•Choice of features is crucial tothe success of the entire pipeline

Feature Extraction

Acquire Raw Data

Supervised Learning

•Train a supervised model using labeled data

•Classification or Regression model

Feature Extraction

Supervised Learning

Acquire Raw Data

Learning

•Q: How do we determine the quality of the model we’ve just trained?

•A: We can evaluate it on test / hold-out data, i.e., labeled data not used for training

•If we don’t like the results, we iterate…

Feature Extraction

Supervised Learning

Evaluation

Acquire Raw Data

Predict

• Once we’re happy with our model,

we can use it to make predictions on

future observations, i.e., data without

a known label

Feature Extraction

Supervised Learning

Evaluation

Predict

Acquire Raw Data

Classification

•Goal: Learn a mapping from observations to discrete labels given a set of training examples (supervised learning)

Classification Examples

•Spam Classification

- Emails → {spam, ham}

•Anomaly detection

- Network activity → {anomoly, not anomoly}

•Fraud detection

- Shopping activity → {fraud, not fraud}

•Clickthrough rate prediction

- User viewing an ad → {click, no click}

Classification Example – 20 Newsgroups

•1000s of documents from 20 Usenet Newsgroups

- alt.atheism, soc.religion.Christian, talk.politics.guns

- comp.sys.ibm.pc.hardware, comp.sys.mac.hardware

•Train a model to classify documents from the 20 Newsgroups data set into two categories according to whether or not the documents are computer related.

•Goal: classify each document as:IS_COMPUTERS or NOT_COMPUTERS.

Classification Pipeline – Newsgroups20

•Raw data consists of a set of labeled training observations

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict>

>>I have been at a shooting range where

>>gang members were "practicing"

shooting.

In article <C5qsBF.IEK@ms.uky.edu>

billq@ms.uky.edu (Billy Quinn) writes:

>I built a little project using the radio shack

5vdc relays to switch

Classification Pipeline

Observation Labeltalk.politics.guns/54279 NOT_COMPUTERS

sci.electronics/53909 IS_COMPUTERS

…. …..

In article <C5qsBF.IEK@ms.uky.edu>

billq@ms.uky.edu (Billy Quinn) writes:

>I built a little project using the radio shack

5vdc relays to switch

shooting.

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

•Feature extraction usually converts each observation into a vector of real numbers (features)

•Choosing a good description for an observation will have huge baring on the success or failure of a classifier.

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

shooting.

7830 7831 7832 7833 7834 7835 7836 7837 7838 7839

case class newsgroupsCaseClass

(id: String, text: String, topic: String)

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

•Show only documents that are related to computer topics

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

•Supervised Learning: Train classifier using training data

- Common classifiers include Logistic Regression, SVMs, Decision Trees, Random Forests, etc.

•Training (especially at scale) often involves iterative computations, e.g., gradient descent

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

shooting.

7830 7831 7832 7833 7834 7835 7836 7837 7838 7839

Classifier

Logistic Regression

•Goal: Find linear decision boundary

- Parameters to learn are feature weights and offset

- Nice probabilistic interpretation

- Covered in more detail later in course

2017-11-02 28/77

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

[Distributed Machine Learning with Apache Spark, Berkeley ‘16 ]

Evaluation

•How can we evaluate the quality of our classifier?

•We want good predictions on unobserved data

- ’Generalization’ ability

•Accuracy on training data is overly optimistic since classifier has already learned from it

- We might be ‘overfitting’

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

Overfitting and Generalization

•Fitting training data does not guarantee generalization, e.g., lookup table

•Which figure below is a better classifier?

- Left: perfectly fits training samples, but it is complex and overfits the data

•Favour simple models over complex ones, ceteris paribus

2017-11-02 30/77

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

•How can we evaluate the quality of our classifier?

•Idea: Create test set to simulate unobserved data

•Evaluation: Split dataset into training / testing datasets

- Various ways to compare predicted and true labels

- Evaluation criterion is called a ‘loss’ function

- Accuracy (or 0-1 loss) is common for classification

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

•Split data set into separate training (70%) and test (30%) data sets

2017-11-02 32/77

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

•Predict: Final classifier can then be used to make predictions on future observations

2017-11-02 33/77

Acquire Raw Data

Feature Extraction

Supervised Learning

Evaluation

Predict

Linear Regression

Regression

•Goal: Learn a mapping from observations (features) to continuous values/labels given a training set (supervised learning)

Linear Least Squares Regression

•For each observation we have a feature vector, x, and label, y

𝐱𝑇 = 𝑥1 𝑥2⋯𝑥𝑑

•We assume a linear mapping between features and label:

𝑦 ≈ 𝑤𝑜 + 𝑤1𝑥1 +⋯𝑤𝑑−1𝑥𝑑−1

Linear Least Squares Regression

•We can augment the feature vector to incorporate offset:

𝐱𝑇 = 1 𝑥1 𝑥2⋯𝑥𝑑

•We can then rewrite this linear mapping as a scalar product:

𝑦 ≈ 𝑦 =

𝑖=0

𝑤𝑖𝑥𝑖 =𝐰𝑇𝐱

Least Squares Regression

•Given n training samples with d features, we define:

𝐗 ∈ ℝ𝒏×𝒅 : matrix storing points

𝐲 ∈ ℝ𝒏 : real-valued labels

𝒚 ∈ ℝ𝒏 : predicted labels, where 𝒚 = 𝐗𝐰

𝐰 ∈ ℝ𝒅 : model parameters

Evaluating Predictions

•What is an appropriate evaluation metric or ‘loss’ function?

•Absolute loss:𝑦 − 𝑦

•Squared loss:𝑦 − 𝑦 2

Learning a Model an Optimization Problem

•Assume we have n training points, where x(i)

denotes the ith point

•Idea: Find the model w that minimizes squared loss over the training points:

min𝐰

𝑖=1

𝐰𝑇𝐱 𝒊 − 𝑦 𝒊 2

where 𝑦 = 𝐰𝑇𝐱 𝑖

Least Squares Regression

•Least Squares Regression: Learn the mapping (w) from features to labels that minimizes residual sum of squares:

min𝐰

𝐗𝐰 − 𝐲 22

•Equivalent by definition of the Euclidean norm:

min𝐰

𝑖=1

𝐰𝑇𝐱 𝒊 − 𝑦 𝒊 2

Closed Form Solution

•Solve by setting derivative to zero

- (left as exercise)

- Hint: find the minimum by differentiating the loss (error) function and setting to zero. Then you will need two linear algebra steps.

•Solution:

w = ( XT X )−1 XT y

( XT X )−1 should not be zero.

Overfitting and Generalization

•We want good predictions on new data, i.e., ’generalization’

•Least squares regression minimizes training error, and could overfit

- Simpler models are more likely to generalize (Occam’s razor)

- Can we change the problem to penalize for model complexity?

- Intuitively, models with smaller weights are simpler

•Can we penalize overly complex models?

•Ridge Regression does this by adding a regularization term

Ridge Regression

•Ridge Regression: Learn mapping (w) that minimizes the residual sum of squares along with a regularization term:

min𝐰

𝐗𝐰 − 𝐲 22 + 𝜆 𝐰 2

Free parameter that trades off training error and model complexity

Training Error Model Complexity

Hyperparameters

𝜆 𝐰 22

•How do find good values for this hyperparameter?

- Need to tune ‘hyperparameters’

•Problem: If we try out different hyperparametervalues and evaluate them using the test set, there is a danger of overfitting the data. Why??

- Problem arises as the test set should reflect unobserved data

•Solution: Use a 2nd hold out dataset to evaluate different tunings for hyperparameters

Evaluating with the Validation Set

•Hyperparameter tuning

- Training: train different models

- Validation: evaluate different models

- Test: evaluate the accuracy of the final model

2017-11-02 46/77[Distributed Machine Learning with Apache Spark, Berkeley ‘16 ]

Hyperparameter Search

•Grid Search: Exhaustively search through hyperparameter space

- Define and discretize search space (linear or log scale)

Evaluation

•Evaluate final model

- Training set: train various models

- Validation set: evaluate various models

- Test set: evaluate final model’s accuracy

2017-11-02 49/77[Distributed Machine Learning with Apache Spark, Berkeley ‘16 ]

Predict

•The trained model can then be used to make predictions on unseen observations

2017-11-02 50/77

Distributed Machine Learning

Computational Complexity - Review

Time Complexity

•The (worst-case) time complexity of a problem is the time complexity of the fastest algorithm that solves the problem.

•Ο(n): there is an algorithm that solves the problem faster than this time

- Has to be true for all possible inputs, that is, the worst-case

for ( i=0 ; i<n ; i++ )

for( j=0 ; j<n ; j++ )

for( k=0 ; k<n ; k++ )

sum[i][j] += entry[i][j][k];

What is the time complexity of the following code snippet?

Orders-of-Growth

Common Orders-of-Growth

•O(1) – constant

•O(log(n)) – logarithmic time

•O(n) – linear

•O(n log(n)) – linearithmic time

•O(n2) – quadratic time

•O(2n) – exponential time

Orders-of-Growth Example

Big-Oh (O)

2017-11-02 57/77

•𝑓 𝑥 = 𝑂 𝑔 𝑥

- ignore constant and lower-order terms

•Big-Oh determines the upper bound on a function

𝑓 𝑛 = 𝑂 𝑔 𝑛 𝑖𝑓𝑓 ∃ 𝑐 > 0 𝑎𝑛𝑑 𝑛0 > 0 𝑤ℎ𝑒𝑟𝑒 𝑓 𝑛 ≤ 𝑐𝑔 𝑛 ∀ 𝑛 ≥ 𝑛0

Space Complexity

•The space complexity of an algorithm is an expression for the worst-case amount of memory that the algorithm will use.

Linear Regression:Computational Complexity

•Can we count the number of arithmetic operations for?

- Assume we have d features and n samples

w = ( XT X )−1 XT y

Linear Regression:Computational Complexity

•If we count the number of arithmetic operations for:

w = ( XT X )−1 XT y

•Cost of operations (n is #samples, d is #features):

- XT X : O(nd2)

- Matrix inverse of XT X: O(d3)

- XT y : O(nd)

- Product XT y and Inverse Matrix: O(d2)

•The Time Complexity is:O(nd2 + d3)

Ignore

lower-order

Linear Regression: Space Complexity

•Storage costs:XTX and its inverse: O(d2) floatsX : O(nd) floats

•Space complexity: O(nd + d2) floats

Big n and Small d

•Assume O(d3) computation and O(d2) memory is feasible on a single machine

•Storing X and computing XTX are the bottlenecks

•How can we distribute storage and computation?

- Partition rows of X over hosts

- Compute XTX as a sum of outer products

Matrix Multiplication using Inner Products

•Each entry of output matrix is result of inner product of inputs matrices

3 4 62 3 1

∙2 31 −14 5

=34 3511 8

3 × 2 + 4 × 1 + 6 × 4 = 34

3 4 62 3 1

∙2 31 −14 5

=34 3511 8

3 × 3 + 4 × −1 + 6 × 5 = 35

3 4 62 3 1

∙2 31 −14 5

=34 3511 8

2 × 2 + 3 × 1 + 1 × 4 = 11

3 4 62 3 1

∙2 31 −14 5

=34 3511 8

2 × 3 + 3 × −1 + 1 × 5 = 8

Matrix-Matrix Multiplication

•Output matrix is the sum of outer products between corresponding rows and columns of input matrices

3 4 62 3 1

∙2 31 −14 5

6 94 6

Matrix Multiplication via Outer Products

3 4 62 3 1

∙2 31 −14 5

6 94 6

+4 −43 −3

3 4 62 3 1

∙2 31 −14 5

=34 3511 8

6 94 6

+4 −43 −3

+24 304 5

=34 3511 8

Computational Complexity with Map-Reduce

Big n and Big d

•With big d, storing and computing are still the bottlenecks

•O(d3) computation in the reduce step is not easily parallelized

Big n and Big d not Computationally Feasible

•Distributing storage and processing doesn’t change the cubic computational complexity

•Options to scale include:

1. Exploit data sparsity to reduce dimensionalityor

2. Make computation (and storage) linear in (n, d)

Iterative Algorithms are more Efficient

•We need methods that are linear in time and space

•Gradient descent is an iterative algorithm that requires O(nd) computation and O(d) local storage per iteration

Gradient Descent

2017-11-02 76/77

References

•Distributed Machine Learning with Apache Spark, UCLA/Berkeley Course 2016

•Andrew Ng, Machine Learning CS 229, Stanford

•Deep Learning Book, Section 4.5

• Sze et Al, “Efficient Processing of Deep Neural Networks:

A Tutorial and Survey”, https://arxiv.org/abs/1703.09039

ID2223 Lecture 2: Distributed ML and Linear …...[Distributed Machine Learning with Apache Spark,...

Documents

Transcript of ID2223 Lecture 2: Distributed ML and Linear …...[Distributed Machine Learning with Apache Spark,...

Distributed machine learning examples

Distributed Numerical and Machine Learning Computations ...

Linear Encoders for Numerically Controlled Machine Tools · Linear encoders for numerically controlled machine tools Linear encoders from HEIDENHAIN for numerically controlled machine

Distributed lag non-linear models in R

Distributed Lasso for In-Network Linear Regression

03 Machine Learning Linear Algebra

LINEAR SEQUENTIAL MACHINE AND REDUCTION OF LINEAR SEQUENTIAL MACHINE

Basic Machine Learning: Linear Models

Distributed Computation of Linear Matrix Equations: An ...

Distributed Private Machine Learninghelper.ipam.ucla.edu/publications/pbd2018/pbd2018_14607.pdf · Distributed Private Machine Learning Abhradeep Guha Thakurta University of California

Distributed Machine Learning: An Intro. - UESTCdm.uestc.edu.cn/wp-content/uploads/seminar/Distributed ML.pdf · Distributed Machine Learning Frameworks 36 MapReduce ‒Synchronous

Distributed Inference for Linear Support Vector Machinejmlr.csail.mit.edu/papers/volume20/18-801/18-801.pdf · Keywords: Linear support vector machine, distributed inference, Bahadur

Distributed Machine Learning · Distributed Machine Learning libraries •Exploit distributed computing engines to provide fault tolerant and scalable implementations for most commonly

Distributed Inference for Linear Support Vector Machine · for linear support vector machine (SVM) for the binary classi cation task. Despite a vast literature on SVM, much less is

Machine learning‐based distributed model predictive ...

Scaling Distributed Machine Learningmuli/file/mu_defense_animation.pdf · Hadoop/Spark BSP data ... Scaling Distributed Machine Learning Large Scale OptimizationDistributed Systems

Distributed Lag Linear and Non-Linear Models in R: The Package … · Distributed lag non-linear models (DLNMs) represent a modeling framework to exibly describe associations showing

Distributed Inference for Linear Support Vector Machine

Linear Sequential Machines1 LINEAR SEQUENTIAL MACHINE AND REDUCTION OF LINEAR SEQUENTIAL MACHINE.

Scaling Distributed Machine Learningmuli/file/mu_defense.pdf · 2017-02-05 · Scaling Distributed Machine Learning Large Scale OptimizationDistributed Systems for machine learning