Data Science and Big Data Analytics

Chap 3: Data Analytics Using R

Charles TappertSeidenberg School of CSIS, Pace

University

Chap 3 Data Analytics Using R

This chapter has three sections1. An overview of R2. Using R to perform exploratory data

analysis tasks using visualization3. A brief review of statistical inference

Hypothesis testing and analysis of variance

3.1 Introduction to R

Generic R functions are functions that share the same name but behave differently depending on the type of arguments they receive (polymorphism)

Some important functions used in chapter (most are generic)

head() displays first six records of a file summary() generates descriptive statistics plot() can generate a scatter plot of one variable against

another lm() applies a linear regression model between two variables hist() generates a histogram help() provides details of a function

3.1 Introduction to RExample: number of orders vs

sales

lm(formula = (sales$sales_total ~ sales$num_of_orders)

intercept = -154.1 slope = 166.2

3.1 Introduction to R

3.1.1 R Graphical User Interfaces Getting R and RStudio

3.1.2 Data Import and Export Necessary for project work

3.1.3 Attributes and Data Types Vectors, matrices, data frames

3.1.4 Descriptive Statistics summary(), mean(), median(), sd()

3.1.1 Getting R and RStudio

Download R and install (32-bit and 64-bit) https://www.r-project.org/

Download RStudio and install https://www.rstudio.com/products/RStudio/#

Desk

3.1.1 RStudio GUI

3.2 Exploratory Data Analysis

Scatterplots show possible relationships

x <- rnorm(50) # default is mean=0, sd=1y <- x + rnorm(50, mean=0, sd=0.5)plot(y,x)

3.2 Exploratory Data Analysis

3.2.1 Visualization before Analysis 3.2.2 Dirty Data 3.2.3 Visualizing a Single Variable 3.2.4 Examining Multiple Variables 3.2.5 Data Exploration versus

Presentation

3.2.1 Visualization before Analysis

Anscombe’s quartet – 4 datasets, same statistics

should be x

3.2.1 Visualization before Analysis

Anscombe’s quartet – visualized

3.2.1 Visualization before Analysis

Anscombe’s quartet – Rstudio exercise

Enter and plot Anscombe’s dataset #3 and obtain the linear regression line

More regression coming in chapter 6

)

x <- 4:14xy <- c(5.39,5.73,6.08,6.42,6.77,7.11,7.46,7.81,8.15,12.74,8.84)ysummary(x)var(x)summary(y)var(y)plot(y~x)lm(y~x)

3.2.2 Dirty DataAge Distribution of bank account

holders

What is wrong here?

3.2.2 Dirty DataAge of Mortgage

What is wrong here?

3.2.3 Visualizing a Single Variable

Example Visualization Functions

3.2.3 Visualizing a Single Variable

Dotchart – MPG of Car Models

3.2.3 Visualizing a Single Variable

Barplot – Distribution of Car Cylinder Counts

3.2.3 Visualizing a Single Variable

Histogram – Income

3.2.3 Visualizing a Single Variable

Density – Income (log10 scale)

In this case, the log density plot emphasizes the log nature of the distribution

The rug() function at the bottom creates a one-dimensional density plot to emphasize the distribution

3.2.3 Visualizing a Single Variable

Density – Income (log10 scale)

3.2.3 Visualizing a Single Variable

Density plots – Diamond prices, log of same

3.2.4 Examining Multiple Variables

Examining two variables with regression

Red line = linear regression Blue line = LOESS curve fit

x

3.2.4 Examining Multiple Variables

Dotchart: MPG of car models grouped by cylinder

3.2.4 Examining Multiple Variables

Barplot: visualize multiple variables

3.2.4 Examining Multiple Variables

Box-and-whisker plot: income versus region

Box contains central 50% of dataLine inside box is median valueShows data quartiles

3.2.4 Examining Multiple Variables

Scatterplot (a) & Hexbinplot – income vs education

The hexbinplot combines the ideas of scatterplot and histogramFor high volume data hexbinplot may be better than scatterplot

3.2.4 Examining Multiple Variables

Matrix of Scatterplots

3.2.4 Examining Multiple Variables

Variable over time – airline passenger counts

Data visualization for data exploration is different from presenting results to stakeholders Data scientists prefer graphs that are

technical in nature Nontechnical stakeholders prefer simple,

clear graphics that focus on the message rather than the data

3.2.5 Exploration vs Presentation

3.2.5 Exploration vs Presentation

Density plots better for data scientists

3.2.5 Exploration vs Presentation

Histograms better to show stakeholders

Model Building What are the best input variables for the model? Can the model predict the outcome given the

input? Model Evaluation

Is the model accurate? Does the model perform better than an obvious

guess? Does the model perform better than other models?

Model Deployment Is the prediction sound? Does model have the desired effect (e.g., reducing

cost)?

3.3 Statistical Methods for Evaluation

Statistics helps answer data analytics questions

3.3.1 Hypothesis Testing 3.3.2 Difference of Means 3.3.3 Wilcoxon Rank-Sum Test 3.3.4 Type I and Type II Errors 3.3.5 Power and Sample Size 3.3.6 ANOVA (Analysis of Variance)

3.3 Statistical Methods for EvaluationSubsections

Basic concept is to form an assertion and test it with data

Common assumption is that there is no difference between samples (default assumption)

Statisticians refer to this as the null hypothesis (H0)

The alternative hypothesis (HA) is that there is a difference between samples

3.3.1 Hypothesis Testing

3.3.1 Hypothesis TestingExample Null and Alternative

Hypotheses

3.3.2 Difference of MeansTwo populations – same or different?

Student’s t-test Assumes two normally distributed

populations, and that they have equal variance

Welch’s t-test Assumes two normally distributed

populations, and they don’t necessarily have equal variance

3.3.2 Difference of MeansTwo Parametric Methods

Makes no assumptions about the underlying probability distributions

3.3.3 Wilcoxon Rank-Sum Test

A Nonparametric Method

An hypothesis test may result in two types of errors Type I error – rejection of the null

hypothesis when the null hypothesis is TRUE

Type II error – acceptance of the null hypothesis when the null hypothesis is FALSE

3.3.4 Type I and Type II Errors

3.3.4 Type I and Type II Errors

The power of a test is the probability of correctly rejecting the null hypothesis

The power of a test increases as the sample size increases

Effect size d = difference between the means

It is important to consider an appropriate effect size for the problem at hand

3.3.5 Power and Sample Size

3.3.5 Power and Sample Size

A generalization of the hypothesis testing of the difference of two population means

Good for analyzing more than two populations

ANOVA tests if any of the population means differ from the other population means

3.3.6 ANOVA (Analysis of Variance)