Based on the book “Building Data Mining Applications for CRM” By Alex Berson Stephen Smith Kurt...

Based on the book “Building Data Mining Applications for CRM”

By

Alex Berson

Stephen Smith

Kurt Thearling

Data Mining Applications for CRM

Data Mining Applications for CRM

Summary of Topics1. Customer Relationship Management-Framework and Architecture2. Reinforcing CRM with Data Mining 3. Data Mining –An Overview4. Key Terms5. Data Mining Methodology 6. Classical Techniques: Statistics, Neighborhoods, and Clustering7. Next Generation Techniques: Trees, Networks, and Rules8. CRM -The Business Perspective9. Deploying Data Mining for CRM10. Data Quality11. Next Generation of Information Mining and Knowledge

Discovery for Effective CRM12. CRM in the e-Business World

Topic 1: Customer Relationship Topic 1: Customer Relationship Management-Framework and Management-Framework and

ArchitectureArchitecture CRM is an enterprise approach to customer service that uses

meaningful communication to understand and influence consumer behavior. The purpose of the process is twofold:

a: To impact all aspects to the consumer relationship (e.g., improve customer satisfaction, enhance customer loyalty, and increase profitability) and

B: To ensure that employees within an organization are using CRM tools. The need for greater profitability requires an organization to proactively pursue its relationships with customers.

Customer Relationship Management-Customer Relationship Management-Framework and ArchitectureFramework and Architecture

Which customers are most profitable to me? Why? What promotions are most effective? For which customers? What kind of customers will be interested in my new product? What customers are at risk to defect to my competitor? How do I identify prospects with the greatest profit potentials?

Customer information is rapidly becoming a company’s most

important asset to answer these questions. However, to answer these

questions in broad generalities is not enough. Each customer must be

analyzed and potentially treated uniquely. Customer Relationship

Management provides the framework for analyzing customer

profitability and improving marketing effectiveness.

Customer Relationship Customer Relationship Management Management -Framework and -Framework and

ArchitectureArchitectureMany organizations have collected and stored a wealth of data about their customers, suppliers, and business partners. However, the inability to discover valuable information hidden in the data prevents these organizations from transforming this data into knowledge. The business desire is, therefore, toextract valid, previously unknown, and comprehensible information from large databases and use it for profits. To fulfill these goals, organizations need to follow these steps:

- Capture and integrate both the internal and external data into a comprehensive view that encompasses the whole organization.- “Mine” the integrated data for information.- Organize and present the information with knowledge for decision-making.

Data, Information, and Data, Information, and DecisionDecision

Data Resource Management (DRM)

MIS (OLTP) & OOAD

KM (Knowledge Mgt), KWS (Knowledge Work Systems)

DSS; ESS, EIS (Executive Information Systems)

Data Warehousing/Data Mart/Data Mining/OLAP (Executive, Collaborative and individual levels)

Business Intelligence

Data

Information (Data + Process)

Knowledge/Business Intelligence

Decision (Information +

Knowledge)

Data/Information/Decision /Business Intelligence

Customer Relationship ManagementCustomer Relationship Management ----Framework Framework and Architectureand Architecture

From the architecture point of view, the entire CRM framework can

be classified into three key components:

Operational CRM – The automation of horizontally integrated business processes, including customer touch-points, channels, and front-back office integration.

Analytical CRM- The analysis of data created by the Operational CRM

Collaborative CRM- Applications of Collaborative services including e-mail, personalized publishing, e-communities, and similar vehicles designed to facilitate interactions between customers and organizations.

CRM ArchitectureCRM Architecture

ETLTools

Data Sources

Market DataStore

CommunicationChannels

Call Center

Call Center

CampaignMgt

Direct Mails

MarketingData Marts

Contact History

External Data

TransactionHistory

Customer ProfileAnd account

AnalyticsData Mart

ReportingData Mart

Decision Support Applications

CampaignMgt

Data MiningAnalytics

ReportingData Mart

Contact Mgt

Customer ServiceCenter

Internet

E-mail

Other

Business Rules and Metadata Management

Workflow Management

Campaign MGT Software-Managing Campaign MGT Software-Managing CampaignsCampaigns

Accommodation of many new touch points besides direct mail, for ex., the Web, direct TV ad., hard copy advertising customer services, street brochure dispatch, and signage.

Focus on profitability (not only on which customer was most profitable, but also on what was the most profitable promotion that could be sent., e.g., send $.025 postcard rather than the $25 rebate if both have the same effect).

Optimization of the sequence of promotion delivery. Tools for constructing experiments that allow the marketing professionals

to test out the effectiveness of new promotions and new segmentation techniques, for ex., using different contents and timing for signage advertising.

Accommodation by the system of predictive modeling from data mining , which provides insights into future customer behavior and future customer profitability.

Web-Enabled Information DeliveryWeb-Enabled Information Delivery

WebServer

QueryEngine

AnalyticsDrill DownAgents

UnstructuredContent

SQL

HTML

CGI

HTML

WebBrowser

StructuredContent

How about the web Log, or “blog” which has become a popular source for information acquisition.

Topic 2:Topic 2: Reinforcing CRM with Data Mining

Companies worldwide are beginning to realize that surviving an intensively competitive and global marketplace requires closer relationships with customers. In turn, enhanced customer relationships can boost profitability three ways: a) by reducing costs by attracting more suitable customers, b) by generating profits through cross-selling and up-selling activities, and c) by extending profits through customer retention. Slightly expanded explanations of these activities follow:

Reinforcing CRM with Data Mining

Attracting more suitable customers: Data mining can help firms understand which customers are most likely to purchase specific products and services, thus enabling businesses to develop targeted marketing programs for higher response rates and better returns on investment.

Better cross-selling and up-selling: Businesses can increase their value proposition by offering additional products and services that are actually desired by customers, thereby raising satisfaction levels and reinforcing purchasing habits.

* Better retention: Data-mining techniques can identify which customers are more likely to defect and why. A company can use this information to generate ideas that allow them to maintain these customers.

DW Technologies and Tools-An DW Technologies and Tools-An OverviewOverview

Data Modeling

Data AcquisitionOLAP

Extraction

SourceSystems

Data Storage

Data Loading

Load Image Creation

Information Delivery

AlertSystems

DataMining

Report Writer

StagingArea

DW/Data Marts

QualityAssurance

Transformation

DW Information FlowDW Information Flow

Data Warehouse DatabaseData Warehouse DatabaseThe central data warehouse database is a cornerstone of data warehousing environment. On the architecture diagram, the database is almost always implemented on the relational database management system (RDBMS) technology. Now the now approaches include the following:

Multidimensional database (MDDBs)- This is tightly coupled with the online analytical processing (OLAP) tools that act as clients to the multidimensional data stores.

An innovative approach to speed up a traditional RDBMs by using new index structures to bypass relational table scans.

Parallel relational database designs that require a parallel computing platforms, for ex., symmetric multiprocessor (SMP), massively parallel processors (MPPs), and or clusters of uni-or multiprocessors.

Information Delivery Tool Information Delivery Tool TaxonomyTaxonomy

Tools are generally divided into five main groups :

Data query and reporting tools.

Application development tools.

Executive Information System (EIS) tools.

Online analytical processing tools.

Data mining tools.

Topic 3:Topic 3: Data Mining: An OverviewData Mining: An Overview

Data mining can help reduce information overload and improve decision making. This is achieved by extracting and refining useful knowledge through a process of searching for relationships and patterns from the extensive data collected by organizations. The extracted information is used to predict, classify, model, and summarize the data being mined. Data-mining technologies, such as rule induction, neural networks, genetic algorithms, fuzzy logic, and rough sets, are used for classification and pattern recognition in many industries.

Data Mining: An OverviewData Mining: An Overview

A supermarket organizes its merchandise stock based on shoppers' purchase patterns.

An airline reservation system uses customers' travel patterns and trends to increase seat utilization.

Web pages alter their organizational structure or visual appearance based on information about the person who is requesting the pages.

Individuals perform a Web-based query to find the median income of households in Iowa.


Data mining builds models of customer behavior by using established statistical and machine-learning techniques. The basic objective is to construct a model for one situation in which the answer or output is known and then apply that model to another situation in which the answer or output is sought. The best applications of the above techniques are integrated with data warehouses and other interactive, flexible business analysis tools. The analytic data warehouse can thus improve business processes across the organization in areas such as campaign management, new product rollout, and fraud detection.


Data mining integrates different technologies to populate, organize, and manage the data store. Because quality data is crucial to accurate results, data-mining tools must be able to clean the data, making it consistent, uniform, and compatible with the data store. Data mining employs several techniques to extract important information. Operations are the actions that can be performed on accumulated data, including predictive modeling, database segmentation, link analysis, and deviation detection.

Taxonomy of Data Mining ToolsTaxonomy of Data Mining Tools

We can divided the entire data mining tool market into three main groups: General-purpose tools, integrated DSS/OLAP/data miningtools, and rapidly growing, application-specific tools.

The General-purpose tools which occupy the larger and more mature segment of the market include the following:

SAS Enterprise Minor IBM Intelligent Minor Unica PRW SPSS Clementine SGI Mineset Oracle Darwin Angoss KnowledgeSeeker

Taxonomy of Data Mining ToolsTaxonomy of Data Mining ToolsThe integrated data mining tool segment addresses a very real and

compelling business requirement of having a single multi-function,

decision-support tool that can provide management reporting, online

analytical processing, and data mining capabilities within a common

framework. Examples of these integrated tools include Cognos

scenario and Business Objects.

The application-specific tools segment is rapidly gaining momentum.

Among these tools are the following:KDI (focuses on retail)

Options & Choices (focuses on insurance industries)

HNC (focuses on fraud detection)

Unica Model 1 (focuses on marketing)

Database Mining Workstation Database Mining Workstation (HNC)(HNC)

HNC is one of the most successful data mining companies. Its Database

Mining workstation (DMW) is a neural network tool that is widely-accepted

For credit card fraud analysis applications. DMW consists of Windows–based

software applications and a custom processing board. Other HNC products

include Falcon and ProfitMax processing applications for financial services,

and the Advanced Telecommunications Abuse Control System (ATACS)

fraud-detection solution that HNC plans to deploy in the Telecommunications

Industries.

Taxonomy of Data Mining ToolsTaxonomy of Data Mining Tools

There are specific tools, for example, for the following applications:

Financial Data Analysis: neural networks have been used in forecasting stock prices, option trading, rating bonds, portfolio management, commodity-price prediction, and mergers and acquisition analysis. Using IBM Intelligent minor, Mellon Bank developed a credit card-attrition model to predict which customers will stop using Mellon’s credit card in the next few months.

Telecommunications Industry: The hyper-competitive nature of the industry has created a need to understand customers, to keep them, to model effective ways to market new products.

Taxonomy of Data Mining ToolsTaxonomy of Data Mining Tools Retail Industry: Retail data mining can help identify customer-

buying behaviors, discover consumer-shopping patterns and trends.

Healthcare and biomedical research: The analysis of large quantities of time-stamped data will provide doctors with important information regarding the progress of the decease. For ex., NeuroMedicalSystems used neural networks to perform a pap smear diagnostic aid.

Science and engineering: To improve its manufacturing process. Boeing has successfully applied machine-learning algorithms to the discovery of informative and useful rules from its plant data.

Data Mining vs. Data WarehouseData Mining vs. Data Warehouse

Major challenge to exploit data mining is identifying suitable data to mine.

Data mining requires single, separate, clean, integrated, and self-consistent source of data.

A data warehouse is well equipped for providing data for mining.

Data quality and consistency is a pre-requisite for mining to ensure the accuracy of the predictive models. Data warehouses are populated with clean, consistent data.


Data Mining does not require that a Data Warehouse be built. Often, data can be downloaded from the operational files to flat files that contain the data ready for the data mining analysis.

Data Mining can be implemented rapidly on existing software and hardware platforms. Data Mining tools can analyze massive databases to deliver answers to questions such as, “ Which customers are most likely to respond to my next promotional mailing, and why?”


Advantageous to mine data from multiple sources to discover as many interrelationships as possible. Data warehouses contain data from a number of sources.

Selecting relevant subsets of records and fields for data mining requires query capabilities of the data warehouse.

Results of a data mining study are useful if there is some way to further investigate the uncovered patterns. Data warehouses provide capability to go back to the data source.

Data Mining vs. OLAP

They are two separate breeds of analysis with

entirely different objectives, not to mention

tools, skill sets, and implementation methods.


With canned reports, ad hoc querying, and OLAP, the

end user defines a hypothesis and determines which data

to examine. With data mining, the tool identifies the

hypothesis, and it actually tells the user where in the data

to start the exploration process.


Rather than using SQL to filter out values and methodically

reduce the data into a concise answer set, data mining uses

algorithms that exhaustively review the relationships among

data elements to determine if any patterns exist. The whole

purpose of data mining is to yield new business information

that a business person can act on.

OLAP vs. Data Mining ToolsOLAP vs. Data Mining Tools

Are ad hoc, shrink wrapped tools that provide an interface to data

Are used when you have specific known questions

Looks and feels like a spreadsheet that allow rotation, slicing and graphic

Can be deployed to large

number of users

Methods for analyzing multiple data types

-- Regression Trees -- Neural networks -- Genetic algorithms

Are used when you don’t know what the questions are

Usually textual in nature

Usually deployed to a small number of analysts

OLAP Tools Data Mining Tools

Topic 4: Key TermsTopic 4: Key Terms

Application Service Providers:

Offer outsourcing solutions that supply, develop, and manage application specific software and hardware so that customers' internal information technology resources can be freed up.

Business Intelligence:

The type of detailed information that business managers need for analyzing sales trends, customers' purchasing habits, and other key performance metrics in the company.

Key TermsKey Terms

Categorical Data:

Fits into a small number of distinct categories of a discrete nature, in contrast to continuous data, and can be ordered (ordinal), for example, high, medium, or low temperatures, or nonordered (nominal), for example, gender or city.

Classification:

The distribution of things into classes or categories of the same type, or the prediction of the category of data by building a model based on some predictor variables.

Key TermsKey Terms

Clustering:

Groups of items that are similar as identified by algorithms. For example, an insurance company can use clustering to group customers by income, age, policy types, and prior claims. The goal is to divide a data set into groups such that records within a group are as homogeneous as possible and groups are as heterogeneous as possible. When the categories are unspecified, this may be called unsupervised learning.

Genetic Algorithm:

Optimization techniques based on evolutionary concepts that employ processes such as genetic combination, mutation, and natural selection in a design.

Key TermsKey Terms Online Profiling:

The process of collecting and analyzing data from Web site visits, which can be used to personalize a customer's subsequent experiences on the Web site. Network advertisers, for example, can use online profiles to track a user's visits and activities across multiple Web sites, although such a practice is controversial and may be subject to various forms of regulation.

Rough Sets:

A mathematical approach to extract knowledge from imprecise and uncertain data.

Key TermsKey Terms Rule Induction:

The extraction of valid and useful if-then-else rules from data based on their statistical significance levels, which are integrated with commercial data warehouse and OLAP platforms.

Visualization:

Graphically displayed data from simple scatter plots to complex multidimensional representations to facilitate better understanding.

Topic 5: Data Mining MethodologyTopic 5: Data Mining MethodologyThe methodology used today in data mining, when it is well thought

out and well executed, consists of just a few very important concepts.

Finding a pattern in the data and building a model. In general, it means any sequence or pattern of data that occurs more often than one would it to if it were a random event.

Sampling or not having to use all of the data in order to make significant conclusions about what might be happening with other parts of the data.

Validating the predictive models that arise out of data mining algorithm. Finally, coming down to finding the pattern or model that is the beat.

The four parts of data mining technology –patterns, sampling, validation,

and choosing the model.

Pattern and ModelPattern and ModelPattern: An event or combination of events in a database that occurs more often than expected. Typically, this means that its actual occurrence is significantly different than what would be by random chance. (for ex., 121212…?

Model: A description that adequately explains and predicts relevant data but is generally much smaller than the data itself. For real-world applications, a model can be anything from a mathematical Equation, to a set of rules that describes customer segments, to the computer representation of a complex neural network architecture, which translates to several sets of mathematical equations.

Predictive model: A model created or used to perform prediction. In contrast to models created solely for pattern detection, exploration or general organization of the data.

Explanatory: For every increase in 1 % in the interest,auto sales decrease by 5 %.

Predictive: predictions about future buyer behavior.

Traditional DW

Operational

OLAP

(OLTP)

Data Mining

Descriptive: The dealer sold 200 cars last month.

Types Of ModelsTypes Of Models

A high-level View of Modeling A high-level View of Modeling ProcessProcess

HistoricalData

PredictionRecord ???

Model

Model Building

123

The Needs for SamplingThe Needs for Sampling

Containing costs

Speeding up the data gathering

Improving effectiveness

Reducing bias

Sampling DesignSampling Design

Four steps:

Determine the data to be collected or described

Determine the population to be sampled

Choose the type of sample

Decide on the sample size

Two Types of Data Mining Modeling- Two Types of Data Mining Modeling- Verification and DiscoveryVerification and Discovery

The verification model utilizes a process that looks in a database to detect trends and patterns in data that will help answer some specific questions about the business.

In this mode, the user generates a hypothesis about the data, issues a query against the data and examines the results of the query looking for verification of the hypothesis or the user decides that the hypothesis is not valid.

Verification ModelVerification Model

In this model, very little information is created in this extraction process: either the hypothesis is verified or it is not.

Common tools used in this mode are: queries, multidimensional analysis and visualization. What all have in common are that the user is essentially ‘guiding’ the exploration of the data being inspected.

Discovery ModelDiscovery Model

A more popular model is the Discovery Model that utilizes a process that looks in a database to discover and/or predict future patterns. The discovery model is divided into two modes: “Descriptive” and “Predictive”.

Discovery Model- Descriptive ModeDiscovery Model- Descriptive Mode

The Descriptive mode finds hidden patterns without a predetermined idea or hypothesis about what the patterns may be. In other words, the Data Mining software or program takes the initiative in finding what the interesting patterns are, without the user thinking of the relevant questions first. In this mode information is created about the data with very little or guidance from the user. The exploration of the data is done in such a way as to yield as large a number of useful facts about the data in the shortest amount of time.

Discovery Model- Predictive ModeDiscovery Model- Predictive Mode

In the Predictive mode patterns discovered from the database are used to predict the future patterns or trends. Predictive modeling allows the user to submit records with some unknown field values, and the system will guess the unknown values based on previous patterns discovered from the database.

In comparing the two models, one can state that “Verification” can be very inefficient, timely and costly. Whereas, “Discovery” modeling can be very efficient, cost effective, less dependent on user input and increases modeling accuracy.

Predictive ModellingPredictive Modelling

Similar to the human learning experience– uses observations to form a model of the important

characteristics of some phenomenon.

Uses generalizations of ‘real world’ and ability to fit new data into a general framework.

Can analyze a database to determine essential characteristics

(model) about the data set.


Model is developed using a supervised learning approach, which has two phases: training and testing.

– Training builds a model using a large sample of historical data called a training set.

– Testing involves trying out the model on new, previously unseen data to determine its accuracy and physical

performance characteristics.


Applications of predictive modelling include customer retention management, credit approval, cross selling, and direct marketing.

Two techniques associated with predictive modelling:

A. classification

B. value prediction, distinguished by nature of the variable being predicted.

Predictive Modelling - Predictive Modelling - ClassificationClassification

Used to establish a specific predetermined class for each record in a database from a finite set of possible, class values.

Two specializations of classification: tree induction and neural induction.

Example of Classification using Example of Classification using Tree InductionTree Induction

Example of Classification using Example of Classification using Tree InductionTree Induction

Customer renting property> 2 years

Rent property

Customer age>45

No Yes

No Yes

Rent property

Buy property

Example of Classification using Example of Classification using Neural InductionNeural Induction

Example of Classification Using Example of Classification Using Neural InductionNeural Induction

Each processing unit (circle) in one layer is connected to each processing unit in the next layer by a weighted value, expressing the strength of the relationship. The network attempts to mirror the way the human brain works in recognizing patterns by arithmetically combining all the variables with a given data point.

In this way, it is possible to develop nonlinear predictive models that ‘learn’ by studying combinations of variables and how different combinations of variables affect different data sets.

Predictive Modelling - Value Predictive Modelling - Value PredictionPrediction

Used to estimate a continuous numeric value that is associated with a database record.

Uses the traditional statistical techniques of linear regression and non-linear regression.

Relatively easy-to-use and understand.


Linear regression attempts to fit a straight line through a plot of the data, such that the line is the best representation of the average of all observations at that point in the plot.

Problem is that the technique only works well with linear data and is sensitive to the presence of outliers (i.e.., data values, which do not conform to the expected norm).


Although non-linear regression avoids the main problems of linear regression, still not flexible enough to handle all possible shapes of the data plot.

Statistical measurements are fine for building linear models that describe predictable data points, however, most data is not linear

in nature.


Data mining requires statistical methods that can accommodate non-linearity, outliers, and non-numeric data.

Applications of value prediction include credit card fraud detection or target mailing list identification.

Database SegmentationDatabase Segmentation

Aim is to partition a database into an unknown number of segments, or clusters, of similar records.

Uses unsupervised learning to discover homogeneous sub-

populations in a database to improve the accuracy of the profiles.


Less precise than other operations thus less sensitive to redundant and irrelevant features.

Sensitivity can be reduced by ignoring a subset of the attributes that describe each instance or by assigning a weighting factor to each variable.

Applications of database segmentation include customer profiling,

direct marketing, and cross selling.

Example of Database Segmentation Example of Database Segmentation using a Scatter plotusing a Scatter plot


Associated with demographic or neural clustering techniques, distinguished by: Allowable data inputs Methods used to calculate the distance between records Presentation of the resulting segments for analysis.

Example of Database Segmentation Example of Database Segmentation using a Visualizationusing a Visualization

Link AnalysisLink Analysis

Aims to establish links (associations) between records, or sets of records, in a database.

There are three specializations– Associations discovery– Sequential pattern discovery– Similar time sequence discovery

Applications include product affinity analysis, direct marketing, and stock price movement.

Link Analysis - Associations DiscoveryLink Analysis - Associations Discovery

Finds items that imply the presence of other items in the same event.

Affinities between items are represented by association rules. – e.g. ‘When customer rents property for more than 2 years and

is more than 25 years old, in 40% of cases, customer will buy a property. Association happens in 35% of all customers who rent properties’.

Link Analysis - Sequential Pattern Link Analysis - Sequential Pattern DiscoveryDiscovery

Finds patterns between events such that the presence of one set of items is followed by another set of items in a database of events over a period of time.

– e.g. Used to understand long term customer buying behaviour.

Link Analysis - Similar Time Sequence Link Analysis - Similar Time Sequence DiscoveryDiscovery

Finds links between two sets of data that are time-dependent, and is based on the degree of similarity between the patterns that both time series demonstrate. – e.g. Within three months of buying property, new home

owners will purchase goods such as cookers, freezers, and washing machines.

Deviation DetectionDeviation Detection

Relatively new operation in terms of commercially available data mining tools.

Often a source of true discovery because it identifies outliers, which express deviation from some previously known expectation and norm.

Deviation DetectionDeviation Detection

Can be performed using statistics and visualization techniques or as a by-product of data mining.

Applications include fraud detection in the use of credit cards and insurance claims, quality control, and defects tracing.

A Summary: Data-Driven TechniquesA Summary: Data-Driven Techniques Data Visualization

Decision Trees

Clustering

Factor Analysis

Neural Network

Association Rules

Rule Induction

* Based on Sakhr Youness’s book “ Professional Data Warehousing with SQL Server 7.0 and OLAP Services

Data VisualizationData VisualizationA pie chart showing the sales of a product by region issometimes much more effective than presenting the samedata in a text or tabular form.

39%

9 %11 %

20 %

21 %

Northeast

East

West

South

North

Decision TreeDecision Tree

Cluster AnalysisCluster Analysis

Have Children

Married

Last car isA used one

Own car

First segment (high income>8,000)

Second Segment (8000>middle income >3000)

Third Segment (low income < 3000)

Factor AnalysisFactor Analysis

Unlike cluster analysis, factor analysis builds a model from data. The technique finds underlying factors, also called “latent variables” and provides models for these factors based on variables in the data. For ex., a software company is considering a survey to find out the nine most perceived attributes of one of their products. They might categorize these products to categories such as service for technical support, availability for training and a help system.

Factor analysis is used for grouping together products based on a similarity of buying patterns so that vendors may bundle several products as one to sell them together at a lower price than their added individual prices..

Neural NetworksNeural Networks

Association RulesAssociation Rules

Association models are models that examine the extent to which values of one field depend on, or are produced by, values of another field. These models are often referred to as Market Basket Analysis when they are applied to retail industries to study the buying patterns of these customers, especially in grocery and retail stores that issue their own credit cards. Charging against these cards gives the store the chance to associate the purchases of customers with their identities, which allows them to study associations among other things.

Rules InductionRules Induction

This is a powerful technique that involves a large number of rules using a set of “if..then” statements in the pursuit of all possible patterns in the dataset. For ex., if the customer is a male then, if he is between 30 and 40 years of ages, and his income is less than $50,000 and more than $20,000, he is likely to be driving a car that was bought as new.

A Summary: Theory-Driven A Summary: Theory-Driven TechniquesTechniques

Correlations

T-Tests

Analysis of Variables

Linear Regression

Logistic Regression

Discriminate Analysis

Forecasting Methods

Validating & Picking the Validating & Picking the ModelModel

Validating any model that comes out of a data mining tool is going to be the

most important thing that you can do. The validation required for data

mining is that after you build the model on some historical data, you apply

the model to similar historical data from which the model was not built.

Because the data is historical, you already know the outcome so that the

accuracy of the predictive model can be measured.

One of the most important things that needs to be done when you are

building a predictive model is to make sure that you have picked up the

essential patterns in the data that will hold true the next time you apply

your model.

Three Additional Ways in Which Three Additional Ways in Which Data mining Supports CRM Data mining Supports CRM

Initiatives.Initiatives.1. Database marketing

2. Customer acquisition

3. Campaign optimization

Database MarketingDatabase Marketing

Data mining helps database marketers develop campaigns that are closer to the targeted needs, desires, and attitudes of their customers. If the necessary information resides in a database, data mining can model a wide range of customer activities. The key objective is to identify patterns that are relevant to current business problems. For example, data mining can help answer questions such as "Which customers are most likely to cancel their cable TV service?" and "What is the probability that a customer will spend over $120 from a given store?" Answering these types of questions can boost customer retention and campaign response rates, which ultimately increases sales and returns on investment.

Database MarketingDatabase Marketing

Database marketing software enables companies to send customers and prospective customers timely and relevant messages and value propositions. Modern campaign management software also monitors and manages customer communications on multiple channels including direct mail, telemarketing, e-mail, the Internet, point of sale, and customer service. Furthermore, this software can be used to automate and unify diverse marketing campaigns at their various stages of planning, execution, assessment, and refinement. The software can also launch campaigns in response to specific customer behaviors, such as the opening of a new account.

Database MarketingDatabase Marketing Generally, better business results are obtained when data mining and

campaign management work closely together. For example, campaign management software can apply the data-mining model's scores to sharpen the definition of targeted customers, thereby raising response rates and campaign effectiveness. Furthermore, data mining may help to resolve the problems that traditional campaign management processes and software typically do not adequately address, such as scheduling, resource assignment, and so forth. Although finding patterns in data is useful, data mining's main contribution is providing relevant information that enables better decision making. In other words, it is a tool that can be used along with other tools (e.g., knowledge, experience, creativity, judgment, etc.) to obtain better results. A data-mining system manages the technical details, thus enabling decision makers to focus on critical business questions such as "Which current customers are likely to be interested in our new product?" and "Which market segment is best for the launch of our new product?"

Customer AcquisitionCustomer Acquisition

The growth strategy of businesses depends heavily on acquiring new customers, which may require finding people who have been unaware of various products and services, who have just entered specific product categories (for example, new parents and the diaper category), or who have purchased from competitors. Although experienced marketers often can select the right set of demographic criteria, the process increases in difficulty with the volume, pattern complexity, and granularity of customer data. Highlighting the challenges of customer segmentation has resulted in an explosive growth in consumer databases. Data mining offers multiple segmentation solutions that could increase the response rate for a customer acquisition campaign. Marketers need to use creativity and experience to tailor new and interesting offers for customers identified through data-mining initiatives.

Campaign OptimizationCampaign Optimization

Many marketing organizations have a variety of methods to interact with current and prospective customers. The process of optimizing a marketing campaign establishes a mapping between the organization's set of offers and a given set of customers that satisfies the campaign's characteristics and constraints, defines the marketing channels to be used, and specifies the relevant time parameters. Data mining can elevate the effectiveness of campaign optimization processes by modeling customers' channel-specific responses to marketing offers.

Topic 6: Topic 6: Classical Techniques: Classical Techniques: Statistics, Neighborhoods, and Statistics, Neighborhoods, and

ClusteringClusteringStatistics can help to answer several important questions about the

data :

What patterns are there in my database?

What is the chance that an event will occur?

What patterns are significant?

What is a high-level summary of the data that gives me some idea of what is contained in my database?

StatisticsStatistics --Histogram --Histogram

The first step in understanding statistics is to understand how the

data is collected into a higher-level form—one of the most notable

Ways of doing this is with the histogram.

0

10

20

30

40

50

60

70

80

90

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

# of customers orAmount of sales

HistogramHistogram

Number of customers

Ages1 11 21 31 41 51 61 71 81

500

1000

1500

2000

2500

3000

Linear Regression Is Similar to the Task of FindingLinear Regression Is Similar to the Task of Findingthe Line that Minimizes the Total Distance to a Set the Line that Minimizes the Total Distance to a Set

of Data.of Data.

Predictor (Consumer annual income)

Prediction(Average Consumer bank balance)

Linear RegressionLinear Regression

The predictive model is the line shown in the previous chart. The line

will take a given value for a predictor and map it into a given value

for a prediction. The actual equation would look something like

Prediction = a + b* predictor. This is just the equation for a line Y =

A + b*X. As an example for a bank, the predicted average consumer

bank balance might equal to $1,000 + 0.01 * customer’s annual

income.


Linear regression attempts to fit a straight line through a plot of the data, such that the line is the best representation of the average of all observations at that point in the plot.

Problem is that the technique only works well with linear data and is sensitive to the presence of outliers (i.e.., data values, which do not conform to the expected norm).


Although non-linear regression avoids the main problems of linear regression, still not flexible enough to handle all possible shapes of the data plot.

Statistical measurements are fine for building linear models that describe predictable data points, however, most data is not linear

in nature.


Data mining requires statistical methods that can accommodate non-linearity, outliers, and non-numeric data.

Applications of value prediction include credit card fraud detection or target mailing list identification.

The Nearest Neighbor Prediction

One of the classic areas that nearest neighbor has been used for

prediction has been in text retrieval. The end user defines a document

(for ex., a Wall Street Journal) to be retrieved, then the nearest

neighbor characteristics with these documents that have been

marked are more likely to be retrieved.

Another good example is that the supermarkets tend to put similar

produces in the same area, for ex., an apple closer to an orange than

to tomato. Thus, if you know the predictive value of one of the

objects, you can predict it for the nearest neighbors.

Data Clustering

Clustering analysis is an important means of processing multimedia

data. It is basically the organization of a collection of patterns into

clusters of similar objects. Patterns within valid cluster are more

similar to each other than they are to a pattern in a different cluster.

Data Clustering

Clustering can allow us to carry out the following activities

that can help in query processing:

Representing patterns in the data so that we can reduce the size of the media;

Defining a way of measuring the proximity of different patterns in the data so that we can find the instances that match our example.

Clustering or grouping the data in preparation for matching; Data abstraction, particularly of features that we can store as

metadata; Assessing the output by estimating how good the selection is.

Clustering and Nearest NeighborClustering and Nearest Neighbor

A simple example of clustering would be the clustering that most

people perform when they do the laundry- grouping the permanent

press, dry cleaning, whites, and brightly colored clothes is important

because they have similar characteristics.

A simple example of the nearest neighbor prediction algorithm Is

when you look at the people in your neighborhood. You may notice

that, in general, you all have somewhat similar income.

Statistical Analysis of Actual Sales (dollars Statistical Analysis of Actual Sales (dollars and quantities) relative to these Signage and quantities) relative to these Signage

Variables-a predictiveVariables-a predictive modelingmodeling example. example.

Content Frequency Depth Focus Depth Scale Length Location

Statistical Analysis : Correlation, Regression, Experiment Design,

Optimization. Now it goes into real time analysis.

SignageSignage

Topic 7: Topic 7: Next Generation Techniques: Next Generation Techniques: Decision Trees, Networks, and RulesDecision Trees, Networks, and Rules


Rent property

Customer age>45

No Yes

No Yes

Rent property

Buy property

A: Decision TreeA: Decision Tree

A: Decision TreeA: Decision Tree

CART and CHAIDCART and CHAID

CART, which stands for Classification and Regression Trees, is a data

exploration and prediction algorithm developed by Leo Breiman, Jerome Friedman, Richard Olshen and Charles Stone. It is nicelydetailed in their 1984 book, Classification and Regression Trees (Breiman, Friedman, Olshen, and Stone, 1984. These researchers fromStandard University and the University of California at Berkeley Showed how this new algorithm could be used on a variety of different problems from the detection of chlorine from the data contained in a mass spectrum. One of the great advantages of CARTis that the algorithm has the validation of the model and the

discovery of the optimally general model built deeply into the algorithm. Another popular decision tree technology is CHARD (Chi-SquareAutomatic Interaction Detector). CHARD is similar to CART in that it builds a decision tree, but it differs in the way that it chooses its splits.

B: Neural NetworksB: Neural Networks

A neural network is loosely based on the way some people believe

That the human brain is organized and how it learns. There are two

Main structures of consequence in the neural networks:

The node, which loosely corresponds to the neuron in the human brain

The link, which loosely corresponds to the connections between neutrons (axons, dendrites, and synapses) in the human brain.

Neural NetworksNeural Networks

‘When customer rents property for more than 2 years and is more than 25 years old, in 40% of cases, customer will buy a property. Association happens in 35% of all customers who rent properties’.

Example of Classification using Example of Classification using Neural InductionNeural Induction

Each processing unit (circle) in one layer is connected to each processing unit in the next layer by a weighted value, expressing the strength of the relationship. The network attempts to mirror the way the human brain works in recognizing patterns by arithmetically combining all the variables with a given data point.

In this way, it is possible to develop nonlinear predictive models that ‘learn’ by studying combinations of variables and how different combinations of variables affect different data sets.

How Does a Neural Induction How Does a Neural Induction Make a prediction?Make a prediction?

The value age of 47 is normalized to fall between 0.0 and 1.0, it has the value of 0.47, and the income is normalized to the value of 0.65. This simplified neural network makes the prediction of no default for a 47-year old making $65,000. The links are weighted at 0.7 and 0.1, and the resulting value, after multiplying the node values by the link weights, is 0.39.

0.47

0.65

0.39

Age

Income

Weighted = 0.7

Weighted = 0.1

default

0.47(0.7) + 0.65(0.1) = 0.39

C: Rule InductionC: Rule Induction

This is a powerful technique that involves a large number of rules using a set of “if..then” statements in the pursuit of all possible patterns in the dataset. For ex., if the customer is a male then, if he is between 30 and 40 years of ages, and his income is less than $50,000 and more than $20,000, he is likely to be driving a car that was bought as new.

What Is A Rule?What Is A Rule?

If breakfast cereal purchased, the 85% 20%

milk is purchased.

If bread purchased, then Swiss choose 15% 6%

will be purchased.

If 42 years old and purchased pretzels 95% 0.01%

and dry roasted peanuts, then beer will

be purchased.

Rule Accuracy Coverage

Tools and technologies will be applied to real business problems across a variety of industries. They are:

Customer Profitability – provides a blueprint for how to define and use customer profitability as the bedrock for your CRM processes.

Customer Acquisition – shows how to use data mining to acquire new customers in the most profitable way possible.

Customer Cross-selling – details how the technology architecture can be used to increase the value of existing customers by applying more to them.

Customer Retention – uses a case study from the telecommunications industry to show how to execute successful CRM systems to retain your profitable customers.

Customer Segmentation – provides the business methodology of how to segment and manage your customers in a consistent and repeatable way across the enterprise.

Topic 8: Topic 8: CRM -The Business CRM -The Business PerspectivePerspective

The Business-Centric View of Data The Business-Centric View of Data Mining ProcessMining Process

ROI Definition

Display

ROI

Predicted ROI

BusinessProblem

PredictiveModel

Data Definition

Data

Define Value

Define Value

Understand

Data Mining

Application

Customer ProfitabilityCustomer Profitability

Customer profitability is the bedrock of data mining. Data mining

earns its keep by helping you to understand and improve Customer

Profitability. How does the organization define what a profitable

customer is versus an unprofitable customer? Keeping a customer

loyal can have profound effects on per-customer profitability. The

compounding effect of customer loyalty on customer profitability also

increases because sales costs are lower and revenue generally has

increased. Data Mining can be used to predict customer profitability,

Under a variety of different marketing campaigns.

CurrentValue

LifetimeValue

PotentialValue

PotentialLifetimeValue

CustomerServiceLevel

BestServiceLevel

1 High High High High Gold Gold

2 High Low High High Gold Gold

3 High Low High Low Gold Bronze

4 Low Low Low High Bronze Gold

5 Low Low High High Bronze Gold

6 Low Low Low Low Bronze Bronze

Segment

A Customer Value Matrix Showing Recommended Service Level

A Customer Value MatrixThis should be one of the first things that we should do

with data mining.

Segment 1 is our best customers. They will remain your best customers through their lives and their current value matches their potential.

Segment 2 is similar, except that they are likely to have low lifetime value, despite their high value today, probably because they are not loyal and likely switch to a competitor at some time in their customer life.

Segments 4 and 5 represent customers who, with the right care and service, can be transitioned to high-value customers, either short-term or long-term .

Segments 6 represents your low-value customers that you will treat with some of your least expensive services.

Customer AcquisitionCustomer Acquisition

The traditional approach to customer acquisition involved a marketing manager developing a combination of mass marketing (magazine advertisements, billboards, etc.) and direct marketing ( Telemarketing, mail, etc.) campaigns based on their knowledge of theParticular customer base that was being targeted.

A marketing manager selects the demographics (Age, Gender, interest in particular subjects, etc.) and then works with a data vendor (sometimes known as a service bureau) to obtain Lists of customers who meet those characteristics.

Although a marketer with a wealth pf experience can often choose relevant demographic selection criteria, the process becomes more difficult as the amount of data increases.

Data Mining can help this process.

Defining Some Key Customer Defining Some Key Customer Acquisition ConceptsAcquisition Concepts

The responses that come in as a result of a marketing campaign are called “response behaviors”. Binary response behaviors (either a yes or no) are the simplest kind of response.

Beyond binary response behaviors are a type of categorical response behaviors which allows for multiple behaviors to be defined. The rules that define the behaviors are based on the kind of business you are involved in.

There are usually several different kinds of positive response behaviors thatcan be associated with an acquisition marketing campaign. They are:

Customer inquiry; Purchase of the offered product or products; Purchase of a product different from the one offered.

Response Analysis Broken Down By Behaviors

Behavior Measures 12/1/05 12/5/05 12/7/05 12/9/05 Total

Inquiry # of Responses 1,556 1,340 328 352 3,576

Purchase A

# of Responses 210 599 128 167 1.104

Purchase B

# of Responses 739 476 164 97 1,476

Purchase C

# of Responses 639 647 113 105 1,504

Cross-SellingCross-Selling

Cross-selling is the process by which you offer your existing customers new

products and services. Customers who purchase baby diapers might also be

interested in hearing about your other baby products.

One form of cross-selling, sometimes called “up selling”, takes place when the

new offer is related to existing purchases by the customer. For., ex., an up-

sell opportunity might exist for a telephone company to market a premium

long-distance service to existing long-distance customers who currently have

the standard service.

How Cross-Selling WorksHow Cross-Selling WorksAssume that you are a marketing manager for a mid-size bank. You have the following products available for your customers:

Value checking account Standard checking account Gold credit card Platinum credit card Primary mortgage Secondary mortgage

Of these products, you’re responsible for marketing the mortgage products to your Customers. Your goal is to find out which customers might be interested in a mortgage offering at least 60 days before they would apply for the loan. It is important that any predictions are made with sufficient lead time (in this case, two months), so that any Interactions with the customers take place before they are committed to a relationship with your competition.

How Cross-Selling WorksHow Cross-Selling Works

You have already done some thinking about your customers and their motivations in this area and came up with several scenarios, which you presented to your boss when pitching this new campaign:

Customer preparing to buy a new home. These customers might be building up cash reserves in their checking and/or savings account in order to put together a down payment.

Customer preparing to refinance an existing home. These customers might be paying off credit card debt (thus making them more acceptable from a risk point of view), and hold a mortgage whose interest rate is higher than the current interest rate.

Customer preparing to add a second mortgage. These customers might have increasing credit card debt, an on-time payment history for their credit cards and existing mortgage (which means that they are a good risk), and enough equity in their house to cover the outstanding credit card balance.

Data Mining Process for Cross-SellingData Mining Process for Cross-Selling

The actual data mining process contains three distinct steps when doing cross-selling process:

Modeling of individual behaviors Scoring data with predictive models Optimization of the scoring matrices

Model: A description that adequately explains and predicts relevant data thatbut is generally much smaller than the data itself. For real-world applications, a model can be anything from a mathematical Equation, to a set of rules that describes customer segments, to the computer representation of a complex neural network architecture, which translates to several sets of mathematical equations.

Predictive model: A model created or used to perform prediction. In contrast to models created solely for pattern detection, exploration or general organization of the data.

Customer RetentionCustomer Retention

As industries become more competitive and the cost of acquiring new

customers increases, the value of retaining current customers also increases.

for instance, in the cellular phone industry, it is estimated that the cost of

attracting and signing up a new customer is $300 or more when the costs of

disconnected hardware and sales commissions are included. The cost of

retaining a current customer, however, can be as low as the price of a phone

call or the cost of updating their cellular phone to the latest technology

offering. Although expensive, this is still significantly cheaper than signing

up a wholly new customer.

A Case Study- Cellular Phone IndustryA Case Study- Cellular Phone Industry

Customer churn is the term used in the cellular telephone industry to denote

the movement of cellular telephone customers from one provider to another.

In many industries, this is called customer attrition, but because of the highly

volatile and growing market, and the somewhat limited competition, many

customers churn from one provider to another frequently in search of better

rates or for the perks of signing up with a new provider. Attribute rates in

cellular phone industry hover around 2.2% per month. In other words,

about 27% of a given carrier’s customers are lost each year when the

contracts need to be renewed. Losing these customers can be very expensive

because it costs fom $300 to $600 to acquire a new customer in sales support,

marketing, advertising, and the commissions. Many of these new customers

are less profitable than the ones that were lost.

The Data Mining ModelThe Data Mining Model

Segment Create Apply Churn Description of the customers in the segment

Number size size rate (criteria defining the segment)

29 1061 403 84% Contract type “N” – no contract

Length of service is less than 23 months

28 899 360 65% Contract type “Y” – indicating a 12 month

contract requiring 3 months notice to discontinue

at the end of 12 months

18 902 7564 50% Contract type “Y”

Customer type “R” indicating a residential not

Business

The Data Mining ModelThe Data Mining ModelThe total Customer churn volume expected in each segment is taken as a percentage of the the total expected to churn across the total population (thetotal of all segments), and this is shown on a cumulative basis alongside the cumulative percentage of the base. The analysis shows the following:

5.2 % of the base contains 27.7% of the expected total churn.



Therefore, marketing campaigns targeted at 5.2 % of the base 14,581 subscribers will address 7848 of the likely churners, a lift factor of 5.4. A liftis a number representing the increase in responses from a targeted marketingapplications using a predictive model over the response rate achieved when no model is used.

A Decision Tree for Mobile PhonesA Decision Tree for Mobile Phones

Contract Type = ”D”

Length of Service > 12 months Length of Service > 12

months

No Yes

No Yes

Segment 28 Segment 29

Segment AnalysisSegment Analysis

Segment Propensity Segment Churn Cumulative Cumulative Cumulative % Cumulative %

Number to Churn base volume base Churn of Churn of base

29 82 % 403 337 403 337 1.2 % 0.1 %

28 64 % 360 233 763 571 2.0 % 0.3 %

Customer SegmentationCustomer Segmentation

Segmentation is the act of breaking down a large of customer population into

segments in which those consumers within the segments are similar to each

other, and those that are in different segments are different from each other.

for ex., even the simple act of organizing the customers in your database by

the state they live in is an act of segmentation. Distinguishing between male

and female customers is also an act of segmentation.

Segmentation allows people to differentially treat consumers in different

segmentations. That is why men are advertised to during football games,

women are advertised to during sitcoms.

How Is Data Mining Used for Segmentation- How Is Data Mining Used for Segmentation- Clustering & Decision TreeClustering & Decision Tree


Rent property

Customer age>45

No Yes

No Yes

Rent property

Buy property

How is Data Mining Used for How is Data Mining Used for Segmentation-Clustering?Segmentation-Clustering?

If decision trees are used to create segments, then the data is

guaranteed to the mutually exclusive and collectively exhaustive (no

customer falls into more than one segment and every customer is

guaranteed to be contained in one of the segments).

Topic 9:Topic 9: Deploying Data Mining for Deploying Data Mining for CRMCRM

Define the problem Define the user Select the data Prepare the data Mine the data Deploy the model Take business action Implement Quality Assurance Educate and train users

Define the ProblemDefine the Problem

A successful data mining initiative always starts with

a well-defined project. To insure that the project produces incremental value, include an assessment of the status quo

solution and a review of technology, organization, and business processes.

Many times, data mining systems will be deployed to optimize

existing CRM process. If a CRM system and process does already exist, it should be understood.

Define the UserDefine the UserAfter the problem is defined, it should be possible to define who will

be using the system when it is completed. These could include using

the data mining application itself all the way to supporting, and then

measuring the customer value matrix and the computation of the

ROI (Return on investment) of the existing system.

Building a profile of each user: you should know, for instance, at least

the following information about these users:

Their technical expertise Their use of the system (every day, once a month, or occasionally) The understanding of data mining Their desire for details

Select the DataSelect the Data

This step involves defining your data source . (not every data source and record is required.) The data is usually extracted from

the source system to a separate server.

The three types of customer data that does the following:

Describe who the consumer is. Describe what marketing or sales promotions were made to the

customer. Describe how the consumer reacted to those promotions by

transacting with the company.

The three types of customer dataThe three types of customer data

Who is theCustomer?

What did you doTo the customer?

Descriptive Promotional

Transactional

How did theCustomer react?

Direct mail, email, sales

Purchase, Web hit,Business reply card,survey

Prepare the DataPrepare the Data

This step represents up to 80 percent of the total project effort. For data mining, the data must reside in one flat table (each record has many columns). In addition to being the most time consuming, the step is also the most critical. The resulting models are only as good as the data used to create them.

Prepare the Data- Assessing Prepare the Data- Assessing Levels of Data IntegrityLevels of Data Integrity

This step involves defining your data source . (not every data source and record is required.) The data is usually extracted from

the source system to a separate server.

The three types of customer data that does the following:

Describe who the consumer is. Describe what marketing or sales promotions were made to the

customer. Describe how the consumer reacted to those promotions by

transacting with the company.

Mine the DataMine the Data

Typically the easiest and shortest phase, this step involves applying statistical and AI tools to create mathematical models. Data mining typically occurs on a server separate from the data

warehousing and other corporate systems.

Deploy the ModelDeploy the Model

Model deployment is the process of implementing the mathematical models into operational systems to improve business results.

Take Business ActionTake Business Action

Use the deployed model to achieve improved results to the business problem identified at the beginning of the process.

Steps to Implement Data MiningSteps to Implement Data MiningDiscovery (patterns, relations

Associations, etc.)Prior Knowledge

Information Model

Deployment

Validation

Implement Quality AssuranceImplement Quality Assurance

All throughout the launch process, quality assurance should be of

highest priority. A good first step is to specifically assign the QA role

and only the QA role to one of the team members on the data mining

projects. This QA role will exist to not only validate the success of the

data mining model, but also to double-check the work from other

parties.

Educate and Train UsersEducate and Train Users

Make sure to work with the users to educate them about what models

and metrics are, and how they can access and visualize the results

of the system. Pay particular attention to the following:

Description of the consumer base and the data that is available. How the data mining results are integrated into the customer

relationship management system. The way the metrics are calculated for understanding the results

of the data mining system. For instances, how is customer profit or customer response calculated.

Topic 10: Data Quality-Indicators of Topic 10: Data Quality-Indicators of Quality DataQuality Data

1. The data is accurate – This means that a client’s name is spelled correctly.

2. The data is stored according to data type – for ex., character, integer.

3. The data has integrity – Referential integrity rules will be properly defined in the logical data model and implemented in the physical data model. The data will not be accidentally destroyed and altered.

4. The data is consistent – The form and content of the data should be consistent. This allows for data to be integrated and shared by multiple departments across multiple applications and multiple platforms.

5. The databases are well designed – A well-designed database will perform satisfactorily for its intended applications.

Data Quality-Indicators of Quality Data Quality-Indicators of Quality DataData

6. The data is not redundant – In actual practice, no organization has ever totally eliminated redundant data. For certain performances, data is purposely maintained in more than one place.

7. The data follows the business rules – for ex., a loan balance may never be negative.

8. The data corresponds to established domains – Referential integrity rules will be properly defined in the logical data model and implemented in the physical data model. The data will not be accidentally destroyed and altered.

9. The data is timely – Timeliness is subjective and can only be determined by the users of the data.

10. The databases are well understood – It does no good to have accurate and timely data if the users do not know what they mean.

Data Quality-Indicators of Quality Data Quality-Indicators of Quality DataData

11. The data is integrated – Database integration requires the knowledge of the characteristics of the data, what the data means, and where the data resides. The information would be kept in the dictionary/repository.

12. The data satisfies the needs of the business – The data has value to the enterprise.13. The user is satisfied with the quality of the data and the

information derived from that data.14. The data is complete– All the line items for an invoice have been

captured so that the bill states the full amount that is owned.15. There are no duplicate records. 16. There is data anomalies– A data anomaly occurs when the data

field defined for one purpose is used for another.

Types of Source System ExtractsTypes of Source System Extracts

In order to update the data warehouse, it is necessary first to identify

what data is required from the operational system in order to capture any new or changed data during a given time variant period interval. Let us begin with identifying the triggers in the operational environment that determine when an extract should be taken. These triggers can be categorized as:

Point-in-time snapshots Significant business events Delta data

Point-in-Time SnapshotsPoint-in-Time Snapshots

The simplest way to preserve history is to take a point-in-time snapshot of the operational data. Snapshots are typically scheduled for very specific points in time, such as the end of a calendar week or month, and they preserve the historical relationship between different data elements and subject areas. A snapshot is a very effective way to determine delta between different points in time. For example, if monthly point-in-time snapshots of customer data are taken from an operational source and loaded into the data warehouse, a determination of the changes in the customer base can

easily be determined from month to month, or any time period.

Significant Business EventsSignificant Business EventsSignificant business event (SBE) are actually an event-oriented variation of point-in-time snapshots. Unlike a point-in-time, which is an easily predetermined date such as the end of a calendar week or month, and SBE cannot be accurately predicted; it must occur for the snapshot to then take place. For ex., an SBE might be the successful completion of a billing cycle. There may be several billing cycles during a calendar month, each with a predetermined cutoff date. Nevertheless, the successful completion of the billing cycle is dependent on a number of factors and can not be predetermined; it may take anywhere from one to three days to complete the cycle. The actual completion event is determined by a set of success criteria, usually judged by the person or group responsible for quality assurance or other similar business function. Once the cycle has been deemed successful, bills can be mailed to the customers. It is at this point, this significant business event, that a snapshot of the billing data and any related or artifact data can take place.

Delta DataDelta Data

Delta data is also known as “new and changed data”; that is, data that represents changes (delta) from one point in time to the next. Delta can be captured in a number of ways:

1. Operational events-An operational event creates delta data

by passing a record of the event either to a holding file or to

a data warehouse updating process. This will result in an

accurate record of the changes that take place in the

operational system at a limited cost in terms of processing.

Delta DataDelta Data

2. Changed Data Capture-It refers to the process of reading a database or other operational change logs and extracting the appropriate changed data for updating the data warehouse.

3. Date Last Changed-A very efficient means of extracting new and changed data from operational sources is to interrogate a date last modified in the operational tables or files.

Topic 11:Topic 11: Next Generation of Next Generation of Information Mining and Knowledge Information Mining and Knowledge

Discovery for Effective CRMDiscovery for Effective CRM

In the current and emerging competitive and highly dynamic business

environment, only the most competitive companies will achieve

sustained market success. In order to capitalize on business

opportunities, these organization will distinguish themselves by the

capacity to leverage information about their marketplace, customers,

and operations. A central part of this strategy for long-term

sustaining success will be an active information repository- an

advanced data warehouse, in which information from various

applications or parts of the business is coalesced and understood.

Information MiningInformation Mining

The shortest path from complex data to knowledge discovery is

Information mining instead of data mining to reflect the rich variety

Of forms that information required for business intelligence can take.

Information mining implies using powerful and sophisticated tools to

Do the following:

Uncover associations, patterns, and trends

Detect deviations

Group and classify information

Develop predictive models

Information MiningInformation Mining

From a technical perspective, the real keys to successful information

Mining are its algorithms: complex mathematical processes that

Compare and correlate data. Algorithms enable an information

mining application to determine who the best customers for the

Business are or what they like to buy. They can also determine at

what time of day, in what combinations, or how an organization can

Optimize inventory, pricing, and merchandising in order to retain

These customers and cause them to buy more, at increased profit

Margins. A large volume of information is stored in anon-numeric

Forms: documents, images and video files.

Text Mining and Knowledge Text Mining and Knowledge ManagementManagement

Text Mining is a subset of information mining technology that, in turn, is a

component of a more general category of Knowledge Management (KM).

Knowledge, in this case, refers to the collective expertise, experiences, know-

how, and wisdom of an organization. In a business world, knowledge is

represented not only by the structured data found in traditional database,

but in a wide variety of unstructured sources such as word documents,

memos and letters, e-mail messages, news feeds, Web pages, and so forth.

Text Mining and Knowledge Text Mining and Knowledge ManagementManagement

Unlike data mining, text mining works with information stored in an

Unstructured collection of text documents. Specifically, online text

Mining refers to the process of searching through unstructured data

On the internet and deriving some meaning from it. Text mining goes

beyond applying statistical models to data files; in fact, text mining

Uncovers relationships in a text collection, and leverages the

creativity of the knowledge work to explore these relationships and

Discover new knowledge.

Text Mining TechnologiesText Mining Technologies

There are two key key technologies that make online text mining possible:

Internet Searching - It has been around for a quite few years. Yahoo, Alta Vista, and Excite are three of the earliest. Search engines (and discovery services) operate by indexing the context in a particular Web site and allows users to search the indexes. Although useful, first generations of these tools often were wrong because they did nit correctly index the content they retrieved. Advances in text mining applied to the internet searching resulted in online text mining, representing the new generation of Internet search tools. With these products, users can gain more relevant information by processing smaller amount of links, pages and indexes.


Text Analysis - It has been around longer than Internet searching. Indeed, scientists have been trying to make computers understand natural languages for decades; text analysis is an integral part of these efforts. The automatic analysis of text information can be used for several different general purposes:

1. To provide an overview of the contents of a large document collection, for ex., finding significant clusters of documents in a customer feedback collection could indicate where a company’s products and services need improvement.

2. To identify hidden structures between groups of objects; this may help to organize an intranet site so that related documents are all connected by hyperlinks.


3. To increase the efficiency and effectiveness of a search process to find similar or related information; for ex., to search articles from a news service and discover all unique documents that contain hints on possible trends or technologies that have so far not been mentioned in their articles.

4. To detect duplicate documents in an article.

Text Mining Technologies-Text Mining Technologies-ApplicationsApplications

1. E-mail management. A popular use of text analysis is for messae routing in which the computer “reads” the message to decide who should deal with it. (Spam control is another good example)

2. Document Management. By mining the different documents for meaning as they are put into a document repository, a company can establish a detailed index that allows the location of relevant documents at any time.

3. Automated help desk. Some companies use text mining to respond to customer inquiries. Customers’ letters and e-mails are processed by a text mining applications.

4. Market research. A market researcher can use online text mining to gather statistics on the occurrences of certain words,c phases, concepts, or themes on the World Wide Web. This information can be useful for establishing market demographics and demand curves.

5. Business intelligence gathering. This is the most advanced use of text mining. (See next slide)

BloggerBlogger

Blogger is one of the most popular online blogging tool, works with

any browser, and is free, well designed and easy to use. Millions of

people are changing their information acquisition habits, and the web

Log, or “blog” has become a popular source.

Title-Publishing a blog with blogger/by Elizabeth Castro,

Berkeley, Calif, Peachpit, 2005 Title- Blog: Understaning the information that’s changing your

world/ Hugh Howitt, Nashiville, Tenn, Nelson Books, c2005 Webblogs (isbn 0321321235)

Semantic Networks and Other Semantic Networks and Other TechniquesTechniques

A key element of building an advanced system for textual information. analysis, summarization, and search is the development of a Semantic Network for the investigated text. A Semantic Network is a set of the mostsignificant concepts—words and word combinations– derived from the analytical texts, along with the semantic relationships between these concepts in the text. A semantic network provides a concise and very accurate summary of the analyzed text.

Other techniques, For ex., Cambio uses absolute positioning, pattern recognition, fixed and floating tags. The SemioMap software extracts all relevant phrases from the text collection. It builds a lexical networkof co-occurrences by grouping related phases and enhancing the most salient features of these groupings.

Text Mining ProductsText Mining ProductsCompany Products

Aptex Software, Inc. SelectResponse

Autonomy Agentware

Data Junction Cambio

Excalibur Technologies Corp. RetrievalWare

Fulcrum Technologies Inc. DOCSFulcrum SearchSearver

IBM Corp. Intelligent Minor for Text

InsightSoft-M Cross-reader

Intercon System, Ltd. Dataset

Megaputer Inc. Text Analyst

Semio Corp. SemioMap

Verity, Inc. KeyView, Intranet Spider

Text Mining Products-An ExampleText Mining Products-An Example

Autonomy (Agentware) offers three kinds of products relating to online text mining:

Knowledge Server – Provides users with a fully automated and precise means of categorizing, cross-referencing, and presenting information.

Knowledge Update – Monitors specified Internet and intranet sites, news feeds, and internal repositories of documents, and creates a personalized report on their contents.

Knowledge Builder – A toolkit that allows companies to integrate Agentware capabilities into their own systems.

Topic 12:Topic 12: CRM in the e=Business CRM in the e=Business WorldWorld

As e-business continues to mature and affect radical changes throughout all

aspects of the businesses, the focus of new e-business-enabled application

software will shift away from narrowly defined commerce platforms toward

a broader vision of managing customer relationships.

A new model that Forrester Research calls eRelationship Management (eRM)

is defined as follows:

“A Web-centric approach to synchronizing customer relationships across

communication channels, business functions, and audiences”

CRM in the e=Business WorldCRM in the e=Business World

To implement this new e-business CRM model, companies should do the

following:

Create a dynamic customer context that can address every customer interaction that is different from a view of the customer constructed from data contained in the applications. This can be achieved by collecting and organizing customer data, calculating high-level matrices for each customer (I.e., customer profitability, satisfaction, and churn potential), and assembling and delivering dynamic context to customer touch points.

Generate consistent, custom responses by delivering a consolidated rules engine for routing, workflow, personalization, smart navigation, and consistent treatment of customers

Build and maintain a Content Directory to point to company, products, and business partner content; and give to employees, business partners, and customers.

Based on the book “Building Data Mining Applications for CRM” By Alex Berson Stephen Smith Kurt...

Documents

Transcript of Based on the book “Building Data Mining Applications for CRM” By Alex Berson Stephen Smith Kurt...