Open06

Comparative Study on Classification

Techniques to Identify 3G Customers

GUO Jing

GD405, Motorola (China) Electronics Ltd. Beijing Second Branch, No.39A

Zi Zhu Yuan Road, Hai Dian District, Beijing China, 100089

E-mail: [email protected]

1. Introduction

Along with the worldwide spreading of 3G network, the mobile market is

undergoing a major transition. In many countries, penetration rates are beginning to

peak, and therefore, the operators are moving from a strategy of new customer

acquisition to one of retention and increasing customer spending and profitability.

Identifying which customers are likely to switch to using their 3G network will

definitely be a major approach to achieve this goal.

Many mobile operators already have tremendous customer usage and demographic

data in their system. However, data is characterized as recorded facts, while

information is the set of patterns or expectations that underlie the data. There is a

huge amount of information locked up there, in the operators’ databases - information

that is potentially important but has not yet been discovered or articulated. Our

mission is to bring it forth. As stated, the data, in many cases, already exist. It is a

matter of knowing which kind of data contributes more useful information and finding

the methodologies to make use of the data to present the information.

Data mining is the extraction of implicit, previously unknown, and potentially

useful information from data. The idea is to build computer programs that examine

through data bases automatically, seeking regularities or patterns. Strong patterns, if

found, will likely generalize to make accurate predictions on future data.

The data mining task in this paper aims at solving a given classification problem for

which the objective is to accurately predict as many current 3G customers as possible

from the “holdout” sample provided (See Table 1). An original sample dataset of

20,000 2G network customers and 4,000 3G network customers has been provided

with more than 200 data fields. The target categorical variable is “Customer_Type”

(2G/3G). Of course, in the total 24,000 instances and 251 attributes, there exist

problems. Many of the attributes will be uninteresting and banal. Others will be

spurious, contingent on accidental coincidences in the particular dataset used. In a

word, the real data is imperfect.

Table 1 2G/3G Network Customers’ Distribution in Training/Test Set and

Prediction Set

Training /Testing Set Prediction Set

2G user number 3G user number 2G user number 3G user number

15K 3K 5K 1K

For this classification case, various models were built up to find strong patterns,

which will make accurate enough predictions on customer type (2G/3G). In this

paper, we will discuss the employed approaches, full technical details of algorithms as

well as the trained classification model. Although the trained model will never be the

elixir to all kinds of cases, the whole procedure of solving a practical problem will

surely leave us a trace to the whole forest. Finally, we discuss on the insight gained

from the model in terms of identifying current 2G customers with the potential to

switch to 3G.

It should be mentioned that all the analysis results and constructed pattern obtained

in this paper are based on a well-known data mining tool, Weka, which is a collection

of state-of-art machine learning algorithm and data preprocessing tools (See Figure 1).

Figure 1 The Weka Explorer Interface

2. Data Preprocessing

We have 24,000 instances at hand, each of which contains 251 attributes.

Obviously there exist some problems in the raw data. Data type varies (nominal or

numeric), missing values exist and not all of attributes contain information we need.

Therefore, we take several steps to make sure the data for later machine learning is

reasonable and clean.

2.1 Remove the banal and uninteresting attributes

Table 2 The List of Removed Attributes in the Data Preprocessing Step

Removed Attributes Reasons

SERIAL_NUMBER {SERIAL_NUMBER<=3000|3G}{SERIAL_NUMBER>=3000|2G}This attribute is meaningless.

HS_CHANGE, TOT_PAST_DEMAND, All the instances share the same value in

VAS_DRIVE_FLAG, VAS_VMN_FLAG, VAS_INFOSRV, VAS_SN_FLAG, VAS_CSMS_FLAG, DELINQ_FREQ, AVG_VAS_IDU, AVG_VAS_WLINK, AVG_VAS_MILL, AVG_VAS_IFSMS, AVG_VAS_#123#, AVG_VAS_CG, AVG_VAS_IEM, AVG_VAS_ISMS, AVG_VAS_SS, STD_VAS_IDU, STD_VAS_WLINK, STD_VAS_MILL, STD _VAS_IFSMS, STD _VAS_#123#, STD _VAS_CG, STD _VAS_IEM, STD_VAS_ISMS, STD _VAS_SS

the training/test set or the prediction set with regard to this attribute.

VAS_VMP_FLAG, TELE_CHANGE_FLAG

Expect for less than 5 instances, nearly all the instances share the same value with regard to this attribute.

TOT_DIS_1900, TOT_USAGE_DAYS There are other similar attributes (AVG_DIS_1900, AVG_USAGE_DAYS) in the data set.

2.2 Remove certain instances that are regarded as

outliers

In the training and testing set, there are 108 numeric attributes out of total 251 ones

that are regarded interesting. However, certain instances appear to have merely 0

values for most of the numeric instances. We can simply remove those outliers in the

first place by setting a lower enough threshold 0.001. The instances with 0 values

(below the threshold 0.001) for important numeric attributes like AVG_CALL

(Average number of calls in the last 6 months) will be deleted. In this way, we could

clean and reduce the data to 17577 instances out of 18000 in the training and testing

set.

2.3 Deal with missing values

The next enhancement to the data mining problem deals with the problems of

missing values. Missing values are endemic in real-world data sets. Most machine

learning methods make the implicit assumption that there is no particular significance

in the fact that a certain instance has an attribute value missing: the value is simply

not known. However, some attribute like OCCUP_CD (occupation code) shows great

importance in the attribute selection procedure, at the same time contains more than

60% missing values. With regard to this kind of attribute, we should find the value of

the highest frequency (e.g. OTH appears to be the most common value in the attribute

OCCUP_CD) and replace the missing values with it. Since instances with missing

values often provide a good deal of information, taking this approach is much better

than simply ignoring all instances in which some of the values are missing.

3. Construct the Prediction Model

Experience shows that no single machine learning scheme is appropriate to all data

mining program. The universal learner is an idealistic fantasy. Real datasets vary,

and to obtain accurate models the bias of the learning algorithm must match the

structure of the domain. Therefore, after attribute selection, we build the prediction

model by learning a serial of models and combining them together to make trust

worthy and wiser decisions on true positives.

It should be mentioned that, since error rate on the training set is too optimistic to

be a good indicator of future performance, in this section we always measure a

classifier’s performance using 10 times cross-validation.

3.1 Attribute Selection

Most machine learning algorithms are designed to learn which are the most

appropriate attributes to use for making decision. For example, decision tree methods

choose the most promising attribute to split on at each point and should - in theory -

never select irrelevant or unhelpful attributes. Having more features should surely - in

theory - result in more discriminating power. However, in practice, adding irrelevant

or distracting attributes to a data set often ”confuses” machine learning systems.

In the data preprocessing step, we have managed to remove some obviously

uninteresting attributes. However, there are still hundreds of attributes in the dataset

some of which may be irrelevant and could probably cause negative effects to the

machine learning schemes. Weka’s attribute selection panel provides us a tool to

specify an attribute subset evaluator and a search method. Subset evaluator takes a

subset of attributes and returns a numeric measure that guides the search. For

instance, CfsSubsetEval assesses the predictive ability of each attribute individually

and the degree of redundancy among them, preferring sets of attributes that are highly

correlated with the class but have low inter-correlation. Search methods traverse that

attribute space to find a good subset whose quality is measured by the chosen attribute

subset evaluator. For example, BestFirst performs greedy hill climbing with

backtracking, while RankSearch sorts attributes using a single-attribute evaluator and

then ranks promising subsets using an attribute subset evaluator. By this means, we

do a lot of experiments to specify certain attributes that should be retained for model

training as shown in the following list.

Table 3 The List of Retained Attributes in the Attribute Selection Step

AGE VAS_CND_FLAG AVG_MINS_INTT3

MARITAL_STATUS VAS_CNND_FLAG AVG_VAS_GAMES

HIGHEND_PROGRAM_FLAG VAS_NR_FLAG AVG_VAS_GPRS

NUM_ACT_TEL VAS_VM_FLAG AVG_VAS_CWAP

NUM_DELINQ_TEL VAS_AR_FLAG STD_VAS_GAMES

HS_AGE AVG_CALL_OB STD_VAS_GPRS

HS_MODEL AVG_CALL_MOB STD_VAS_CWAP

LOYALTY_POINTS_USAGE AVG_MINS_MOB STD_VAS_ESMS

BLACK_LIST_FLAG AVG_BILL_AMT CUSTOMER_TYPE

3.2 Decision Tree

The first machine learning scheme that we use to train the customer type model

derives from the simple divide-and-conquer algorithm for producing decision trees.

We employ the popular decision tree algorithm C4.5 to construct model, which, with

its commercial successor C5.0, has emerged as the industry workhorse or off-the-shelf

machine learning. Compared with algorithm ID3, C4.5 has made a series of

improvements, which include methods dealing with numeric attributes, missing

values, noisy data, and generating rules from trees. Figure 2 shows the whole C4.5

training procedure and provides us a platform to think in terms of how data flows

through the system.

Figure 2 The Weka Knowledge Flow Interface

It is a surprising fact that the decision tree we construct with C4.5 is of great scale.

Most popular attribute seems to be HS_MODEL (Handset Model) where the tree

splits. There are 332 distinct handset models in the data set which could be merged

into several categories described in Table 4 and Figure 3. We hope that, by this

means, the decision tree induced from training data can be simplified, without loss of

accuracy.

Table 4 Handset Models Merge into Different Categories

Handset Model Category Characteristic

baicj, bagia, bagib, bagic, bagid, baicb, bgcbj, bbcea, beccj (Total number: 9)

3G Very high proportions of customers with these handset models are 3G customers.

bahji, baggb, bdaac, bgbba, baggg, bagfc... (Total number: 145)

2G Few customers own handset of these kinds, and according to the training and test set, these handsets can only support 2G services.

Begac, bbacj, begaf, bdaab, bbafa, baibj… (Total number: 148)

0to1 Normal proportion patterns: 2G vs. 3G; instance number ranges from 0 to 100.

Bbbei, bbcdd, bbadi, bagfh, bgcac, bajci (Total number: 25)

1to5 Normal proportion patterns: 2G vs. 3G; instance number ranges from 101 to 500;

bgcab, baiai, baiaj, bbcch, bahfi (Total number:5)

5to10 Normal proportion patterns: 2G vs. 3G; instance number ranges from 501 to 1000.

Figure 3 Handset Models Merge into Different Categories (in the histogram: blue stands for

3G instances while red stands for 2G instances)

Experiments have shown that this hardly affects the classification accuracy of C4.5

(evaluated by 10 times cross-validation). What the technique does affect is decision

tree size. The resulting trees are invariably much smaller than the original ones, even

though they perform almost the same. This comparison is shown in Table 5.

Table 5 Accuracy and Tree Size Comparison of Different Decision Tree Models

Before Merging Handset Models

After Merging Handset Models

Correctly Classified Instances 16124 (89.5778 %) 16110 (89.5%)

Confusion Matrix 3G 2G 1505 1495 | 3G 381 14619 | 2G

3G 2G 1614 1386 | 3G 504 14496 | 2G

Number of Leaves 773 240

Size of the tree 857 470

3.3 Lazy Classifier

We did plenty of experiments to find the second appropriate approach solving the

given classification problem to be Lazy learners, which store the training instances

and do no real work until classification time. IB1 is a basic instance-based learner

which finds the training instance closest in Euclidean distance to the given test

instance and predicts the same class as this training instance. If several instances

qualify as the closest, the first one found is used. IBk is a k-nearest-neighbors which

can be specified explicitly or determined automatically using leave-one-out cross-

validation, subject to an upper limit given by the specified value. In addition,

predictions from more than one neighbor can be weighted according to their distance

from the test instance. We set k to 3 to trade off between accuracy and complexity.

Results could be found in Table 6.

Table 6 Accuracy Comparison of Different Lazy Classifier

1 nearest neighbour 3 nearest neighbour

Correctly Classified Instances 14919 (85.0522 %) 15488 (88.1152 %)

Confusion Matrix 3G 2G1538 1413 | 3G1209 13381 | 2G

3G 2G1501 1477 | 3G

612 13987 | 2G

3.4 Boosting

When wise people make critical decisions, they usually take into account the

opinions of several experts rather than relying on their own judgment or that of a

solitary trusted adviser. In data mining, an obvious approach to making decisions

more reliable is to combine the output of different models. The boosting method for

combining multiple models exploits this insight by explicitly seeking models that

complement one another. Boosting uses voting to combine the output of individual

models which are of the same type (here we employ C4.5). In boosting each new

model is influenced by the performance of those built previously, which means that

boosting encourages new models to become experts for instances handled incorrectly

by earlier ones. Boosting perhaps is the most powerful method to achieve true

positives. Results of boosting with learner AdaBoostingM1 is shown in Table 7.

Table 7 Results of Boosting with learner AdaBoostM1

Boosting with C4.5

Correctly Classified Instances 15571 (88.5874 %)

Confusion Matrix 3G 2G1775 1203 | 3G803 13796 | 2G

4. Conclusions

We have constructed many models with different machine learning algorithms.

According to 10 times cross-validation performance results, identifying true negatives

(2G customers) is much easier than identifying true positives (3G customers). Since

the number of 3G instances in the training and test set is much smaller than the 2G’s,

there is not enough information identifying 3G customers out of massive 2G

customers. The data set at hand is limited, but we can make use of all kinds of

potentially useful data in the future to build a strong enough dataset in the first place.

As mentioned before, the real data is imperfect: Some parts are garbled, and some

are missing. Anything discovered will be inexact: There will be exceptions to every

rule and cases not covered by any rule. Therefore, algorithms need to be robust

enough to cope with imperfect data and to extract regularities that are inexact but

useful. There is no perfect thing in the universe, what we could do is to dig a little

deeper to make better prediction and make a step closer to the truth.

We combined all the meaningful models together to vote for the final decision on

classification. It is an exiting fact that the overlapped part of the classification results

derived from different algorithms and concepts is great (see Table 8), which also

proves that the models we gained is dependable to make future prediction. Finally we

successfully identified 884 3G customers out of the prediction set. The decision trees

derived from the given training and test set is of great scale, so we visualize one of

them in text form as an example, which can be found in Appendix A.

Table 8 A segment of Prediction Results Decided by Three Different Algorithms: AdaBoostM1,

LazyIBk, C4.5

AdaBoostM1 LazyIBk C4.5 Comparison

3G 3G 3G

2G 2G 2G

2G 2G 2G

2G 2G 2G

2G 2G 2G

2G 2G 2G

2G 2G 2G

2G 2G 2G

2G 3G 3G +

2G 2G 2G

2G 2G 2G

2G 2G 2G

2G 2G 2G

2G 3G 3G +

3G 2G 2G +

2G 2G 2G

2G 2G 2G

3G 3G 3G

2G 3G 3G +

2G 2G 2G

2G 3G 3G +

2G 2G 2G

2G 2G 2G

2G 2G 2G

3G 3G 3G

2G 2G 2G

3G 3G 3G

The “false positives” which belong to 2G category but are classified into 3G

category answers our remaining question - how to identify current 2G customers with

the potential switching to 3G. According to the trained model, the reason why those

false positives are recognized to be 3G customers is that they must have something in

common with the real 3G customers. Some of them may use brand new handset

model which is developed to fulfill 3G purpose, some of them may play a lot of

games, while others would like to download via GPRS. The false positives in the

confusion matrix could tell us the truth and simply give us valuable information to

which customers the operators should pay more attention, when they intend to gain

more 3G customers and increase their customer spending and profitability.

Appendix A

=== Run information ===

Scheme: weka.classifiers.trees.J48 -C 0.25 -M 2

Relation: Classify_2G_3G-weka.filters.unsupervised.instance.RemoveWithValues-S0.0010-C71-Lfirst-last-

weka.filters.unsupervised.attribute.Remove-R2,4-6,8-15,17,19-21,23,25-28,31-46,49-50,53-54,56-63,65-75,77-

88,91-104,106-115,117,119-121,123-193,195,197,200-219

Instances: 17577

Attributes: 27

AGE

MARITAL_STATUS

HIGHEND_PROGRAM_FLAG

NUM_ACT_TEL

NUM_DELINQ_TEL

HS_AGE

HS_MODEL

LOYALTY_POINTS_USAGE

BLACK_LIST_FLAG

VAS_CND_FLAG

VAS_CNND_FLAG

VAS_NR_FLAG

VAS_VM_FLAG

VAS_AR_FLAG

AVG_BILL_AMT

AVG_CALL_OB

AVG_CALL_MOB

AVG_MINS_MOB

AVG_MINS_INTT3

AVG_VAS_GAMES

AVG_VAS_GPRS

AVG_VAS_CWAP

STD_VAS_GAMES

STD_VAS_GPRS

STD_VAS_CWAP

STD_VAS_ESMS

CUSTOMER_TYPE

Test mode: 10-fold cross-validation

=== Classifier model (full training set) ===

J48 pruned tree

------------------

HS_AGE <= 3

| HS_MODEL = 5to10

| | AVG_VAS_GPRS <= 0

| | | HS_AGE <= 1

| | | | AVG_VAS_GAMES <= 6292.17

| | | | | HIGHEND_PROGRAM_FLAG = TRUE: 3G (4.0/1.0)

| | | | | HIGHEND_PROGRAM_FLAG = FALSE

| | | | | | HS_AGE <= 0

Time taken to build model: 24.43 seconds

=== Stratified cross-validation ===

=== Summary ===

Correctly Classified Instances 15700 89.3213 %

Incorrectly Classified Instances 1877 10.6787 %

Kappa statistic 0.5748

Mean absolute error 0.156

Root mean squared error 0.3041

Relative absolute error 55.4236 %

Root relative squared error 81.0696 %

Total Number of Instances 17577

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure Class

0.963 0.45 0.913 0.963 0.937 2G

0.55 0.037 0.753 0.55 0.636 3G

=== Confusion Matrix ===

a b <-- classified as

1637 1341 | a = 3G

536 14063 | b = 2G

Open06

Documents

Transcript of Open06