1
CPS 196.03: Information Management and Mining
Association Rules and Frequent Itemsets
2
Let Us Begin with an Example
A common marketing problem: examine what people buy together to discover patterns.
1. What pairs of items are unusually often found together at Kroger checkout?• Answer: diapers and beer.
2. What books are likely to be bought by the same Amazon customer?
3
Caveat A big risk when data mining is that
you will “discover” patterns that are meaningless.
Statisticians call it Bonferroni’s principle: (roughly) if you look in more places for interesting patterns than your amount of data will support, you are bound to find “false patterns”.
4
Rhine Paradox --- (1) David Rhine was a parapsychologist in
the 1950’s who hypothesized that some people had Extra-Sensory Perception.
He devised an experiment where subjects were asked to guess 10 hidden cards --- red or blue.
He discovered that almost 1 in 1000 had ESP --- they were able to get all 10 right!
5
Rhine Paradox --- (2) He told these people they had ESP
and called them in for another test of the same type.
Alas, he discovered that almost all of them had lost their ESP.
What did he conclude? Answer on next slide.
6
Rhine Paradox --- (3) He concluded that you shouldn’t
tell people they have ESP; it causes them to lose it.
7
“Association Rules”
Market BasketsFrequent ItemsetsA-priori Algorithm
8
The Market-Basket Model A large set of items, e.g., things
sold in a supermarket. A large set of baskets, each of
which is a small set of the items, e.g., the things one customer buys on one day.
9
Association Rule Mining
tran1 cust33 p2, p5, p8tran2 cust45 p5, p8, p11tran3 cust12 p1, p9tran4 cust40 p5, p8, p11tran5 cust12 p2, p9tran6 cust12 p9
transaction
id customer
id products
bought
salesrecords:
• Trend: Products p5, p8 often bought together
market-basketdata
10
Support Simplest question: find sets of items
that appear “frequently” in the baskets.
Support for itemset I = the number of baskets containing all items in I.
Given a support threshold s, sets of items that appear in > s baskets are called frequent itemsets.
11
Example Items={milk, coke, pepsi, beer, juice}. Support = 3 baskets.
B1 = {m, c, b} B2 = {m, p, j}B3 = {m, b} B4 = {c, j}B5 = {m, p, b} B6 = {m, c, b, j}B7 = {c, b, j} B8 = {b, c}
What are the possible itemsets? The Lattice of itemsets
How would you find the frequent itemsets?
12
Example Frequent itemsets: {m}, {c}, {b},
{j}, {m, b}, {c, b}, {j, c}.
13
Applications --- (1) Real market baskets: chain stores
keep terabytes of information about what customers buy together. Tells how typical customers navigate
stores, lets them position tempting items. Suggests tie-in “tricks,” e.g., run sale on
diapers and raise the price of beer. High support needed, or no $$’s .
14
Applications --- (2) “Baskets” = documents; “items” =
words in those documents. Lets us find words that appear together
unusually frequently, i.e., linked concepts. “Baskets” = sentences, “items” =
documents containing those sentences. Items that appear together too often could
represent plagiarism.
15
Applications --- (3) “Baskets” = Web pages; “items” =
linked pages. Pairs of pages with many common
references may be about the same topic. Ex: think of our two data mining textbooks
“Baskets” = Web pages p ; “items” = pages that link to p . Pages with many of the same links may be
mirrors or about the same topic. Ex: think of people with similar interests
16
Important Point “Market Baskets” is an abstraction
that models any many-many relationship between two concepts: “items” and “baskets.” Items need not be “contained” in baskets.
The only difference is that we count co-occurrences of items related to a basket, not vice-versa.
17
Scale of Problem WalMart sells 100,000 items and
can store billions of baskets. The Web has over 100,000,000
words and billions of pages.
18
Association Rules If-then rules about the contents of
baskets. {i1, i2,…,ik} → j means: “if a basket
contains all of i1,…,ik then it is likely to contain j.
Confidence of this association rule is the probability of j given i1,…,ik.
19
ExampleB1 = {m, c, b} B2 = {m, p, j}B3 = {m, b} B4 = {c, j}B5 = {m, p, b} B6 = {m, c, b, j}B7 = {c, b, j} B8 = {b, c}
An association rule: {m, b} → c. Confidence = 2/4 = 50%.
+__ +
20
Interest The interest of an association rule
is the absolute value of the amount by which the confidence differs from what you would expect, were items selected independently of one another.
21
ExampleB1 = {m, c, b} B2 = {m, p, j}B3 = {m, b} B4 = {c, j}B5 = {m, p, b}B6 = {m, c, b, j}B7 = {c, b, j} B8 = {b, c}
For association rule {m, b} → c, item c appears in 5/8 of the baskets.
Interest = | 2/4 - 5/8 | = 1/8 --- not very interesting.
22
Relationships Among Measures Rules with high support and
confidence may be useful even if they are not “interesting.” We don’t care if buying bread causes
people to buy milk, or whether simply a lot of people buy both bread and milk.
But high interest suggests a cause that might be worth investigating.
23
Finding Association Rules A typical question: “find all association
rules with support ≥ s and confidence ≥ c.” Note: “support” of an association rule is the
support of the set of items it mentions. Hard part: finding the high-support
(frequent ) itemsets. Checking the confidence of association rules
involving those sets is relatively easy.
24
Finding Association Rules Two-step approach:
1. Frequent Itemset Generation– Generate all itemsets whose support minsup
2. Rule Generation– Generate high confidence rules from each
frequent itemset, where each rule is a binary partitioning of a frequent itemset
Frequent itemset generation is still computationally expensive
25
Computation Model Typically, data is kept in a “flat
file” rather than a database system. Stored on disk. Stored basket-by-basket. Expand baskets into pairs, triples, etc.
as you read baskets.
26
Computation Model --- (2) The true cost of mining disk-resident
data is usually the number of disk I/O’s.
In practice, association-rule algorithms read the data in passes --- all baskets read in turn.
Thus, we measure the cost by the number of passes an algorithm takes.
27
Main-Memory Bottleneck In many algorithms to find frequent
itemsets we need to worry about how main memory is used. As we read baskets, we need to count
something, e.g., occurrences of pairs. The number of different things we can
count is limited by main memory. Swapping counts in/out is a disaster.
28
Finding Frequent Pairs The hardest problem often turns
out to be finding the frequent pairs.
We’ll concentrate on how to do that, then discuss extensions to finding frequent triples, etc.
29
The Lattice of ItemSetsnull
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCDE
Given d items, there are 2d possible candidate itemsets
30
Naïve Algorithm A simple way to find frequent pairs
is: Read file once, counting in main
memory the occurrences of each pair.• Expand each basket of n items into its
n (n -1)/2 pairs. Fails if #items-squared exceeds
main memory.
31
Details of Main-Memory Counting
There are two basic approaches:1. Count all item pairs, using a triangular
matrix.2. Keep a table of triples [i, j, c] = the
count of the pair of items {i,j } is c. (1) requires only (say) 4 bytes/pair;
(2) requires 12 bytes, but only for those pairs with >0 counts.
32
4 per pair
Method (1) Method (2)
12 peroccurring pair
33
Details of Approach (1) Number items 1,2,… Keep pairs in the order {1,2},
{1,3},…, {1,n }, {2,3}, {2,4},…,{2,n }, {3,4},…, {3,n },…{n -1,n }.
Find pair {i, j } at the position (i –1)(n –i /2) + j – i.
Total number of pairs n (n –1)/2; total bytes about 2n 2.
34
Details of Approach (2) You need a hash table, with i and j as
the key, to locate (i, j, c) triples efficiently. Typically, the cost of the hash structure can
be neglected. Total bytes used is about 12p, where p is
the number of pairs that actually occur. Beats triangular matrix if at most 1/3 of
possible pairs actually occur.
35
A-Priori Algorithm --- (1) A two-pass approach called a-priori
limits the need for main memory. Key idea: monotonicity : if a set of
items appears at least s times, so does every subset. Contrapositive for pairs: if item i does
not appear in s baskets, then no pair including i can appear in s baskets.
36
Found to be Infrequent
null
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCDE
Illustrating Apriori Principlenull
AB AC AD AE BC BD BE CD CE DE
A B C D E
ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE
ABCD ABCE ABDE ACDE BCDE
ABCDEPruned supersets
37
Consider the following market-basket data
Market-Basket transactions
TID Items
1 Bread, Milk
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke 4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke
Illustrating Apriori Principle
38
Illustrating Apriori Principle
Item CountBread 4Coke 2Milk 4Beer 3Diaper 4Eggs 1
Itemset Count{Bread,Milk} 3{Bread,Beer} 2{Bread,Diaper} 3{Milk,Beer} 2{Milk,Diaper} 3{Beer,Diaper} 3
Itemset Count {Bread,Milk,Diaper} 3
Items (1-itemsets)
Pairs (2-itemsets)
(No need to generatecandidates involving Cokeor Eggs)
Triplets (3-itemsets)Minimum Support = 3
If every subset is considered, 6C1 + 6C2 + 6C3 = 41
With support-based pruning,6 + 6 + 1 = 13
39
A-Priori Algorithm --- (2) Pass 1: Read baskets and count in main
memory the occurrences of each item. Requires only memory proportional to #items.
Pass 2: Read baskets again and count in main memory only those pairs both of which were found in Pass 1 to be frequent. Requires memory proportional to square of
frequent items only.
40
Picture of A-Priori
Item counts
Pass 1 Pass 2
Frequent items
Counts ofcandidate pairs
41
Detail for A-Priori You can use the triangular matrix
method with n = number of frequent items. Saves space compared with storing
triples. Trick: number frequent items 1,2,
… and keep a table relating new numbers to original item numbers.
42
Frequent Triples, Etc. For each k, we construct two sets
of k –tuples: Ck = candidate k – tuples = those
that might be frequent sets (support > s ) based on information from the pass for k –1.
Lk = the set of truly frequent k –tuples.
43
C1 L1 C2 L2 C3Filter Filter ConstructConstruct
Firstpass
Secondpass
44
A-Priori for All Frequent Itemsets
One pass for each k. Needs room in main memory to
count each candidate k –tuple. For typical market-basket data and
reasonable support (e.g., 1%), k = 2 requires the most memory.
45
Frequent Itemsets --- (2) C1 = all items L1 = those counted on first pass to be
frequent. C2 = pairs, both chosen from L1. In general, Ck = k –tuples each k –1 of
which is in Lk-1. Lk = those candidates with support ≥ s.
Top Related