1
Plan B Defense –Context-Inclusive Approach to Speed-up Function
Evaluation for Statistical Queries: An Extended Abstract
Vijay GandhiMasters Student
AdvisorDr. Shashi Shekhar
Committee MembersDr. Bradley Carlin
Dr. Jaideep Srivastava
Department of Computer ScienceUniversity of Minnesota, USA
Biography Bachelor of Computer Science & Engineering, Madras University Master of Science, Computer Science, University of Minnesota (current) 3 years of work experience in the field of Data warehousing and Business
Intelligence at Oracle Corporation
Publications
Context-Inclusive Approach to Speed-up Function Evaluation for Statistical Queries: An Extended Abstract, Vijay Gandhi, James Kang, Shashi Shekhar, Junchang Ju, Eric Kolaczyk, Sucharita Gopal. IEEE International Conference on Data Mining Workshop on Spatio and Spatio-Temporal Data Mining (SSTDM), 2006
Parallelizing Multi-scale and Multi-granular Spatial Data Mining Algorithm, Vijay Gandhi, Mete Celik, Shashi Shekhar. PGAS Programming Models Conference 2006
Context-Inclusive Approach to Speed-up Function Evaluation for Statistical Queries, Vijay Gandhi, James Kang, Shashi Shekhar, Junchang Ju, Eric Kolaczyk, Sucharita Gopal. Submitted to the Journal on Knowledge and Information Systems, 2007
2
Scope of the talk NSF Project on Land-use classification Joint collaboration with Boston University Main goal: Reduce the execution time of the algorithm Contributions:
Approach Description Results Effort (in hrs)
Black box tuning Limiting factor was modified to control
the precision
Reduced CPU time by 50% with
98% accuracy
20
Code Conversion Code was converted from MATLAB to C
Reduced CPU time by about 50%
120
Parallel implementation
Converted code from C to UPC on
Cray X1
Obtained near linear scalability
20
Algorithm Changes
Context-Inclusive approach
100
3
4
Overview Motivation Background Problem Statement Related Work Contribution Validation Conclusion & Future Work
5
Motivation Land-cover Change
Loss of land - 217 square miles of Louisiana’s coastal lands were transformed to water after Hurricanes Katrina and Rita.
Deforestation – Brazil lost 150,000 sq. km. of forest between
May 2000 and August 2006 Urban Sprawl
Mississippi River Delta, Louisiana(Red represents land loss between 2004
and 2005. Courtesy: USGS)
Deforestation, Ariquemes, Brazil(Courtesy: Global Change Program,
University of Michigan)
Urban Sprawl in Atlanta(Red indicates expansion between
1976 and 1992)
6
Multiscale Multigranular Image Classification (MSMG) Input: Class hierarchy, Likelihood of specific classes
Conifer Hardwood Brush Grass
Likelihood of specific-classesLand-use Class Hierarchy
Output: Classified images at multiple scales
Scale: 2x2 Scale: 4x4 Scale: 64x64. . .
Scale: 1x1
Source courtesy: Boston University
Background Algorithm
Divide input image into quad segments recursively
Calculate the log-likelihood for each class using Expectation Maximization
Decide the class for each segment
ProfilingInput Image: 128 x 128
Function # Calls % Time
gen_loglike 294,915 80
7
Background Gen_loglike
gen_loglike calculates the log-likelihood of a non-specific class It uses Expectation Maximization # gen_loglike = f ( # general classes, image size, spatial scale)
Number of iterations depends on the number of general class, image size, and spatial scale
8
10
Problem Statement Given:
Algorithm for Multiscale Multigranular Image Classification Input image with likelihood of specific classes, class hierarchy
Find: Classification at each quad segment
Objective: Minimize computation time
Constraints: Use the Expectation Maximization (EM) algorithm to calculate
quality measure of each non-specific class High Accuracy
Classification Examples Likelihood of classes are compared to
find the best candidate
C
C1 C2
0.9 0.9
0.9 0.9
LikelihoodC1
0.1 0.1
0.1 0.1
LikelihoodC2
Best Candidate
C1
0.9 0.9
0.9 0.9
0.1 0.1
0.1 0.1
C2
0.1 0.9
0.1 0.9
0.9 0.1
0.9 0.1
C
0.9 0.1
0.1 0.9
0.1 0.9
0.9 0.1
C
0.8 0.8
0.1 0.1
0.2 0.2
0.9 0.9
C
0.75 0.75
0.1 0.1
0.25 0.25
0.9 0.9
C1
3.6
0.4
2.0
2.0
2.2
2.3
0.4
3.6
2.0
2.0
1.8
1.7
11
12
Algorithm: Expectation Maximization Given:
Class hierarchy, Likelihood of specific classes
Find: Best Class and corresponding likelihood for a region (e.g. 2x2 region)
Likelihood of a specific class = sum of corresponding likelihood
(Likelihood of C1 = 2.2; Likelihood of C2 = 1.8 ) Log-likelihood of best specific class = -3.4296 (C1)
Likelihood of non-specific class (EM):
1. Initialize the proportion of each corresponding specific class
2. Multiply each likelihood by corresponding specific class proportion
3. Add the likelihood at corresponding pixel
4. Divide the value in step 1 by corresponding value in Step 2
5. Average the likelihood for each specific class
6. Repeat Step 2 to Step 5 until required accuracy
Likelihood of classes C1 and C2
at a 2x2 region
C
C1 C2
Class hierarchy
Lij(C1) Lij(C2)
0.8 0.8
0.1 0.1
0.2 0.2
0.9 0.9
Example
13
Execution Trace: Expectation Maximization
Given: Class hierarchy, Likelihood of
specific classes
Find: Best Class for the 2x2 region
Likelihood of C:
1. Iteration 1: EM(p1n, p2n)
2. Multiply: L1ij(C1) = Lij(C1) . p1n; L2ij(C2) = Lij(C2) . p2n
3. Add: Lij = L1ij(C1) .+ L2ij(C2)
4. Divide: L1ij(C1) = L1ij(C1)./Lij; L2ij(C2) = L2ij(C2)./Lij
5. Average: p1n+1 = Avg(L1ij(C1)); p2n+1 = Avg(L2ij(C2))
0.1 0.1
0.45 0.45
0.4 0.4
0.05 0.05
EM(0.5, 0.5)
0.5 0.5
0.5 0.5
0.2 0.2
0.9 0.9
0.8 0.8
0.1 0.1
0.55, 0.45
Likelihood of classes C1 and C2
at a 2x2 region
C
C1 C2
Class hierarchy
Lij(C1) Lij(C2)
0.8 0.8
0.1 0.1
0.2 0.2
0.9 0.9
After 17 iterations: Mixing proportions = (0.6042, 0.3958)
Likelihood of C:
Log-likelihood of best specific class = -3.4296 (C1) Log-likelihood considering penalties:
C (penalty: 4.3922) = -7.1235 C1 (penalty: 4.0456) = -7.4752
Best class: Class with maximum log-likelihood C1
Execution Trace: Expectation Maximization
Likelihood of classes C1 and C2
at a 2x2 region
C
C1 C2
Lij(C1) Lij(C2)
0.8 0.8
0.1 0.1
0.2 0.2
0.9 0.9
0.31664 0.31664
0.03958 0.03958
0.12084 0.12084
0.10875 0.10875
Log-likelihood of C = -2.7314
14
18
Related Work
EM Performance improvement
Single candidate[ Aitken’s Method ][ Triple Jump – Huang et al.]
Multiple candidate[Context-Inclusive]
19
Context-Exclusive Approach Instance Tree
Each candidate model is analyzed independently until convergence
The candidate model with maximum likelihood is selected
Instance Tree
Context-Exclusive Approach:1. Select the best specific class, Brush
(Specific classes do not require EM)2. Vegetation is evaluated until convergence (46)3. Forest is evaluated until convergence (34)4. Non-Forest is evaluated until convergence (3)5. Select the best class (Non-Forest)
1.2.
3.4.
1
2
3
4
Land-use Class Hierarchy
Total iterations: 46 + 34 + 3 = 83
20
Limitations of Context-Exclusive Approach Computational Scalability
For 512 x 512 pixels - 7 hours of CPU time Where is the computational bottleneck?
80% of total execution time is spent in computing maximum likelihood
Number of function calls is dependent on the number of pixels, and spatial scale
CPU Time for example datasets
As spatial scale increases, the computation time increases exponentially
Contributions – Context Inclusive Approaches Context-Inclusive Approaches – Ideal and Heuristic Context-Inclusive Approach – Ideal
If we may calculate a theoretical upper bound on likelihood for each class, it may be used to filter candidates
If upper bound calculation is not significant and has good filtering property, Context-Inclusive approach may perform better
21
Land-use Class Hierarchy
Context-Inclusive Approach - Ideal:1. Select the best specific class, Brush (Specific classes do not require EM)2. Calculate upper bound for each non-specific class (Non-forest , Forest , Vegetation )3. Assume Non-forest is evaluated first (3 iteration)4. We may prune Forest because upper bound of Forest is less than likelihood of Non-forest (0 iteration)5. We may prune Vegetation because upper bound of Vegetation is less than likelihood of Non-forest (0 iteration)6. Non-forest is selected
Total EM iterations: 3+ 0 + 0 = 3
Context-Inclusive Approach – Ideal Lemma
Context Inclusive is correct such that each region is classified as with the best candidate class from the user-defined concept hierarchy. Likelihood value of a non-specific class can never go beyond its upper bound Upper bound can be used to compare the likelihood of non-specific classes
Discussion Is there any method to calculate the upper bound? C. Biernacki. An Asymptotic Upper Bound of the Likelihood to Prevent Gaussian
Mixtures from Degenerating,Preprint, 2005 Expensive
22
23
Context Inclusive Approach - HeuristicInstance Tree is evaluated with context
Each candidate model is analyzed until it is better than the current best
Uses a instance-level syntax tree
Context-Inclusive Approach - Heuristic:1. Select the best specific class, Brush (Specific classes do not require EM)2. Vegetation is evaluated until convergence (46)3. Forest is evaluated (4)4. Non-Forest is evaluated (1)5. Non-Forest is the best-so-far
1.2.
3.4.
1
2
3
4 Land-use Class Hierarchy
Total iterations: 46 + 4 + 1 = 51
24
Context-Exclusive vs. Context-Inclusive HeuristicAlgorithm 1 Context-Exclusive Approach
1: Function ContextExclusive(set Cand)2: Select the best specific class3: for each candidate model c Cand do4: repeat5: Refine quality measure for each candidate model c Cand6: until EM converges7: end for8: Select candidate model with the maximum quality measure9: return c
1: Function ContextInclusive(set Cand)2: Select the best specific class3: for each remaining candidate model c Cand do4: repeat5: Refine quality measure for each candidate model c Cand 6: until EM converges OR quality measure exceeds best so far7: end for8: Select candidate model that is best so far9: return c
Algorithm 2 Context-Inclusive Approach - Heuristic
26
Experimental Design
Input: Synthetic dataset and Real dataset Language: MATLAB Platform: UltraSparc III 1.1 GHz, 1 GB RAM Measurements: Number of Iterations, CPU Time, Accuracy
ImageClassification
Benchmark Datasets
Limiting Factor
Measurements
Experimental Design
Experimental Questions: How does Context-Exclusive compare to Context-Inclusive Heuristic
approach? Accuracy Computational Efficiency
Candidates:Context-Exclusive,Context-Inclusive Heuristic
CompareClassifications
ClassificationAccuracy
Synthetic
Real
27
Experiments – Dataset S Synthetic Dataset 128 x 128 pixels, 7 Classes Input: Class hierarchy, Likelihood of specific classes
Conifer Hardwood Brush Grass
Likelihood of specific-classesLand-use Class Hierarchy
Output: Classified images at multiple scales
Scale: 2x2 Scale: 4x4 Scale: 64x64. . .
Scale: 1x1
28
Experiments – Dataset R Real Dataset; Plymouth County, Massachusetts 128 x 128 pixels, 12 Classes Input: Class hierarchy, Likelihood of specific classes
Land-use Class Hierarchy
Output: Classified images at multiple scales
Scale: 2x2 Scale: 4x4 Scale: 64x64. . .
Barren Brush Pitch Pine Bogs
…
Scale: 1x1
How accurate is Context-Inclusive as compared to Context-Exclusive?
Accuracy (Limiting Factor = 0.00001)
Accuracy of Above 99% for Synthetic Dataset About 98% for Real Dataset
30
31
How does computation of Context-Exclusive Compare that of Context-Inclusive? Number of Iterations (Limiting Factor: 0.00001)
Iterations reduced 67% for Synthetic Dataset 61% for Real Dataset
Dataset S Dataset R
32
CPU Time (Limiting Factor = 0.00001)
CPU Time reduced 53% for Synthetic Dataset 47% for Real Dataset
How does computation of Context-Exclusive compare to that of Context-Inclusive?
Dataset S Dataset R
33
Conclusion Context-Inclusive approach for function evaluation Experimental results supporting contributions Reduced the CPU Time by 50% without sacrificing
accuracy Future work: Context-Inclusive with Upper bound
SummaryApproach Description Results Effort (in hrs)
Black box tuning Limiting factor was modified to control
the precision
Reduced CPU time by 50% with 98%
accuracy
20
Code Conversion Code was converted from MATLAB to C
Reduced CPU time by about 50%
120
Parallel implementation
Converted code from C to UPC on
Cray X1
Obtained near linear scalability
20
Algorithm Changes Context-Inclusive approach
Reduced CPU time without sacrificing
accuracy
100
34
Acknowledgements
James Kang Dr. Junchang Ju Dr. Eric Kolaczyk Dr. Sucharita Gopal
35
40
Forest
Non-Forest
VegetationL1
Context-Exclusive Approach Instance Tree
Each candidate model is analyzed independently until convergence
The candidate model with maximum likelihood is selected
Instance Tree
Iterations
Qua
lity
Mea
sure
Context-Exclusive Approach:1. Vegetation is evaluated until convergence, L12. Forest is evaluated until convergence, L23. Non-Forest is evaluated until convergence, L3
L2
L3
41
Contributions Context Inclusive Approach Instance Tree is evaluated with context
Each candidate model is analyzed until it is better than the current best
Uses a instance-level syntax tree
Context-Inclusive Approach:1. Vegetation is evaluated until convergence, L12. Forest is evaluated until L23. Non-Forest is evaluated until L3
Forest
Non-Forest
Vegetation
Iterations
Qua
lity
Mea
sure
L1L2
L3
Top Related