FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi...
Transcript of FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi...
![Page 1: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/1.jpg)
FOREPOST: A Tool For Finding Performance Problems Automatically with Feedback-
Directed Learning Software Testing
Qi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik*
College of William and Mary
*University of Illinois at Chicago
ICSE 2016
Austin, TX, U.S.
1
![Page 2: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/2.jpg)
Performance Testing
Inputs Subjects
2
Profiling tools
![Page 3: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/3.jpg)
Performance Testing
Inputs Subjects
3
Performance bottlenecks
Profiling tools
1. Agilefant.model.WidgetCollection.getName() 273.2s2. Agilefant.db.hibernate.UserTypeFilter.deepCopy() 213.5s3. Agilefant.model.Team.setId() 192.3s4. Agilefant.model.Backlog.setChildren() 123.9s5. ……
![Page 4: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/4.jpg)
4
78 Million customer profiles
![Page 5: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/5.jpg)
5
78 Million customer profiles
902 Days !!!
![Page 6: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/6.jpg)
6
Facts:
1. California and Texas
![Page 7: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/7.jpg)
7
Facts:
1. California and Texas2. Southwestern
![Page 8: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/8.jpg)
8
Facts:
1. California and Texas2. Southwestern 3. Oklahoma
![Page 9: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/9.jpg)
9
Facts:
1. California and Texas2. Southwestern 3. Oklahoma
WILDFIRE => bottlenecks
![Page 10: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/10.jpg)
• Using machine learning algorithms to extract rules for selecting test input data
• Using Independent Component Analysis (ICA) to identify performance bottlenecks automatically
Feedback-ORiEnted PerfOrmanceSoftware Testing (FOREPOST)
10
![Page 11: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/11.jpg)
• Using machine learning algorithms to extracts rules for selecting test input data
• Using Independent Component Analysis (ICA) to identify performance bottlenecks automatically
Input: test input data and binary codeOutput: a set of rules, a ranked list of methods
11
Feedback-ORiEnted PerfOrmanceSoftware Testing (FOREPOST)
![Page 12: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/12.jpg)
FOREPOST Foundation
ICSE’12EMSE’16
12
![Page 13: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/13.jpg)
13
FOREPOST
![Page 14: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/14.jpg)
14
FOREPOST
![Page 15: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/15.jpg)
15
FOREPOST
![Page 16: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/16.jpg)
16
FOREPOST
![Page 17: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/17.jpg)
17
FOREPOST
![Page 18: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/18.jpg)
• Test input data comes from existing repositories or databases
• Test scripts
18
![Page 19: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/19.jpg)
19
FOREPOST
![Page 20: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/20.jpg)
The Eclipse Test & Performance Tools Platform (TPTP) Profiling tool
20
![Page 21: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/21.jpg)
The Eclipse Test & Performance Tools Platform (TPTP) Profiling tool
21
![Page 22: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/22.jpg)
22
FOREPOST
![Page 23: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/23.jpg)
23
![Page 24: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/24.jpg)
24
FOREPOST
![Page 25: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/25.jpg)
Examples of rules:• (childOrAdultCareDetails.numberPersonsCaredForChild>3)
=> Good• (browse_title_ACADEMY_AFRICAN_2 > 5) => Bad• (viewPrdct_K9RT01 > 5) and (viewItem_EST16 > 5) => Bad• (storeStory3 > 100) and (storeProject6 < 25) => Bad
25
![Page 26: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/26.jpg)
26
![Page 27: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/27.jpg)
27
FOREPOST
![Page 28: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/28.jpg)
28
FOREPOST
![Page 29: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/29.jpg)
Bottlenecks => the most significant methods that occur in good traces but are not invoked or have little significance in bad traces
29
![Page 30: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/30.jpg)
30
Traces
Elap
sed
Exe
cuti
on
Tim
e
Bad
Good
Bad Bad
Good
![Page 31: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/31.jpg)
31
bottleneck
Bad
Good
Bad Bad
Good
Elap
sed
Exe
cuti
on
Tim
e
Traces
![Page 32: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/32.jpg)
32
![Page 33: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/33.jpg)
33
Experimental Design
JPetStore
Agilefant
Dell DVD Store
Renters
![Page 34: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/34.jpg)
34
Experimental Design
• Effectively finding input test data • FOREPOST & Random
• Identifying performance bottlenecks effectively• Injected performance bottlenecks• Real performance bottlenecks
![Page 35: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/35.jpg)
35
Experimental Results
• Effectively finding input test data that expose performance bottlenecks
• Identifying performance bottlenecks effectively
![Page 36: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/36.jpg)
36
![Page 37: FOREPOST: A Tool For Finding Performance Problems ...denys/pubs/talks/ICSE16-FOREPOST-DEMO.pdfQi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik* College of William and Mary *University](https://reader034.fdocuments.net/reader034/viewer/2022050518/5fa231df18162832555dbfe0/html5/thumbnails/37.jpg)
http://www.cs.wm.edu/semeru/data/ICSE16-FOREPOST/
Thank you!
Tool, source code, more materials:
37