Python Meetup Talk 21072009

IntroductionData Mining

And the results areA vision over the present and the future

Mining Software Repositories

Improving software

Pere Urbon Bayes

Data Management Group

Dept. Arquitectura de Computadors

Universitat Politecnica de Catalunya

[email protected]

July of 2009

Pere Urbon Bayes Mining Software Repositories



Index

Introduction

Data Mining

The results

The future




MotivationsThe SituationObjectives

The problem

Companies need to own highly available and reliable software.

The software of low quality harms both, clients and producers.

Unfortunately, avoiding defects is a difficult task to undertake.

Project Leaders need to keep an eye inside to many projects.

Software engineer tend not to document software in deep.

The complexity of software projects is growing every day.





The software development process





Support tools

Tools used to support software development:

Version Control server.

Bug Tracker server.

Project Management server.

Life cycle management software.

...

This set of tools store a huge amount of information during theprocess, Why not to use this information to improve our software?





Objective and Applications

Objectives:

Analyse the use of data mining technology, to data stored insupport tools, with the aim to improve software quality.

Develop an experimental prototype tool.

Applications:

Reduce the error rate.

Provides a non-exploited source of documentation.

Provide a new source of support tools for IDE’s.




IntroductionThe use of

Data mining

Type of database analysis that attempts to discover useful patternsor relationships in a group of data. The analysis uses advancedstatistical methods, such as cluster analysis, and sometimesemploys artificial intelligence or neural network techniques. Amajor goal of data mining is to discover previously unknownrelationships among the data, especially when the data come fromdifferent databases.





Methods

Types of:

Traditional Data Mining (K-Means, C4.5, Bayesian Networks).

Relational Data Mining (ILP, Markov logic networks,

Relational bayesian methods, Dependency Networks).

Categories:

Clusterers

Classifiers

Associative rules

Network Models.





Data mining

Type of database analysis that attempts to discover useful patternsor relationships in a group of data. The analysis uses advancedstatistical methods, such as cluster analysis, and sometimesemploys artificial intelligence or neural network techniques. Amajor goal of data mining is to discover previously unknownrelationships among the data, especially when the data come fromdifferent databases.





Issue detection

LOC DefectAppearence2Month RevisionsAuthor

LineAddedIRLAdd ReportedI2Month Revision2Month

LineAddedIRLDel Revision3Month Releases

AlterType DefectAppearence3Month ReportedI1Month

AgeMonths ReportedI3Month ReportedIssues

RevisionAge Revision5Month ReportedI5Month

DefectReleases DefectAppearence5Month

Revision1Month DefectAppearance1Month

Question: Has this file a non detected error. The exact number oferrors can be predicted to.





Another types of objectives

Predict bugs related to a software developer.

Prediction of bugs in software components.

This techniques could be used in different topics:

Software understanding.

Software evolution.

Software visualization.

Change propagation.

Impact analysis.

Software complexity.

Fault prediction.




Error predictionSoftware

Error prediction

Eclipse Project Firefox Project

Correctly classified 94.65% 94.822%

Statistics Kappa 0.893 0.8883

Precision 0.9465 0.9482

Recall 0.945 0.949

AUC ROC 0.9682 0.9808

Eclipse-Firefox Firefox-Eclipse

Correctly classified 82.0065% 87.975%

Statistics Kappa 0.5976 0.7595

Precision 0.818 0.894

Recall 0.82 0.88

AUC ROC 0.805 0.83




Error predictionSoftware

The end App




Software librariesAn envision

The Prototype

Software being used:

Programming: JAVA

Database: MySQL and MonetDB.

Data Mining: Weka 3.6 and Proximity 4.3

XML: Apache Xerces 2.9.1

SVN, CVS : svnkit 1.3.0, for CVS netbeans-cvs lib and acustom rcs file parser.

Presentation: Prefuse Visualization Toolkit and WekaDrawing facilities.





Could python give use the same?

Machine Learning:

Orange: With 1.0 this lib has many interesting and usefulmethods, Classification, Regression and Clustering. The mostsimilar to Weka.

PyML: Only has classifier facilities.

Shogun: Only for Support Vector Machines.

RPy: An interface to R.

Databases:

The most important relational databases are available viaDB-API.

ZODB: Zope Object Database.Metakit: An embedded database with a not defined paradigm.Pygr: Python graph database framework for bioinformatics.





Could python give use the same?

Presentation:

Graph Drawing: NetworkX, with nice result. There are someother but they look incomplete.

GUI: PyQT, wxWindows, pyGTK. It’s your taste XD!.

SVN, CVS processing:

SVN: pysvn - Python interface to Subversion.

CVS: It seams nothing is available.

GIT: PyGit - Pythonic git bindings targeted towardsporcelains.

XML Processing could be done using built-in support and with anySAX or DOM parser.





The future

Known issues:

Data preprocessing performance.

Database performance, is the relational model valid?

Dynamic procedure addition.

The Todo List:

Develop new procedures over different related topics, likesoftware visualization, change support, etc.

Develop a more mature software. Python could help in someparts. This software must be easily extensible.

Improve the hole process performance.





The end

Question?

Pere Urbon BayesData Management Group

Dept. Arquitectura de ComputadorsUniversitat Politecnica de Catalunya

[email protected]


Python Meetup Talk 21072009

Technology

Transcript of Python Meetup Talk 21072009