Nasir Eisty Assistant Professor Testing and Code Review ...

Testing and Code Review Practices in Research

Software

Nasir EistyAssistant ProfessorDept. of CSSECal Poly State UniversitySan Luis Obispo, California

2

Who Am I?

Metric Process

Research Software

3

Software Quality

4

• Software Testing

Part #1

• Peer Code Review

Part #2

Contents

5

Recommendation

Part #1

Testing Research Software: A survey

6

Online Survey

7

Part #1 (Testing) Outline

• Demographics

• Level of knowledge research software developers have on testing

• Current testing practices in research software community

• Difficulties to test research software

• Compatibility of Commercial/IT testing techniques

• Improvement of the testing process in research software

8


• Demographics





• Improvement of the testing process in researach software

9

020406080

100120

Roles

10

Coun

t

0

10

20

30

40

50

60

70

Less than 1 year 1 year to lessthan 5 years

5 years to 10years

More than 10years

Years Worked

11

Coun

t

0102030405060708090

Project Stage

12

Coun

t

01020304050607080

1 to 5 6 to 10 11 to 15 16 to 20 More than 20

# Developers

13

Coun

t


• Demographics






14

0

10

20

30

40

50

60

Low Average High Very High

Knowledge of Testing

15

Coun

t

0

10

20

30

40

50

60

Very Low Low Average High Very High

Understanding Testing Concepts USED

16

Coun

t

0

10

20

30

40

50

60

70

Very Low Low Average High Very High

Understanding Testing Concepts NEEDED

17

Coun

t


• Demographics






18

0

10

20

30

40

50

60

Level 0 Level 1 Level 2 Level 3 Level 4

Testing Goals

19

Level 4 – Testing is a mental discipline that helps all researchers develop higher quality softwareLevel 1 – The Purpose of testing is to show correctnessLevel 3 – The Purpose of testing is not to prove anything specific, but to reduce the risk of using the software

Coun

t

020406080

100120140160

Unit Integration System Acceptance Module

Testing Methods Used

20

Coun

t

0

10

20

30

40

50

60

70

Always Most of the time Sometimes Rarely

Usefulness

21

Coun

t


• Demographics






22

0

10

20

30

40

50

60

Not Complex SlightlyComplex

ModeratelyComplex

VeryComplex

ExtremelyComplex

Complexity to Test

23

Coun

t

0 5 10 15 20 25 30

OtherCost

Legacy codeCodebase

Comparing with realityAffects CI

CultureSlow

Lack of knowledgeExternal dependencies

Lack of resourcesTest case design

Challenges

24Count


• Demographics






25

0

5

10

15

20

25

30

35

Always Most of thetime

Sometimes Rarely Never

Commercial/IT Testing Methods - Team

26

Coun

t

05

10152025303540

Always Most of thetime

Sometimes Rarely Never

Commercial/IT Testing Methods- Individual

27

Coun

t

05

1015202530354045

Very High High Average Low Very Low

Value Seen in Comm/IT Testing Methods

28

Coun

t

0 5 10 15 20

OtherRuntime restrictions

Difficult to useCost

InfrastructureLack of knowledge

MindsetLack of resources

Not useful

Challenges to Adapt Comm/IT Methods

29Count

0 2 4 6 8 10

Other

Benchmarks

CI issues

Slow

Lack of expertise

Challenges Not Met by Comm/IT Methods

30Count


• Demographics






31

0 2 4 6 8 10 12

ResourcesMake simpler

AutomationContinuous integration

CultureImprove code quality

AcknowledgementInfrastructure

More testsTraining

Testing Improvements

32Count

Discussion

• Researchers pose a clear goal of testing their project

• Complexity associated with the process needs further attention

• Make a culture of testing in the research software community.

• Providing proper training and resources can improve the testing process in research software.

33

Part #2

Peer Code Review in Research Software

34

Interview & Survey

35

Part #2 (Code Review) Outline

36

• Demographics

• Current code review practices in research software

• Impacts of the code review process in research software

• Difficulties develpers face during code review

• Potential areas of improvement in the review process


37

• Demographics





0

5

10

15

20

25

30

Less than 1 year 1 year to lessthan 5 years

5 years to 10years

More than 10years

Years Worked

38

Coun

t

01020304050607080

Add newcode

Fix bugs Reportbugs

Maintainproject

Makestrategicdecisions

Other

Role

39

Coun

t

05

1015202530354045

1 -Reviewee

only

2 3 4 - Equally 5 6 7 -Reviewer

only

Balance as a Reviewee and Reviewer

40


41

• Demographics





0

10

20

30

40

50

60

70

Less than10%

11% - 25% 26% - 50% 51% - 75% More than75%

Percentage of Code Undergo Review

42

Coun

t

05

101520253035404550

Less than 1 1 to 5 6 to 10 10 to 15 More than 20

Time Spent on Code review

43

Coun

t

Hour

0

5

10

15

20

25

30

35

Less than 1hour

Less than 1day

1 - 3 days 4 - 7 days More than 7days

Time For a First Response

44

Coun

t

05

10152025303540

Less than 1 day Less than 1week

Less than 1month

More than 1month

Time For a Final Decision

45

Coun

t

0 20 40 60 80 100 120

Other

Design

Code mistakes

Software quality

Problems Identified

46Count

0 10 20 30 40 50

Other

Problems identified

Positive feeling

Good feedback

Improved code quality

Knowledge sharing

Positive Experience

47Count

0 5 10 15 20 25

OtherUnresponsive author

Difficult taskHard to find reviewers

BottleneckDisagreements

Misunderstand criticismTakes too long

Negative Experience

48Count


49

• Demographics





0 10 20 30 40 50 60

Other

Sustainability

Improve maintainability

Knowledge sharing

Improve code quality

Why Code Review is Important

50Count

0 5 10 15 20 25

OtherDocumentation

Better styleImproves reliabilityKnowledge sharing

Improve designBetter maintainability

More EyesImprove readability

Correctness

How Code Review Improves Code

51Count

05

10152025303540

Stronglydisagree

Somewhatdisagree

Neitheragree nordisagree

Somewhatagree

Stronglyagree

Decrease Code Complexity

52

Coun

t


53

• Demographics





0 5 10 15 20 25 30 35

Other

Identify problems

Administrative issues

Understanding system

Understanding code

Challenges

54Count

0 10 20 30 40 50 60 70

Other

Takes too long

Developer egos

Participation

Finding right people

Phrasing comments

Time

Barriers

55Count


56

• Demographics



• Difficulties developers face during code review


0 5 10 15 20 25

OtherFaster response

More timeMore training

IncentiveMore people

ToolingFormalizing process

Improvements

57Count

Discussion• Research software developers employ an informal code

review process

• Code review has an overall positive impact

• Most common difficulty reported by participants is finding time to to do it and understand other people’s code.

• Formalizing the review process by including more people, more training, and providing compensation could potentially improve the code review process.

58

Recommendations - Testing

59

• Provide enough training on software testing to all kinds of research software developers ranging from graduate students to experienced researchers

• Incorporate more tests that can solve specific needs of the research software

• Provide infrastructure support, for example, a public service for testing including many-tier pricing structure for machine time and a sophisticated testing dashboard

60


• Provide automation for setting tests and analysis of the results

• Improve continuous integration system to facilitate a better way of testing, especially, the incoming tests during down time

• Make a culture of testing in the team and encourage others by sharing the benefits from the experience of testing

• Improve the quality of the code so that developers can write tests easily

61



• Provide proper acknowledgement of developers for contributions in testing

• Make the testing process simpler so that it is easy to adopt in the project

• Provide enough resources to developers so that they can utilize the resources to develop test suits

62

Recommendations – Code Review

63


• Make code review process more formal with a structured guideline for each step of the process

• Try to ensure at least one science review and one technical review

• Include automatic tools in the code review process and train your peer reviewers the best practices to use the tool

64

• Encourae more people to participate in the review process and allocate some time to do the review

• Provide incentives or rewards to reviewers to participate in code review

• Allocate sufficient time in the development process to perform code review

• Provide faster feedback to any incoming review request

65


• Train reviewers on how to phrase good feedback

• Train developers to forget their egos and accept comments from the reviewers to improve their code

• Make the overall code review process faster

• Provide necessary support from the administrative level that encourages people to participate in the code review process

66


AcknowledgementDr. Jeffrey CarverUniversity of Alabama

Dr. George ThiruvathukalLoyola University Chicago

Dr. David BernholdtOak Ridge National Laboratory

Dr. Hai Ah NamLos Alamos National Laboratory

Dr. Danny PerezLos Alamos National Laboratory

Dr. J. Dave MoultonLos Alamos National Laboratory

Dr. Roland HaasNational Center for Supercomputing Applications

Dr. Gabrielle AllenNational Center for Supercomputing Applications

Dr. Daniel KatzNational Center for Supercomputing Application

NSF grant 1445344

67

68

Nasir [email protected]

Nasir Eisty Assistant Professor Testing and Code Review ...

Documents

Transcript of Nasir Eisty Assistant Professor Testing and Code Review ...