Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners

Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners

@brunocartaxo @gustavopinto @scbs@soueltonvieira

moving to1

Before we start..

2

Evidence-Based Medicine

EBM is the conscientious, explicit, judicious and reasonable use of modern, best evidence in making decisions about

the care of individual patients.

It integrates the clinical experience and patient values with the best available

research

It aims to increase the use of high quality clinical research in clinical

decision makingActa Informatica Medica’2008 3

Evidence-Based Medicine

One of the greatest achievements of EBM has been the development of systematic

reviews and meta-analyses, methods by which researchers identify multiple studies on a

topic, separate the best ones and then critically analyze them to come up with a summary of the

best available evidence.

Acta Informatica Medica’2008 4

Evidence-Based Practice

5

EB Psychology EB Psychiatry

EBM

EB Education

extends

Evidence-Based Practice

6

EB Psychology EB Psychiatry

EBM

EB Education

extends

EB Software Eng.

Evidence-Based Software Engineering

Evidence-Based Software Engineering (EBSE) is a way

to integrate the best research evidence with practice.

Barbara Kitchenham Keele University

ICSE’20047

1. Convert the need for information into answerable questions

2. Identify the best evidence with which to answer these questions

3. Appraise the evidence critically: assess its validity and usefulness

4. Implement the results of this appraisal in software engineering practice

5. Evaluate the performance of this implementation

The 5 Steps of Evidence-Based Practice

}Systematic Literature Reviews

(SRs)

8

How to find the best evidence?

Software Engineering conferences

Software Engineering journals

9

How to find the best evidence?

X, Y, and Z: A Systematic Literature Review

10

It involves reading the right papers and then changing

behavior in the practice of the discipline.

Trisha Greenhalgh Oxford University

Evidence-Based practice is not only about reading and summarizing papers

11

Fábio Queda CIn @ UFPE

There is a lack of connection between systematic reviews

and software engineering practice

IST’201112

SRs do not provide guidelines for practitioners

Most of SR authors affirmed that they

hadn’t a direct impact on industrial practice

Lack of connection with industry is the

6th top barrier

13

In addition

14

Evidence-Based Medicine (EBM) researchers argue that SRs:

Focus on a narrow question

15

Are time consumingthey usually require

between 6 months and 2 years to complete

(1,139 hours, on average) Policymakers often require access to contextualized resources that

address a broader scope of scientific evidence quickly

On the other hand…

16

Rapid Reviews Briefings and Summaries

56% of the rapid reviews were conducted

in the last 3 years

“Rapid reviews simplify the

process of SRs to produce

information in a timely manner”

“[Briefings] translates existing SRs into actionable

messages in the form of short accessible briefings”

17

EBM Researchers

Health decision-makers

18

EBSE Researchers

Software Industry

How to transfer knowledge from systematic reviews to

software engineering practice?

19



moving to21

“Evidence Briefings”

22

An one-page document, extracted from a systematic

review, that contains findings useful for

practitioners

Approach

23

1

Systematic Reviews Selection 2

Systematic Review Data Extraction 3

Evidence Briefings Generation

4

Evidence Briefings Evaluation

Approach

24

1




4


25

1 Systematic Reviews Selection120

Systematic Reviews

26


Systematic Reviews


YAY! citation++

27


Systematic Reviews

32 Systematic Reviews

with guidelinesFábio Queda CIn @ UFPE

YAY! citation++

28


Systematic Reviews


with guidelines

24 Systematic Reviews with search strings


YAY! citation++

29


Systematic Reviews


with guidelines

24 Systematic Reviews with search strings

12 Selected SRs


YAY! citation++

# Subject2 Global software development3 Agile software development1 Software testing1 Software requirements1 Model based software development1 Software development productivity1 Cost and effort estimation1 Code duplication1 Software engineering knowledge management

Approach

30

1




4


31

2 Systematic Reviews Data Extraction

Paper title:Original The effectiveness of pair programing: A meta-analysis

Briefing The effectiveness of pair programming

32


Paper title:

Research goals:

Original The effectiveness of pair programing: A meta-analysis


Template This briefing reports evidence on <GOAL> based on scientific evidence from a systematic review.

Briefing This briefing reports evidence on the effectiveness of pair programming around quality duration and effort based on scientific evidence from a systematic review.

33


Paper title:

Research goals:

Research findings:

Original The effectiveness of pair programing: A meta-analysis


Template This briefing reports evidence on <GOAL> based on scientific evidence from a systematic review.

Briefing This briefing reports evidence on the effectiveness of pair programming around quality duration and effort based on scientific evidence from a systematic review.

Finding 1 Pairing up of individuals seems to elevate the junior pairs up to near senior pair performance

Finding 2 If you do not know the seniority or skill levels of your programmers, but do have a feeling for task complexity, then employ pair programing either when task complexity is low and time is of the essence, or when task complexity is high and correctness if important

Approach

34

1




4


35

3 Evidence Briefings Generation: Principles

Similarity: elements that are similar are more likely to be organized together Proximity: closer elements are more likely to be perceived as a group Continuation: elements will be grouped as a whole if they are co-linear Unity: elements that have a visual connection should belong to a uniform group

36

3 Evidence Briefings Generation

1. The title of the briefing 2. The goal of the briefing 3. The findings extracted

from the original review 4. An informative box

with general information

5. The reference to the original review

6. The logos of our research group and university

37

3 Evidence Briefings Generation

1. The title of the briefing 2. The goal of the briefing 3. The findings extracted

from the original review 4. An informative box

with general information

5. The reference to the original review

6. The logos of our research group and university

The template of Evidence Briefings is licensed under CC-BY license!

Approach

38

1




4


RQ: How practitioners and researchers perceive the content and format of

Evidence Briefings?

39

4 Evidence Briefings Evaluation

40

4 Evidence Briefings Evaluation: Practitioners

473 StackExchange Users who asked questions

related to the SRs

41

4

1 SRs Search Strings

Queried a StackExchange website2

Removed false-positive questions3

Classified questions using open card sort4

programmers.stackexchange.com

sqa.stackexchange.com

pm.stackexchange.comreverseengineering.stackexchange.com

softwarerecs.stackexchange.com 1,738 related questions

473 related questions

Kappa: 0.72

Evidence Briefings Evaluation: Practitioners

42

4

473 StackExchange Users who asked questions

related to the SRs


43

4

473 StackExchange Users who asked questions related

to the SRs

only 146 of them had public profile

(LinkedIn, Github, Twitter, etc)


44

4

The 22 authors of the 12 Systematic Reviews

Evidence Briefings Evaluation: Researchers

45

4 Evidence Briefings Evaluation: Survey

Survey principles:

Reciprocity (e.g., we raffled a 100 USD Amazon card gift)

Authority & Credibility (e.g , Ph.D., University professors)

Liking (e.g., personalized emails)

Scarcity: (e.g., we defined a deadline)

Brevity (e.g., we asked closed questions as much as possible)

Social Benefit (e.g., 1 USD for the Brazilian Red Cross)

Evidence Briefings Evaluation: Survey

46

4

Survey principles:

Reciprocity (e.g., we raffled a 100 USD Amazon card gift)

Authority & Credibility (e.g , Ph.D., University professors)

Liking (e.g., personalized emails)

Scarcity: (e.g., we defined a deadline)

Brevity (e.g., we asked closed questions as much as possible)

Social Benefit (e.g., 1 USD for the Brazilian Red Cross)

47

4

Survey with Practitioners (17 questions, 2 open-ended)

Survey with Researchers (8 questions, 2 open-ended)

Evidence Briefings Evaluation: Survey

7 respondents (31% of response rate)

32 respondents (22% of response rate)

Results

48

Survey with Practitioners

Survey with Researchers

Results

49



50

Survey with Practitioners: Demographics

Q1. What is your current position?

Q2. How many years of experience do you have in your current position?

51


Q1. What is your current position?

Q2. How many years of experience do you have in your current position?

52


Q3. Where do you work?

Q4. What is your level of educational attainment?

53


Q3. Where do you work?

Q4. What is your level of educational attainment?

54

Survey with Practitioners: Acquiring Knowledge

Q5. How often do you use StackExchange websites? Q6. How often do you read software engineering research papers?

55

Q7. Have you ever read a systematic review paper?

Q8. For what reason you read a systematic review paper?

Survey with Practitioners: Acquiring Knowledge

56

Q10. To what degree do you think the information available in the briefing we sent to you can answer your question on StackExchange?

Survey with Practitioners: Briefings’ Content

57

Q10. To what degree do you think the information available in the briefing we sent to you can answer your question on StackExchange?

Q11. Why?


“The question is too specific”

“The question expected more than one answer”

“The question touched a slightly different issue”

“The briefing lacks details”

58


Q12. Regardless the briefing answers or not your question, how important do you think is the research presented on the briefing?

59

Q12. Regardless the briefing answers or not your question, how important do you think is the research presented on the briefing?


Q13. Why?“Agile is not a one size fits all methodology. To make it work you need to see what works for you and your

team. [...] Making bold high level statistical statements about Agile software development will only

hurt it where as it can shine in truly Agile organizations.”

60

Q14. How do you compare the answers from the StackExchange community to the findings presented in the briefing?


61

Survey with Practitioners: Briefings’ Format

Q15. How easy was to find the information in the briefing? Q16. Is the briefing interface clear and understandable? Q17. Does the briefing look reliable?

62

Q15. How easy was to find the information in the briefing? Q16. Is the briefing interface clear and understandable? Q17. Does the briefing look reliable?

Survey with Practitioners: Briefings’ Format

Results

63



64

Survey with Researchers: Sharing KnowledgeQ1. How important for you is to share research results to practitioners?

?

65

Survey with Researchers: Sharing KnowledgeQ1. How important for you is to share research results to practitioners?

66

Survey with Researchers: Sharing KnowledgeQ2. How often do you share research results to practitioners?

67


68


Q3. How do you do that?

“Teaching”

“Seminars”

“Writing”

“Advisory work”

“Social Networks”

69

Survey with Researchers: Briefings’ ContentQ4. How does the briefing that we sent to you cover the main findings of your paper?

70

Survey with Researchers: Briefings’ ContentQ4. How does the briefing that we sent to you cover the main findings of your paper?

Q5. Why?

YAY! :-)

71

Survey with Researchers: Briefings’ FormatQ6. How easy was to find the information in the briefing? Q7. Is the briefing interface clear and understandable? Q8. Does the briefing look reliable?

Discussions

72

Revisiting Findings

73

Practitioners rarely use research papers as mediums to acquire knowledge.

Software engineering practice still has many beliefs with no evidence basis.

Revisiting Findings

74

Both researchers and practitioners positively evaluated the evidence briefings

Revisiting Findings

75

Both researchers and practitioners positively evaluated the evidence briefings

The briefings well covered the main findings of the original systematic reviews

The Yin-Yang of Research and Practice

76

Researchers want to transfer knowledge. But not all of them do so.

Practitioners want to be more aware of software engineering research. But few of them do so.

Implications

77

Researchers

78

Researchers

79

Practitioners

80

Tool builders

81

Tool builders

82

Educators

83

Conference and Publicity Chairs

84

One last thing..

85

86

Search Strings are your best

87

Assess your search strings on

StackExchange on early stages of a systematic

review planning!

88

Are your search strings well-designed?

“quality + model”, “quality + model driven” and “model driven + experience”

(software AND ((cost OR effort OR productivity) WITH (factors OR indicators

OR drivers OR measure)))

89

Thanks!



moving to90

Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners

Software

Transcript of Evidence Briefings: Towards a Medium to Transfer Knowledge from Systematic Reviews to Practitioners