What Drives Successful Product Development and Innovation in the Software Development Process? The Product Development Success Index (PDSI)

PRESENTED BY: CHARLES COLBY, ROCKBRIDGE

SUNIL MITHAS, ROBERT H. SMITH SCHOOL

TONY ORLANDO, 3 PILLARRIICCARDO DE MARCHI TREVISAN, 3

PILLAR ERIN NORMAN, ROCKBRIDGE

2015 Frontiers in Services Conference

Rockbridge Associates, Inc.

2

Introduction to 3Pillar Global

• 3Pillar Global is a product lifecycle management and development company that helps clients implement products that drive revenue.

• Services include strategy and consulting, customer experience design, platform development, software engineering based on agile practices, and devops.

• Our clients include mid-sized to large companies in the media/entertainment, edtech, financial services, information services, and health and wellness sectors.

3

Background and Environment for this Study

• Service sector companies increasingly seek to develop new software applications to…

• Drive their core business• Add customer value• Create new revenue streams• Create entirely new products that monetize information

captured in their processes

• Efforts are usually driven by a functional area within the company headed by a Chief Innovation Officer or Chief Technology Officer

• Companies have varying degrees of success in this product development process

4

Objectives of this Study

• Determine what drives success in new software product development – the focus is on companies where software development is not their core mission (e.g., a bank, TV network, healthcare provider)

• Create an instrument that can be used to…• Evaluate success• Benchmark success against other companies• Diagnose issues and identify priorities for improvement

This was a collaborative effort between 3Pillar Global and the Center for Excellence in Service at the Robert H. Smith School of Business. Rockbridge Associates was a research partner that spearheaded much of the effort.

5

Methodology

• Exploratory• In-depth meetings with 3Pillar management/experts • 6 in-depth interviews conducted by phone with heads of

corporate functions • Survey of relevant literature• Creation of a Product Development Success framework

• Quantitative• 20-minute online survey of 204 professionals responsible for

new software product development (145 from online panel, 59 from 3Pillar lists)

• Respondents worked in range of service businesses• Had decision input for development function• Mid-sized to large companies (Minimum $1M)

6

Product Development & Innovation Success Framework

Business Outcomes

Macro Factors• New entrants/products• Globalization• Regulation/legal constraints• Speed of technological

development• Economy

Long Range Industry Shifts• Move towards product standardization• Big Data opportunities• Search for new revenue streams• Increasing investment uncertainty

Product Development Success Index

Perceived Development & Innovation Success

Organizational Characteristics• Revenue• Size• Industry• Geographic reach• Strategic vision

New Software Product Development Function Characteristics• R&D budget• Number of employees• Third party services used

Culture

Feedback

Communication

Staffing

Collaboration

Time/Budget Focus

The Product Development Success Index (PDSI)

8

Survey Topics

Topic MeasuresProduct Development Success

53 items, scaled on a 1 to 7 scale covering culture, process and resources

Product Development Process

Degree of structure, development time, planning/review cycle, testing, capture of high-value data, product customization, revenue streams

Macro Factors Competitive environment, industry growth, globalization, geographic coverage, regulation

Product Development Outcomes

Confidence in process, success in achieving dev goals, innovation leader vs. follower, impact on brand, impact on efficiency, barriers

Business Traits Revenue, employment, industry, revenue growth, customer churn

Function Traits Employee size, budget, strategy focus, reliance on outsourcing, decision-making process

9

Index Development

· Goal was to create an instrument that was concise, provided diagnostic detail, and met thresholds for reliability and validity

· Input consisted of the battery of 53 success attributes (49 after dropping 4 items with high item non-response)

· Exploratory factor analysis used to identify underlying dimensions of success, account for redundancy, and eliminate items with low communality

· Resulting index included:· 25 items· 6 dimensions (3 to 7 items per dimension)· Explained 67% of variance· Cronbach Alphas ranged from .74 to .89· Aggregated index had a .69 correlation with perceived overall success of

the product development function

10

7359 87

Highly Successful

(19%)

Somewhat Successful

(34%)

Marginally Successful

(28%)

Rarely Successful

(19%)

The Product Development Success Index (PDSI) is calibrated to a scale of 0 to 100• The PDSI correlates with success in meeting (1) software product

development goals and (2) hard business outcomes, such as revenue, internal efficiency and customer retention.

• Companies are divided into four tiers for purposes of studying the correlates and consequences of success.

11

The overall PDSI score is comprised of six dimensions

· Culture: attract and retain top talent, have creative leadership, encourage interaction, provide career growth, and reward and recognize innovation

· Feedback: degree of input on products collected from customers and end-users

· Communication: interaction between business and technical team members as well as within the development groups

· Staffing: appropriate mix of staff from in-house, outsourced, domestic and foreign workers

· Collaboration: extent to which team members assist each other and can cut across hierarchy to interact

· Time/Budget Focus: succeeding on traditional project metrics

12

The 6 sub-indices vary in importance

Sub-Index Importance to Product Development Success Index

· Importance is based on correlation with overall perceived success (normalized pairwise estimates)

· Dimension scores are weighted accordingly

Percentages do not add up to 100% due to rounding. See notes for more exact figures.

Across the sample, the biggest pain-point is improving culture

13

Performance on Sub-Indices vs. Importance

Top Challenge: Recognizing and

Rewarding Innovation

Correlates and Consequences of Product Development Success

15

What do the successful PD functions look like?

Most Successful Least Successful

Large Divisions (1,000+ Staff) Small (<100)R&D Budget $161M R&D Budget $39M

Larger technology budgets Smaller budgetsInformation Services Education and Hospitality

Revenue > Cost Cutting Cost Cutting > RevenueSell high margin products Not a focus

Select high quality vendors Not a focusBusiness execs & IT

collaboratively drive strategy Business execs drive strategy

Business execs & IT collaboratively drive technical IT drives technical decisions

Seek to lead their industry Tend to follow or track with peersConfident in Innovation Strategy Not Confident

16

Measures of Agile Development

Q1: For each one, please indicate how well the statement describes your organization.Q2a: How frequently does your team review and re-plan the next highest priorities for ongoing software development efforts?Q2c: How frequently during the development process do end users test working software, including new features?Q3a: How often do you discuss and/or implement improvements to your software development process?Q2: How long does it take in your organization to go from an idea to software that is working in a production environment?Q2b: What percentage of your new software, including new features, is covered by automated tests that run every time a feature is added or modified?

Component Frequency/RatingTime from idea to working software Three weeks or lessNew software covered by automated tests 76%+Frequency of review and re-planning priorities

Every three weeks or more often

Frequency of end user testing Every three weeks or more oftenFrequency of process improvement Every three weeks or more oftenTeam Members Exchange Information and Learn From Each Other

Describes company (6 or 7 on a 7-point scale)

Business and Technical Teams Collaborate of Software Development Projects

Describes company (6 or 7 on a 7-point scale)

17

Agile practices correlate strongly with PD success

Number of Agile Practices Used

Total(n=204)

RarelySuccessful

(39)B

MarginallySuccessful

(57)B

Somewhat Successful

(70)C

HighlySuccessful

(38)D

0.02 0.010.080.07 0.110.1814%

8% 7%

17%

24%18%

10%23%

16%

21%26%

13%

25%

34%

26%20%

28%

32%

14%

3%14%

41%

14% 6%

0 Agile Practices

1 Agile Practice

2 Agile Practices

3 Agile Practices

4 Agile Practices

5 Agile Practices

6 Agile Practices

7 Agile Practices

Q1: For each one, please indicate how well the statement describes your organization.Q2a: How frequently does your team review and re-plan the next highest priorities for ongoing software development efforts?Q2c: How frequently during the development process do end users test working software, including new features?Q3a: How often do you discuss and/or implement improvements to your software development process?Q2: How long does it take in your organization to go from an idea to software that is working in a production environment?Q2b: What percentage of your new software, including new features, is covered by automated tests that run every time a feature is added or modified? *Caution, small base size

AB

A

D

D

CDBC

18

Companies need to cross thresholds on their way to success – consensus, financial, knowledge

Q34: Why does your organization not move faster in software product development and technology? (Select all that apply)*Caution, small base size

Constraints on Speed of Software Product Development

Total(n=204)

Rarely Successful

(n=39*)A

Marginally Successful

(n=57)B

Somewhat Successful

(n=70)C

Highly Successful

(n=38*)D

Financial constraints 47% 51% 58%D 41% 34%

Disagreement among decision makers

29% 49%BC 23% 23% 29%

Lack of time 29% 31% 30% 30% 24%

Ineffectiveness of consensus decision making

25% 51%BCD 18% 20% 18%

Fear of risk 23% 41%CD 25% 19% 11%

Lack of vision 19% 41%BCD 12% 16% 11%

Lack of knowledge 19% 38%BCD 14% 19%D 5%

Legal constraints 19% 13% 21% 20% 21%

None of these 15% 5% 14% 16% 24%A

19

More successful companies operate industries with more dynamic competitive activity

Q9: How frequently do the following actions occur in your industry? Seven-point scale from “Never” to “Constantly”*Caution, small base size

Software products with brand new capabili-ties are released

There are new entrants to the industry

Firms leave the industry

33%

33%

18%

37%

23%

7%

56%

27%

21%

68%

55%

32%

Series5Highly Successful(n=36-38*) DSomewhat Successful(n=69) CMarginally Success-ful(n=56-57)BRarely Successful(n=37-38*)A

Industry Changes(% rated 6-7 on a 7-point frequency scale)

ABAB

BC

BB

Perceived Development & Innovation Success

21

PDSI correlates with success on a variety of goals connected to new software product development

Q16: How successful is your organization in achieving the following goals? *Caution, small base size

Index Score versus Software Product Development Goals(% rated 6-7 on a 7-point success scale)

Total(n=196-204)

Rarely Successful(n=38-39*)

A

Marginally Successful(n=52-57)

B

Somewhat Successful(n=69-70)

C

Highly Successful

(n=37*)D

Overall success of new product innovation 37% 8% 18% 43% AB 87% ABC

Capabilities meet customer needs 42% 10% 21% 59% AB 74% AB

Stay competitive with dev 42% 5% 24% A 56% AB 84% ABC

Products competitive in market 41% 10% 23% 50%AB 82%ABC

Keep software products current 37% 3% 25% A 46% AB 76% ABC

Products create new revenue 31% 5% 19% A 37% AB 68% ABC

Quickly roll out new products 31% 8% 14% 33% AB 76% ABC

Based on radically new tech 30% 5% 16% 29% A 79% ABC

22

Product development success also tracks with business outcomes like revenue growth & customer retention

Number of Positive Business Outcomes

Total(n=204)

Rarely Successful(n=39*)

A

Marginally Successful

(n=57)B

Somewhat Successful

(n=70)C

Highly Innovative(n=38*)

D

15%5% 14% 10%

34%20%

8%

19% 29%

16%23%

23%

18%27%

21%19%

31%18%

17% 13%16%

21% 21%11% 11%

8% 13% 11% 6% 5%

0 Positive Outcomes1 Positive Outcome2 Positive Outcomes3 Positive Outcomes4 Positive Outcomes5 Positive Outcomes

Q18: What impact do new software products developed by your organization have on the reputation of your brand?Q19: How, if at all, have software products developed by your organization in the past three years affected the internal efficiency of your organization? Q20: How would you describe the change in your company’s total revenue over the last three years?Q21: How would you describe the change in your company’s total number of customers over the last three years?Q22: Thinking of the last 12 months, which best describes the change (if any) in your company’s customer retention rate?*Caution, small base size

ABCA

Key Takeaways

24

The overall PDSI score is comprised of six dimensions

· Key opportunity is to build a culture of talented people where innovation is rewarded

· Most successful companies tend to be larger, better funded, focused on quality, and strive to be leading edge

· Success is related to a dynamic competitive landscape· Agile development practices truly matter · Success requires overcoming hurdles: consensus -> financial ->

know-how· It is possible to score and diagnose success

For more information, visit:Visit: http://productdevelopmentsuccess.com/

25

Author Contact Information

· Charles Colby, Principal, Chief Methodologist and Founder, Rockbridge Associates, Inc.: ccolby@rockresearch.com

· Sunil Mithas, Professor, Robert H. Smith School of Business & Research Director, Center for Excellence in Service: smithas@rhsmith.umd.edu

· Tony Orlando, Executive Vice President, 3 Pillar Global: tony.orlando@3pillarglobal.com

· Riccardo de Marchi Trevisan, Business Development Executive: riccardo.demarchi@3pillarglobal.com

· Erin Norman, Director, Rockbridge Associates, Inc.: ENorman@rockresearch.com

Appendix: Detailed Index Scores