Jankowski - Findings from an ongoing examination of metrics on innovation
Performance Metrics for the Open Innovation Model - David Scarlatti
-
Upload
dscarlatti -
Category
Documents
-
view
31 -
download
2
Transcript of Performance Metrics for the Open Innovation Model - David Scarlatti
T802
MSc in Technology Management
Performance metrics for the Open Innovation model
David Scarlatti
X7922828
February 2010
T802
MSc in Technology Management
Performance metrics for the Open Innovation model
David Scarlatti
X7922828
February 2010
This dissertation is submitted in partial fulfilment of the requirements for the MSc in
Technology Management
David Scarlatti - X792282
Abstract
Innovation is seen as a key element of an organisation’s competitiveness. Members of
senior management teams and managers in charge of Research and Development have
struggled for years to find the most appropriate way to measure innovation. While best
practices for measuring innovation are still under development, a new model for
innovation has emerged, known as Open Innovation. This new model is expanding quickly
and is causing researchers to question traditionally accepted principles. For those who are
embracing the new model, a key question arises: Can we measure innovation in the same
way that we have until now? Open Innovation proponents seem to have paid little
attention to this topic to date. The aim of this research is to produce recommendations
concerning metrics for evaluating the effectiveness of an ‘Open Innovation’ approach to
research. A major objective is to explore the use of performance metrics in research
pertaining to Open Innovation. The validity of existing innovation performance metrics is
examined using the principles of Open Innovation and by applying defined criteria of
applicability. I conclude that while most metrics currently in place remain useful, they need
some adaptation. In many cases, additional supporting metrics, especially those related to
external resources and activities, must be added. A “Decalogue”–outlining the changes
needed in the current set of metrics–is proposed as a useful tool for managers in charge of
evaluating this new approach to innovation. The corollary of the Decalogue is that one
should avoid using a current innovation performance measurement system for the new
Open Innovation model without adapting the relevant metrics.
David Scarlatti - X792282
iii
Table of Contents
Figures ........................................................................................................................................... v
Tables ........................................................................................................................................... vi
Glossary ........................................................................................................................................vii
Chapter 1 Introduction............................................................................................................. 1
1.1 Background on the problem/issue .............................................................................. 1
1.2 Justification for the research ....................................................................................... 1
1.3 Aim and objectives....................................................................................................... 2
1.4 Scope of the research .................................................................................................. 2
1.5 Outline of the dissertation........................................................................................... 3
Chapter 2 Research definition.................................................................................................. 5
2.1 The practical problem/issue ........................................................................................ 5
2.2 Existing relevant knowledge ...................................................................................... 11
2.3 Research questions.................................................................................................... 20
Chapter 3 Methodology ......................................................................................................... 21
3.1 Methods and techniques selected............................................................................. 21
3.2 Justification ................................................................................................................ 21
3.3 Research procedures ................................................................................................. 22
3.4 Ethical considerations................................................................................................ 22
Chapter 4 Analysis and interpretation ................................................................................... 23
4.1 Summary of data collected........................................................................................ 23
4.2 Data Analysis.............................................................................................................. 29
4.3 Interpretation in relation to the research questions................................................. 33
4.4 Interpretation in relation to the aim ......................................................................... 35
Chapter 5 Conclusions............................................................................................................ 37
5.1 Conclusions regarding the research questions.......................................................... 37
5.2 Conclusions regarding the research aim.................................................................... 37
David Scarlatti - X792282
iv
5.3 Further work .............................................................................................................. 38
5.4 Implications of this research...................................................................................... 38
References................................................................................................................................... 40
David Scarlatti - X792282
v
Figures
Figure 1‐1 Scope of this research.................................................................................................. 3
Figure 2‐1 Problem Definition Flow .............................................................................................. 5
Figure 2‐2 The Closed Innovation Model (Chesbrough 2003) ...................................................... 6
Figure 2‐3 The Open Innovation Model (Chesbrough 2003) ........................................................ 7
Figure 2‐4 Reasons for measuring Innovation ............................................................................ 13
Figure 2‐5 Challenges measuring Innovation.............................................................................. 14
Figure 2‐6 Approaches facilitating Innovation Measurement .................................................... 18
Figure 4‐1 The measurement and control process (KerssensvanDrongelen, Cook 1997).......... 23
Figure 4‐2 Balance in Open Innovation (Lichtenthaler 2008) ..................................................... 32
David Scarlatti - X792282
vi
Tables
Table 2‐1 Contrasting Principles of Closed and Open Innovation (Chesbrough 2003)................. 8
Table 2‐3 Contrasting Principles between Closed and Open Innovation (Chesbrough 2003) and
measurement questions ............................................................................................................. 10
Table 4‐1 The dimensions of R&D performance (Ojanen, Vuola 2003)...................................... 25
Table 4‐2 Grouped Innovation metrics ....................................................................................... 26
Table 4‐3 Innovation metrics for individuals............................................................................... 29
Table 4‐4 Innovation metrics regarding R&D value .................................................................... 30
David Scarlatti - X792282
vii
Glossary
Business process At its most generic, any set of activities performed by a business that is initiated by an event; transforms information, materials or business commitments; and produces an output. Value chains and large‐scale business processes produce outputs that are valued by customers. Other processes generate outputs that are valued by other processes.
Effectiveness The degree to which the process’s stated objectives have been met.
Firm A term equivalent to ‘organisation’ in Open Innovation literature.
Innovation The implementation of a new or significantly improved product (good or service) or process; a new marketing method; or a new organisational method used in business practices, workplace organisation or external relations.(OECD 2005)
IP Intellectual Property. Any product of someone's intellect that has commercial value, especially copyrighted material, patents, and trademarks.
IPR Intellectual Property Rights. A general term for the assignment of property rights through patents, copyrights and trademarks. These property rights allow the holder to exercise a monopoly on the use of the item for a specified period of time.
KPI Key Performance Indicator. A metric that is especially important for measuring the performance of a process.
Measure In science, the process of obtaining the magnitude of a quantity (e.g., length or mass), relative to a unit of measurement (e.g., meter or kilogram). In management, a way to find out whether a goal is being met.
Metric A measure for something; a means of deriving a quantitative measurement or approximation for an otherwise qualitative phenomenon.
PMS Performance Management System. A system for evaluation against a set of determined criteria with respect to the economy, efficiency and effectiveness with which an organisation executes a particular activity. Organisations may set regular objectives for particular metrics, against which the organisation’s performance is evaluated.
R&D Research & Development. An organisation’s activity undertaken with the intent of discovering or developing new products or services. R&D includes pursuing enhanced versions of (or qualities in) existing products or services, or it may include discovering or developing new or more efficient processes of production.
David Scarlatti - X792282
1
Chapter 1 Introduction
1.1 Background on the problem/issue Today, most organisations rely on the innovation process to remain competitive and to grow in
a healthy manner (Kumpe, Bolwijn 1994). Such reliance makes innovation and its
measurement an extremely important topic. The same attention should be paid to this area of
business practise as is paid to other business processes, such as production or marketing
(Kerssens‐van Drongelen, Bilderbeek 1999, Andrew, Haanaes 2008).
If innovation is important in general, it is especially important in technology fields. Technology
is an enabler of innovation, and innovation confers a competitive advantage in technology
industries (Cumming 1998).
Open Innovation was first introduced by Chesbrough (Chesbrough 2003a), who referred to a
new model for innovation at the firm level. He observed that some companies started to
compete and subsequently found success in the marketplace using approaches to innovation
that had never been used before. The Open Innovation model contrasts with the traditional
“closed” model because it removes any strong barrier delimiting or shielding the internal
research and its development process.
Those in charge of funding, evaluating or executing innovation must use a measurement
system. Metrics, targets and results are required to make sound decisions about the process.
Measuring innovation is not an easy task; ongoing research continues in its quest to identify
good practices and metrics (Pearson, Nixon & Kerssens‐van Drongelen 2000).
Furthermore, for the novel Open Innovation model, much research is still required for the
purpose of properly defining the model (Chesbrough 2006b) and the appropriate ways of
measuring its effectiveness.
1.2 Justification for the research This research investigates the innovation measurement process for an organisation that
embraces the Open Innovation model. Many candidate organisations for such an investigation
exist, including companies along the lines of Eli Lilly, IBM, Procter & Gamble and Dow
Chemical.
Having a good set of measures (i.e., key performance indicators) can help organisations to
make the right choices about investments and organisational changes. Companies also rely on
David Scarlatti - X792282
2
their measurement system to demonstrate that positive results are happening as a
consequence of the actions taken.
Open Innovation is a very new model and, as such, there is still a great opportunity to learn
and explore its measurement metrics and its role as a driver for decision making.
Considering that a single error in product development investment could have an economic
impact several times greater than the research effort described here, it is worth investing in
research on Open Innovation.
Additionally, I am personally interested in the Open Innovation model because I believe that it
could have a major impact on the way businesses will operate in the coming years.
1.3 Aim and objectives
Aim
The aim of this project is to produce recommendations for metrics designed to evaluate the
effectiveness of an ‘Open Innovation’ approach to research.
Objectives
The objectives for this project are as follows:
1. To survey current innovation effectiveness metrics to create a catalogue of known
options.
2. To identify the basic criteria for the applicability of metrics based both on the Open
Innovation principles and on the examples available in the literature.
3. To correlate the deliverables of Objective 1 and Objective 2 to generate a proposal
about the use of effectiveness metrics for using Open Innovation in research.
1.4 Scope of the research Innovation can occur in very different ways. The research presented here focuses on
innovation in the research department or research unit of an organisation. Moreover, the
results presented here have a clear bias towards technology‐oriented organisations, which
makes perfect sense because this research falls under the “Technology Management” domain.
However, this research still may be of interest for those involved in measuring innovation in
other contexts, but such individuals must translate and amend concepts from the research and
David Scarlatti - X792282
3
development discipline in technology to the appropriate business role and market. Figure 1‐1
shows the scope of this research in terms of the intersection of the three concepts.
INNOVATIONTECHNOLOGY
R&D
SCOPE
Figure 1‐1 Scope of this research
From a historical point of view, Open Innovation is a very young model, so this research is not
limited to any specific period of time. Geographically, it is true that most of the available
information comes from Europe and the USA, so there is an implicit limitation to these areas.
However, this limitation arises only because these are the geographic areas where Open
Innovation is undergoing active development.
1.5 Outline of the dissertation This dissertation is organised into five chapters.
Chapter one serves as an introduction to the problem that originated the research, and it
defines the aim and three objectives of the research.
Chapter two explains, in detail, the problem that this research tries to solve, and it presents
existing knowledge relevant to the problem and its possible solutions. From the initial
problem, three specific research questions are outlined.
Chapter three focuses on the methodology used for this research and justifies the selection of
the method.
David Scarlatti - X792282
4
Chapter four presents the information collected, and it presents the interpretation of that data
as they relate to the research questions and the aim.
Chapter five serves as wrap‐up and includes the conclusion, as well as some considerations for
future work.
David Scarlatti - X792282
5
Chapter 2 Research definition
2.1 The practical problem/issue The objective of this research is to produce recommendations for the implementation of an
Open Innovation performance measurement system.
The Open Innovation model represents a drastic change from the traditional view of the
business rules related to inventions, research and product development. However, because a
lot of literature is already available on the subject, this research will not study the intricacies of
Open Innovation per se. Instead, this paper will focus on the problem of its measurement,
which has far less coverage in the available literature.
This section follows on others’ results ((McGrath, Romeri 1994), (Chiesa et al. 2008), (Bremser,
Barsky 2004), (Schumann, Ransley & Prestwood 1995), (Kerssens‐van Drongelen, Bilderbeek
1999, Davila, Epstein & Shelton 2005), (Mark Rogers 1998); see section 2.2 “Existing relevant
knowledge” ) that demonstrate the following:
1. Measuring innovation is widely accepted as a good practice
2. Measuring innovation is not an easy task
Previous research has also tried to demonstrate the following principle:
3. The Open Innovation model, because of its novelty and completely new principles,
requires more research into how it should be measured.
For a clear understanding of the problem, this section flows in the manner described in Figure
2‐1.
Why measure innovation?
Challenges measuring innovation
Current approaches for measuring innovation
What about measuring Open Innovation?
What is Open Innovation?
Figure 2‐1 Problem Definition Flow
David Scarlatti - X792282
6
Fully studying the problem of measuring Open Innovation is important to gain an
understanding of the differences between this new model and the traditional one.
Traditionally, innovation has been used by companies to create barriers to entry in their
markets. Research and development departments would maintain operational secrecy.
Talented personnel were hired by firms and retained to avoid the transfer of knowledge to
competitors. Strong intellectual property clauses in contracts protected the company from
partners, customers or employees using the company’s knowledge for their own benefit.
Open Innovation was first introduced by Chesbrough (Chesbrough 2003a), who referred to a
new model for innovation at the company level. The concept refers to a fresh approach,
focused on removing the barriers that delimit or defend the internal research and its
development process. The process is in contrast with the old, “closed” model. The author
observed how some companies started to compete and subsequently found success in the
marketplace using this new approach.
The closed model is usually represented with a clear boundary isolating the internal and the
external world, as shown in Figure 2‐2.
Figure 2‐2 The Closed Innovation Model (Chesbrough 2003)
David Scarlatti - X792282
7
The Open Innovation model presents a porous boundary for the organisation, a boundary that
allows knowledge to enter and exit at practically any stage of the traditional R&D cycle, as
shown in Figure 2‐3:
Figure 2‐3 The Open Innovation Model (Chesbrough 2003)
The model above does not assume that benefits only arise from one’s own knowledge; rather,
the model recognises that one can also benefit from several other factors:
• The available knowledge of others
• The ability of other organisations to use a business’s knowledge
This new model offers principles that contrast with those of the well‐established, traditional
closed model. Table 2‐1 summarises these contrasting principles according to Chesbrough
(2003a).
David Scarlatti - X792282
8
Closed Innovation Principles Open Innovation Principles
The smart people in our field work for us.
Not all of the smart people work for us, so we
must find and tap into the knowledge and
expertise of bright individuals outside our
company.
To profit from R&D, we must discover,
develop and ship it ourselves.
External R&D can create significant value;
internal R&D is needed to claim some portion of
that value.
If we discover it ourselves, we will get it
to market first.
We don't have to originate the research to profit
from it.
If we are the first to commercialise an
innovation, we will win.
Building a better business model is better than
getting to the market first.
If we create the most and best ideas in
the Industry, we will win.
If we make the best use of internal and external
ideas, we will win.
We should control our intellectual
property (IP) so that our competitors
don't profit from our ideas.
We should profit from others' use of our IP, and
we should buy others' IP whenever it advances
our own business model.
Table 2‐1 Contrasting Principles of Closed and Open Innovation (Chesbrough 2003)
We will revisit Table 2‐1 later in the report to determine how these principles impact the
measurement of innovation.
Chesbrough predicted that even the more traditional sectors and companies will evolve from
the old, closed model to an Open Innovation model (Chesbrough 2003a). The concept can
already be found today in the IT industry (e.g., IBM, Linux, etc.)(Andrews 2003), biotech
companies (e.g., Eli Lilly, Pfizer, etc.)(Linder, Jarvenpaa & Davenport 2003) and in less
advanced technology‐related companies (e.g., Procter & Gamble) (Huston, Sakkab 2006,
Sarkar, Costa 2008).
David Scarlatti - X792282
9
The differences between this model and the closed model are so important that, even at the
strategic level, that the new model cannot be explained by dominant business theory; basic
concepts such as entry‐level barriers or the competitive advantage of resource ownership are
questioned (Chesbrough, Appleyard 2007). The new Open Innovation models escape from the
classic Porter's vision of business strategy and subsequent schools. At the same time, the new
model’s survival may depend on these classic theories. Appleyard calls for a new "Open
Strategy" concept that gains the advantages of the "emerging anomalies" and still creates a
sustainable business. The open strategy urges a balance between openness and value capture.
Appleyard links sustainability with value capture and community engagement following some
existing business models for open‐source software.
Open Innovation represents an important shift in the innovation world, and it is a sufficiently
new concept to merit substantial attention from researchers.
During the past year, the European Innovation Scoreboard (EIS) was reviewed to better serve
the European Community as a true measurement of innovation at the EU level. However, the
report indicates that “To reflect and measure new forms of innovation in future editions of the
EIS, we must develop and incorporate new indicators measuring Open Innovation…”
(Hollanders, van Cruysen 2008).
In 2008, NESTA, the National Endowment for Science Technology and the Arts, published a
report as part of the Innovation Index project and proposed measures of firm‐level Innovation.
The report highlights interest from participants for measuring “Open Innovation”–“… a
respondent in the aerospace and defence sector expressed interest in a future measure of
Open Innovation…”. However, the authors report “It is likely, though, that the output of Open
Innovation can be measured using the same metrics as for traditional innovation” (Adams et
al. 2008). The argument is valid as long as one accepts that measuring innovation can be
limited to measuring the outputs of the process. We have found, however, a multitude of
metrics related to the input of the process as well as the way in which it is performed.
Although the need to measure Open Innovation is broadly recognised, there is a lack of
research on the applicability of existing methods and metrics for this new model. Because
Open Innovation principles are so different from previous models, it is expected that at least
some adaptation of the existing methods and metrics would be needed.
David Scarlatti - X792282
10
Table 2‐2 introduces a number of questions relevant to the measurement of Open Innovation:
Closed Innovation Principles
Open Innovation Principles Measurement‐related Questions
The smart people in our field work for us.
Not all of the smart people work for us, so we must find and tap into the knowledge and expertise of bright individuals outside our company.
Which metrics related to human resources should we use?
How do we measure knowledge outside of the firm?
Can we use metrics to motivate external people?
To profit from R&D, we must discover, develop and ship it ourselves.
External R&D can create significant value; internal R&D is needed to claim some portion of that value.
How do we measure R&D value?
How do we measure the portion of each R&D?
How do you measure access to external R&D?
If we discover it ourselves, we will get it to market first.
We don't have to originate the research to profit from it.
What measures of the discovery phase are relevant in Open Innovation?
If we are the first to commercialise an innovation, we will win.
Building a better business model is better than getting to market first.
Are speed measures relevant?
What substitutes for speed measures are of interest?
If we create the most and best ideas in the industry, we will win.
If we make the best use of internal and external ideas, we will win.
What uses we can make of these ideas? How do we mix internal and external ideas?
How can we measure the use of others’ ideas?
We should control our intellectual property (IP) so that our competitors don't profit from our ideas.
We should profit from others' use of our IP, and we should buy others' IP whenever it advances our own business model.
How do we trade with IP?
What do we do about patent metrics?
How do we measure licensing?
Table 2‐2 Contrasting Principles between Closed and Open Innovation (Chesbrough 2003) and measurement questions
The questions shown in Table 2‐2 point to a general research question: Can Open Innovation
be measured in the same way as traditional, closed Innovation?
David Scarlatti - X792282
11
The research presented here analyses this issue and, based on any contrast found between the
new and old models, elaborates on recommendations for the implementation of metrics for
Open Innovation.
2.2 Existing relevant knowledge This section summarises knowledge from existing published research.
Innovation measurement
Existing literature helps one to understand the motivation to measure innovation. There is
agreement among past and current literature on the need to measure the research and
development process and, taking a wider viewpoint, to measure innovation.
One needs measurement to improve things. The measurement process allows researchers to
set up a reference base, to determine whether changes have had an impact (positive or
negative) and to determine the size of the impacts (e.g., “Do you execute more projects on
time using this project planning tool?”). In the research and development process, there was a
time when improvements were made more easily, in large steps, but once performance
approached its limit, companies needed measurement metrics to carefully track new
improvements (McGrath, Romeri 1994). The performance measurement for R&D was
accepted later than such measurements were accepted for other processes (Chiesa et al.
2008). Nevertheless, there was a clear, strong case for implementing such measures in R&D
(Bremser, Barsky 2004).
The introduction of quality management systems and certifications (such as ISO9001) has been
another precursor of measurement in research and development departments (Schumann,
Ransley & Prestwood 1995).
Modern techniques for people management, such as variable compensation linked to
objectives, also called for measurement. To successfully tighten the compensation packages of
research and development staff, there must be agreement on measurable objectives. (Davila,
Epstein & Shelton 2005). Davila, Epstein & Shelton (2005) define innovation and present two
different strategies a company can follow to influence the innovation process: Play to Win and
Play Not to Lose. The book offers tips for innovation strategy definition and for structuring a
company for innovation. More relevant to this research is the chapter “How to measure
Innovation”, which introduces soft metrics versus hard financial metrics. These metrics are
linked to an incentives programme to support innovation.
David Scarlatti - X792282
12
Support for these different academic trends exists among current senior management in this
field. Among respondents to a 2008 Boston Consulting Group survey on measuring innovation,
74% agreed with the need for and relevance of such measurements (Andrew, Haanaes 2008).
The Boston Consulting Group report also highlighted the reality of innovation measurement in
companies (2008 survey). There is agreement on the need to measure innovation, but such
measurements are rarely performed. Moreover, there is a generalised dissatisfaction with the
results of efforts made to measure innovation. The 2008 report describes the cost and
competitive implications of innovation measurement. In addition, some of the most innovative
companies have some of the most rigorous measurement systems, but the report fails to
outline its selection process for identifying the most innovative companies.
Most companies are not happy with their measurement system because they system uses only
a few metrics and does not have a tie‐in to reward and incentive systems. Innovation output is
more accurately measured than the process or the input. The most commonly measured
components are presented and can be compared with Damanpour and Wischnevsky's (2006)
proposal. Similar measures are time to market, effectiveness and efficiency, time to volume,
and lifecycle performance. Some components added by this report are profitability (similar to
the R&D index from (McGrath, Romeri 1994)), idea generation and the selection and health of
portfolio management.
The most commonly used metrics relate to funding/revenue and time performance. The report
shows the gap between the number of available theories and their implementation at various
companies. Research into measurement frameworks is required to facilitate methodology
adoption by companies.
Even governments in different parts of the world agree on the need to measure innovation:
• “The centrality of the need to advance innovation measurement cannot be
understated” (Innovation Measurement. Tracking the State of Innovation in the
American Economy, The Advisory Committee on Measuring Innovation in the 21st
Century Economy).
• “It has been long understood that the generation, exploitation and diffusion of
knowledge are fundamental to economic growth, development and the well being of
nations. Central to this is the need for better measures of innovation” (OECD 2005).
David Scarlatti - X792282
13
It is possible to group the reasons for measuring Innovation into four main categories (see
Figure 2‐4).
Figure 2‐4 Reasons for measuring Innovation
The literature indicates a general agreement regarding the difficulties involved in measuring
innovation (Chiesa et al. 2008) (McGrath, Romeri 1994). Some authors have recognised
peculiarities in the R&D process (KerssensvanDrongelen, Cook 1997), and there is agreement
that this complexity requires the use of several combined metrics (Schumann, Ransley &
Prestwood 1995) (Mark Rogers 1998). Again, the research presented here found that senior
management agreed on this: “… innovation performance measurement belongs in the ‘too
hard’ basket…” (Adams et al. 2008).
KerssensvanDrongelen and Cook (1997) explain how the changing role of R&D, from a support
process to a driver of competitive advantage, calls for the proper performance measurements.
The authors review the R&D performance measurement literature and position it in the
context of control theory. Furthermore, they identify four main problems experienced when
measuring R&D performance: R&D contribution isolation, time lag, correct norms and
acceptance. They also discuss two main issues, the purpose of measurement and contingency
factors. The purpose of measurement is then broken down into two groups: “motivating” and
David Scarlatti - X792282
14
“diagnosing”. Contingency identifies several issues: the type of R&D, company size,
aggregation level, etc. KerssensvanDrongelen and Cook (1997) introduce an interesting
categorisation of metrics for their research and align it with the four categories found in the
Balanced Scorecard. The authors further define the system requirements, design parameters
and principles required for a performance measurement system. Interestingly, because the
type of R&D is seen as a key concept for designing the system, one can view Open Innovation
as a type of R&D.
The main challenges can be grouped into four categories (see Figure 2‐5).
Figure 2‐5 Challenges measuring Innovation
Most approaches proposed for innovation measurement focus on a few sets of dimensions
and metrics (Brown, Svenson 1988, Driva, Pawar & Menon 2000, Godener, Soderquist 2004,
Hauser 1998, Pawar, Driva 1999, Werner, Souder 1997b, Werner, Souder 1997a, Pappas,
Remer 1985). Only a few have tried to define a complete framework or Performance
Management System (PMS) (Chiesa et al. 2008), frequently based on the Balanced Scorecard
(BSC) (Bremser, Barsky 2004).
Chiesa et al. (2008) state that, in addition to the R&D function, the use of a Performance
Measurement System (PMS) is a critical success factor. They explore some of the reasons why
it can be a difficult task, but although they do recognise some approaches, they see most of
David Scarlatti - X792282
15
them as being limited to a set of dimensions and metrics. The authors further reference a few
approaches to a full PMS system, based on the Balanced Scorecard of Kaplan and Norton. They
complement this approach by proposing a framework applicable to any kind of R&D as part of
technology innovation. They define the steps necessary to apply the framework, and they offer
a practical example for a Clinical Research Organisation (CRO). The full set of PMS elements
includes a set of dimensions and metrics, together with a structure and a process for operating
the system. The framework takes into account contextual factors to define the elements,
namely, strategy, entities, types of activity, PMS objectives, and available resources.
Chiesa et al. outline the relationships among each factor in the framework, but do not always
specify or clarify these relationships. The steps given in the example are not generalised
enough to learn how each factor can influence each element in the PMS. The study is a good
example, but it is not a complete guide.
Bremser and Barsky (2004) focus on the framework aspects of the measurement systems and
work to integrate the Stage‐Gate and BSC approaches. The main contribution of the paper is
the evaluation of the Balanced Scorecard in an R&D process. Bremser and Barsky (2004) map
the functions of the performance measurement system using the characteristics of the BSC
and demonstrate its suitability. Some examples of metrics that are useful for this research are
reported.
Some organisations have developed their own frameworks. The European Foundation for
Quality Management (EFQM), published in 2005, is an adaptation of the Excellence Model for
innovation (EFQM 2004). The Oslo Manual proposes a conceptual framework for collecting
quantitative data (OECD 2005).
Categorisation or classification of innovation has been used to simplify the process of
measuring it (Damanpour, Wischnevsky 2006). Damanpour sets innovation as a prerequisite
for competitive advantage and growth, and the authors highlight how previous research on
the determinants, processes and consequences of innovation has produced inconsistent
results. Damanpour states that current approaches to explain inconsistencies based on
innovation types (product vs. process, technical vs. administrative, radical vs. incremental) fail,
and proposes a model based on organisational types (Innovation Generating Organisations vs.
Innovation Adopting Organisations). However, Damanpour restricts his model to the study of
one of the aforementioned types, namely technical innovation. Damanpour does not mention
applicability to administrative innovations.
David Scarlatti - X792282
16
Damanpour explains that Innovation Generating Organisations deal with creative processes
and Innovation Adopting Organisations deal with a problem‐solving process. These different
approaches are highlighted as the root cause of the differences between the two approaches.
Innovation Adopting Organisations are supposed to use what Innovation Generation
Organisations deliver, which is closely related to the use of technology both as a product and
as an application, as shown in Open University’s course T840. What happens if a product for
one organisation becomes an application for another, following a hierarchy of components?
Damanpour applied the model to explain how innovation is influenced by other factors such as
an organisation’s size or age and how radical it is in terms of innovation. Interestingly for this
research, Damanpour also describes the implications of these factors on the measurement of
innovation. Damanpour focused only on the outcome measurement, discarding input
measurement (R&D expenditure) or intermediate outcomes (Patens), in direct disagreement
with Katila (2008), who sees patents as an outcome measure.
Damanpour reviewed an interesting number of measures, each with reference to the rates and
speed of innovation (e.g., speed of development, earliness of adoption, percent adopters over
a period of time, timeliness, speed of implementation, extent of implementation, rate of
adoption, and innovation impact). The authors also explained the relation between Innovation
Generating Organisations and Innovation Adopting Organisations. The aforementioned lists,
however, must be revisited from the point of view of Open Innovation to assess whether these
factors are still relevant.
Damanpour introduces organisations in which there is coexistence between different units
that are adopting and generating innovation for the Innovation Integration Unit (IIU), a
managing organisation focused on strategic integration. It seems from these findings that an
Innovation Integration Unit would need different metrics for performance measurement.
At the other extreme of generic frameworks, there are very specific proposals such as the use
of patents in radical technology innovation (Katila 2008) or the definition of the R&D
effectiveness index for product development (McGrath, Romeri 1994).
Katila proposes the use of patents and patent citations for measuring innovation. However,
she proposes this strategy for a particular kind of innovation, namely, radical innovation. More
specifically, she focuses on technological radical innovation and excludes the other three kinds
of radical innovation: industry, organisation and user radical innovations.
David Scarlatti - X792282
17
Katila sometimes refers to measuring the ‘radicality’ of innovation, a notion that is well suited
for measuring innovation because it is assumed that more radical innovations are also better
innovations. However, this may not be the case for all organisations. Katila occasionally implies
that incremental innovation can also be measured with patents, but this thought is not
explicitly mentioned. Some limitations are recognised, and Katila points out that probably the
best use of innovation measurements is for comparison within an industry (i.e., benchmarking
use). One of the limitations mentioned is that measurements that are “…only applicable to a
fraction of the total output” can be greater than expected because the size of the fraction is
not mentioned and is likely small for some industries. Moreover, technological innovations are
not always patented. Either there are other options available for their protection (trade
secrets) or there is no protection at all (standards). Specifically for Open Innovation, patents
are an option, but are not always preferred.
Conversely, some technological innovations may be part of a group of other inventions. For
example, a new cell phone can include hundreds (if not thousands) of patents behind it. The
correction of some limitations is palliated using patent citation, based on citation only in
subsequent patents. This endogamy is a weakness of the system, but it clearly measures only
the output of the innovation process.
It seems that patents and patent citations could be used as a metric but that other metrics will
also be needed if the organisation wants to balance incremental and radical innovation. The
need for other metrics also applies if the organisation wants to develop other kinds of
innovation (industrial, organisational, user, etc.). This is true for measuring inputs and the
process itself and is not limited only to the measurement of outputs.
McGrath focuses on product development and indicates that it is much more difficult to
measure than other business processes. A company should criticise the percentage of R&D
spending as a measure and should advocate for measuring the effectiveness of R&D (instead
of the level of investment). McGrath proposes a very specific measure: the R&D Effectiveness
Index. The R&D Effectiveness Index focuses on the economic output of the process and is
based on the strong correlation between growth and profitability, as demonstrated through a
benchmarking study.
David Scarlatti - X792282
18
Figure 2‐6 summarises the existing approaches:
Figure 2‐6 Approaches facilitating Innovation Measurement
Open Innovation
The father of Open Innovation, Henry Chesbrough, defined the concept in his seminal works in
2003 (Chesbrough 2003a), (Chesbrough 2003b). Chesbrough presents the "Open Innovation”
(OI) model in opposition to the traditional closed approach. He shows how many well‐
established, giant firms have been challenged by newcomers that bypass the supposed "entry
barrier," thanks to this new approach to innovation. Chesbrough states that the Open
Innovation model will be adopted steadily in almost all industries by all companies, and he
defines three areas to show how the adoption of the model is happening in each of them:
funding, generating innovation and commercialising innovation.
By that time, others had already started researching other ways of sourcing innovation (Linder,
Jarvenpaa & Davenport 2003), but Chesbrough blended both sourcing and exploiting. Since
then, the model initially adopted by technology firms (e.g., IBM, Intel, Lucent, Xerox and
others) (Andrews 2003) has been adopted by more and more companies in very different
David Scarlatti - X792282
19
sectors (e.g., Procter & Gamble, General Motors, Eli Lilly Colgate‐Palmolive and others)
(Gwynne 2007).
Different ways have been identified to use the new model. Such areas for implementation
include corporate venture capital, internal spin‐off programmes, external licensing
programmes, "use it or lose it" initiatives, programmes to license technologies, portals to
outplace technologies and other portals to solicit input technologies (Andrews 2003). Andrews
pinpoints the two sides of the Open Innovation approach, namely the access to external
knowledge versus the monetisation of internal knowledge. Open Innovation practitioners
identified by Andrews include IBM, Intel, Lucent, Xerox, Procter & Gamble, Merck, Pfizer and
Millennium. Andrews proposes the following metrics:
• Identify the sources of the most important ideas in your industry over the past five
years.
• Identify how many came from inside your company, how many came from inside other
current industry participants, and how many came from outsiders.
• Identify what percentage of your own pipeline of future projects come from outside
versus from within. How does that ratio compare with the first ratio?
Using this approach, relations with customers, competitors and suppliers may be modified. For
example, areas for modification can include the following: customer integration, supplier
integration, competitive alliances, customer of customer integration, integration of second‐tier
suppliers, cross‐industry innovation, university‐industry cooperation and globalisation of
innovation (Enkel, Gassmann 2007)
A new concept has even been created to enable the model: knowledge brokers (Gwynne
2007), (Sousa 2008). Gwynne introduces the idea of the “Knowledge Broker” as a catalyst in
the Open Innovation model. Examples he mentions include NineSigma, yet2.com or
InnoCentive. Gwynne points to one very specific problem of Open Innovation, namely, how to
integrate the technology coming from outside the organisation. He worked with early adopters
of the Open Innovation model including Procter & Gamble, General Motors, IBM, Eli Lilly,
Colgate‐Palmolive, and Philips.
Soon it became apparent that, in an Open Innovation model, there is a key issue: balancing
value capture against value creation (Andrews 2003), (Laursen, Salter 2006). These two
dimensions of value (capture and creation), plus the duality of internal and external resources,
David Scarlatti - X792282
20
configures a map that is very relevant when thinking about measurements. This concept is
discussed later in this paper.
It is important to note that Open Innovation exhibits a diffuse definition, as recognised by the
concept’s own originator (Chesbrough 2006b). The latest Chesbrough work claims that the
new model requires some changes even at the strategy level for full exploitation, which he
calls “Open Strategy” (Chesbrough, Appleyard 2007).
Interestingly, current key trends in the research on Open Innovation do not focus on
measurement. Rather, the main five themes are business models (balancing value creation and
value capture), inclusion of external technologies, identification/assessment of knowledge,
Start‐ups and Intellectual Property Management (Chesbrough 2006a). One exception to this
lack of focus on measurement and metrics is presented in Chesbrough (2004), where the need
for a change in metrics is clearly stated, but is focused in the issue of managing “false
negatives” (i.e., projects that would have been discarded under a closed model but may be
pursued under an open model).
2.3 Research questions The general question from the introduction section, “Can Open Innovation be measured in the
same way as traditional closed Innovation?” can now be split into more specific questions:
1. Does Open innovation require the same sort of measurement approaches as
traditional innovation?
2. What metrics are useful when the Open Innovation model is applied?
3. What changes are needed regarding metrics when the Open Innovation model is
embraced?
David Scarlatti - X792282
21
Chapter 3 Methodology
3.1 Methods and techniques selected First, a comprehensive literature review responds to the first objective of the research:
• Determine the current practices regarding the measurement of innovation.
A cross‐sectional case study has been used to explore Open Innovation, with a focus on the
measurement practices in place or processes related to the practices identified in the first
literature review. A review of the currently available examples of Open Innovation has been
used to respond to the second objective of the research:
• Determine, based on Open Innovation principles and examples available in the
literature, the basic criteria for the applicability of measurement practices.
Analysis and synthesis of the previous deliverables have generated the deliverable for
Objective 3:
• Generate a proposal for Open Innovation measurement practices based on
correlation of the deliverables from Objective 1 and Objective 2.
3.2 Justification This research is empirical and based on observation. Some sources for review on the general
problem of measuring innovation have been identified as a necessary starting point, but the
available references examining the specific problem of measuring Open Innovation are very
limited at the time of this writing . Other, alternative approaches have also been evaluated, as
discussed in the next two paragraphs.
Experiments do not seem appropriate because it would be very difficult within the research
timeframe to conduct any reasonable experiments. The typical period for the measurement of
a business process is one year because this is the typical budgeting cycle, even in R&D
concepts. It would make sense to make observations over longer periods of time as innovation
matures (3‐5 years). Similarly, it does not seem possible to find an organisation that would be
willing to test any conclusions regarding enhancements in the measurement process.
Surveys would have presented unmanageable risk because the population is unknown and
dispersed. An attempt was undertaken, using unstructured interviews at a workshop on Open
Innovation conducted in Madrid in May 2009, but it was difficult to collect more than informal
David Scarlatti - X792282
22
feedback, and that feedback was difficult to compare or analyse. The survey was only useful
for qualitative input about the interest in the research topic, not as a data collection
technique.
Case studies, however, seemed to fit well with the scope of this research. Thanks to the
Internet, it was possible to access information regarding real examples of Open Innovation
adoption at different companies. As stated, Open Innovation is new enough to discard any
longitudinal case study. Due to the general applicability of the research, the exemplar case
study also does not apply. Therefore, a cross‐sectional case‐study has been selected and takes
into account a variety of organisations as they operate today.
3.3 Research procedures The data collected for this research come from two main sources:
• Public‐domain company records on Open Innovation adoption. The use of such records
avoids potential confidentiality issues.
• Literature available regarding the measurement of innovation.
As mentioned in section 3.2, an initial set of unstructured interviews was attempted, with
limited benefit. Therefore, no additional questionnaires or interviews were undertaken.
Use of the “ISI Web of Knowledge” tool has allowed me to identify relevant existing literature
on both the measurement of innovation and the themes of Open Innovation.
Other relevant sources have been used to collect information:
• The OpenInnovation.eu platform founded by Prof. Dr. Wim Vanhaverbeke, in
partnership with Henry Chesbrough.
• The UK Innovation Index Project, led by the National Endowment for Science,
Technology and the Arts (NESTA).
• The European Industrial Research Management Association (EIRMA).
3.4 Ethical considerations The main ethical consideration had to do with the disclosure of private information used
during the research. However, the final data used are publicly available, so no disclosure of
private information has taken place, to the best of my knowledge. Details of the responses to
the informal survey have not been included, so no ethical issues are apparent.
David Scarlatti - X792282
23
Chapter 4 Analysis and interpretation
4.1 Summary of data collected
Current practices
This section provides an overview of current practices as cited in the literature on measuring
innovation. These practices are later analysed to determine their suitability for the Open
Innovation approach.
Fundamentally, every approach follows the same process for measurement, which is well
defined by KerssensvanDrongelen and Cook (1997) and is shown in Figure 4‐1.
Figure 4‐1 The measurement and control process (KerssensvanDrongelen, Cook 1997)
The main differences are more related to what is measured (i.e., the metrics used). Metrics are
sometimes referenced as Key Performance Indicators (KPIs). Multiple factors influence the
selection of metrics, and some research has already addressed the identification of these
factors. These factors are what KerssensvanDrongelen and Cook (1997) call “contingency
factors” or what Chiesa et al. (2008) refer to as “contextual factors”. Our research accepts the
David Scarlatti - X792282
24
type of research as a factor influencing metrics and will focus on the particular case of Open
Innovation.
When a complete framework is used, these metrics are aligned with dimensions (usually a
dimension can be measured with a set of metrics) and a formal process with norms and rules
about who does the measurement, how, with what frequency, and so on. A good example of a
framework is the Balanced Scorecard (Bremser, Barsky 2004).
Reviewing the vast array of metrics proposed (see section 2.2 Existing relevant knowledge), we
find metrics that can be grouped into different types, according to different characteristics. For
further analysis, it seems useful to examine groups of metrics, instead of each one individually.
Several groups were identified for this research:
Absolute metrics and ratio metrics: Absolute metrics are expressed in units of some magnitude
(e.g., Euros or months) and can be compared to target or reference values. Ratio metrics are
expressed in units per another unit or as percentages. Ratio units are normally used for better
comparison over a period of time or between different organisations.
Input, intermediate or outcome metrics: This classification is more relevant for our research.
Metrics can be found relative to the input of the innovation process (e.g., a budget for R&D or
expenditure), the output (e.g., the number of new products) or,most relevant for us, the
process itself or intermediate processes (e.g., the number of patents).
According to the type of magnitude, we found metrics for speed, time, financials or
performance‐related measurements (e.g., the execution of projects or market share).
Quantitative and qualitative (or objective and subjective): Some metrics are an exact measure
of a magnitude, but others are estimations or rankings based on expert judgement or
subjective validation.
Metrics at the organisation level, group level or individual level: Some metrics can be valid at
all three levels, whilst others only make sense at the aggregated or individual level.
Metrics related to the creativity phase or to the value capture phase: These are related to the
phases for innovation from Davila, Epstein & Shelton (2005).
Furthermore, metrics can serve one or both main objectives and can be used to monitor
and/or to motivate (reward).
David Scarlatti - X792282
25
Interestingly, other authors have found very similar grouping categories for R&D measures
(Ojanen, Vuola 2003) (Table 4‐1).
Table 4‐1 The dimensions of R&D performance (Ojanen, Vuola 2003)
Based on the literature review, a collection of metrics has been selected. The metrics are
grouped consistent with listed characteristics (Table 4‐2). This table serve as a reference for
the analysis of the different groups of metrics and is helpful to understand which groups are
affected in an Open Innovation approach and to understand the nature of the effects.
David Scarlatti - X792282
26
METRIC
Absoluteor
Ratio
Phase:Input
iNtermediateOutcome Magnitude
quanTitativequaLitative
Levels:Organization
GroupIndividua
Creativityor
Valuecreation
Function:MonitoringMotivating
Time used to adopt an innovation A N Time T O/G C MTime used to develop a product A N Time T O/G C MTime used to commercialize A N Time T O V MProject related measures (EVM) R N Index T O/G C MImpact on market (market share…) R O %/€ T O V MProfitability A O € T O V MIdea generation and selection A N ideas T O/G/I C RTime to market A N Time T O V MR&D effectiveness and efficiency R N Index T O C MHealth of the innovation portfolio R O % T O C MLife cycle performance R N Index T O C RTime to volume A N Time T O V MCustomer satisfaction A O Index L O/G V RTotal fund invested in growth projects A I € T O C Mperformance, projected versus actual R N % T O/G C R# of projects that meet planned targets A N projects T O/G C Mdevelopment time (Average) A N Time T O/G C M
revenue realized form offering in the past X years A O € T O V MCannibalization of existing products sales by new offerings A O € T O V M% of ideas funded R N %/ideas T O C M# of projects killed A N projects T O/G C M# of new products per dollar spend on R&D R O products T O C MTRL’s Technology maturation achieved A N Index T O/G C R# product at a given stage of the stage-gate approach A N products T O/G C MTime among stages A N Time T O/G C M% passing each stage R N %/projects T O/G C M#acceptable rate R N products T O/G C MSafety incidents (rare) A N incidents T O/G/I C RSkill coverage of competencies R I % L O/G/I C RTraining hours A N Time T O/G/I C R# patents A N patents T O/G/I C R# patent citations A N citations T O/G/I C R# patents per employee R N index T O/G C REffectiveness Index (ratio output/input) R O index T O V MPerformance: present values of an accomplishment A O € T O V MR&D contribution to profit/R&D cost A O %/€ T O V MMarket share gained due to R&D A O %/€ T O V MScores on surveys-customer satisfaction A O index L O/G/I V RScores on surveys-employee satisfaction A O index L O/G/I V R% of customer driven projects R N %/projects T O/G C MEngineering hours on projects/total engineering hours R N %/hours T O/G C M% projects terminated R N %/projects T O/G C MHours on projects / total R&D hours R N %/hours T O C M
Current time to market/ reference time to market R N %/Time T O V MRate of reuse of standard designs or proven technology R N index T O C MSum of revised project duration / sum of planned duration R N %/Time T O/G C M# of times rework A N reworks T C R% budget spent internally and externally on basic and applied research R I %/€ T O V M% of projects in co-operation with third party R N %/projects T O C M%of project evaluation ideas applied in new projects R N %/ideas T O C MTalent, in recruitment, training … A I talent L O/I C RKnowledge Management Tools use R N %/time T O/G/I C MCommunication Effectiveness A N index L O C RBalance of innovation portfolio R O % T O C M
Table 4‐2 Grouped Innovation metrics
David Scarlatti - X792282
27
Basic Criteria of Applicability
Based on the Open Innovation reference material, this section identifies the criteria that
should be used to judge the applicability of current measurement practices to the Open
Innovation model.
Open Innovation principles introduced in previous sections will serve as a guide for this criteria
identification:
Not all of the smart people work for us, so we must find and tap into the knowledge and
expertise of bright individuals outside our company.
This principle refers to the fact that in the Open Innovation model, the employees of the
company (“people who work for us”) are not the only appropriate participants in the
innovation process. People from outside the company can–and should be invited to–
participate in the innovation process. However, we must ensure that the people from outside
who participate in innovation are the right ones (i.e., smart, bright, knowledgeable).
According to this principle, people‐related metrics, which usually refer to internal staff, should
be used or adapted to include external actors.
At the same time, metrics related to knowledge from people must cover not only internal
knowledge but also the knowledge accessed from people outside of the company.
Because people‐related metrics can be aimed at motivating people, the metrics should also
aim to motivate external people.
Metrics regarding “bright” people are needed to ensure that the organisation is accessing the
right people. “Bright” can be associated with talent, so talent measurement is probably
important.
External R&D can create significant value; internal R&D is needed to claim some portion of
that value.
This principle recognises that both external and internal R&D must be used in the innovation
process. Internal R&D is easy to locate and to manage in the traditional way, but external R&D
is beyond the boundaries of the company. Therefore, a different approach is needed for
external R&D. The first thing to establish is access to the external R&D. Then, one can develop
ways to measure it.
David Scarlatti - X792282
28
Metrics about the value of R&D must be applicable to external R&D.
Metrics must differentiate between external and internal R&D, which is especially relevant for
ratios and comparison metrics.
Metrics related to access to external R&D are needed as well.
We don't have to originate the research to profit from it.
This principle is based on the fact one can access external R&D at any stage. R&D initiated by
others can be relevant to an organisation at any point in time. One has access to the external
R&D, and it can be used to generate profit.
Metrics are needed to measure R&D that is not generated internally.
Metrics must differentiate internally originated researched from other types of research. Profit
should also be measured and classified according to whether it originated from internal or
external research.
Building a better business model is better than getting to market first.
This principle invites a business to focus not only on being the first but also on thinking about
the right business model to profit from the innovation.
Time‐to‐market metrics must be balanced with others. Measures of the success of the
business model are needed. Speed metrics may refer to only a single phase of the innovation
process instead of the whole process.
If we make the best use of internal and external ideas, we will win.
Ideas can be used in different ways, and one can use both internal and external frameworks.
“Use” can mean incorporate into an existing product or it can simply mean the sale of the idea.
Paying for a third‐party idea can multiply the value of your own product.
Metrics of usage are needed. Again, distinction between internal and external ideas is needed.
Metrics for creation must be balanced with metrics for usage.
We should profit from others' use of our Intellectual Property, and we should buy others'
Intellectual Property whenever it advances our own business model.
David Scarlatti - X792282
29
Intellectual property must be managed in the same way as any other asset. One can sell and
buy it, and it is even possible to rent it. This principle again focuses on profiting from
intellectual property, regardless of how it is used.
Metrics regarding intellectual property are needed. Metrics must differentiate one’s own
intellectual property versus other companies’ intellectual property. Metrics regarding the
profit related to intellectual property are needed.
4.2 Data Analysis In this section, the basic criteria and current practices are correlated to find support for
current measurement practices for Open Innovation. I also propose any enhancements for
cases where there is insufficient existing support.
Current practices show the low use of people‐related metrics. At the individual level, 11 of 54
metrics can be applied (Table 4‐3).
Absolute/Ratio
Phase of processInput /
iNntermediate / Outcome Magnitude
quanTitativequaLitative
LevelsOrganization/
Group/Individual
PhaseCreativity/
Value creation
FunctionMonitoring/Motivating
Idea generation and selection A N ideas T O/G/I C RSafety incidents (rare) A N incidents T O/G/I C RSkill coverage of competencies R I % L O/G/I C RTraining hours A N Time T O/G/I C R# patents A N patents T O/G/I C R# patent citations A N citations T O/G/I C RScores on surveys-customer satisfaction A O index L O/G/I V RScores on surveys-employee satisfaction A O index L O/G/I V R# of times rework A N reworks T O/G/I C RTalent, in recruitment, training … A I talent L O/I C RKnowledge Management Tools use R N %/time T O/G/I C M
Table 4‐3 Innovation metrics for individuals
These metrics are most usable for external staff, but some of them need to be adapted when
applied to external people. Clear example is metrics based on surveys; the surveys need to be
extended to external collaborators. Other metrics call for special attention due to the issue of
their collecting and treating personal data from people who are not employees. In general,
metrics are applicable but may be complex if a company needs to account for external people
who do not collaborate (i.e., external people not willing to report the training they have
received).
David Scarlatti - X792282
30
Especially difficult is the measurement of talent. Are external collaborators really “bright”
people? What are the parameters by which this determination is made? Knowledge‐related
measures must be applied.
The convenience of these metrics for motivating external collaborators is still to be proven
because the reward mechanism will be different (i.e., the measured individuals are not on the
payroll).
Value of the R&D is usually measured in Euros. The problem with using these metrics for
external R&D is that the economic value is usually associated with products. However,
products will include both internal and external R&D. To use these metrics properly, it is
necessary to account for both internal and external contributions at development time, which
enables managers to later split this economic value. Typical metrics found regarding R&D value
are shown in Table 4‐4.
Absolute/Ratio
Phase of processInput /
iNntermediate / Outcome Magnitude
quanTitativequaLitative
LevelsOrganization/
Group/Individual
PhaseCreativity/
Value creation
FunctionMonitoring/Motivating
Impact on market (market share…) R O %/€ T O V MProfitability A O € T O V MTotal fund invested in growth projects A I € T O C Mrevenue realized form offering in the past X years A O € T O V MCannibalization of existing products sales by new offerings A O € T O V MPerformance: present values of an accomplishment A O € T O V MR&D contribution to profit/R&D cost A O %/€ T O V MMarket share gained due to R&D A O %/€ T O V M% budget spent internally and externally on basic and applied research R I %/€ T O V M
Table 4‐4 Innovation metrics regarding R&D value
The ability to distinguish between external and internal value is especially relevant for metrics
like the “Balance of the innovation portfolio.” Such an ability also permits a company to
enhance metrics like “Percent of projects in co‐operation with a third party”. Balance can be
pursued on the new axis of internal/external value, and percentage of co‐operation can be
measured not only in terms of the number of projects but also in terms of the value they
provide.
What is difficult to find in a traditional measurement system are metrics regarding access to
external R&D, which is a clear gap to close when adopting Open Innovation.
David Scarlatti - X792282
31
Apart from splitting profit metrics into internal and external R&D, it is important to
differentiate metrics for the input (i.e., originating) phase of the process. Outputs of Open
Innovation are not expected to be different (new products, methods, patents), but it is the
input and intermediate phases that are executed differently. Useful tools for this splitting are
available systems for classifying the maturity level of a technology. One such system is NASA’s
Technology Readiness Levels (Mankins 1995). Metrics can be used to compare the maturity
level at the starting point and at the ending point for both internal and external R&D.
Measuring profit in terms of R&D sold (e.g., licensing and patent selling) is also important. One
can measure profit in Euros, so it is possible to easily compare the sale of R&D with the sale of
traditional products.
Metrics are also needed for inputs that come from outside the organisation (i.e., ideas
proposed to the company). The number of ideas generated should be compared with the
number of ideas selected from outside the company.
Typically, there is strong emphasis on time‐to‐market metrics. To balance these metrics with
the Open Innovation focus on business models, other independent metrics are needed (i.e.,
market adoption of products) to determine whether an organisation is the first to arrive.
“Time among stages” and “Percent passing each stage” are useful to measure speed at other
phases, and these metrics are perhaps more relevant than the duration of the whole process.
Usage metrics are needed at different stages (e.g., percent of ideas used, percent of products
used, percent of patents used and percent of trade secrets used), and such metrics should
include information on whether the usage is internal or external. New ratios are potentially
valuable in this area. For example, metrics may include internally created ideas versus
externally acquired ideas, or they may include internal ideas used versus external ideas used.
These ratios can help to detect situations where more weight is on the worst, less effective
approach.
Metrics need to reflect the transition from Closed Innovation to Open Innovation, and they
help to detect deviations. Such deviations include desorbing innovation (i.e., too little external
technology acquisition) and absorbing innovation (i.e., too little external technology
exploitation) (Lichtenthaler 2008). Lichtenthaler presents how the Open Innovation strategies
can vary from a proper balance between external technology exploitation and external
technology acquisition. He identifies groups of companies, ranging from Closed Innovators to
Open Innovators, with extreme deviations like Desorbing Innovators and Absorbing
David Scarlatti - X792282
32
Innovators, see Figure 4‐2. In his research, Lichtenthaler shows how the high profitability of
the open innovators suggests that sticking to the closed innovation model may lead to a
substantial weakening of a firm’s competitive position in the future. The right balance between
exploitation of internal and external R&D is the desired condition. Ratios can be measured in
terms of economic value and / or monetary investment.
Figure 4‐2 Balance in Open Innovation (Lichtenthaler 2008)
Intellectual Property related metrics (patents, licenses) are usually in place, but when adopting
the Open Innovation model, it is necessary to address the balance of the Intellectual Property
produced, the Intellectual Property used, the Intellectual Property sold, and the Intellectual
Property bought. Again, different ratios can be defined that are useful for detecting deviations,
like selling all of the IP a company produces or only using the IP bought. In the Open
Innovation model an organisation is supposed to blend its own knowledge with the acquired
knowledge and to then sell only a part of the knowledge it generates.
David Scarlatti - X792282
33
4.3 Interpretation in relation to the research questions
Does Open innovation require the same sort of measurement approaches as
traditional innovation?
Existing measurement approaches, based on metrics and a performance management system,
are completely valid for the Open Innovation model. There are only three considerations to
keep in mind:
• Measuring the output of the innovation process is completely valid because the
objectives of the process do not change. However, new ways of generating profit can
require additional metrics regarding economic results because there is the possibility
to sell IP or benefit in different ways from other companies’ products (including using
them in one’s own innovations).
• Measuring the process requires a new axis, namely, that of externality. Most of the
constituents coming from inside the company can come now from outside the
company as well, calling for new metrics (ratios) that compare internal and external
sources.
• Externality poses challenges when the objective pursued by a metric cannot be
achieved, as is the case with external sources. Therefore, new metrics are needed to
complement the initial ones.
What metrics are useful when the Open Innovation model is applied?
Following the approach of treating the metrics according to the groups proposed in section 0,
the answer to this question includes the following six parts:
Absolute metrics and ratio metrics: Absolute metrics remain useful, but the focus is on a new
breed of ratio metrics that will compare the internal and the external contributions of each
innovation element. In addition to the results, each metric of the process can be split into the
internal (traditional) source and the new sources.
Input, intermediate or outcome metrics: Output metrics are useful because they permit the
measurement of how the whole innovation process is working. Eventually, if all is reduced to
the economic success of the company, even using the same single metric (revenue) would be
useful. However, under a richer measurement system, the input and intermediate metrics are
very useful to determine how the Open Innovation model works. Inputs to the process are
now broader because the technology may be at different maturation levels, probably because
David Scarlatti - X792282
34
the technology was invented and developed by others who entered the innovation process at
later stages. Intermediate metrics, which can measure the process until the output of the new
technology to third parties, are considered the best option. Efficiency cannot be measured
only in terms of time‐to‐market metrics. Some phases may move at greater speeds than
others. Finally, there is a need to measure the IP (acquired or generated) at all stages .
Quantitative and qualitative (or objective and subjective): The fact that some metrics must be
translated to external sources, where direct access to the data is difficult, will drive an increase
in the use of qualitative metrics. These metrics are still valuable to manage the innovation
process when external elements not under the direct control of the company must be
evaluated.
Metrics at the organisation level, group level or individual level: These categories of metrics
are slightly less useful in the Open Innovation model because the concept of organisation must
be extended to all third parties involved. Also, the individuals and groups of individuals now
include persons from multiple organisations. Individual metrics valid for inside the company
cannot be practically (or even legally) measured.
Metrics can relate to the creativity phase or to the value‐capture phase: Open Innovation
impacts both phases of the process. During the creativity phase, one can count the ideas
coming out of the organisation. At value capture, one can decide to transfer to others an
invention at a given price or royalty. Therefore, it is interesting to maintain metrics for both
phases and to discriminate between external and internal contributions.
Metrics can serve one or both main objectives: to monitor and/or to motivate (reward):
Metrics for monitoring are useful and may need to be complemented with additional metrics
that measure specific results or contributions. Metrics for motivating face the challenge of
motivating external people who are not part of the organisation. This issue find similarities in
the purchasing process or to the supplier management field, namely, how does one motivate
suppliers? (Carr, Pearson 1999)(Krause, Ellram 1997).
What changes are needed in terms of metrics when the Open Innovation model is
adopted?
Consistent with my interpretation of previous research questions, it is clear that the changes
imply adaptation or at least changes to external metrics along with the addition of new ones.
New metrics for new results, like the profit coming from IP sold or the cost of buying others’
licenses, are needed. It will be necessary to split some metrics to reflect the contribution of
David Scarlatti - X792282
35
external sources. This way, one can trace the contribution from each part when analysing the
revenue coming from products. Finally, current metrics that are useful inside the organisation
will need complementary parameters that generate the same impact outside the organisation.
Development of these metrics will requires consideration of the different relationships
involved, and any developments should always consider the legality of the measurement
process.
The rest of the performance management system can remain. A few changes regarding
responsibilities may be required, depending on who is in the best position to measure external
contributions.
4.4 Interpretation in relation to the aim The aim of this project was to produce recommendations for metrics used to evaluate the
effectiveness of an ‘Open Innovation’ approach to research. All of the findings presented can
be condensed in this “Decalogue” that a company embracing Open Innovation model should
follow when implementing a performance measurement system:
1. Add new metrics to measure new profitability models. It is necessary to measure the
revenue derived from licensing or from selling innovation to others.
2. Split all relevant measures into those of internal and external origin. For example, a
product can make use of 10 patents; measure how many are internal and how many
are external.
3. Introduce ratio metrics for comparing internal and external contributions. For
example, determine which new products are developed in‐house versus which are the
result of product acquisition.
4. If not present, introduce a maturity level model to measure the process step by step.
5. Measure the external contributions at different stages of maturity. For example,
identify the basic ideas coming from outside, the basic inventions sold to others and
the final solutions from others that may be incorporated into a company’s products.
6. Measure efficiency at different stages of the model. For example, measure the ideas
discarded versus the ideas selected, the projects approved versus the projects
cancelled and the products in portfolio versus the products discontinued.
7. Categorise the contributions to revenues as being those of internal and external origin.
David Scarlatti - X792282
36
8. To motivate external parties, use different metrics than the ones used to motivate
internal parties.
9. Check the legality of maintaining databases regarding external people.
10. Introduce a “talent” (i.e., knowledge) metric for your external parties.
There is also a corollary to this Decalogue:
Avoid using your current innovation performance measurement system without
adaptation for the new Open Innovation model.
David Scarlatti - X792282
37
Chapter 5 Conclusions
5.1 Conclusions regarding the research questions
Does Open innovation require the same sort of measurement approaches as
traditional innovation?
The main conclusion is that the same sort of approaches can be used but it is important to
adapt the metrics to capture the whole value of the new approach. If a company embraces the
new model but uses exactly the same performance measurement system and metrics, then
there is a high risk that the real value of the new model will not be properly reflected. Such a
scenario leads to the wrong conclusions about the suitability of the new model for the
company.
What metrics are useful when the Open Innovation model is in use?
Most metrics remain useful in the Open Innovation model, but some types must be carefully
considered:
• Ratios between external and internal contributions.
• Metrics regarding new ways of profiting (e.g., licensing, venturing, spin‐offs etc.).
• Metrics regarding the efficiency at intermediate steps of the technology maturation
cycle.
• Qualitative metrics for external sources not under the control of the company.
• New metrics for motivating external people.
• Intellectual Property metrics.
What changes are needed to metrics when the Open Innovation model is embraced?
It is clear that the changes needed are related to the set of metrics already in place. It will be
necessary to add new metrics for new results, to split some metrics to reflect the contribution
of external sources and to complement some metrics to achieve the same–or at least similar–
impact inside and outside the organisation.
5.2 Conclusions regarding the research aim The aim of this project is to produce recommendations for metrics used in evaluating the
effectiveness of an ‘Open Innovation’ approach to research.
David Scarlatti - X792282
38
The first clear recommendation is to use metrics regarding innovation in your organisation for
the purpose of properly managing the technology that a company develops or acquires.
The second recommendation is to adapt the metric system when one embraces the Open
Innovation model. These changes are designed to be effective in measuring innovation and in
acquiring appropriate data for making decisions.
The third recommendation is to apply the “Decalogue” presented in section 4.4.
5.3 Further work A valuable research objective would involve taking a real performance measurement system
from a company embracing Open Innovation and applying the changes proposed herein. Then,
one could compare the metrics results from the old performance measurement system with
those from the new scheme. The purpose for such work would be to determine whether the
new measurement system better captures the value of the Open Innovation model.
Other possible paths for further research include broadening the scope of the research. The
initial scope shown in Figure 1‐1can be expanded in two directions:
• Innovation can occur outside the R&D department. For example, the Marketing
department could adopt an open model.
• The innovation may occur not only in the technology (product or process) sector, but
also in the methods used for completing any task or in the skills used in the business.
5.4 Implications of this research One interesting finding from this research is a determination of how beneficial new paradigms
for innovation might be for other technology management activities. Such paradigms would
necessarily be accompanied by the right set of metrics or measurement processes to provide
upfront demonstration of the benefits they can provide.
Another interesting question involves the fact that openness, in general, implies a blurring of
the boundaries between in‐company and out‐of‐company. Ethical and legal considerations are
needed to treat, for example, the personal data of collaborators who are not proper
employees. Special attention to confidentiality issues may be necessary in this open
environment.
David Scarlatti - X792282
39
A question arises once we accept that changes are needed in the measurement system when
the Open Innovation model is embraced: How can these changes be better planned and
implemented from the technology management point of view?
This research and work by others regarding Open Innovation still needs to be reviewed in light
of new publications on real‐life experiences of adopting the new model. While this research
was conducted, the number of publications regarding Open Innovation was growing steadily.
From a personal point of view, I am very happy to have followed this process. After working in
a research and development centre for 5 years, I now recognise that daily business pressures
can cause me to forget some of the basic principles of the research method. We should never
forget these rules while doing research and development. I have found that industrial activity
can benefit a great deal from a more academic point of view.
David Scarlatti - X792282
40
References
Adams, R., Neely, A., Yaghi, B. & Bessant, J. 2008, Proposal for Measures of Firm‐Level Innovation Performance in 12 Sectors of UK Industry, NESTA, UK.
Andrew, J.P. & Haanaes, K. 2008, Measuring Innovation 2008. Squandered Opportunities, The Boston Consulting Group.
Andrews, P. 2003, Open innovation: Using research from everywhere for new product and service development, IBM BCS.
Bremser, W.G. & Barsky, N.P. 2004, "Utilizing the balanced scorecard for R&D performance measurement", R & D Management, vol. 34, no. 3, pp. 229‐238.
Brown, M.G. & Svenson, R.A. 1988, "Measuring R‐And‐D Productivity", Research‐Technology Management, vol. 31, no. 4, pp. 11‐15.
Carr, A.S. & Pearson, J.N. 1999, "Strategically managed buyer–supplier relationships and performance outcomes", Journal of Operations Management, vol. 17, no. 5, pp. 497‐519.
Chesbrough, H. 2004, "Managing open innovation", Research‐Technology Management, vol. 47, no. 1, pp. 23‐26.
Chesbrough, H.W. 2006a, "New puzzles and new findings" in Open innovation: Researching a new paradigm, eds. H.W. Chesbrough, W. Vanhaverbeke & J. West, Oxford University Press, Oxford, pp. 15‐34.
Chesbrough, H.W. 2006b, "Open Innovation: A new paradigm for understanding industrial innovation." in Open innovation: Researching a new paradigm, eds. H.W. Chesbrough, W. Vanhaverbeke & J. West, Oxford University Press, Oxford, pp. 1‐12.
Chesbrough, H.W. 2003a, "The era of open innovation", MIT SLOAN MANAGEMENT REVIEW, vol. 44, no. 3, pp. 35‐41.
Chesbrough, H.W. 2003b, Open innovation: The new imperative for creating and profiting from technology, Harvard Business School Press, Boston, MA.
Chesbrough, H.W. & Appleyard, M.M. 2007, "Open innovation and strategy", CALIFORNIA MANAGEMENT REVIEW, vol. 50, no. 1, pp. 57.
Chiesa, V., Frattini, F., Lazzarotti, V. & Manzini, R. 2008, "Designing a performance measurement system for the research activities: A reference framework and an empirical study", J.Eng.Technol.Manag., vol. 25, no. 3, pp. 213‐226.
Cumming, B.S. 1998, "Innovation overview and future challenges", European Journal of Innovation Management, vol. 1, no. 1, pp. 21‐29.
David Scarlatti - X792282
41
Damanpour, F. & Wischnevsky, J.D. 2006, "Research on innovation in organizations: Distinguishing innovation‐generating from innovation‐adopting organizations", J.Eng.Technol.Manag., vol. 23, no. 4, pp. 269‐291.
Davila, T., Epstein, M.J. & Shelton, R. 2005, Making Innovation Work: How to Manage It, Measure It, and Profit from it, Wharton School publishing.
Driva, H., Pawar, K.S. & Menon, U. 2000, "Measuring product development performance in manufacturing organisations", International Journal of Production Economics, vol. 63, no. 2, pp. 147‐159.
EFQM 2004, EFQM Framework for Innovation , European Foundation for Quality Management.
Enkel, E. & Gassmann, O. 2007, Driving Open Innovation in the Front End, EURAM (Paris, France).
Godener, A. & Soderquist, K.E. 2004, "Use and impact of performance measurement results in R&D and NPD: an exploratory study", R & D Management, vol. 34, no. 2, pp. 191‐219.
Gwynne, P. 2007, Open Innovation's Promise and Perils.
Hauser, J.R. 1998, "Research, development, and engineering metrics", Management Science, vol. 44, no. 12, pp. 1670‐1689.
Hollanders, H. & van Cruysen, A. 2008, Rethinking the European Innovation Scoreboard: A New Methodology for 2008‐2010, INNO METRICS.
Huston, L. & Sakkab, N. 2006, "Connect and develop: Inside Procter & Gamble's new model for innovation", HARVARD BUSINESS REVIEW, vol. 84, no. 3, pp. 58.
Katila, R. 2008, "Measuring Innovation Performance" in Business Performance Measurement: Unifying Theory and Integrating Practice Cambridge University Press, , pp. 304.
Kerssens‐van Drongelen, I.C. & Bilderbeek, J. 1999, "R & D performance measurement: more than choosing a set of metrics", R & D Management, vol. 29, no. 1, pp. 35‐46.
KerssensvanDrongelen, I.C. & Cook, A. 1997, "Design principles for the development of measurement systems for research and development processes", R & D Management, vol. 27, no. 4, pp. 345‐357.
Krause, D.R. & Ellram, L.M. 1997, "Critical elements of supplier development The buying‐firm perspective", European Journal of Purchasing & Supply Management, vol. 3, no. 1, pp. 21‐31.
Kumpe, T. & Bolwijn, P.T. 1994, "Toward the Innovative Firm ‐ Challenge for Research‐And‐Development Management", Research‐Technology Management, vol. 37, no. 1, pp. 38‐44.
Laursen, K. & Salter, A. 2006, "Open for innovation: The role of openness in explaining innovation performance among UK manufacturing firms", Strategic Management Journal; 64th Annual Meeting of the Academy‐of‐Managemen.JOHN WILEY & SONS LTD,
David Scarlatti - X792282
42
CHICHESTER; THE ATRIUM, SOUTHERN GATE, CHICHESTER PO19 8SQ, W SUSSEX, ENGLAND, pp. 131.
Lichtenthaler, U. 2008, "Open Innovation in Practice: An Analysis of Strategic Approaches to Technology Transactions", IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, vol. 55, no. 1, pp. 148.
Linder, J.C., Jarvenpaa, S.L. & Davenport, T.H. 2003, Toward an Innovation Sourcing Strategy.
Mankins, J.C. 1995, Technology Readiness Levels, Advanced Concepts Office, Office of Space Access and Technology, NASA.
Mark Rogers 1998, The Definition and Measurement of Innovation, Melbourne Institute.
McGrath, M.E. & Romeri, M.N. 1994, "The R&D effectiveness index", The Journal of Product Innovation Management, vol. 11, no. 3, pp. 213.
OECD 2005, OSLO MANUAL: GUIDELINES FOR COLLECTING AND INTERPRETING INNOVATION DATA, OECD and Eurostat, Europe.
Ojanen, V. & Vuola, O. 2003, Categorizing the Measures and Evaluation Methods of R&D Performance, Telecom Business Research Center Lappeenranta, Lappeenranta.
Pappas, R.A. & Remer, D.S. 1985, "Measuring R‐And‐D Productivity", Research Management, vol. 28, no. 3, pp. 15‐22.
Pawar, K.S. & Driva, H. 1999, "Performance measurement for product design and development in a manufacturing environment", International Journal of Production Economics, vol. 60‐1, pp. 61‐68.
Pearson, A.W., Nixon, W.A. & Kerssens‐van Drongelen, I.C. 2000, "R&D as a business ‐ what are the implications for performance measurement?", R & D Management, vol. 30, no. 4, pp. 355‐366.
Sarkar, S. & Costa, A.I.A. 2008, "Dynamics of open innovation in the food industry", TRENDS IN FOOD SCIENCE \& TECHNOLOGY, vol. 19, no. 11, Sp. Iss. SI, pp. 574‐580.
Schumann, P.A., Ransley, D.L. & Prestwood, D.C.L. 1995, "MEASURING ‐ R‐AND‐D PERFORMANCE", Research‐Technology Management, vol. 38, no. 3, pp. 45‐54.
Sousa, M. 2008, Open innovation models and the role of knowledge brokers.
Werner, B.M. & Souder, W.E. 1997a, "Measuring R&D performance ‐ State of the art", Research‐Technology Management, vol. 40, no. 2, pp. 34‐42.
Werner, B.M. & Souder, W.E. 1997b, "Measuring R&D performance ‐ US and German practices", Research‐Technology Management, vol. 40, no. 3, pp. 28‐32.