Internal and external evaluation of scientific research

Presentation at Hanken, Helsinki 22.4.2016

PhD Antti HautamäkiResearch professor (emeritus), University of Jyväskylä

Adjunct professor, University of HelsinkiConsulting Sustainable Innovation

Basic research and its impact• Science is systematic research which produces

new knowledge (results)• To be effective science must follow its own

interests and methods• The way from research results to practical

applications is long and indirect• A major problem is how to stimulate impact

effectively and maintain a consistent high quality of the research

Tensions in science debate• Truth vs. practice• Pure science vs. usefulness• Basic science vs. applied science• Quality vs. relevance• Autonomy vs. external steering• Collegiality vs. managerialism• Budgetary funding vs. competed funding• Science logic vs. market logic

Scientific method (ideal, Ch. Peirce)

• The scientific method• Is fallible• Purposely tests itself• Criticizes itself• Corrects itself• Improves itself

• Other “methods” are based on tenacity, authority, or a priori assumptions

Basic elements of scientific method

• PROBLEMS: identification and classification of problems

• DATA: systematic observation, measurement, and experiment

• HYPOTHESES: formulation, testing, and modification of hypotheses

• EVALUATION: iteration, replication, peer review, open data

Qualitative research• Differs from standard hypothetic-deductive

method, where hypotheses are tested by their deductive consequences and predictions

• Target is more to understand phenomena (meanings) than to explain them

• A variety of methods: action theory, grounded theory, ethnography, hermeneutics, narrative analysis, observation, phenomenology etc.

• Challenges: generalization, objectivity, sampling

Progress of science (L. Laudan: Progress and Its Problems, 1977)

• Progressiveness of a theory or an approach is related to its problem solving effectiveness

• Empirical problems are often wh-questions related to the world around us: what, why, when, how,..

• Conceptual problems are a) internal related to theoretical difficulties (concepts, methods) or b) external rising from conflicts between different theories

• Anomalous problems for a theory are empirical problems which a competitor has solved

Measure of progress• The aim of science is to maximize the scope of

solved empirical problems, while minimizing the scope of anomalous and conceptual problems

• Theories are produced in research traditions (schools), which are sets of general assumptions about the entities and processes in a domain of study

• The central cognitive test for any theory is whether it provides satisfactory solutions to important problems

Instrumentalism in science• According to instrumentalism science is first of

all providing workable solutions to practical problems (vs. realism = uncovering the reality)

• Solutions might be formulated as “technical norms”:• If you perform action A, then you will get

result B• Technical norms are “value-neutral”• But values are involved in selecting which

problems are worth to solve!

Humboldtian university• Humboldt University in Berlin (est. 1810) was the

model of modern large-scale universities • W. von Humboldt's model was based on two ideas

of the Enlightenment: the individual and the world citizen

• Principles:• Academic freedom and autonomy of universities• The pursuit of knowledge as a base for culture,

civilization and education (German “Bildung”)• The unity of teaching and research

Neo-humboldtian university (A.Hautamäki & P. Ståhle, Ristiriitainen tiedepolitiikkamme, 2012)

• Bias of Humboldtian university: the pursuit of truth without paying attention to societal and economic issues and challenges

• In Neo-humboldtian university wicked problems are taken to be an organizing principle for research, education and problem solving

• Wicked problems are the grand challenges of contemporary world such as climate change, health and food issues, polarization, and security

• They are complex and difficult to solve

Impact analysis

Human and financial

resources

ActivitiesProcesses

Goods andservicesproduced

Initialimpact

Longer-termimpact

Impact of science• Scientific impact

• Cumulation and growth of knowledge• Citation impact quantifies the citation usage

of scholarly works; Impact factor measures the citation impact of an academic journal

• Societal impact• Our society is increasingly focused on

requiring justification for the investments made in scientific research: need for societal impact of science

Scientific impact• Academic publications in journals with high

impact factors• Quality as measured by citation• Peer review is central in journals and funding

decisions• Bias: support “normal science”, not open to

radical openings, innovative approaches or interdisciplinary research

• Societal impact is neglected

Societal impact• Some dimensions of impact:

• Development of skills and knowledge for employment

• Business–university collaboration in research and innovation

• Graduate recruitment: the interface between students, universities and employers

• Universities in their local communities: enabling economic growth

• How to measure impact, what are good indicators?• Case Finland (next two slides)

Universities core funding 2013-2016 in Finland (and changes for 2017-2020)

+ 2%

-2%

-1%

1 % = 16 m€

+1%

Strategic Research Council Academy of Finland

• New funding for strategic research (55 m€ /y)• “The SRC funds high-quality research that has great

societal impact. The research should seek to find concrete solutions to grand challenges that require multidisciplinary approaches. An important element of such research is active collaboration between those who produce new knowledge and those who use it.”

• “The Government determines the research needs and decides the final themes, which the SRC then formulates into research programmes and funding calls.”

• SRC programmes run for 3–6 years

Readings• Berman R.P. (2012). Creating the Market University, How Academic Science

Became an Economic Engine. Princeton & Oxford: Yale University Press.• Geiger R.L. (2004) Knowledge and Money, Research Universities and the

Paradox of Marketplace. Stanford: Stanford University Press.• Hautamäki A. (2010). Sustainable innovation. A New Age of Innovation and

Finland’s Innovation Policy. Sitra Reports 87. Helsinki.• Hautamäki Antti ja Ståhle Pirjo: Ristiriitainen tiedepolitiikkamme, Suuntana

innovaatiot vai sivistys? Helsinki: Gaudeamus 2012 • Hautamäki Antti, Ståhle Pirjo, Oksanen Kaisa ja Tukiainen Taina (2016).

Vaikuttavaa tutkimusta – Kokeiluehdotuksia tutkimuksen vaikuttavuuden ja kaupallistamisen edistämiseksi. Työ- ja elinkeinoministeriön julkaisuja 2/2016.

• Oksanen Kaisa ja Hautamäki Antti (2014): “Uushumboldtilainen yliopisto – suomalainen ratkaisu yliopistojen kehittämiseen.” Tiedepolitiikka 1/2014, 46-52.