pg. 1 Paul H. Cleverley (Robert Gordon University) 2016

pg. 2 Paul H. Cleverley (Robert Gordon University) 2016

Deep Learning in Organizations

Paul H. Cleverley, Robert Gordon University (January 2016)

Background

The term ‘Deep learning’ has been increasingly used recently to describe how organizations augment

their learning by applying machine learning techniques on large volumes of information (big data) to

discover patterns. Building on the 2015 article ‘emergence of the corporate brain’, this article applies

a systems thinking approach to balance the technology focus on deep learning, with one that focuses

on people, how the organization thinks deeply. The focus is on Enterprise search & discovery capability

although some concepts may be transferable to other areas.

The fields of information searching (people centric) and information retrieval (technology centric),

although porous, have begun to converge. However, there is a dearth in the research and practitioner

literature for how these two fields relate to organizational learning as applied to enterprise search &

discovery capability (Figure 1).

Figure 1 – Information searching, information retrieval and organizational learning Metacognition Socrates is cited as saying “I know that I am intelligent because I know that I know nothing”, “I cannot teach anyone anything, I can only make them think.” He recognized the importance of ‘learning how to learn’, our higher order executive processes of planning, checking and reflection – thinking about thinking. Simplifying, metacognition is both knowledge of cognition and regulation of cognition, it is the literacy of self-awareness and has been linked to increased intelligence. Metacognitive capability includes the extent to which we can ‘listen inward’ and monitor what we think (our inner voice) for the tell-tale signs of cognitive biases and closed mindedness, i.e. are the skills being applied likely to lead to a desirable outcome. Metacognition empowers us to learn, can support transformational change and the good news is that we can improve our metacognitive capabilities. The importance of metacognition is illustrated by this quote from the Carnegie Report on Educating Lawyers, “The essential goal of professional schools must be to form practitioners who are aware of what it takes to become competent in their chosen domain and to equip them with the reflective capacity and motivation to pursue genuine expertise. They must become ‘metacognitive’ about their

pg. 3 Paul H. Cleverley (Robert Gordon University) 2016

own learning.” This includes understanding the strengths and weaknesses in your own learning styles and that of your colleagues. Different levels of learning

Definitions for Learning and Organizational Learning lack consensus. They could be considered to be

a qualitative increase in information and understanding; the detection and correction of error. Trends

in teaching have seen a shift in emphasis over the past few decades from what the teacher does to

what the learner does. Three levels of learning are shown (Table 1) within the organization (modified

from Lublin 2003, Entwistle 2000, Senge 1990, Marton and Saljo 1984, Argyris and Schon 1978).

Table 1 – Levels of learning in the organization

Deep learning (similarities with double loop learning, lateral thinking & organizational metacognition)

Surface learning (similarities with single loop learning, continuous business improvement methodologies)

Strategic learning (similarities with single loop learning)

Actively seek to understand and interact with information Use evidence, inquiry through dialogue and reflection Holistic systems thinking view Notice key ideas & concepts Relate new ideas to previous knowledge and relate concepts to experiences Question norms & conclusions Motivated by interest

Passively accept information Failure to reflect, discussion dominates Reductionist narrow silo view Learn to repeat what is learnt Memorise by rote Unable to distinguish principles from examples Anxiety and Cynicism Maintain norms, daily goals Motivated by fear of failure

Knowing requirements for success i.e. personal, team, project & org. targets & KPI’s Prioritise time under overload to greatest extent. Commercially aware Ensure they have the right training and materials Strong network builders Alert to cues of management preferences Motivated by achievement

Some combination of deep learning and strategic learning has been suggested to provide both an

intelligent engagement in a subject and organizational success. Business improvement methodologies

such as Six Sigma are generally perceived by many researchers (not all) as focusing on incremental

single loop learning. Here the focus is ensuring the process functions in the most effective way (often

cost reduction based), but will not necessarily produce an alternative framework or deep learning.

Critical thinking, governance and decision making require good metacognition for good judgement.

These concepts can be applied to the enterprise where organizational learning has been shown to

improve business performance.

Metacognition in organizations

Organizational metacognition (which may be synonymous in part with deutero-learning in

organizations) is how an organization ‘learns to learn’, how it knows what it knows. It is how

organizations learn how to conduct organizational learning (such as single & double loop learning).

Metacognition in organizations enables critical thinking. It describes how organizations and teams

develop an awareness of their own thinking, where awareness of ignorance can motivate learning.

Learning what facilitates and inhibits learning enables organizations to develop new strategies to

develop their knowledge. For example, identification of a gap (figure 2) between perceived

performance (such as satisfaction) and actual performance (outcomes) creates an awareness that

pg. 4 Paul H. Cleverley (Robert Gordon University) 2016

makes the organization understand that learning needs to occur (need to bridge the gap) driving

appropriate changes to the environment and processes. Organizational metacognition is considered

a key norm to the prescriptive concept of the learning organization, its significance has been

recognized by the military, industry and disaster response amongst other areas.

Figure 2 – Awareness of a gap may avoid oblivious satisfaction & the illusion of knowledge

An organization can be obliviously satisfied due to a state of ignorance; where an illusion of knowledge

has been constructed by the organization.

Information Search & Discovery

As we become more reliant on the Internet for information, some research has shown that Internet

search (e.g. Google) has actually changed how many of us think. Instead of remembering the

information itself, we just remember where to find it. Researchers have not proposed this is

necessarily a bad thing, it may be analogous to extended group working where we know who on our

team (or in the wider organization) has the ‘knowledge’ we need. However, it has been suggested that

a downside may be decreased innovation (surface not deep learning) where some searchers may

simply use the ‘top ranked’ information and not stray from the beaten path.

An Enterprise Search Centre of Excellence within an organization can use search logs to continuously

tune the search engine ranking for typically known item (lookup) searches, in a form of single loop

learning. Poor exploratory search has been known to have caused business mistakes, missed evidence

of fraud and caused fatalities. If searchers (e.g. information professionals, scientists and engineers)

are not aware of how well they are performing exploratory search tasks (and management are not

aware either) that is an issue. If they are not aware ‘that they are not aware’ that is an even bigger

issue. This is where organizational metacognition comes in.

This brings me to some of the case study research I have been undertaking over the past few years on

enterprise search & discovery capability (primarily in the oil and gas sector) to reveal to organizations

‘how well they are doing’ a type of formative assessment. In one experiment with experienced staff,

60% expressed satisfaction at their performance of exploratory search tasks, although they only found

27% of the high value items. There was no association between self-reported search expertise (search

literacy) and how well they performed the tasks. When these assessment results were fed back to the

staff and senior management, they were surprised at their poor performance which led to an update

in their personal and organizational mental models.

pg. 5 Paul H. Cleverley (Robert Gordon University) 2016

So why do we have a situation in which experienced staff (using an established internal search tool,

conducting a familiar task) deliver such poor outcomes? The absence of effective feedback loops

perhaps caused by poor metacognition in the organization may have created the conditions for this

situation to occur. You cannot make interventions if you don’t know you need to make interventions.

My research findings have led to the development of a new theory, Relative Satisfaction Theory (RST).

This theory proposes that whilst user satisfaction may not be a good indicator of search task

performance or search literacy, the difference between user satisfactions for tasks (where tasks

represent varying levels of information overload) can provide an indication of relative task

performance and search literacy.

The research findings also appear to show a gap between latent information needs of scientists and

engineers and the way many enterprise search user interfaces are designed and deployed in

organizations. In particular is the surfacing of intriguing associations - designing for serendipity.

Dialogue - Organizational metacognitive capability

Reflective practice in organizations is the willingness to question habitual ways of thinking and acting.

Effective metacognition enables this practice. Some researchers state that “unless the team can learn,

the organization cannot learn”. How teams learn as a collective (shared or distributed meta-cognition

and socially regulated learning) may be crucial to understand how organizations learn. For the

purposes of this article, the scope of organizational metacognition covers both metacognition as

practiced by individuals (within the culture of the organization) and team metacognition.

Organizational metacognition can therefore be viewed as that which is practiced by individuals within

organizations and organizational units of different size and numbers (e.g. teams, departments and

communities). This is illustrated in Figure 3, showing both its individual and collaborative nature where

dialogue “reveals the incoherency of our thought” mitigating confusion.

Figure 3 – Organizational Metacognition (inquiry and reflective practice) at different layers in the

organization towards building a shared mental model for action

Dialogue differs from discussion. The latter is focused on presenting and defending views in order to

make decisions, whereas the former aims to explore and challenge habitual traits, possibly surfacing

a new shared view. Instead of discussing strategies, solutions, perspectives and opinions, dialogue

pg. 6 Paul H. Cleverley (Robert Gordon University) 2016

focuses on the assumptions (which are often hidden) on which they are based, applying a logical

scientific and social approach of investigation. This is a key element to the Theory of Change. In this

way through this process of collaborative inquiry, the organizational unit develops an intelligence

which is greater than the sum of its parts.

One of the themes I have observed during my action research over the past few years in the oil and

gas industry is how many meetings on ‘search & discovery’ within companies appear dominated by a

narrow focus and discussion not dialogue. Effectively surface learning. Opinions are freely given but

assumptions (on which those opinions are based) are not always made explicit. Whilst there were

plenty of good examples of deep questioning and inquiry, there were also plenty of examples of

virtually no questioning at all. The social biases of avoiding tension and rewarding consensus thinking

appear alive and well in organizations large and small (at least in the oil and gas sector). Furthermore,

‘critical thinking’ may often be conflated with the undesirable behaviour of ‘being critical’. The latter

is often backward looking motivated by emotion, the former is a reasoned logical argumentation with

a clear purpose. This may shut down dialogue and some opportunities for deep learning may be lost.

Reasons for poor organizational metacognition

A variety of factors that may lead to conditions (an organizational culture) in which poor organizational

metacognition thrive may include:

1. Lack of awareness of the impact of cognitive biases (heuristics) and flaws in reasoning

• Action biases: e.g. Overoptimistic and overconfidence bias, a person’s subjective judgements

are reliably greater than the objective accuracy of those judgements. Fallacy of centrality

(leaders overestimate the likelihood they would know about a phenomenon if it was

occurring). Immediate gratification bias (focus on short term not long term).

• Self-serving biases: Naïve realism (the belief we see reality as it really is). Leaders and staff are

motivated to deliver a favourable outcome for their department or themselves, at the

expense of the enterprise as a whole. Power structures.

• Pattern recognition biases: e.g. Give more weight to the first piece of information offered

(anchoring), bias towards recent events, familiarity and confirmation bias (seek out and cherry

pick information to confirm what we already believe).

• Status quo biases: Fear of change, support of existing systems. For example, even a new

project “The new search engine deployment is better than the old one” can fall into a status

quo bias (no need to change anything about the new deployment because it’s better than the

last one). Plato’s cave allegory.

• Social biases: e.g. Office politics, In-group bias (favour information from within the group,

dismiss information from outside the group), homophily (birds of a feather flock together),

Bandwagon/Herd mentality – tendency to go with the majority views, avoiding tension

rewards for consensus thinking (Groupthink).

• Flaws in reasoning: e.g. Use of anecdotal evidence, false premises (e.g. search & discovery is

a technology problem to be solved), false dilemmas (e.g. information should be either

manually tagged or automatically tagged (but not both) the Tyranny of the OR), Post hoc

fallacy (i.e. cause and effect are close in space and time, “I cannot find information using the

search engine, it must be the search engine that is the problem”). Self-fulfilling prophecies:

e.g. an IT manager has an assumption that the social system in the organization is unreliable,

pg. 7 Paul H. Cleverley (Robert Gordon University) 2016

the technical system is reliable. So virtually all the resources available for managing content

are put into buying and deploying new technology. When the system fails to deliver on its

promises (after little investment in the social side, so those involved in manual content

curation are unmotivated), the manager feels their view has been vindicated and justifies

investing more into technology and automation to manage content.

2. Environment of work overload, under resourced and no time to think

3. Organizational design issues

• Conflicting tasks and targets (IT functions often driven by cost, business by value creation)

• Responsibilities that focus on the part (silo thinking) not the whole in mind (systems thinking).

4. Defensive behaviours (an unwillingness to be open and share assumptions), self-justification.

5. The opinions of senior staff always treated with unequal weight, assumptions not made explicit.

6. Lack of motivation, cynicism, going through the motions

What can organizations do to mitigate these effects?

When it comes to enterprise search & discovery capability, there is some evidence that deep learning

is not occurring within many organizations. A number of tactics may mitigate these barriers:

1. Simply being mindful of these barriers may be a first step to enable an organization to improve its

metacognitive capabilities. Raising awareness of learning styles and barriers through training or

storytelling may help although it is unlikely to solve all problems such as poor leadership.

2. Conducting Formative Assessments (i.e. search tests where feedback is provided to learners and

other parts of the ‘system’ e.g. the search engine) may improve capabilities. Metacognitive prompting

& scaffolding mechanisms (e.g. After Action Reviews) have also been shown to improve team learning.

3. Adopting an experimentation mind-set may aid the sensemaking process, designing for serendipity

and proactively probing the environment gathering information rather than waiting for the

information to come to you. For example A/B testing (two versions of the same user interface with a

minor change randomly split between users) has been used by e-commerce providers for many years

to surface ‘what works best’. Perhaps assuming undesirable things are occurring (or opportunities are

being missed) that you do not know about is a useful position to take; assume ignorance is inevitable.

Summary

Factors & conditions for poor enterprise search & discovery capability may include the level of

executive support, organizational culture and amount of resources invested. It may include the

volume and type of content indexed, ranking of search results, analysis of search logs by a centre of

excellence and what user interface functionality and reasoning algorithms have been deployed. It may

include deployment technology infrastructure. It may include content management and access control

and even information searching behaviour and information literacy.

Over time however, enterprise search & discovery capability may have more to do with the capacity of

the organization for deep learning.

pg. 8 Paul H. Cleverley (Robert Gordon University) 2016

More information

A PDF of this article with references is in Slideshare http://www.slideshare.net/phcleverley/ More

information at Blog site www.paulhcleverley.com and email: p.h.cleverley@rgu.ac.uk.

Further Reading

Argote, L., Miron-Spektor, E. (2009). Organizational Learning from Experience to Knowledge. Conference for

New Perspectives in Organizational Science.

https://server1.tepper.cmu.edu/Seminars/docs/ARGOTE_SPEKTOR_OS_%20in-press.pdf

Argyris, Chris; Schon, Donald A. (1978-05-18). Organizational Learning: A Theory of Action Perspective: 001.

Reading, Mass: Addison Wesley Longman Publishing Co. ISBN 9780201001747.

Briñol, Pablo and DeMarree, Kenneth (2012-04-27). Social Metacognition. Taylor & Francis.

ISBN 9781135234096.

Blummer, B., Kenton, J.M. (2014). Improving Student Information Search: A Metacognitive Approach. Chandos

Publishing, UK.

Choo, C.W. (2016). The Inquiring Organization. How organizations acquire knowledge and seek information.

Oxford University Press, UK.

Cleverley, P.H., Burnett, S., Muir, L. (2015). Exploratory information searching in the enterprise: A study of user

satisfaction and task performance. Journal of the Association for Information Science and Technology (JASIST).

http://onlinelibrary.wiley.com/doi/10.1002/asi.23595/abstract

Cleverley, P.H., Burnett, s. (2015). Creating sparks: comparing search results using discriminatory search term

word co-occurrence to facilitate serendipity in the enterprise. Journal of Information and Knowledge

Management, 14. http://www.worldscientific.com/doi/abs/10.1142/S0219649215500070

Duncan, R., and A. Weiss. "Organizational Learning: A Theory of Action Perspective." MA: Addison-Weslay.

Morgan, G. and Ramirez (1979): 1-28.

Edwards, M. (2011). Google changing how humans think. http://www.canadianbusiness.com/lifestyle/google-

changing-how-humans-think/

Entwistle N. J. (2000). Approaches to studying and levels of understanding: the influences of teaching and

assessment. In J. C. Smart (Ed.), Higher Education: Handbook of Theory and Research (Vol. XV) (pp. 156-218).

New York: Agathon Press.

Flavell, J.H. (1979). Metacognition and cognitive monitoring: A new area of cognitive developmental inquiry.

American Psychologist, 34(10), 906-911. http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=1980-

09388-001

Halpern, D.F. (2014). Thought and Knowledge: An introduction to critical thinking. Fifth Edition. Taylor Francis,

USA. http://tandfbis.s3.amazonaws.com/rt-media/pdf/9781848726291/chpt_1.pdf

Harrison, P. (2004). Unleashing Deep Learning Through Questioning. Education in a changing environment, 13-

14th September 2004.

Hilden, S., Tikkamaki, K. (2013). Reflective Practice as a Fuel for Organizational Learning. Adm Sci, 3(3), 76-95.

http://www.mdpi.com/2076-3387/3/3/76/htm

Janis, I.L. (1972). Victims of groupthink: A psychological study of foreign-policy decisions and fiascoes. American

Psychological Association. http://psycnet.apa.org/index.cfm?fa=search.displayRecord&uid=1975-29417-000

Jansen, B.J., Rieh, S.Y. (2010). The Seventeen Theoretical Constructs of Information Searching and Information

Retrieval. Journal of the American Society for Information Science and Technology. 61(8), 1517-1534

pg. 9 Paul H. Cleverley (Robert Gordon University) 2016

Johnston, C.A. (1998). Using the Learning Combination Inventory.

http://www.ascd.org/publications/educational-leadership/dec97/vol55/num04/Using-the-Learning-

Combination-Inventory.aspx

Kahneman, D. (2011). Thinking fast and slow. Farrar, Straus and Giroux, New York, USA.

"Kellogg School of Management | Northwestern University". www.kellogg.northwestern.edu. Retrieved 2015-

12-11.

King, D., Lawley, S. (2013). Organizational Behaviour. Oxford University Press, UK.

Knowles, G. (2011). Six Sigma. Bookboon.com

Livingstone, J. A. (1997). Metacognition: An overview. http://gse.buffalo.edu/fas/shuell/cep564/metacog.htm

Ljungberg, J. (2009). The Proceedings for the 3rd European Conference on Information Management and

Evaluation. University of Gothenburg, Sweden, 17-18th September 2009. Academic Publishing Limited, Reading,

UK.

Looney, J.P.; Nissen, Mark E. (2007-01-01). "Organizational Metacognition: The Importance of Knowing the

Knowledge Network". 40th Annual Hawaii International Conference on System Sciences, 2007. HICSS 2007:

190c–190c. doi:10.1109/HICSS.2007.419.

Lublin, J. (2003). Deep, surface and strategic approaches to learning

http://www2.warwick.ac.uk/services/ldc/development/pga/introtandl/resources/2a_deep_surfacestrategic_a

pproaches_to_learning.pdf

Marton, F., & Säljö, R. (1984). Approaches to Learning. In F. Marton, D. Hounsell, & N. Entwistle (Eds.), The

Experience of Learning (pp. 36-55). Edinburgh: Academic Press.

McCarthy, Alma; Garavan, Thomas N. (2008-06-19). "Team Learning and Metacognition: A Neglected Area of

HRD Research and Practice". Advances in Developing Human Resources. doi:10.1177/1523422308320496.

ISSN 1523-4223

McGraw, B. (2011).Does you organization have cognitive biases that influence management decisions?

http://fearnoproject.com/2011/06/24/does-your-organization-have-cognitive-biases-that-influence-

management-decisions/

Meyer, John W. "Conclusion: Institutionalization and the rationality of formal organizational structure."

Organizational environments: Ritual and rationality (1992): 261-282.

"Understanding Organizations as Learning Systems". MIT Sloan Management Review. Retrieved 2015-12-14.

Newell, J., Dahm, K., Harvey, R., Newell, H. (2004). Developing Metacognitive Engineering Teams.

http://users.rowan.edu/~newell/Publications/28%20Metacognitive.pdf

"Organizational Learning - Knowledge Center". www.12manage.com. Retrieved 2015-12-14.

Preskill, Hallie (1998-10-20). PRESKILL: EVALUATIVE INQUIRY FOR. Thousand Oaks, Calif: Sage Publications,

Incorporated. ISBN 9780761904540.

Riding, R. J. (2000-01-01). Cognitive Styles. Greenwood Publishing Group. ISBN 9781567504590.

Salem-Schatz, S., Ordin, D., Mittman, B. A guide to the after action review. VA Center for Implementation Practice and Research Support http://as.vanderbilt.edu/overview/faculty/facultycouncil/archive/sitemason.vanderbilt.edu/files/cHpJCw/Guid

e%20to%20the%20After%20Action%20Review.pdf

Savolainen, T., Haikonen, A. (2007). Dynamics of organizational learning and continuous improvement is six

sigma implementation. The TQM Magazine, 19(1), 6-17

pg. 10 Paul H. Cleverley (Robert Gordon University) 2016

Schon, Donald A. (1975-01-01). "Deutero-learning in organizations: Learning for". Organizational Dynamics 4 (1):

2–16. doi:10.1016/0090-2616(75)90001-7.

Senge, Peter M. (2006-04-06). The Fifth Discipline: The art and practice of the learning organization: Second

edition (2nd Revised ed.). London: Random House Business. ISBN 9781905211203.

Simonin, Bernard L.; Helleloid, Duane (1993-08-01). "Do Organizations Learn? an Empirical Test of Organizational

Learning in International Strategic Alliances.". Academy of Management Proceedings 1993 (1): 222–226.

doi:10.5465/AMBPP.1993.10316992. ISSN 0065-0668.

Siroker, D., Koomen, P. (2013). A/B Testing. The most powerful way to turn clicks into customers. Wiley, USA.

Sole, D., Wilson, D.G. (2011). Storytelling in organizations: The power and traps of using stories to share

knowledge in organizations.

http://www.providersedge.com/docs/km_articles/storytelling_in_organizations.pdf

Spitzer, D.R. (2007). Transforming Performance Measurement. Rethinking the way we measure and drive

organizational success. AMACOM, USA.

Sullivan, W. et al (2007). Carnegie Report. http://lawteaching.org/conferences/2011/handouts/1e-

TeachingforLifeLongLearning.pdf

Tampoe, Mahen (1994-08-01). "Exploiting the core competences of your organization". Long Range Planning 27

(4): 66–77. doi:10.1016/0024-6301(94)90057-4.

Theory of Change. http://www.theoryofchange.org/what-is-theory-of-change/

Thompson, L., Cohen, T.R. (2012). Metacognition in Teams and Organizations. In Social Metacognition, Chapter

15 (Ed. Brinol and Demarree). Taylor Francis, USA.

Travis J. Wiltshire, Kelly Rosch, Logan Fiorella and Stephen M. Fiore (2014). Training for collaborative problem

solving: Improving team process and performance through metacognitive prompting. Proceedings of the Human

Factors and Ergonomics Society Annual Meeting, 58(1154)

https://www.academia.edu/8923220/Training_for_Collaborative_Problem_Solving_Improving_Team_Process

_and_Performance_through_Metacognitive_Prompting

Visser, Max (2007-04-01). "Deutero-Learning in Organizations: A Review and a Reformulation". Academy of

Management Review 32 (2): 659–667. doi:10.5465/AMR.2007.24351883. ISSN 0363-7425.

Weick, Karl E. (1995-07-20). Sensemaking in Organizations. Thousand Oaks, Calif.: SAGE Publications, Inc.

ISBN 9780803971776.

Wijnhoven, Fons (2001-06-01). "Acquiring Organizational Learning Norms A Contingency Approach for

Understanding Deutero Learning". Management Learning 32 (2): 181–200. doi:10.1177/1350507601322002.

ISSN 1350-5076.

"Business professor receives funds to study disaster response". news.emory.edu. 2014-12-30. Retrieved 2015-

12-15