jobknowledge.eu

Labour Market Driven Learning Analytics

Dr. Gábor KismihókSenior Researcher

University of AmsterdamAmsterdam Business School

g.kismihok@uva.nl@kismihok

www.eduworks-network.euwww.jobknowledge.eu

jobknowledge.eu

Background

Multidisciplinary (HRM-OB/Data Science/Knowledge Management/ Education)Focus:

• Data science and HRM/Lifelong Learning• Learning Analytics• Adaptive, personalised assessment• Self regulated learning

• Job Analysis/Job Knowledge• Employability of individuals• HR practicies in the 21st century

jobknowledge.eu

Learning Analytics

Learning analytics is “the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (SOLAR 2012).

4 levels of LA• Describe• Diagnose• Predict• Recommend

Expertise: Educational scientist, computer scientist, data scientist, managers, teachers, students, labour market representatives

jobknowledge.eu

Hot topics

Person

• Learner profiles• Personalisation• Individual

feedback/benchmarking• Teaching analytics

Organisation

• Student retention• Curriculum design• Workplace and

professional learning• Institutional readiness

Technology

• Tools and interventions• Visualisation• MOOCs• Predictive modelling• Learning environments

Pedagogy

• Learning design• Personalisation• Feedback• Blended learning

Ethics

• Data ownership• Data management• Transparency of algorithms• Quality of

recommendations

jobknowledge.eu

What type of data?

• Performance• Grades• Assignments (text)

• Behavioral• Clicks• Content views• Social media

• Physiological• Pulse• Brain activity

• Labour Market data• Vacancy data• Economic indicators/surveys

jobknowledge.eu

Examples

Purdue University (US) – Course Signals• Predicting student drop outs based on LMS activity• Teachers and students are notified and personal intervention is planned• 21% retention rate improvement (Kimberley and Pistili 2012)

OU Analyse• Predicts students at risk• Predictive models on the basis of VLE and demographic data• Also explains the reason, recommends activities• Open dataset

Social Networks Adapting Pedagogical Practice (SNAPP) • real-time social network analysis and data visualisation of forum discussion activity• identification of isolated students, non-functioning groups or groups need to work

together

jobknowledge.eu

Examples

Predictive Analytics Reporting (PAR) Framework (US)• Non-profit provider of analytics-as-a-service• Central analytics service for HE institutions• Cross institutional analyses

Kahn Academy Analytics (US/Global)• Learning content is mapped to skills • Learning content is offered on the basis of effort, engagement, difficulty, etc…

jobknowledge.euLabour market oriented learning trajectories

Match learners’ pathways to those of alumni

Help incoming students find a long term focus during their university

education

Mirror alumni datato current students

based on desired/acquired

occupations

jobknowledge.eu

Goal setting pilot

• Goal setting improves performance• Create a goal setting interface for students to manage and track

(learning) goals• Focus of research:

• Goal commitment/Shared goals• Matching goals to behavioral and performance data• Apply advanced analytics

jobknowledge.eu

Goal App Features

• Set goals (Specific, Measurable, Attainable, Relevant, Time Based)• Set sub-goals • Option to make goals private or public• Feedback on goals• Can view and commit public goals• Tag goals• Learning records• Dashboard• Reminders

First findings:• Difficult to think about goals• Goals should be generated on the basis of labour market data (vacancies)

virgo.ic.uva.nl:3000

jobknowledge.eu

Learning Analytics is a great opportunity to shorten the gap between education and the

labour market

jobknowledge.eu

Challenges

xkcd.com

jobknowledge.eu

Challenge 1Changing nature of the society

and the Labour Market

jobknowledge.eu

Jobs (society) are changing

What are the relevant job information types, what is a job in the 21 century?

• increased idiosyncratic nature of work and the crafting of jobs by job holders• 161 job information types in 50 studies - Volume and semantics

Shared economyNew selection and recruitment methods

Many vacant jobs simply do not show up on the webRole of vacancy announcements in the future

How to target the output of Education?

jobknowledge.eu

Challenge 2Landing analytics at

organisations

xkcd.com

jobknowledge.eu

Stakeholder analysis

(Szörényi, 2014)

LA stakeholder map at the UvA

jobknowledge.euPositioning a learning analytics project• Centralized vs. decentralized• Research vs. practice• Imposed vs. desired• Technology vs. Pedagogy• Make or buy (resource oriented)• Adoption of best practices or local

identification thereof

There is a lot of ambiguity and fear, but little experience with LA

jobknowledge.euSandbox – safe environment to play

https://www.flickr.com/photos/mzn37/

jobknowledge.eu

Challenge 3Terminology

jobknowledge.euDefinitions of constructs (like skills) are very context sensitiveData generation is not unified, terminology is weak (in the hands of HR managers and their objectives)

Definition is influenced by• The data producer (person or machine)• Country, region• Organisation• Occupation • Language

Universal taxonomy, ontology

jobknowledge.eu

Challenge 4Data and algorithms

jobknowledge.eu

Call for Transparency

• Trust is a big issue• Data/algorithms are often not public - Black box society

Web-data:• Traceability is an issue• Scraping policies of data providers are not always visible• Paid websites are rarely scraped

jobknowledge.eu

Educational data

• Many data silos• Who owns what data?• Highly political issue

• Organizational resistance • Gatekeepers resistance

• Complex infrastructure

jobknowledge.eu

Challenge 5Ethics

jobknowledge.eu

Ethics and Privacy

• One of the greatest barrier• JISC reports 86 ethical, legal and logistical issues

https://analytics.jiscinvolve.org/wp/2015/03/03/a-taxonomy-of-ethical-legal-and-logistical-issues-of-learning-analytics-v1-0/

• Algorithms/codes are often hidden – Black Box society

• Personal/sensitive data, lasting effects of recommendations• How this data will be used on learners?

• What is the objective of the data collection? (not known at the time the data is collected)

jobknowledge.eu

What is more important? Personal development or privacy?

jobknowledge.eu

Educational data

Is it ethical to use these data to improve the learning process?

jobknowledge.eu

Educational data

Is it ethical NOT to use these data to improve the learning process?

jobknowledge.eu

Summary Power is in numbers

Bigger data, better matching, better insights, better research

LA is a new areaMany opportunities for innovative ideas and services More evidence (research) needed what works and what doesn’t

Inductive vs DeductiveNeed to document failures (not only success) properly

Technology is not a bottleneckOrganizational awareness is growing - LA is on the agenda in many stakeholder groupsLegal and ethical concerns are critical

jobknowledge.eu

Further information

• LACE Evidence HUB http://evidence.laceproject.eu/

• LEAP Inventory http://cloudworks.ac.uk/cloudscape/view/2959

• SOLAR Community https://solaresearch.org/

• Learning Analytics and Knowledge Conference (LAK)

• Learning Analytics Summer Institute (LASI)

• Journal of Learning Analytics http://learning-analytics.info/

jobknowledge.eu

Participate!

Facilitate discussions among key stakeholders

jobknowledge.eu

Invitations

• Eduworks communityNewsletter: http://www.eduworks-network.eu/Fb: https://www.facebook.com/eduworksnetwork

• Eduworks events, stakeholder meetingshttp://www.eduworks-network.eu/upcoming-events

• Eduworks Dutch Stakeholder meeting (October 2016)

• Contribute to our Sandbox!

jobknowledge.eu