PowerPoint Presentation

Implementing analyticsLearning Analytics and Business Intelligence7/7/2016

1

07/07/20161Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Session outlineIntroductionsOverview of the Learning Analytics Service Next steps / get involvedThe user voice Prifysgol Aberystwyth UniversityOverview of the Business Intelligence projectNext steps / get involved7/7/2016

2

Myles 10.30 10.35Overview of LA service Shri (5 mins)User voice David Matthews (10 mins)Overview of BI-Myles (5 mins)The user voice-James (10 mins)Group exercise-Shri (20 mins)Whats coming next- Shri and Myles (5 mins)

07/07/20162Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Learning Analytics

Shri 10.35 10.4007/07/20163Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

What do we mean by Learning Analytics?The application of big data techniques such as machine based learning and data mining to help learners and institutions meet their goals:

For our project:Improve retention (current project)Improve attainment (current project)Improve employability (future project)Personalised learning (future project)

What do we mean by learning analytics. The service we are developing will collect data and undertake statistical analysis of historical and current data derived from the learning process to create models that allow for predictions that can be used to improve learning outcomes.Models are developed by mining large amounts of data to find hidden patterns that correlate to specific outcomesE.g. Mine VLE event data to find usage patterns that correlate to course gradesThe service will provide predictive models initially for retention (identify students at risk of failing) and attainment (identifying students at risk of not achieving a specified level of attainment). In the future we will look to offer predictive models to support employability and personal/adaptive learning.

07/07/20164Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Effective Learning Analytics ChallengeRationaleUniversities and colleges wanted help to get started and have access to a standard set of tools and technologies to monitor and intervene. Priorities identifiedCode of Practice on legal and ethical issuesDevelop a basic learning analytics service including an app for studentsProvide a network to share knowledge and experienceTimescale2015-16 - Test and develop the tools and metrics2016-17 - Transition to service (Freemium)Sept 2017 Launch. Measure impact on retention and achievement

ShriThe effective learning analytics challenge was initiated from consultation with stakeholders, senior manager and practitioners who felt the sector need support to get up to speed with learning analytics. They prioritised three main areas, a Code of Practice to address legal and ethical issues of using learning analytics; a set of basic learning analytics tools to allow institutions to get started and make informed decisions; and a network to allow institutions to share practice and learn from each other.The current project has procured suppliers to provide a learning analytics service which are currently being tested by several institutions. This will be developed into a full service next year and provided as a new Jisc service from Sept 2017.07/07/20165Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Toolkit and CommunityBlog: http://analytics.jiscinvolve.orgReportsCode of Practice for Learning AnalyticsThe current state of play in UK higher and further educationLearning Analytics in Higher Education: A review of UK and international practiceMailing: analytics@jiscmail.ac.uk Network Meetings

The project consists of the learning analytics architecture (next slide), a toolkit and community.These consist of a blog with reports and information to assist institutions with readiness to implement learning analytics and technical implementation of the Jisc service.There are three reports all linked from the blog a Code of Practice for Learning Analytics, A report from 18 months ago that reviewed current state of learning analytics in the UK and a more recent report on the evidence base for the effectiveness of learning analytics with 12 international case studies.If you want to be involved and keep informed about the development of the service then join the analytics jiscmail listWe also hold quarterly network meetings which are promoted via the blog and jiscmail list07/07/20166Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Learning Analytics Architecture

Overview of learning analytics architecture. Red items are components that will include the tools in the project (Tribal student insight, Unicon/Apereo LAP and Student Success Plan, Student App) but also alternative third party or institutional tools.07/07/20167Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Current EngagementExpressions of interest: 85Engaged in activity: 35Discovery to Sept 16: agreed (28), completed (18), reported (17)Learning Analytics Pre-Implementation: (12)Learning Analytics Implementation: (7)

We have ~400 people on the Jiscmail list and a pipeline of interested institution's (50+ HE, 20+FE). We are actively engaging with 35 institutions, 28 in discovery institutional readiness and 12 in beta implementations. 07/07/20168Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Technical Progression to dateDeployment of our LA Data Processing Agreement (DPA)Latest student data specification (UDD v1.2.4) https://github.com/jiscdev/analytics-udd12-36 months of UDD (student) data + aligned Activity data (VLE, Attendance etc)Ongoing Technical Trials: Learner Records Warehouse(s), Bb & Moodle VLE plugin(s), UDD data validation/ APIsPredictive Model development service pilots start Q4 2016BlackBoard Learn VLE activity data plugin evaluation for historical data capture, with Moodle equivalentStudent App Beta v1.0 due for release July 2016 (iOS/ Android)Student App evaluations currently being formulated for 3 HEIs

Overview of learning analytics architecture. Red items are components that will include the tools in the project (Tribal student insight, Unicon/Apereo LAP and Student Success Plan, Student App) but also alternative third party or institutional tools.07/07/20169Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Press Coverage

From Sept 16 well be introducing a new institutional readiness process to help institutions get ready for implementing learning analytics. This will consist of an overview workshop to introduce the service and an diagnostic assessment tool, institutions will complete the assessment tool and then undertake appropriate actions to address recommendations. For institutions who are ready to start implementation there will be set of guidelines to get set-up with data collection and visualisations, ready to implement a predictive analytics solution and the student app.Details will be announced via the jiscmail list so join it to participate.07/07/201610Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Future Engagement/ get involvedFrom Sept 2016Readiness Toolkit with a diagnostic set of questions and support materials leading to implementation. Start-up guidelines to get ready for learning implementation.Further details will be announced via analytics @jiscmail.ac.uk

From Sept 16 well be introducing a new institutional readiness process to help institutions get ready for implementing learning analytics. This will consist of an overview workshop to introduce the service and an diagnostic assessment tool, institutions will complete the assessment tool and then undertake appropriate actions to address recommendations. For institutions who are ready to start implementation there will be set of guidelines to get set-up with data collection and visualisations, ready to implement a predictive analytics solution and the student app.Details will be announced via the jiscmail list so join it to participate.07/07/201611Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

The value of Learning AnalyticsLearning Analytics at Aberystwyth University

Monitoring AttendanceRecording attendance prior to 2014 involved registers and manual entry onto SAMS (in-house Student Attendance Monitoring System)SAMS simply recorded attendance and produced limited reports on an individual students attendanceThe Computer Science department developed and first ran MOPS (Monitoring of Performance System) 2013/14MOPS used data from SAMS to produced reports of students with poor attendanceMOPS then managed intervention workflows (typically meetings) and recorded outcomes

7/7/2016

13

Initial FocusInitial focus was on Monitoring attendance - seen as the key indicator of engagementEnabling early intervention Focus on student retentionSystems were developed in-house (SAMS, MOPS)An accurate personalized Timetable was an essential pre-requisite of this work7/7/2016

14

Automatic Attendance MonitoringBut - high manual overhead of recording attendanceInvestigated card readers. Problem = high unit cost (500+)2014 Computer Science dept developed prototype proximity card reader2015 CS worked with AU IS to turn this into production 2015-16 Deployed across core teaching roomsBy Sept 2016 will be deployed across all teaching rooms7/7/2016

15

Attendance Stats (up to end Semester 1 2016)7/7/2016

16

Attendance StatsMore data now collected on attendance & less workreporting on attendance at the student, module, scheme, and department level new insightsAlso being used forMonitoring international students to ensure compliance with Tier 4 visas Monitor exam attendance Time and attendance to record some employee working hours - replacing time-consuming timesheets.

7/7/2016

17

Components of the System7/7/2016

18

3 Pillars to Successful ImplementationA University-wide attendance policy with the expectation that students should attend all timetabled sessions (introduced 16/17). Universal automated attendance monitoring points to capture attendance data with minimal manual intervention. Systems to monitor and report attendance, alongside managing intervention workflows.7/7/2016

19

FutureAberystwyths involvement with the JISC project and future plans7/7/2016

20

Why the JISC Learning Analytics Project?Investigated commercial suppliers. Issues with specification and priceJISC solution designed by the sector and for the sectorMain motivators: Continuous quality improvementImprove retention (reduce drop out, improve completion)Improve the student experience / satisfactionImprove student outcomes (degree class)Personalization / Student engagement with their learning

7/7/2016

21

Work with JISC Project so farProviding data from Student record system in-house ASTRAVLE BlackboardAttendance monitoring from SAMSFeeding in to JISCs UDD data specificationPlan to pilot the JISC tools 16/17 & provide feedback: Test predictive modelUnicon Learning analytics processor and dashboardsStudent Success Plan (intervention management tool)Student Learning Analytics App

7/7/2016

22

Other Plans 2016/17The following are being deployed University-wide from September 2016: Attendance monitoring points in all teaching locationsConsistent attendance policyNew personal tutor system incorporating use of analytics in discussionsNew version of MOPS Tutor dashboard showing attendance & VLE useDisplay of attendance data in the Student Record system (web-based app) see next page7/7/2016

23

Attendance Data on Individuals Student Record

7/7/2016

24

Future Considerations

Post Brexit will we need more formal monitoring for more of our students?Data Protection legislation = EU law

HE Bill in England delay? What does this mean for the metrics we need?Relationships with Student Unions / Representatives will they continue to support or does political uncertainty = uncertainty in their support?

7/7/2016

25

HESA and Jisc Business Intelligence

Myles 10.50 10.55Jisc and HESA are collaborating to develop new national shared services for business intelligence, making better use of the national data landscape, reducing repetitive activities across universities, brining the benefits of BI to all Univerisits regardless of capability / expertise26

27About HESA

MylesHESA is a not for profit subscription organisation, so similar to Jisc in that sense. As well as a mandatory subscription, members are mandated to provide data collections covering the broad themes of Student, Staff, Destinations (of graduates) and Estates data. This is annual but in year collection is under consideration. HESA cleanse the data and provide back full data sets, published statistics and undertake bespoke analysis. Jisc and HESA membership is similar.27

MylesHeidi Plus is depicted on the left highlight the trucks driving in to the HESA data warehouse. HESA mandates that all publicly funded HEPs provide performance data on students, destinations of leavers, staff, finance and estates. Currently an annual collection they are moving to more frequent in year collections. The data is cleansed and a new team undertake dashboard development. Quality is assured as the dashboards are offered throught the radio mast in the middle a new national BI dashboard delivery service offered to all HESA customers (currently 180 HEPs and associated organisations and departments). Built with Jisc and launched as a HESA service in November. Includes legal framework and national training programme. Replaces a system with 6.5K users. Lowers the bar to usage through the interactive dashboards so could take BI to a woder range of staff than is currently possible.Heidi Lab is depicted to the right. A Jisc led alpha July 15 July 16. Highlight the trucks again and note its a two way street a data sharing agreement allows HESA data into the Heidi Lab secure data processing environment. Agile analysis teams are created from multiple universities and given access. They identify commonly felt problems spaces, explore the wider national data landscape, acquire non-HESA data and cleanse, link and transform it creating new proof of concept dashboards. Highlight the trucksa driving from the Lab to the Radio Mast. Successful dashboards will be branbded produced by Jisc and delivered via Heidi Plus.Piece in the middle is the beta service what comes next Heidi Plus is sustained by HESA as a service. We have proved there is real merit in Heidi Labs and will launch a beta service July 16 July 17.

28

James 10.55 11.0029

Heidi Plus The new business intelligence service for UK Higher EducationReplaces Heidi (which will be decommissioned in November 2016)Launched in November 2015 offering:Improved data content and functionalityDelivery of data sets through commercial data explorer toolNew visualisations and dashboards New training programme and support materialsAvailable to HE institutions with a full HESA subscriptionOver 80% of current Heidi subscribers have started the Heidi Plus application process (40% completed)

HESAs current data delivery service is known as HEIDI (Higher Education Information Database for Institutions) developed in house in 2007. Jisc and HESA collaborated to replace this with a more up to date service. We procured Tableau, market leading data exploration software and now offer Heidi Plus Feedback has been extremely good across the sector

30

Heidi Lab overview

Myles 11.00 11.05Heidi Lab as a Jisc Alpha project (proof of concept) engaged with 290 individuals from 130 universities to develop a successful model of agile analysis. 50 analysts (planners, directors of planning from 44 universities volunteered to join cross institutional agile analysis teams for three Heidi Lab cycles of 3 months each at just 0.2 FTE. Teams were supported as they identified and refined widely felt problem areas (see example on the slide covered student, staff, research, estates etc) linked to national policy. They explored the data landscape for supportive insights, recording the issues encountered in our data catalogue. Finally they produced interactive dashboards using Tableau software as proofs of concept to offer through Heidi Plus

31

Secure data processing environment

Technical infrastructure bound by legal agreements to ensure data and dashboards are secure

32

Information improvement manager UEL with;Kent, Middlesex, Brunel, Royal HollowayStrategic planning and BI manager Sunderland with;Glasgow, Glasgow Caledonian, St Andrews, SunderlandDirector of planning, Kent with;Birkbeck, Cardiff, Oxford, Southampton, SouthamptonStrategic Planning Manager, MMU with;Leicester, Leicester, Cambridge, Bishop GrossetesteWinter teams

Led by a senior staff member with knowledge of the information needs of a wide range of staff and institutions as well as national policy and what is up stream33

Upskilling of staff resource across sector Opening up of collaborative relationships across other organisations Value, saving and efficiency gains from the creation and delivery but also the actions subsequently taken due to the insights gained across research, student, staff and estates and possibly internationallyOpening up access to disparate data sets and making sense of them in an HE contextPossible national licensing deals for paid access to data

Team member experiences

James 11.05 11.1534

Team Laura Q & A

Claire Daniells University of PlymouthFrances Leach MMUNatalie Butler - Leeds BeckettNicola Witts MMURhodri Rowlands Sheffield HallamShri Footring JISCScott Wilson JISCLaura Knox St Andrews

13.00 13.20

35

User StoriesI want to: Understand the destinations of my students post-graduation (in particular further study and employment)So that I can: ensure the credibility and sustainability of our curriculumI want to: understand the demographics of students who progress on to further studySo that I can: better understand the quality and demography of students applying to PGT level study at my or competitor institutionsI want to: understand the geographical locations of my graduating students who enter employmentSo that I can: ensure the curriculum is adding value and is credible in the context of the relevant labour market (local/national etc).

I want to: understand the gaps in the labour market (local, national and international)So that I can: ensure the curriculum is adding value and is credible in the context of the relevant labour market (local/national etc).

There were more but these give a high level overview

36

White Paper and TEFGraduate employmentHighly skilled employmentLEO dataset

New DHLEConsidering the need to understand graduate migration in greater depth, including the wider social impacts of graduates and travel to work patterns. Linked dataAdditional Context

In addition to general state of play for HE in terms of student perceptions of value for money, increased competition and squeezed public funding..

White paper: LEO 'we hope this will also be used by providers evaluating their provision and considering how they can tailor it to better deliver relevant skills for the labour market'

New DLHE: understand graduate migration in greater depth, including the wider social impacts of graduates and travel to work patterns. And 'there may be linked data options that could be explored to obtain additional depth and quality of information, while minimising the costs of dataacquisition. This might include utilising a geospatial data systemsuch as the Unique Property Reference Number (UKPRN) to derivecontextualinformation about the locationwhere agraduateis living or working'.37

DHLE DataNOMIS (Official Labour Market Statistics)

Possible addition:IPPR Burning Glass - Wheretheworkis.orgAssociation of Graduate RecruitersData Sources

Give overview of content of each..

DLHENOMIS employment by sectors (industry and geo)WTWI: IPPR Burning glass data set (LEP, SOC, employability potential/ave salary)

Explain potential opportunity with ARG for HESA to pick up: aggregate vacancies and hires sliced by career areas, industry types, employer size, and applicant numbers per role.

Note/make plea here in relation to lack of coherent/comprehensive data set

Demand dataBurning Glass has collected more than 1.5 million jobs posted online by employers in the UK since 2012. Burning Glass uses advanced natural language analytics to turn the information in each job posting into usable data. This allows Burning Glass to describe employer demand for specific roles or skills. The demand for entry-level (< 2 years of experience) talent is compared with the available supply of new graduates or trainees. Burning Glass postings data is normalised against vacancy data published by the Office for National Statistics (ONS) and Jobcentre Plus. The data is further validated against the Annual Survey of Hours and Earnings (ASHE) from the ONS. Supply dataWe have used the numbers of learners leaving higher and further education (programme finishers by subject area) as a proxy for the supply of entry-level talent. Supply data are sourced from the following agencies: Higher Education Statistics Agency (UK wide) Skills Funding Agency (England) Scottish Funding Council Skills Development Scotland Department for Employment and Learning of Northern Ireland StatsWales OccupationsWe use the standard occupational classification 2010 (SOC2010) by the ONS, which is the official classification of occupational information for the UK. Within this system, jobs are classified in terms of their skill level and skill content. In this tool, the occupations are shown at minor group or 3-digit level.

38

40

07/07/201642Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Dashboard: Course Market Analysis for InstitutionsWhat is it?An Overview MoviePurpose:This dashboard is designed to support a universitys strategic planner in designing course by allowing comparison across the sector.

Use case:As a Strategic planner when working out which courses to teach I want to examine competition to my course offerings to ensure I target recruitment activity most effectively.

Data sources:National Pupil Database: http://bit.ly/224CU8IKey Information Sets: http://bit.ly/1ZYnG5zNational Pupil Database: http://bit.ly/224CU8IHESA DataWhat needs to be done and issuesTime and Effort to MarketWhere there is scope for improvement:Generally very polishedSome work on the interface required perhaps to sign-post the featuresLicencing issues for league table data need to be negotiated.Data sources would need updating each year particularly the school data.

James to continue with these (as many as time permits)43

Dashboard: University Finder for StudentsWhat is it?An Overview MoviePurpose:This dashboard is targeted at students who are looking for a university course to fit their needs. By needs we don't only mean course but also: cost, employability, location and entry tariff.

Use case:As a student when working out which university course offers best fit my needs, I want to understand factors of relevance to me (course, cost, employability, location, cost of living, rural/urban and entry tariff) to compare and match offers to my circumstances.

Data Sources:Key Information Sets: http://bit.ly/1ZYnG5zHESA DataWhat needs to be done and issuesTime and Effort to MarketThis dashboard supplies a unique perspective on data and services that are already available to students. In some ways this is a crowded marked. So the unique selling point of this product would need to be promoted that is that the data already available to students is amalgamated and drawn together to create a wizard like app for students to find courses.What would need to be done:Identify appropriate vehicle for deliveryMarket uniqueness of the the productNegotiate data licences for league table data

James to continue with these (as many as time permits)

44

Dashboard: Finding Comparable InstitutionsWhat is it?An Overview MoviePurpose:This dashboard can be used to identify a universitys relative performance against a benchmark of similar institutions.

Use case:As a Planning Manager I want to select similar institutions based on metrics I choose so that I can determine the best institutions to compare with my own university to understand if our performance is relatively good or bad

Data Sources:HESA data from HeidiKey Information Sets: http://bit.ly/1ZYnG5zLeague Table Data will require licensingWhat needs to be done and issuesTime and Effort to MarketWhere there is scope for improvement:Data a relatively narrow data set was used for prototyping; a production version could accommodate a far more comprehensive data set.Filters searching and filtering could be enhancedLicencing Makes use of some league table data to benchmark against entry tariff. Licence for this need to be negotiated.

James to continue with these (as many as time permits)45

Library Data LabsTeams working on Library BI Stories at 0.2 FTE, total estimated effort 15 days from July - Oct 2016Both Product Owners and Sector Data Experts invited:Product Owner from the sector to steer which stories are of interestSector Experts to understand what data sources are available & what is in the dataJisc Contracted Data transformation specialist Jisc Agile Scrum Master & Tableau UserTeams receive experience and guidance of Agile workingOption for Tableau Desktop training to help with creating visualisations7/7/2016

46

Myles just to note we are running a set of teams from the library area to prove the concept transfers07/07/201646Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Analytics academy a Jisc beta service - October 2016Business intelligence offers value, savings and efficiencies to Universities through data informed enhanced planning / decision makingMany problem spaces are commonly felt, while the data landscape to support insights is vast. Some universities have little access to good BI at all, while those with capability are often duplicating effort. There is no higher education focused CPD offer to train up BI expertise.Analytics academy addresses these problems by providing expertise and tools for analysts (planning officers and others) to identify suitable problem areas (student, staff, research, estates etc), exploring the data landscape for insights and producing interactive dashboards for the sector

7/7/2016

47

Myles a new Jisc offer to explode whether there is a sustainable service in this07/07/201647Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Keep in touchhttp://www.business-intelligence.ac.ukSubscribe via www.jiscmail.ac.uk/JISC-HESA-BUSINESS-INTEL Twitter @HESA @jisc #hesajiscbi

7/7/2016

48

Questions?

11.25 11.3049