Listening for Impact – Rapid Analysis...

Listening for Impact: Rapid Analysis Report

of 18

Rapid Analysis Report

Authors: Rowan Wilson, Carl Marshall, Fawei Geng Published: 7th Dec 2010

Introduction

Listening for Impact is a 6-month project (Oct 2010 to Apr 2011) funded by the JISC to perform a thorough, yet rapid analysis of the impact of the public Oxford Podcast collection containing nearly 1800 items. By mixing technical approaches and user engagement it aims to increase discoverability and reuse of material within teaching, learning and research. The project attempts to:

1. Address the lack of systematic analysis of the impact of our podcasts offered on our open Oxford University websites and via Oxford on iTunes U

2. Develop and instigate practical approaches to embedding academic podcast resources within teaching, learning virtual environment at Oxford and the wider subject community.

3. Collect and evaluate data to demonstrate the impact of the Oxford Podcast collection, which will strengthen the case for continued sustainability for the podcasting service, the Oxford Podcast collection and consequently, the OpenSpires OER collection at Oxford (the OpenSpires OER collection is a subset of the Oxford Podcast collection).

4. Research and contribute towards the knowledge of embedding online collections in academic disciplines by a case study of impact and user engagement.

Of the five areas of study raised in the project plan,

• what impact the collections have • what factors influence their impact • what the audience is for this material • if it is worth funding this service in the medium and long term • how these collections can best be maintained

…this report makes an initial approach to the first three based on currently available data. The report focuses on the level of take-up and the way the podcasts have been used, utilising the best practices identified by the Toolkit for the Impact of Digital Scholarly Resources (TIDSR).

Methodology To assess impact, it helps to know what you are measuring and from where you are getting the data. Whilst the TIDSR approaches seem geared towards the idea of a single entity of academic interest, Podcasting at Oxford


of 18

does not easily fit this model. Figure 1 illustrates the complexity of the data sourcing: podcasting at Oxford is a collection of portals and file servers.

Figure 1: An illustration of the many entities that relate to podcasting from Oxford

To analyse the available data gathered by the project team, both quantitative and qualitative methods were employed. In terms of quantitative methods, the starting point was to determine “Have we got enough information to be representative of the podcasting activities defined in our scope?” From there we can use a combination of TIDSR approaches (Log File Analysis, Webometrics, Analytics and Referrer Analysis). The quantitative approaches used so far have been sample-based as we determine what can be done whilst seeking a preliminary insight into the wider impact of our podcasts. The qualitative data collection was conducted using the following methods:

1. News coverage analysis: gathering and categorising the news coverage that Oxford podcasting activities have attracted. This was supported by news media databases Nexis1 and Google News2.

2. Analysis of feedback received by academics: analysing the email feedback received by individual the academic colleagues (Oxford University lecturers/tutors) and by the central Podcasting Service from the listeners worldwide.

3. Analysis of feedback received from Students at Oxford • Oxford Freshers’ survey gathering data on the use new Oxford students make of podcasts before

arriving on campus. • Oxford student group survey, gathering data to find out how current Oxford students make use of

podcasts to enhance their learning in particular courses.

1 http://www.lexisnexis.com/uk/nexis/ 2 http://news.google.com/


of 18

4. Focus groups: examining data from the three focus groups organised by the OpenSpires project, which aimed to produce podcasts to contribute to Open Education Resources (OER).

In the following sections, the quantitative data collection and analysis will be discussed first, as it provides a rough picture of how Oxford’s podcasts have been used globally. The aim of the qualitative data analysis is to learn why and how listeners use the podcasts.

Quantitative Analysis To assess impact quantitatively, technical approaches such as log file analysis and web analytics were employed to help to understand the podcast distribution process and what data can be collected during it.

Figure 2: The steps a visitor's computer takes to download a podcast

The above diagram (Figure 2) attempts to illustrate the steps that occur when a podcast is downloaded from Oxford. There are two typical scenarios covered above:

• Locating and downloading a file via the Oxford Web Portal for Podcasts3 in a web browser (shown as the upper image on the visitor’s computer)

• Locating and downloading a file via the iTunes U4 store using the iTunes application (shown as the lower image on the visitor’s computer)

The distributed nature of the podcast hosting environment (as illustrated in Figure 1) means that pure log file analysis is significantly complicated. For the purposes of this report Apache web logs have been analysed from the central repository of podcasts held at the University Computing Services. For greater detail on the nature and extent of the web-related data set please refer to the project web site at http://blogs.oucs.ox.ac.uk/listeningforimpact/.

3 http://podcasts.ox.ac.uk/ 4 http://itunes.apple.com/gb/institution/oxford-university/id381699182


of 18

Analysis News Coverage Analysis Since the launch of Oxford on iTunes U in October 2008, the podcasting activities at Oxford have attracted great attention from various media sources including national and international newspapers (e.g. Guardian, Telegraph, Times, Independent, Australia’s Age) and broadcast companies (e.g. CNN and BBC). Oxford Podcasting featured recently in an Apple iPad TV advert, which aired for the first time in the US on 1 November across all major US networks in prime time. To gather articles related to Oxford’s podcasting, searches were done using Nexis and Google News. Two databases were used, as Nexis did not always list the latest news. After filtering out the duplicated articles, a total of 70 articles were identified by the two databases. Table 1 (below) summarises the media coverage from October 2008 (the launch of Oxford on iTunes U) to the beginning of November 2010. The articles were categorised into web, newspaper, radio and TV. The newspaper articles were then further divided into international, national and local newspapers. Media Number of articles Sub-categories Web 48 Newspaper 20 5 international, 14 national, 1 local Radio and TV 2 1 local radio, 1 local TV Table 1: Media coverage of Oxford on iTunes U since October 2008

After reading all the articles, we have noted that the news articles focused on a number of key events. Table 2 below summarises the number of media articles for each identified event. Event Number of articles Sub-categories Oxford iTunes U launch Oct-2008 15 7 Websites

6 Newspapers: 4 national, 1 local and 1 international 1 Radio 1 TV

Oxford downloads reached no. 1 in the iTunes U chart

9 8 Websites, 1 national newspaper

E-book announcement 4 The article ‘Oxford, Rice, Open University release eBooks on iTunes U’ has appeared on 25 websites

Table 2: Media coverage associated with a number of key events

A number of key points related to learning and teaching emerged, which are grouped according to their frequency/popularity. Free for all: iTunes U has enabled Oxford University to provide free educational materials to the public. This was mentioned by almost all of the articles. “450 hours of free podcasts, lectures, films and admissions guides up on the iTunesu academic portal, available to anyone who wants to download them.”5 5 Source: http://www.digitaltrends.com/international/oxford-v-cambridge-on-itunes/


of 18

Informative: Oxford iTunes U podcasts help prospective applicants to become familiar with the application process and Oxford life. “Potential applicants will be able to access free podcasts about how to apply to the University, including choosing a college and a course, and preparing for interview. They will also be able to see what an Oxford tutorial is like.”6 Popular: the iTunes U effort at Oxford generated a number of headlines based on statistics. “In the first week their [Oxford’s] 200 items were visited by 168,000 visitors and 60,000 downloads occur”7 “Information about the Oxford admissions process has proved very popular, attracting more than 30,000 downloads to date.”8 “Apple’s iTunes U sees more than 300M downloads, offers 350,000 educational files”9 Openness and reaching out: opening up to a wider audience, and contributing to the emerging OER market. “We hope that this service will make Oxford’s diverse range of audio and video material more widely accessible to applicants, alumni, supporters of the university, and the intellectually curious”10 Easy access: “… for the first time this brings everything together on one, easy to use website, that is really easy for people to navigate. It gives people a real sense of breadth and depth of the activities that are going on at the university.”11 Learners are in control: “Providing material online means that students are more in control of how, where, when and the pace of their learning”12 Supplementary: the podcasts can supplement other sources of learning, rather than being a complete substitute. “While most academics think that allowing students to experience content at a time that suits them is helpful, most also agree that it should only be used to supplement regular study. ‘This is by no means a substitute’…”13 Social responsibility, i.e. Educate the world "At the end of the day that's what universities are about… It's about educating people and bringing new ideas to people, so it's part of our overall mission."14

Implication and reflection By reading through all the articles discovered by the project team, it seems that there are a few patterns that may help us to learn how to boost the visibility in the future. First, most of the media tend to produce their news based on the ‘press release’ articles provided by the University press office. For example, the news of the launch of Oxford on iTunes U was used almost verbatim

6 Source: http://www.macworld.co.uk/education/news/index.cfm?newsid=23052&pagtype=allchandate 7 Source: https://elgg.leeds.ac.uk/webteam/weblog/12177.html 8 Source: http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycode=408300&c=2] 9 Source: http://venturebeat.com/2010/08/24/apples-itunes-u-sees-more-than-300m-downloads-offers-350000-educational-files/ 10 Source: http://www.guardian.co.uk/education/2008/oct/07/elearning.students 11 Source: http://www.oxfordmail.co.uk/news/3748048.Get_lectures_on_your_iPod/ 12 Source: http://www.independent.co.uk/life-style/gadgets-and-tech/features/downloadable-lectures-making-beowulf-bearable-976587.html 13 Source: http://www.guardian.co.uk/digitalstudent/podcasts 14 Source: http://www.guardian.co.uk/digitalstudent/podcasts


of 18

by the majority of the 15 examined articles. This highlights the importance of a well-crafted press release featuring the key messages the University wishes to convey. Second, significant milestone based stories attract further news reports. For example, Oxford reached the global number 1 positions in the iTunes U downloads chart, and in doing so attracted much attention from the media. So too did the launch of e-books titles derived from holdings in the Oxford Text Archive. Third, as Oxford already has a strong reputation, “Oxford” was often mentioned in reports on other institutions. We should note the value of our prominence when promoting our future podcasting developments and milestones. Finally, the association with a successful company (i.e. Apple) increases the chance of being publicised in the company’s marketing materials. For example, the quote from Apple of “Apple’s iTunes U sees more than 300M downloads” was reported by a large number of web sites and newspapers. Oxford’s relative prominence in download charts was also mentioned in all these articles.

Focus groups analysis OpenSpires [http://openspires.oucs.ox.ac.uk/] was a JISC-funded project which aimed to produce and distribute Creative Commons licensed podcasts. The project organised three focus groups involving academics to find out what they really thought about the initiative and Oxford podcasting in general. The key points drawn from these focus groups related directly to podcasting are reproduced here:

• Publicly available podcasts meet a frequently stated requirement of most funding bodies - wide dissemination and public impact.

• Podcasts also support one of the core activities of the University - wider engagement with society. • The Creative Commons-licensed podcasts promoted debate and discussion about the openness of the

academic practice and other concerns, i.e. intellectual property issues. • There was a need to get the balance right in terms of amount of material released – enough to attract

interest without giving away absolutely everything. • The variety of material offered highlighted the distinction between 'research-driven' podcasts and

'teaching' podcasts. • Apart from lecture recording, it was felt that a greater variety of podcasting activities might be

undertaken, for example recordings of tutorials, and conversations. • The main Oxford platform - podcasts.ox.ac.uk, can to be improved to allow users to find individual

podcasts more easily.

Analysis of feedback received by academics In the last two years, a number of items produced by Oxford academics have featured in the Top 10 iTunes U download chart. For example, a podcast series made by Marianne Talbot, ‘A Romp through the history of Philosophy’ has repeatedly reached number one in the iTunes U chart.


of 18

Figure 3: A sample week of downloads from iTunes U plotted against a logarithmic scale

Figure 3 illustrates the downloads via iTunes U for one week. Two points can be drawn from the diagram: a small number of items were extremely popular – more than 5000 download per week; whereas around 50 percent of the items were downloaded less than 10 times per week. To identify the factors which make podcast popular the project team asked academic colleagues to share feedback that they had received from their listeners. So far 67 emails have been forwarded to the project team. To analyse the email feedback, all messages were carefully read to extract information based on the following topics, which are related to the research questions of the project:

• Audience: who is listening to these popular podcasts • General: comments or viewpoints on podcasting in general • Impact: what effects the podcasts had on the individual • Factors which may affect the impact of these podcasts: elements that attracted the listeners and

suggestions made by the listeners Although most of the people who sent feedback did not identify where they were from, a number of messages did include some geographical information, which indicated that Oxford’s podcasts on iTunes U were listened or viewed by people from all over the world including Sweden, Norway, Brazil, USA, Canada, China, Korea, and New Zealand. (For further analysis of audience by geographic location and domain please see the ‘Analysis of requesting domains’ section below).

Audience The email feedback received by academics also revealed the occupation of some listeners:

• Professionals (writer, lawyer, an orthopaedic surgeon) o “I have always liked philosophy (being a lawyer I actually need it), but I must say your

[podcast] brought me back the will to study it more carefully.” o “It was both stimulating and thought provoking, particularly as I am trying to write a book [on

the subject]” • Students (prospective Oxford student, graduate students and high school students);

o “I'm not joking but this has become my favourite site in ten seconds flat - can't stop downloading! Where has this been all my life?????? This is ridiculous!”


of 18

o “Because of your lecture, I'm getting better and better knowing quantum mechanics (Absolutely, it is so hard and difficult to understand all contents of quantum mechanics),.. I really really thank you for providing me with a good opportunity to learn quantum mechanics from you.”

• Others who are interested in the subject e.g. retired professionals and teachers. For example, after listening to a philosophy podcast, a high school teacher said in an email that philosophy could benefit both teaching and learning in their secondary school. They sought advice from the podcast producer on how philosophy could be introduced into their teaching. Although the messages received were from a number of countries around the world, it did not appear that any were from the current Oxford students. We suspect this may be because those Oxford students have the opportunity to discuss issues face to face with their tutors. Later in this report we will present a survey that was designed to analyse the impact of podcasts on the current Oxford students.

General Most of the feedback sent by the listeners expressed their enjoyment and gratitude to the people who produced the podcasts. They declared the internet a fantastic tool for spreading knowledge. People also felt privileged that they could gain access to online materials offered by the University of Oxford ‘for free’. Most of the feedback also indicated that further free podcasts would be greatly appreciated. “You ask if there is a demand outside Oxford for podcasts of your early modern lectures. Yes, big demand I would think. ……"

Impact By going through all the available feedback, it seems that the impact can be summarised by the following aspects:

a) Engagement and interaction with the podcasting topic for private study b) The podcasts influenced the listeners to explore the podcasts in other Oxford subjects c) Re-use in teaching situations

The details of each point above is explained below:

Engagement, motivation and interaction with the podcasting topic for private study • Listeners were engaged by the podcasts and interacted with the podcast creator by asking questions and

confirming their understanding of the topics. • Some listeners attempted to demonstrate their understanding by providing their own examples which

illustrated a concept discussed in the podcast • Listeners sought to advance their knowledge of the topics in the podcasts by asking for reading lists,

lecture notes, and recommended textbooks. • A listener was motivated to carry on studying the subject after listening to the podcast:

“I have recently enrolled in an Open Universities in Australia with the plan to complete a BA in Philosophy, but the first unit I have had to complete is a Study Skills unit which has been so boring and mundane I have been questioning whether to continue or not. Your enthusiasm for philosophy is infectious and put me back on course to continue my studies. Thanks again.”

Podcasts influencing listeners to explore further • Feedback indicated that the most popular podcasts had not only attracted people to the particular

subject, but also encouraged the listeners to explore podcasts in other subjects offered by Oxford.


of 18

• While some listeners listened to the podcasts for leisure, a number of students said that the podcasts they listened to helped them to learn more deeply and to prepare for exams such as GMAT, GRE: “listening to the Shakespeare’s play sheds much light upon a number of issues in Japanese play”

Re-‐use in teaching situations • A secondary school teacher wanted to reuse the ideas implemented in the podcast in their own teaching. • Another teacher was impressed by “the excitement and clarity” of how the podcast was delivered. They

said that they would try to deliver their lecture in this way in their class. • A listener said that one of the advantages of learning through podcasts was the capability to learn at

their own pace: “I have quite a bit of work to do to understand some of the trickier derivations – fortunately, I have a ‘pause’ button and much more time than your students do”

Factors which may improve the impact Here we examine the features which attracted listeners/viewers, and the suggestions made by them.

• The podcasts used great examples which made the complex concepts clearer and easier to understand • The podcasts were interesting, engaging, engrossing and stimulating – “The exemplary and passionate

delivery affects learning outcomes” • The podcasts were presented in a relatively unadorned form - “The podcasts are recorded raw, not

dressed up as some have with distracting background music” The following is a wish list compiled from the feedback:

• Podcasts in more subjects • Reading list and handouts • A pdf copy of the text (transcription) • Lecture notes in pdf format • A copy of the solution to the questions discussed in the certain podcasts

Reflection Although listeners cannot interact with the lecturer during the course of the lecture, it seems clear that being able to pause when listening helps listeners review the material follow up with queries by email later. Although learning through podcasting can be passive, the feedback indicated that it was still possible to enable interaction and engagement between the listeners, the podcasting topics and the person who delivered the podcast through back-channels such as email.

Analysis of feedback from Oxford students

Feedback from the Oxford Freshers’ survey Most of the feedback received by the academics that produce podcasts came from people outside of Oxford University. To sample audience impact within the institution, a survey was undertaken. An Oxford University Freshers’ Fair survey was held in early October 2010. Four of the Survey Questions (Q5 – Q8) were about the podcasting service. These questions aimed to find out whether the freshers were aware of and used the Oxford podcasts, as well as their opinions of Oxford’s podcasts. In total, 676 students completed the survey. The results are summarised as follows. Q5. Did you know that Oxford University produces podcasts that anyone can download?


of 18

The results showed that more than half of the respondents knew that Oxford University produces podcasts, which may be attributable to the wide news coverage. Q6. Have you ever heard of the following?

• Oxford iTunes U • Oxford Podcasting Service • podcasts.ox.ac.uk

The results showed 3/4 of them were aware of Oxford’s podcasts available on the iTunes U portal. The lower level of awareness for Oxford Podcasting service and our Web Podcast Portal may be explained by the far greater level of marketing done by Apple (promoting iTunes U) compared to Oxford’s promotion of the web portal. Q7. Did you download any of the following podcasts before you came to Oxford? (Tick all that apply)

• Introductions to the University • Lectures and talks on particular subjects for my own interest • Lectures and talks on particular subjects that my teacher recommended

Overall, only 29.1% of the respondents have downloaded podcasts from Oxford. Of those that downloaded content, 71% accessed podcasts that related to subject they were interested in, and around half downloaded material that would help introduce the University to them. Q8. If you downloaded podcasts, what did you think of them? Only 21.7% of the respondents (147) answered this question. Among them, many comments are brief (a few words, single sentences) and general: e.g. very interesting; very professional, informative, etc. The detailed comments are grouped into the following categories:

Reaction to the podcasts related to academic subjects Students were impressed by the podcasts in a number of subjects: e.g. Nanotechnology, Medieval English, and ‘Philosophy for Beginners’. Some students expressed interest in hearing podcasts in other subjects, e.g. History. By listening to the podcasts, students felt that they were more prepared to study at Oxford. “I found them really interesting because I hadn’t previously had any experience of learning this way” “Very informative and helpful in encouraging the correct mindset and attitude when coming to study at Oxford.” “They gave me a good idea about what to expect from lectures in those particular subject areas.”

Suggestions for improvements While students were generally impressed by the quality and usefulness of the podcasts, they also made a number of suggestions:

• In some cases, video podcasting was preferred to help understand or to relate to the speaker. “I’ve only listened to one about depression and anxiety in children from the Psychology Faculty. It was very interesting, but I would have preferred a Youtube-style video (I like to be able to see the person talking)” “ They were helpful but sometimes the audio only podcasts referred to diagrams or screens that I couldn’t see.”

• Associated materials could help understanding. “They were good. They could be made better if they were accompanied by a .pdf handout, just so that it is easier to follow”

• Other suggestions to make the podcasts more appealing:


of 18

o “Keep it short” o “Make it more interactive” o Not limited to subjects of interests of the researchers who made the podcasts – this seems to

be a call for a broader range of topics within Oxford’s catalogue. o “Go beyond traditional lecture formats” o Provide introduction to Oxford for both undergraduate and postgraduate – this may be a

reflection on Admission’s material being largely oriented towards potential undergraduates. o “Clarify the target audience” – this may be related to the listener not having enough

information prior to downloading on which to make an accurate assessment of nature of the podcast.

o Issues identified when listening § “Sometimes cut out the ppt presentation and the questions from the audience which

couldn’t be heard” – this may be a reflection on our policy of editing out material which may have copyright or consent issues.

§ “Low sound quality“

Admissions and University life Among the written comments, there were a notable number of positive comments about the podcasts for ‘Admissions and University life’. Here are a few examples: “I found the ones from the admissions offices really helpful. As an international student they were a really good way for me to learn about the admissions procedure as it’s not as easy for international students to come to the open days as it is for UK residents.” “I found the Undergraduate admissions podcasts quite useful, especially those concerning interviews. I think video podcasts of mock interviews would be a very helpful addition, hopefully in as many subjects as possible.” “Very helpful and full of information about the application process” “Very helpful for giving an insight into student life and current affairs of my department” “They were helpful and allowed me to know what to expect at Interviews and how lectures work here” “The social anthropology dept. did a series of interviews of current students about life as a grad student at their dept, which was really helpful. ……I don’t know if I will use it now that I’m here, but a few students have been asking about whether lectures will be on iTunes U so perhaps in a few years it will be expected? I hope not… I still believe in education taking place in the classroom, particularly for students AT the institution…” The results from the last question indicated that podcasts regarding admission and University life attracted positive attention from the perspective students. The suggestions made by students may help us to improve the podcasts from this department.

Feedback from the Oxford student survey So far the data suggests that most listeners of Oxford’s podcasts are people outside of University of Oxford. Considering that a core activity of the University is teaching and learning, it is essential to explore whether and how the podcasts influence the students at Oxford. To help address this issue, a survey questionnaire was designed and delivered (initially) to a group of 3rd-year undergraduates in English Faculty at Oxford who attended an optional lecture in Modern English from which


of 18

the course lecturer produces podcasts. The questionnaire was delivered in the class by the tutor and 93% of respondents (28 students) took the survey. Below we explore the survey and initial results.

Questions about podcasting in general Q1 Have you ever listened to podcasts from any of the following sites? (Please tick all that apply)

• Oxford on iTunes U • Oxford’s Podcasting Website (http://podcasts.ox.ac.uk/) • Podcasts via Mobile Oxford (http://m.ox.ac.uk/) • Oxford’s VLE – WebLearn (http://weblearn.ox.ac.uk/) • Elsewhere on the internet • Not applicable – I have not listened to any podcasts

Roughly half of respondents had listened to podcasts from Oxford’s iTunes U. This may be due to the following reasons:

• The course tutor produces podcasts on Oxford’s iTunes U and has told their students about this. • High levels of news coverage for iTunes U

It is also worth noting that nearly half of the respondents did not listen to any podcasts at all, which may be because the students had attended the lectures and were surveyed in advance of any exams before the need for revision was pressing. Q2 If you have not listened to any podcasts, was it because? (Please tick all that apply)

• I didn’t see anything relevant to my learning • I had technical difficulties accessing them • I didn’t know the university publishes podcasts • I was too busy

64.3% of respondents did not answer this question. A small proportion of students who answered the question reported that the reason that they had not listened to any podcasts was either because they felt themselves to be too busy, or because they were unaware of any other Oxford podcasts. This sample may be too small to be indicative. Q3 How do you find your podcasts? (Please tick all that apply)

• Via search engines, such as Google or Yahoo • Recommended by my teachers • Recommended by friends or classmates

Half of the respondents reported that the podcasts were recommended by their teachers. This may be more of a reflection on surveying a class whose tutor podcasts and who had advertised their podcasts to their class, rather than the student population as a whole (where less than 10% of the University’s tutors have produced a podcast). Q4 Which of the following do you do to listen to podcasts? (Please tick that all apply)

• Save to your portable media player (e.g. phone, ipod, etc) and listen to later • Save to your laptop and listen to later • Save to your desktop computer and listen to later • Listen immediately on the laptop without saving for later • Listen immediately on a portable media player (e.g. phone, ipod, etc) without saving for later • Listen immediately on the campus computer without saving for later • Have not yet tried any podcasts


of 18

The results show that most people save the podcasts and listen to them later, and that (for these students) they are often using a laptop (likely their own) over a university-owned computer. Again the sample size is a little too low to be able to extrapolate more general trends. This result also raises a question: how many of the saved/downloaded podcasts were listened to? This is at least as important as the downloading figure that is often quoted to illustrate how well or otherwise podcasts are used. If a user subscribes to a podcast using an RSS client, whenever their media application is running, new items will downloaded automatically without their intervention.

Oxford University Podcasting Q1 Have you ever listened to the following types of podcasts produced by the University? (Please tick all that apply)

• Interview style podcasts • Lectures of your own course • Lectures or talks related to your own subject area from Oxford • Lectures or talks related to your own subject area from elsewhere • Lectures or talks *not* related to your own subject area from Oxford • Lectures or talks *not* related to your own subject area from elsewhere • Podcasts related to life at Oxford University (e.g. sports, admissions, tours, etc) • Not applicable – I have not listened to any podcasts from Oxford University

For this group, half of the respondents had listened to Oxford’s podcasts. As the course tutor produces podcasts for this module, not surprisingly 39.3% of them listened to podcasts in both ‘Lectures of your own course’ and ‘Lectures or talks related to your own subject area from Oxford’ categories. It is interesting to learn that 17.9% students explored other subjects by listening to podcasts that are not related to their course. Q2. Why did you listen to the Oxford’s podcasts (please tick all that apply)?

• The podcasts allowed me to catch up if I missed the actual lecture • The podcasts helped me to make good use of my time • The podcasts stimulated my interest in the subject • The podcast motivated me to advance my study in the subject • The podcasts provided a good introduction to the lecture or tutorial topic • Podcasts were useful for me to revise the lecture topics

Consistent with the last question, 50% respondent did not respond to this question as they did not listen to any Oxford’s podcast. The top three reasons for them to listen to Oxford’s podcasts are:

• The podcasts allowed me to catch up if I missed the actual lecture • The podcasts stimulated my interest in the subject • Podcasts were useful for me to revise the lecture topics

Q3. Please indicate if you agree or disagree with the following statements:

• I always download my podcasts whilst in the University or a College • I always select podcasts relevant to my studies • The podcasts inspire me to explore a subject more • The University should record my lectures • High quality and clear audio is important • High quality and clear video is important


of 18

The response to the first statement shows that only 28.6% of respondents download podcasts whilst in the University or a College. The response to the second statement indicates that students download podcasts for various reasons, which may not always be relevant to their study. In the previous section the feedback from the external listeners clearly showed that most listeners wanted to explore the subject more, and over half the students who took this survey felt the same – the podcasts could be said to be quite inspirational, however this does not distinguish the method of delivery (podcast) from the value of the content (style and substance). In terms of the two statements related to the quality of the media file (video and audio), most respondents strongly agreed that high quality and clear audio is important whereas far fewer rated video availability and quality as important. Finally, 60% of those surveyed believe that the University should record their lectures for them. Q4. Please select the three most important factors which attract you to a podcast

• Is an audio only podcast • Is a video podcast • I can see the person speaking • There are closed-captions (subtitling) in the podcast • There is a transcript for the podcast • There are relevant slides for the podcast • The podcast can be played online where I discover it • There is a detailed text description of the podcast content • The podcast has a creative commons usage licence (allowing you to incorporate in your own work) • The presenter of a podcast is known to me

The top three most important factors which attract this group of students to a podcast are:

• The podcast can be played online where I discover it • Is an audio only podcast • There is a detailed text description of the podcast content

This question highlights that the fact that podcasts can be played immediately and easily is appealing (46.4%) and that audio is the preferred format for respondents. Q5. How long is your ideal podcast? (please pick one)?

• The podcast is less than 5 minutes long • The podcast is less than 20 minutes long • The podcast is less than 60 minutes long • The podcast duration doesn’t matter to me

Although it is often recommended that short podcast may be preferable for learners15, nobody in this group agreed. The majority of the group thought the podcast duration does not matter Q6. If you listened to any podcasts produced by the University, how could they be improved? Below are some quotes from the feedback to this question. Most people (92.9%) surveyed did not respond here.

• “Clearer audio, more lectures recorded” • “More description of their content so can see what it is before downloading/listening”

15 For example http://clt.lse.ac.uk/Projects/Case_Study_Three_report.pdf http://www.economicsnetwork.ac.uk/showcase/johnes_podcasts.htm


of 18

Reflection To summarise, Oxford’s podcasts are well known among students but the uptake of podcasts is relatively low. This may be due to a number of factors including their having the opportunity to meet their lecturers in person and the busy student life. The results show that students believe that listening to podcasts can help them with catching up a missed lecture and exam revision. Whilst students normally prefer to download the podcasts and listen to them later, one of the top three factors attracting students to a podcast is that the podcast can be played online using their browser. Looking at the data from both external users and the current Oxford students, creating more podcasts with associated materials (e.g. lectures notes, brief description of the content) may attract more listeners and have a greater impact. Finally, as the sample of the survey is relatively small, the findings here may not be generalised to the entire student population at Oxford. As the project progresses we will survey more students across different divisions.

Analysis of requesting domains The following diagram was generated based on a sample of web log data from the month preceding project startup using the open source web log analysis tool Analog. The diagram illustrates that our audience is distributed widely throughout global top level domains. The label ‘unknown’ represents the set of IP addresses that cannot be associated with a top level domain via a reverse DNS lookup. Note some data points are not legitimate top level domains (for example ‘local’); this seems to be an artefact of incorrect information being entered with regional internet registries by entities managing IP blocks.

Figure 4: Domain Name Analysis of a sample of data from media.podcasts.ox.ac.uk

The preponderance of unknown domains is clearly problematic. We therefore analysed data from the same sampling period using a ‘GeoIP’ database – essentially a mapping of IP addresses to geographic locations not based upon top level. The figure below illustrates number of requesting IPs by country:


of 18

Figure 5: Unique IP addresses making requests for podcasts grouped by Country

This again seems to make the point that a large proportion of the audience for Oxford’s podcasts is outside the UK, although it is hard to drawn any firm conclusions from a single sample. This is something that merits further investigation as the project continues. Some further analysis was also done on data from the same sampling period to break down the domains within which requesting IPs reside. Figure 6 shows a breakdown of UK-based top level domains:


of 18

Figure 6: UK second level domains by number of unique requesting IP addresses

As might be expected, the largest proportion of requesting IPs from within the .uk top level domains resided within blocks managed by academic institutions. Breaking down requesting IPs from within these blocks further, we find that requests from within Oxford itself are most common by a significant margin. It should be noted that – as this data has been broken down by requesting IP – it may be skewed towards institutions who do not employ network models which mask multiple requestors behind single IPs.

Figure 7: Unique domain names ending in .ac.uk found in the sample data

Reflection Log file analysis has been challenging for a number of reasons

• The extreme size of the log files in question due to the high traffic nature of the site • The large number of requesting IPs with no reverse DNS information


of 18

• The distributed nature of the Oxford podcast hosting environment Despite these difficulties it has been possible to pull some interesting initial results from the sample we have started with, and we are confident that the solutions we are currently building will ameliorate at least the first two of these issues. From the current analysis results it seems that Oxford podcasts have an extremely large audience beyond the UK, particularly in China and the US. UK interest is also strong, with the most requesting IPs residing within academia, and particularly within our home institution. One possible conclusion from these data is that we are succeeding on both the very local and very global scale, but could make improvements in the reach of our material into other UK academic institutions.

Further Work For the purposes of this report we have analysed mostly pre-existent qualitative data and a sample of technical quantitative data from a period of a single month prior to the project’s start. This has produced some interesting results as detailed above, and will serve to inform the work we do in the rest of the project. Here we present some of the results of our reflections on these results:

• The log analysis seems to show that there is strong demand for our materials beyond the UK. We will need to perform more in-depth analysis on these results and monitor the trends in activity as we step through the log files for subsequent months.

• We need to dig deeper into information on access from other Uk educational institutions • Student surveys have provided useful feedback and we should undertake more of them across differing

subject areas, in addition to surveys for visitors from outside academia. • It would be extremely useful to be able to map the results of these data analyses to a calendar of

promotional activities – both by ourselves and by our distribution partner Apple - and incidental triggers for interest in our materials such as anniversaries, festivals and major news events.

Listening for Impact – Rapid Analysis...

Documents

Transcript of Listening for Impact – Rapid Analysis...