This presentation was prepared for the LibrariesThriving webinar series sponsored by Credo Reference, which took place on Jan. 14, 2013. In the presentation, we explore • the nature of evaluation in higher education, and how current trends are affecting

the delivery of information literacy instruction. • the proposition that embedded librarianship offers an effective framework for

delivering information literacy instruction, and also lends itself – more than any other option – to meaningful evaluation.

The presentation also reviews the definition of embedded librarianship really is, and discusses some key factors that contribute to applying it successfully. Finally, other valued roles beyond instruction are discussed.

1

Let’s focus on the connections between three elements: higher education, information literacy instruction, and evaluation.

2

Let’s start with the intersection of evaluation and higher education. Over the past few years, academic institutions have come under increasing scrutiny. They are being called on to demonstrate their effectiveness, justify their value, and adapt to radically new circumstances – and they haven’t always been up to the task. In the abstract of her address to the Library Assessment Conference last Fall, Dr. Judith Eaton, President of the Council for Higher Education Accreditation, had this to say, “Higher education’s work in assessment … is not sufficiently responsive to today’s environment in which greater transparency, reliable evidence of student achievement and clear and readily accessible information about our performance have taken center stage.”

3

And really we have seen only the beginning. Predictions abound that new computing and communications technologies, along with new operational models, are poised to cause a fundamental disruption in higher education. Mark Andreessen, creator of the first graphical web browser and now a successful Silicon Valley venture capitalist, was quoted in Wired magazine last May as saying that education was high on the list of industries to be disrupted.

4

No doubt you’re familiar with examples of the disruption taking place – initiatives like the Khan Academy, the EdX joint venture of Harvard, M.I.T., and now others, the Stanford University experiments that led to Coursera and Udacity – et cetera, et cetera. This list is going to continue to grow.

5

So it’s clear that university administrators have to sharpen the focus on questions like: are we doing a good job? Are we worth what people are paying us? How do we know, and how can we prove it? What changes will enable us to do even better? They’re going to ask us and every other unit on campus too. They recognize that expectations are rising continually, and the status quo is not an option.

6

Now let’s look at evaluation and information literacy instruction.

7

Academic libraries are no strangers to disruption. We have been living with disruption of our traditional operational model for just exactly 20 years now. It was in 1993 that Andreessen’s Mozilla graphical web browser, along with the opening of the World Wide Web to all users, began to disrupt society’s information infrastructure in the most profound way since Gutenberg. The resulting disruption is evident in this graph of reference transactions reported by members of the Association of Research libraries in the first decade of the 21st century. The numbers are down 45% in 10 years.

8

As Michael Stephens put it in Library Journal, People do not think of the library first when they need information. (“Stuck in the Past.” LJ Apr 15, 2011, p. 54)

9

We’ve responded to the disruption in a variety of ways: creating and integrating digital collections; repurposing our buildings; and taking on an increased instructional role are some of the major changes we have introduced. Information literacy instruction has been an effective response by academic librarians to the disruption we have been facing.

10

But we haven’t done so well when it comes to evaluation. The authors of ARL Spec Kit 318 stated in 2010 that they had found “shockingly little work that focuses on investigating whether use of library resources and services correlate with measures of success for library users.” Yet that’s exactly what our institutions are demanding of us, and will continue to demand, as society and new competitors demand it of them.

11

So, still we are left with the questions – the same ones our parent institutions are facing: and not very good ways to answer them.

12

Note that what we need here is not just more evaluation, but different evaluation. To help us analyze the options for evaluating our work, I’d like to cite Philip Kotler, of Northwestern University, Kellogg School of Management. Kotler is widely regarded as the dean of academic thinkers in the field of Marketing. He identifies three levels of evaluation, which he terms Output, Outcome, and Impact. Adapting these three levels to today’s discussion, I’d like to offer these definitions.

13

First, Output. We might expand this to include Inputs and Activities as well. When we measure our budgets, or the number of items in our collections, or the number of items we buy, or catalog, we are measuring inputs, activities, and outputs. These measures can tell us something about our value, indirectly, but their relationship to value is tenuous. They don’t tell us if those items we acquire, catalog, and hold are used; or if the use benefits anyone.

14

Then there are Outcomes. Outcomes tell us how people responded to our actions. In terms of traditional library services, measures like the number of circulations, or database accesses, or reference questions, or interlibrary loan transactions are a form of Outcome measurement. They tell us that the students and faculty we have tried to reach used our services to a greater or lesser degree. Surveys are generally a form of Outcome measurement as well. In surveys, we often ask whether our audiences are familiar with our services, whether they have used them, and what they think of the quality and effectiveness of our work. These are Outcomes.

15

Finally there are the Impacts. Impacts tell the story of how our work has changed the world. Sadly, in traditional library work we rarely are able to gauge the Impacts of our work. We don’t know if a book that was checked out got read. We don’t know if reading that book helped the student get an A on the paper, or an A for the course. We don’t know if a great article retrieved from one of our databases – or even the help of a reference librarian – changed the student’s major, or career choice. Edwin Perry was an academic reference librarian for 43 years. As he wrote in his article in the December 2011 issue of Searcher magazine, as a reference librarian *quote+ “you rarely learn the end result of your work. Did your help really help to produce the desired results? If you are lucky, you will experience a few situations where you get the answer to that question.” *end quote+ … But a few situations aren’t enough for the kind of evaluation demanded of us today. But a few situations aren’t enough for the kind of evaluation demanded of us today. What we need to do is to align our goals with the instructional goals of the courses and programs of our institutions, and ultimately with their overarching goals for student learning and achievement. That way, our work will be evaluated as part of the overall institutional evaluation strategy. So, how do we do that?

16

To develop an answer, let’s start by analyzing information literacy instruction. Then we’ll connect up IL instruction with evaluation. Clearly, information literacy instruction is meeting a need. It’s needed because in some ways that disruption of our society’s information infrastructure that we mentioned earlier has made it harder, not easier, to get the right information at the right time. So, learning about instructional design, developing instructional programs, and delivering them have become hot skills for academic librarians. There are several different ways that we can analyze the options for information literacy instruction. I’m going to label three important dimensions as the Technology dimension, the Mode dimension, and the Structural dimension.

17

Technology is probably the most obvious. We have an ever-proliferating array of technologies to choose from when we are designing IL instruction. Here’s a selection of the ever-multiplying array of tools at our disposal.

18

Compared to the growing array of technologies at our disposal, what I’m calling the Mode dimension is pretty simple. We can either deliver instruction when everyone is together at one time – synchronously – or we can design instructional programs that allow students to work through the instruction whenever they want to – asynchronously. Similarly, we have options for bringing everyone together in one place, or conducting our instruction across a distance, with remote students: co-located, or dispersed.

19

Finally there is what I’m calling the Structural dimension. What I have in mind here is the administrative structure under which we deliver our IL instruction. There are three main options. The first is ad hoc delivery. The most traditional form of ad hoc instruction takes place when we turn a reference interview into a teachable moment, and use the student-initiated interaction to conduct a little training exercise in research methods. Beyond that, we can post tutorials on the web, and promote them, and hope students use them. We can offer workshops and seminars. All of these, because they are more or less informal and voluntary, I group together under the label “Ad Hoc.” Next is the standalone, formal course. This can be either non-credit or for-credit; mandatory or voluntary. But it’s a formal learning program that students register for, that requires them to go through a set of experiences, and includes some sort of formal evaluation. It’s “standalone” because it’s unilaterally administered and led by the library organization. Then, finally, there’s the form of instruction that’s commonly referred to as Embedded. With the Embedded structure, IL instruction is incorporated into an academic program. Typically, this is at the level of a specific subject course – so it’s sometimes called course-embedded. However, it doesn’t have to be at the course level. Thus, there’s some overlap between the standalone and embedded models. That is, you might have a separate information literacy course, but one that is developed to meet the needs of a specific academic program and is tailored to the needs of that program.

20

So, these are the three dimensions we can use to analyze approaches to IL instruction: Technology, Mode of instruction, and administrative Structure. The choices we make on each dimension can affect the success of our work. But the point I want to emphasize is this: They’re substantially independent of each other. In particular, it’s critically important to recognize that embedded librarianship isn’t a technology, or a mode, and it isn’t dependent on any technology or mode of delivery. You can be an embedded librarian for a distance course and do all your work in a course management system, or you can stand up in a classroom and lead a class session. Embedded librarianship is something entirely independent of technology or mode.

21

Obviously, technology, mode of instruction, and structure are all going to affect how you evaluate your instruction. But let’s focus on just the structural dimension, so we can examine how the evaluation differs among the options of Ad Hoc, Standalone, and Embedded. I’ll use a series of tables to illustrate the issues we face. Remember from our earlier discussion that it’s impacts we are truly after. Outputs and outcomes have their place, and I’m not dismissing them, but ultimately what we also need to show is that our work has an impact on student success, in terms of course goals and institutional goals for student achievement.

22

We’ve already discussed the point that traditional ad hoc instruction in the reference interview doesn’t give us much insight into outcomes and impacts. The same is true for ad hoc tours, orientations, and guides. Web-based tutorials do offer at least the possibility of assessing student learning, through the use of self-administered quizzes. However, almost all the ones I’ve seen are pretty simplistic. Usually you can just guess the answer to a multiple-choice question – and if you guess wrong, you get to guess again, so you “pass” the test. Plus, students taking ad hoc tutorials are often anonymous, so there would be no systematic way to relate their use of the tutorials to academic results. That’s the reason why I’ve grayed out the Impacts of web-based tutorials in the table. It’s pretty hard to tell if we’ve “changed the world” – that is, effected a lasting change in the students’ capabilities – from ad hoc instruction.

23

When it comes to standalone instruction, the situation is better. A formal course, whether non-credit or for-credit, enables the librarian to set up some form of evaluation of student learning. A for-credit, graded course might be the strongest candidate for demonstrating an impact, because it affords the librarian the opportunity to make a formal, recorded evaluation of student achievement. If the IL course is truly standalone – not connected to the needs of an academic program, or to overall institutional goals and mission – then we can correlate IL course performance with overall academic performance.

24

With course-embedded instruction, we get essentially the same measures as with Standalone Instruction. But we also get a few more. Since the IL instruction takes place as part of the course, course assignments can be structured for students to apply, and be evaluated on, Information Literacy skills. The immediate application of the skills in context, and evaluation, provides a powerful learning and evaluation opportunity that’s missing from the Standalone model. Also, subject faculty gain direct, immediate insight and appreciation for the impact of IL instruction. When it works well, they see the immediate improvement in student performance. This generates a “virtuous cycle” in which they then become advocates for embedded IL instruction, and reinforce it. There are examples of this in the literature and I’ve encountered them in my own research.

25

And here’s just one vivid example – an observation by a Biology professor I interviewed in the course of my research. She had worked with an embedded librarian in the first-year course and then had the following perception when she saw some of the same students again as sophomores. “The information literacy skills developed in the Great Problems course *a first-year course with an embedded librarian] stay with the students and enable them to perform better *as sophomores+.” This instructor has perceived a direct connection between a librarian’s effective IL instruction and ongoing student academic performance. There are other examples like this in my own research and other literature as well. It seems to me that that is the kind of change we are trying to create in the world as a result of our work – and the kind of impact that we need to show to our academic administrators and institutional leaders. I do want to emphasize that general correlations have their place. But, as I’ll observe in another context in a few minutes, direct perception and support from subject faculty are extremely powerful. You want them on your side!

26

Now let’s explore in more depth the nature of embedded librarianship, and how it affords us opportunities to strengthen both our work and our ability to evaluate our impacts.

27

So here’s my definition of embedded librarianship: An embedded librarian is one who develops strong working relationships with members of a team or community; develops shared objectives with the community and responsibility for achieving them; and makes customized, highly-valued contributions to the team, which play an important role in the success of the team and the institution. Applied to IL instruction, this means that the embedded librarian builds strong working relationships with instructors and students. The relationships lead to shared understanding – the librarian’s understanding of the course and academic requirements; the instructor’s understanding of the role of information literacy in the course. From the basis of this understanding, the two can collaborate effectively to provide a powerful learning opportunity for the students. For the librarian’s part, this means customized, relevant professional work. It’s these elements that lead to achieving the learning objectives of the course and the outcomes that the institution wants to produce for the students.

28

To develop this further, here are five key factors that differentiate embedded librarianship from traditional librarianship.

29

As I talk to academic librarians and read accounts of embedded librarianship, I see a considerable number of so-called “embedded IL instruction” programs that don’t fully achieve the promise of embedded librarianship. There’s a sense that something is missing. So, now I’d like to address what works and what doesn’t. I’ll focus on two key factors. The first is the relationship between the librarian and the subject instructor, and the second is the key role of leadership on the part of senior library adminstrators.

30

Sometimes just because there’s a link to a librarian, library resources, or library tutorials, in the course management system, or the librarian is able to participate in CMS-based activities, the instruction is considered “embedded”. Sometimes a one-shot lecture is moved from the library to the classroom, and it’s considered “embedded”. Sometimes librarians are designated (by the library administrator) as embedded, but instructors fail to engage. I recently read an account of a librarian who quote embedded unquote a video introduction into some courses in a course management system. Only she didn’t tell the instructors ahead of time that she was doing this. Guess what? Some of them weren’t too happy about it. That was not embedded librarianship! So what was lacking in that example, is lacking in a substantial number of IL instructional initiatives that aren’t really embedded: the fundamental collaborative relationship that is the prerequisite for embedded librarianship. Or, the relationship may not have matured into a true collaboration. Here’s a set of questions that can help you evaluate the strength of collaboration between a librarian and a subject instructor.

31

If all the above sounds like a lot of work, it is. So, how are we going to get all of that done? That’s where the second factor comes in. It is effective strategic leadership. This probably involves the senior library administrator, though it may involve others as well. In my research, I’ve seen counterproductive leadership behavior ranging from passivity to outright undermining of embedded librarianship. On the other hand, I’ve seen examples of active and effective strategic leadership. Here are four key elements of this effective leadership.

32

I promised to close with some observations about other roles for librarians in higher education. That might seem a bit disconnected from the emphasis on evaluation of information literacy instruction in higher education, but it’s not. The connection is that final point about evaluating our work by how we contribute to our parent universities. There’s a general formulation of the mission of institutions of higher learning, and it incorporates three things: teaching, research, and service. So far, all we’ve talked about in this presentation is the teaching mission. It’s important to be sure, but what about research, and service? Increasingly, academic librarians are finding ways to embed themselves in these activities.

33

Many academic libraries are getting involved in curating research data. Embedded librarians are in a position to contribute a great deal to these initiatives. Because of their relationships with faculty and researchers, they understand the data and can contribute their understanding to its management. Another contribution that spans teaching, research, and service is knowledge management. A higher education trend we haven’t talked about today is the trend toward interdisciplinarity. In all aspects of academic life, the integration of disciplines is increasingly important, and librarians are well positioned to contribute. Academic librarians are also able to leverage their embedded relationships to take a leadership role in service.

34

These are just a couple of examples of the trend. Barbara Dewey, then of the University of Tennessee, gave an enduring, comprehensive vision of the potential in her 2004 article in “Resource Sharing & Information Networks.” She wrote: “The embedded librarian, who is truly integrated into the academic, administrative, athletic, cultural, research, teaching, and learning arenas of the university, provides quality and depth to the total campus experience.” That’s what we’re after. So, when you’ve established your embedded information literacy instruction program, and you’ve demonstrated its success in terms of impacts on the university mission – don’t think you’re done. As an academic librarian, you have more opportunities awaiting you!

35

And now I’d like to close by thanking you for your attention. I’d also like to thank Laura and the Credo Reference organization for sponsoring today’s session. If you’d like to know more, here are three ways to follow up: 1. Feel free to email me. I love to hear from those of you who are doing this work, or thinking about it. I’d love to hear from you. 2. Check out my blog at www.embeddedlibrarian.com . I only post when I have time and something to say, so if you subscribe to it you won’t be inundated. 3. And finally, you might be interested in my book, published last summer by Information Today. The second half of it is an in-depth guide to developing and sustaining embedded librarianship, based on my research and experience in the field. I hope it may be helpful to you.

36

Credits

37