1

Update on the NISOOpen Discovery

InitiativeMarshall Breedinghttp://www.librarytechnology.org/http://twitter.com/mbreeding November 20, 2013

NISO Virtual Conference: Web-Scale Discovery Services: Transforming Access to Library Resources

Index-based DiscoverySearch:

Digital Collections

Web Site Content

Institutional

Repositories

…E-Journals

Reference Sources

Search Results

Pre-built harvesting and indexing

Consolidated Index

ILS Data

Aggregated Content packages

(2009- present)

Usage-generate

dData

Customer

Profile

Open Access

Bento Box Discovery Model

Search:

Digital Collections

Web Site Content

Institutional

Repositories

E-JournalsSearch Results

Central index & search functionality

Consolidated Index

ILS Data

Aggregated Content packages

Open AccessVuFind /

Blacklight

API

Web-scale search problemSearch:

Search Results

Pre-built harvesting and

indexing

Consolidated

Index

???

Non Participating

Content Sources

Problem in how to deal with resources not provided to ingest into consolidated index

Digital Collections

Web Site ContentInstitution

al Repositori

es

…E-Journals

ILS Data

Aggregated Content packages

Discovery Concerns• Important space for libraries and

publishers• Discovery brings value to library

collections• Discovery brings uncertainty to

publishers• Uneven participation diminishes

impact• Ecosystem dominated by private

agreements • Complexity and uncertainty poses

barriers for participation

5

Heterogeneous Representations• Content objects represented by

– MARC Records for books and journal titles

– Citation data for articles– Full text for articles– Full text for books– Abstracts and Indexing products– Other metadata or enrichment

Discovery index issues• Citations or structured metadata

provide key data to power search & retrieval and faceted navigation

• Indexing full-text of content amplifies access

• Important to understand what is indexed– Currency, dates covered, full-text or

citation– Many other factors 7

Library Perspective• Strategic investments in subscriptions• Strategic investments in Discovery Solutions to

provide access to their collections• Expect comprehensive representation of resources in

discovery indexes– Problem with access to resources not represented in index– Encourage all publishers to participate and to lower

thresholds of technical involvement and clarify the business rules associated with involvement

• Need to be able to evaluate the coverage and performance of competing index-based discovery products

Collection Coverage?• To work effectively, discovery services

need to cover comprehensively the body of content represented in library collections

• Why do some content providers not participate?

• How are A&I resources represented?• Is content indexed at the citation or full-

text level?• What are the restrictions for non-

authenticated users?• How can libraries understand the

differences in coverage among competing services?

Evaluating the Coverage of Index-based Discovery Services

• Intense competition: how well the index covers the body of scholarly content stands as a key differentiator

• Difficult to evaluate based on numbers of items indexed alone.

• Important to ascertain how your library’s content packages are represented by the discovery service.

• Important to know what items are indexed by citation and which are full text

Some Key Areas for Publishers1. Expose content appropriately2. Trust that access to material will be

controlled consistent with subscription terms

3. “Fair” Linking4. Materials not disadvantaged or

underrepresented in library discovery implementations

5. Usage reporting

Facilitate a healthy ecosystem among

discovery service providers, libraries and content

providers

ODI context

ODI Pre-History• June 26, 2011: Exploratory meeting

@ ALA Annual• July 2011: NISO expresses interest• Aug 7, 2011: Proposal drafted by

participants submitted to NISO• Aug 2011: Proposal accepted by D2D• Vote of approval by NISO

membership• Oct 2011: ODI launched• Feb 2012: ODI Workgroup Formed 13

Organization• Reports in NISO through Document to

Delivery topic committee (D2D)• Staff support from NISO through

Nettie Lagace• Co-Chairs

– Jenny Walker (Ex Libris)– Marshall Breeding (Library Consultant)

• D2D Observers: Jeff Penka (OCLC)Lucy Harrison (CCLA)

14

ODI Timeline

Milestone Target Date

Status

Appointment of working group Dec 2011

Approval of charge and initial work plan Mar 2012

Agreement on process and tools Jun 2012

Completion of information gathering Jan 2013

Completion of initial draft Jun 2013

Completion of final draft Sep 2013

Public Review Period commences Sep 2013 15

Balance of ConstituentsLibraries

Publishers

Service Providers

16

Marshall Breeding, Vanderbilt UniversityJamene Brooks-Kieffer, Kansas State University Laura Morse, Harvard UniversityKen Varnum, University of Michigan

Sara Brownmiller, University of Oregon

Lucy Harrison, College Center for Library Automation (D2D liaison/observer)Michele Newberry

Lettie Conrad, SAGE PublicationsRoger Schonfeld, ITHAKA/JSTOR/PorticoJeff Lang, Thomson Reuters

Linda Beebe, American Psychological AssocAaron Wood, Alexander Street Press

Jenny Walker, Ex Libris GroupJohn Law, Serials SolutionsMichael Gorrell, EBSCO Information ServicesDavid Lindahl, University of Rochester (XC)

Jeff Penka, OCLC (D2D liaison/observer)

ODI Project Goals:• Identify … needs and requirements of the three

stakeholder groups in this area of work. • Create recommendations and tools to streamline

the process by which information providers, discovery service providers, and librarians work together to better serve libraries and their users.

• Provide effective means for librarians to assess the level of participation by information providers in discovery services, to evaluate the breadth and depth of content indexed and the degree to which this content is made available to the user.

Subgroups for Info Gathering• Level of Indexing + Communication

of Library Rights• Technical formats• Usage Statistics• Fair Linking

18

Specific deliverables• Standard vocabulary• NISO Recommended Practice:

– Data format & transfer– Communicating content rights– Levels of indexing, content availability– Linking to content– Usage statistics– Evaluate compliance

• Inform and Promote Adoption19

ODI Stakeholder Survey• Collected data from Sept 11 thru Oct

4, 2012• Each subgroup developed questions

pertinent to it area of concern

20

Survey Responses• 782 Librarians• 74 Publishers• 15 Discovery Services• 871 Total

21

Selected results• Libraries: do you use a discovery

service?– Yes: 74%, Planning to soon: 17%, No:

5%, Don’t know: 4%• Smallest discoverable unit:

– Component title: 9%, Article: 25%, Collective work record: 11%, All the above: 50%

• Linking from A&I entry: 75 prefer linking to full text on original publisher’s server 22

Librarian’s preferred Use statistics• Total Number of Searches• List of search query terms• Referring URLs

23

Content providers (74)• Contribute data: Yes-All: 44%, Some:

48%, No: 8%– Current data: 12%, Current + back files:

85• Barriers to contributing:

– IP concerns, technology, staff resources• Challenges in delivery:

– Complicated formats: 15%, transmission of data: 18, allocation of personnel: 23%, can’t automate: 12%, None: 20%

24

Issues surrounding A&I resources• Concern that A&I resources not be

freely available to non authenticated users and only for subscribing institutions

• How to “credit” A&I data that contributes to search results– Example: Index entry produced by

enhancing full-text with A&I data• Preservation of the value added by

A&I in the discovery ecosystem 25

ODO Survey Report• Issued January 2013• NOT the final report for ODI• Survey findings, especially for those

that responded to survey• One source of input for the ODI final

report of findings and recommended practices

26

ODI Final Report• Issued for public Comment• Comment period closed November

18, 2013

27

Report Topics• Introduction

– In scope / out of scope– Terms and definitions

• Evolution of Discovery– Related initiatives

• Recommendations

28

General Recommendations• Create oversight group• Actions for content providers and

discovery service creators to assert conformance

29

Recommendations for Content Providers• Content providers should make items

available to discovery service providers. – Basic: Citations: specific metadata

elements– Enhanced: additional metadata + Full-

text• Provide to Libraries: disclosure of

participation in discovery services30

Recommendations for Discovery Service Creators• Disclosure of content indexed

– Specific metadata fields• Fair / non-biased linking

– Mechanisms for libraries to choose versions preferred for linking

– Annual statement regarding neutrality of linking or relevance

– Provide links to A&I services when applicable

• Usage statistics to Publishers– Searches– Result clicks– Click-throughs– Link source identification

31

Report Highlights• What is in and out of Scope

– Focus on content available to be indexed– Quantity and form of content (citations /

fulltext)– Metadata fields contributed– Role of A&I products– Controlled Vocabularies

• Out of Scope– Relevancy algorithms– User Interface issues– APIs exposed– Fair linking

32

Technical recommendations• Transfer of data from content

providers to discovery service creators– Make use of existing standards and

protocols when possible

33

Current work Next Steps• Review comments received

– Chairs + Workgroup members– Make any needed revisions– Submit for final approval by NISO D2D

34

Connect with ODI• ODI Project website:

http://www.niso.org/workrooms/odi/

• Interest group mailing list:http://www.niso.org/lists/opendiscovery/

• Email ODI:odi@niso.org

35