Download - ResourceSync in 24x7

“Synchronize your resources with ResourceSync”July 10, 2013, Open Repositories 2013, PEI, Canada

Synchronize your resources with ResourceSync

Simeon Warner(Cornell University Library)

1

“Synchronize your resources with ResourceSync”July 10, 2013, Open Repositories 2013, PEI, Canada 2

Team sport


more, still more missing

JISC

Richard JonesGraham Klyne

Stuart Lewis

OCLC

Jeff Young

LOCKSS

David Rosenthal

RedHat

Christian Sadilek

Ex Libris Inc.

Shlomo Sanders

Library of Congress

Kevin Ford


$Alfred P. Sloan

Foundation


Synchronize• keep “in sync” (colloq.)

• Following changes over timeand

• Keeping copies on different systems the same

• Tackle only the unidirectional problem:

From a Source, to a Destination


Resourcesaka Web Resources:

have URI, HTTP GET representation(s)

Many / Few Big / Small Fast / Slow


Why?


Scholarly repositories• Replicate data/articles for mirroring, reuse,

indexing, ...• OAI-PMH for metadata• Many custom solutions

for full content


Linked dataFundamentally distributed but local copy often required. Either:

1. cache

2. sync local copy...

• Many custom solutions

for local copy

Last.FM

MusicBrainz

GeoNames

DBpedia

others...

BBC


Didn’t you sell us OAI-PMH?

Or... will ResourceSync replace OAI-PMH?

Proven metadata transfer protocolWidely adopted in our community

X Predates REST, not “of the web”X Not adopted for content transfer

Can replace, likely coexistence


What?


1. Baseline syncInitial load, copy, or catch-up from source• need list of all resources• optional packaged content

Want to• avoid out-of-band setup & customization


2. Incremental syncKeep up-to-date with changes at a source• need information about changes• optional packaged content• minimal primitives: create/update/delete

Want• allow catch-up after destination offline• lower latency and/or greater efficiency than

repeated baseline sync


3. AuditDestination should be able to verify whether it is synchronized with a source• need list of all resources + fixity info

Want• lower latency and/or greater efficiency than

baseline sync• note: subject to some latency


How?


All ResourceSync documents are

Sitemaps with

minor

extensions


Minor?<urlset xmlns=“http://www.sitemaps.org/schemas/sitemap/0.9” xmlns:rs="http://www.openarchives.org/rs/terms/”> <rs:ln …/> <rs:md …/> <url> <loc>http://example.com/res1</loc> <lastmod>2013-01-02T13:00:00Z</lastmod> <rs:ln …/> <rs:md …/> </url> <url> … </url></urlset>


Baseline sync & Google

Most basic capability is Resource List:• Snapshot of state of resources• URI, datestamp + optional extra fixity info• Destination does GET on each resource

ResourceSync Baseline sync & Audit

Google/Bing/Yahoo!/etc. harvest

19

Modular

Discovery

Four CoreCapabilities

1 2 3 4


ExtensibleExtensible use of Link Relations from Atom• Spec describes use for mirrors, patches,

historical, provenance, conneg...• Use <rs:ln rel=“your-relation-here” .../>

Extensible attributes for fixity etc.• Includes lastmod, fixity, length, type...

Extensible framework -> new capabilities


Push = Lower latency Pull• easy setup, no trust required

Push Changes• lower latency, better scaling• same descriptions as pull• standard transports (XMPP, Websockets...)• can push discovery info to trigger pull


Timeline January 2013

June 2013

July 2013

Fall 2013

• Tools and libraries being developed to ease implementation

First betaVersion 0.9 Update and push spec NISO standardization

• Tutorials at major conferences (OAI8, OR, JCDL,...)

22

23

http://www.openarchives.org/rs/

• Framework• Archives• Push (to come)

• Links to Google group, associated articles, blogs, etc.

24

That’s all

folks