Search engines and digital libraries

8/14/2019 Search engines and digital libraries

http://slidepdf.com/reader/full/search-engines-and-digital-libraries 1/42

© Tefko Saracevic 1

part 1: search engines

part 2: digital

libraries




dictionary definitions

searchCOMPUTING (transitive verb) to examine a computer

file, disk, database, or network for particular

information

enginesomething that supplies the driving force or energy to

a movement, system, or trend

search enginea computer program that searches for particular

keywords and returns a list of documents in whichthey were found, especially a commercial servicethat scans documents on the Internet




about definition of search

engines• oh well …

search engines do not search only for

keywords, some search for other stuff as well

• and they are really not “engines” in theclassical sensebut then mouse is not a “mouse”




use of search engines… among others




YourBrowser

How Search Engines Work(Sherman 2003)

The Web

URL1

URL2

URL3 URL4

Crawler

Indexer

SearchEngine

Database Eggs?Eggs.

Eggs - 90%

Eggo - 81%

Ego- 40%

Huh? - 10%

All AboutEggsby

S. I. Am




how do search engineswork? elaboration

• crawlers, spiders: go out to findcontent in various ways go through the web

looking for new & changed sitesperiodic, not for each query

no search engine works in real time

some search engines do it for themselves,

others not buy content from companies such as Inktomi

for a number of reasons crawlers do notcover all of the web – just a fraction

what is not covered is “invisible web”


http://slidepdf.com/reader/full/search-engines-and-digital-libraries 7/42© Tefko Saracevic 7

elaboration …

• organizing content: labeling, arranging indexing for searching – automatic

keywords and other fields

arranging by URL popularity - PageRank as Googleclassifying as directory

mostly human handpicked & classified

• as a result of different organization we

have basically two kinds of searchengines:

search – input is a query that is then searched &displayed

directory – classified content – a class is displayed–



elaboration (cont.)

• databases, caches: storing content humongous files usually distributed over many

computers

• query processor: searching, retrieval,display takes your query as input

engines have differing rules how handled

displays ranked output some engines also cluster output and provide

visualization

• at the other end is your browser



elaboration…similarities, differences

• all search engines have these basicparts in common

• BUT the actual processes – methodshow they do it – are based on variousalgorithms & they differ

most are proprietary with details kept

mostly secret but based on well knownprinciples from information retrieval orclassification

to some extent Google is an exception –

they published their method



case of

• developed by Sergey Brin andLawrence Page while students atStanford in the beginning run on Stanford computers

• basic approach has been described intheir famous paper“The Anatomy of a Large-ScaleHypertextual Web Search Engine”

well written, simple language, has their pictures in acknowledgement they cite the support by NSF’sDigital Library Initiative i.e. initially, Google cameout of government sponsored research

describe their method PageRank - based on rankinghyperlinks as in citation indexing

“We chose our system name, Google, because it is acommon spelling of googol, or ten on hundredth”

http://www-db.stanford.edu/~backrub/google.html







coverage differences

• no engine covers more than a fraction of WWW estimates: none more than 16%

hard (even impossible) to discern & compare coverage, butthey differ substantially in what they cover

• in addition: many national search engines

own coverage, orientation, governance many specialized or domain search engines

own coverage geared to subject of interest

many comprehensive sources independent of searchengines

some have compilations of evaluated web sources



searching differences

• substantial differences among searchengines on searching, retrieval display

need to know how they work & differ inrespect to defaults in searching a query

searching of phrases, case sensitivity, categories

searching of different fields, formats, types of

resources advance search capabilities and features

possibilities for refinement, using relevancefeedback

display options

personalization options




business model

differencesseveral business models

• public good - have independent

budget e.g. PubMed, Librarians’ Index to Internet

• earn revenue from provision of information all commercial search engines

• using search engines to promote theirother activities

e.g. telephone directories

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi

http://lii.org/

http://lii.org/

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi




sponsorship differences

• need to understand treatment of sponsorship – they influence what theysearch & how they display results

some list separately results fromsponsored sites so you are reasonablyclear what is there because it is sponsored& not

some have display-per-pay - showing first

sites that paid most & do not even tell youthat

some have pay per update of sites

• imperative to find sources that explain

these models for different engines to




limitations

• every search engine has limitation as tocoverage

meta engines just follow coverage limitations & have

more of their ownsearch capabilities

finding quality information

• some have compromised search with

economicsbecoming little more than advertisers

• but search engines are also many times

victims of spamindexingaffectin what is included and how ranked




spamming a search

engine• use of techniques that push rankings

higher than they belong is also called

spamdexingmethods typically include textual as well as

link-based techniques

like e-mail spam, search engine spam is a

form of adversarial information retrieval the conflicting goals of accurate results of search

providers & high positioning by content pagerank




meta search engines

• meta engines search multipleengines

getting combined results from avariety of engines

• do not have their own databases

but have their own business modelsaffecting results

• a number of techniques usedinteresting ones: clustering, statistical

analyses




how to find a search

engine?• variety of resources that list or categorizeengines

• SearchEngines.com

search for engines by topic, geography, referenceSearch Engine Guideengines categorized by topic; other engine information

Search Engine Colossus

international directory of search engines by country,topicfrom 198 countries and 61 territories; engines in choice of

languages

Phil Bradley’s country based search enginover 2000 serach engines from countries all over the globe

http://www.searchengines.com/

http://www.searchengineguide.com/

http://www.searchenginecolossus.com/

http://www.philb.com/countryse.htm

http://www.philb.com/countryse.htm



http://www.searchengineguide.com/

http://www.searchengines.com/




sample of meta engines- with organized results

Dogpile

results from a number of leading search engines;gives source, so overlap can be compared; (hasalso a (bad) joke of the day)

Surfwax

gives statistics and text sources & linking tosources; for some terms gives related terms tofocus

Teomaresults with suggestions for narrowing; links

resources derived; originated at Rutgers

Turbo10

provides results in clusters; engines searched canbe edited

http://www.dogpile.com/

http://www.surfwax.com/

http://teoma.com/

http://turbo10.com/

http://turbo10.com/

http://teoma.com/








meta search engines

(cont.)• Large directory

Complete Planet directory of over 70,000 databases & specialty

engines

• Results with graphical displays Vivisimo

clusters results; innovative

Webbrain results in tree structure – fun to use

Kartoo

results in display by topics of query

http://completeplanet.com/

http://vivisimo.com/

http://www.webbrain.com/html/default_win.html

http://www.kartoo.com/

http://www.kartoo.com/





http://completeplanet.com/




domain engines &

catalogs• cover specific subjects & topics

• important tool for subject searches

particularly for subject specialistvalued by professional searchers

• selection mostly hand-picked ratherthan by crawlers, following inclusioncriteriaoften not readily discernable

but content more trustworthy




domain engines … sample

Open Directory Project large edited catalog of the web – global, run by

volunteers

BUBL LINK selected Internet resources covering all academic

subject areas; organized by Dewey Decimal System– from UK

Profusion search in categories for resources & search engines

Resource Discovery Network – UK “UK's free national gateway to Internet resources

for the learning, teaching and research”

http://dmoz.org/

http://bubl.ac.uk/link/

http://www.profusion.com/

http://www.rdn.ac.uk/



http://www.profusion.com/



http://dmoz.org/




domain engines … sample

Think Quest – Oracle Education Foundation • education resources, programs; web sites created by students

All Music Guide• resource about musicians, albums, and songs

Internet Movie Database• treasure trove of American and British movies

Genealogy links and surname search engineswell.. that is getting really specialized (and popular)

Daypopsearches the “living web” “The living web is composed of sites

that update on a daily basis: newspapers, online magazineand weblogs”

http://www.thinkquest.org/

http://allmusic.com/

http://www.imdb.com/

http://darcisplace.com/darci/search.htm

http://www.daypop.com/

http://www.daypop.com/

http://darcisplace.com/darci/search.htm

http://www.imdb.com/








science, scholarshipengines …sample free

access Psychcrawler - Amer Psychological

Association web index for psychology

Entrez PubMed – Nat Library of Medicinebiomedical literature from MEDLINE & health

journals

CiteSeer - NEC Research Center scientific literature, citations index; strong in

computer science

Scholar Googlesearches for scholarly articles & resources

Infominescholarly internet research collections

Scirusscientific information in journals & on the web

http://www.psychcrawler.com/

http://www.ncbi.nlm.nih.gov/PubMed/


http://citeseer.nj.nec.com/

http://scholar.google.com/

http://infomine.ucr.edu/

http://www.scirus.com/srsapp/

http://www.scirus.com/srsapp/

http://infomine.ucr.edu/

http://scholar.google.com/












science, scholarshipengines …sample

commercial access• an addition to freely accessible engines

many provide search free but access tofull text paid

by subscription or per itemRUL provides access to these & many

more:

ScienceDirect

Elsevier: “world's largest electronic collection of science,technology and medicine full text and bibliographic

information” ACM PortalAsoc. for Computing Machinery: access to ACM Digital Library &

Guide to Computing

http://www.sciencedirect.com/

http://portal.acm.org/portal.cfm

http://portal.acm.org/portal.cfm

http://www.sciencedirect.com/




where to find out?

• information about search engines insources that have updates, news, tipsfor searching and more – a MUST forsearchers : Search Engine Watch

ratings, news, statistics, charts, explanations,tutorials

Search Engine Showdown “The users’ guide to web searching” - run by a

librarian, news links, ratings

Virtual Chase a site about “Teaching Legal Professionals How To

Do Research;,” this section has very good tipsand links for consideration of quality on the web

http://searchenginewatch.com/

http://searchengineshowdown.com/


http://www.virtualchase.com/quality/

http://www.virtualchase.com/quality/








where? ….

SiteLines

a blog, written by Rita Vine, a professionallibrarian, & web search trainer; many

evaluations in archiveResourceShelf

“Resources and News for InformationProfessionals,” edited by Gary Price, a librarian &author of Invisible Web – has extensive archive

WebsearchAbout

not evaluative, but provides news, capabilities,sources, articles about web searching

http://www.workingfaster.com/sitelines/

http://www.resourceshelf.com/

http://websearch.about.com/

http://websearch.about.com/

http://www.resourceshelf.com/

http://www.workingfaster.com/sitelines/




art of searching search

engines




part 2: digital libraries




definition

• digital libraries are viewed from severalperspectivestechnical: “Digital library is a managed

collection of information, with associatedservices, where information is stored in digitalformat and accessible over a network.” (Arms,2000)

institutional: “Digital libraries are

organizations that provide the resources,including the specialized staff, to select,structure, offer intellectual access to,interpret, distribute, preserve the integrity of,and ensure the persistence over time of

collections of digital works so that they are




a bit of context

• short but volatile history research & development took of by start/mid

1990’s

in the next decade phenomenal growthworldwide

large investment in research & building

• number of communities involvedcomputer science, primarily in researchmany subjects: digital libraries in their

domain

library & information science: operations,

studies of users, use, usability




libraries & digital

resources• libraries (particularly research, academic &

special) directed massive funding toward

such resourceselectronic journals

databases

catalogs

digitization of parts of collection

• thus becoming in effect digital libraries –or more accurately hybrid librarieswith graphic and digital versions or types of

resources




emphasis here

• on large academic or research digitallibraries that also are related to

searchingprovide search capabilities or access to

search engines

provide electronic journals that provide full

text of articles after a search• such libraries have become also search

portals of sort, essential for their users in education, research & related activities




sample

New York Public Library Digital“NYPL Digital is your gateway to The New York Public Library’s

rare and unique collections in digitized form.” Includesaccess to searchable databases

U California Berkeley Digital LibrarySUNsite“builds digital collections and services while providing

information and support to digital library developers

worldwide.

The British Library“The world’s knowledge.” Includes “Services fro library and

information Professionals.”

Los Angeles Public Library Kids’ Pathresources for children; search through directory

http://www.nypl.org/digital/

http://sunsite.berkeley.edu/


http://www.bl.uk/

http://www.lapl.org/kidspath/

http://www.lapl.org/kidspath/

http://www.bl.uk/



http://www.nypl.org/digital/




sample …

New Zealand Digital Librarysearching of a number of digital collections, including

humanity development library

Research Library Group“RLG is a not-for-profit organization of over 150 research

libraries, archives, museums, and other cultural memoryinstitutions.” Includes links to a number of searchablecollections

Public Library of Science“PLoS is a nonprofit organization of scientists and physicians

committed to making the world's scientific and medicalliterature a public resource.” Publishes open access journals

http://www.sadl.uleth.ca/nz/cgi-bin/library

http://www.rlg.org/

http://www.plos.org/index.html

http://www.plos.org/index.html

http://www.rlg.org/

http://www.sadl.uleth.ca/nz/cgi-bin/library




Rutgers libraries – digital components

• strategic planning in developing digitalaccess

• rich & complex content of digitalresourcesseveral hundred indexes & databases for

searching

some 20,000 electronic journals

thousand & more digital reference sources

subject research guides

Searchpath & other tutorials

electronic reserve

• affected teachin , learnin , research b

http://www.libraries.rutgers.edu/

http://www.libraries.rutgers.edu.proxy.libraries.rutgers.edu/rul/staff/strategic_planning/strategic_planning.shtml

http://www.libraries.rutgers.edu.proxy.libraries.rutgers.edu/rul/staff/strategic_planning/strategic_planning.shtml

http://www.libraries.rutgers.edu/




some critical issues for

searching• no way yet to do federated searching

in digital libraries

to search several indexes at the same timeeach source has to be searched separately

most have very different search features,capabilities

• finding items in indexes does not meanthat always able to get full text

• thus, searching time-consuming,chaotic




where to find out?

• information about digital librariesLibWeb U California, Berkeley“lists currently over 7200 pages from libraries in over 125

countries”Digital Library Federation“a consortium of libraries and related agencies that are

pioneering the use of electronic-information technologies

to extend their collections and services”

D-Lib Magazine“a solely electronic publication with a primary focus on

digital library research and development, including butnot limited to new technologies, applications, andcontextual social and economic issues”

http://lists.webjunction.org/libweb/

http://www.diglib.org/

http://www.dlib.org/

http://www.dlib.org/

http://www.diglib.org/

http://lists.webjunction.org/libweb/




where? …

Ariadne (UK)“to report on information service developments and

information networking issues worldwide, keepingthe busy practitioner abreast of current digital library

initiatives” Information Technology and Libraries

ALA publication; “related to all aspects of libraries andinformation technology, including digital libraries”

Journal of Digital Information

“Publishing papers on the management, presentationand uses of information in digital environments”

Biblio Tech Review“Information Technology for Libraries” – monthly news

and review magazine

http://www.ariadne.ac.uk/

http://www.ala.org/ala/lita/litapublications/ital/italinformation.htm

http://jodi.tamu.edu/

http://www.biblio-tech.com/




http://jodi.tamu.edu/

http://www.ala.org/ala/lita/litapublications/ital/italinformation.htm






in conclusion

• search engines are great but you haveto KNOW what is under the hoodas to coverage, business model, search

features, outputs … they are NOT for every kind of information

need

• digital libraries are great for searching

but you have to KNOW requirementsfor searching different resources thatare included there is no federated searching as yet, or

for the time to come




art of searching digital

libraries



and rewards …

Search engines and digital libraries

Documents

Transcript of Search engines and digital libraries