Building a Distributed Full-Text Index for the

WebS. Melnik, S. Raghavan, B.Yang, H. Garcia-Molina

• Introduction.

• Testbed architecture.

• Design of the indexer.

• Distributed indexing.

• Introduction.

• Testbed architecture.

• Design of the indexer.

• Distributed indexing.

Pig

Cat

Fish

Cat

Fly

Dog

Pig

Dog

Cat

Fish

Dog

123

Inverted list

Cat-> (1,2), (1,4), (3,2)

Dog->(2,2), (3,1), (3,4)

Fish->(1,3), (3,3)

Pig->(1,1), (2,3)

Inverted

index location

Inverted index consist of an inverted lists for each sorted term.

Inverted list consist of a locations in sorted way.

Location consist of

)page identifier, position in the page.(

Posting consist of (index term, location).

Building an inverted index over a collection of web pages involves:

1 .Processing each page to extract postings.

2 .Building for each term inverted list.

3 .Writing out on disk.

Important problems when building web-scale inverted index:

1 .Scale and growth rate.

2 .Rate of change

• Introduction.

• Testbed architecture.

• Design of the indexer.

• Distributed indexing.

• Distributors.

• Indexers.

• Query servers.

Distributed inverted index organization:

1. Local inverted files.

2. Global inverted files.

Global inverted files

Query server 1Cat->(1,2), (1,4), (3,2)

Dog->(2,2), (3,1), (3,4)a-e

Query server 2 Fish->(1,3), (3,3)

Pig->(1,1), (2,3)f-z

Dog

Cat

Fish

Dog

Fly

Dog

Pig

Pig

Cat

Fish

Cat

2 13

Local inverted files

Query server 1

Cat->(1,2), (1,4)

Dog->(2,2)

Fish->(1,3)

Fly->(2,1)

Pig->(1,1), (2,3)

a-e Query server 2

Cat->(3,2)

Dog->(3,1), (3,4)

Fish->(3,3)

f-z

Dog

Cat

Fish

Dog

Fly

Dog

Pig

Pig

Cat

Fish

Cat

2 13

Local vs. Global

• Resilience to failures.

• Network load.

Testbed environment:

The indexers and the query servers are single processor PC’s with 350-500 MHz processors, 300-500 MB of main memory, and equipped with multiple disks.

All the machines are interconnected by a 100 Mbps Ethernet LAN network.

The WebBase collection:

To study some properties of web pages that are relevant to text indexing, we analyzed 5 samples, of 100,000 pages each, from different portions of the WebBase repository.

Propertyvalue

Average number of words per page438

Average number of distinct words per page171

Average size of each page (as HTML)8650

Average size of each page after removing HTML tags

2815

Average size of a word in the vocabulary8

Table 1: Properties of the WebBase collection

• Introduction.

• Testbed architecture.

• Design of the indexer.

• Distributed indexing.

Design of the Indexer

• Software pipeline.

•The storage of the inverted files generated by the process.

Software pipeline

The process can logically be split into 3 phases:

• Processing -> CPU intensive.

• Flushing -> disk.

• loading -> network.

The goal of our pipelining technique is to design an execution schedule for the different indexing phases that will result in minimal overall running time.

Examples:

L

P

F

Execution of the pipeline

t

Pipeline time

Theoretical analysis vs. experimental results

Design of the Indexer

• Software pipeline.

•The storage of the inverted files generated by the process.

Storage schemes:

We consider ed three storage schemes for storing inverted files as sets of (key, value) pairs in a B-tree:

1 .Full list .

2 .Single payload.

3 .Mixed list.

A qualitative comparison of these storage schemes:

• Index size

• Zig-zag joins

• Hot updates

Zig-zag join using ordered indexes

1 2 3 4 7 9 18

1 7 9 11 1712 19

Experimental results (using mixed list)

Number of pages(million)

Input size (GB)

Index size (GB)

Index size (%age)

0.10.810.056.17

0.54.030.276.70

2.016.111.137.01

5.040.282.786.90

Table 5:Mixed-list scheme index sizes

Only one posting was generated for all the occurrences of a word in a page

• Introduction.

• Testbed architecture.

• Design of the indexer.

• Distributed indexing.

Two problems that must be addressed when building an inverted index on a distributed

architecture:

•Page distribution: The question of when and how to distribute pages to the indexing nodes.

•Collecting global statistics: the question of where, when, and how to compute and distribute global statistics.

Two strategies for page distribution:

• A priori distribution.

• Runtime distribution.

Three advantages of runtime distribution:

• Space.

• Load balancing.

• Effective pipelining.

Collecting global statistics

A dedicated server known as the statistician.

•Parallel computation.

•Minimize the number of conversations among servers.

•Avoid extra disk I/O

•Reduces network overhead.

Two strategies for sending information to the statistician:

• ME Strategy: sending local information during merging.

•FL Strategy: sending local information during flushing.

comparison