Module 3: File Organizations and Indexes

Module Outline

3.1 Comparison of file organizations

3.2 Overview of indexes

3.3 Properties of indexes

3.4 Indexes and SQL

! This is a repetition of materialfrom Information Systems .

Files and Index Structures

Buffer Manager

Disk Space Manager

RecoveryManager

Plan Executor

Operator Evaluator Optimizer

Parser

ApplicationsWeb Forms SQL Interface

SQL Commands

Query Processor

Concurrency ControlDBMS

Database

Index Files

Data FilesSystem Catalog

TransactionManager

LockManager

You are here!

77

I A heap file provides just enough structure to maintain a collection of records (ofa table).

I The heap file supports sequential scans (openScan) over the collection, e.g.

SELECT A,B

FROM R

No further operations receive specific support from the heap file.

I For queries like

SELECT A,B

FROM R

WHERE C > 42

or

SELECT A,B

FROM R

ORDER BY C ASC

it would definitely be helpful if the SQL query processor could rely on a particular

organization of the records in the file for table R.

. File organization for table RWhich organization of records in the file for table R could speed up the

evaluation of the two queries above?

78

This section . . .

I . . . presents a comparison of 3 file organizations:

1 files of randomly ordered records (heap files)2 files sorted on some record field(s)3 files hashed on some record field(s).

I . . . introduces the index concept: A file organization is tuned to make a certain query (class) efficient, but if we

have to support more than one query class, we may be in trouble. Consider:

Q SELECT A,B, C

FROM R

WHERE A > 0 AND A < 100

If the file for table R is sorted on C, this does not buy us anything for query

Q.

If Q is an important query but is not supported by Rs file organization, we

can build a support data structure, an index, to speed up (queries similar to)

Q.79

3.1 Comparison of file organizations

I We will now enter a competition in which 3 file organizations are assessed in 5disciplines:

1 Scan: fetch all records in a given file.2 Search with equality test: needed to implement SQL queries like

SELECT FROM R

WHERE_ _ _

_ _ _C = 42 .

3 Search with range selection: needed to implement SQL queries like (upperor lower bound might be unspecified)

SELECT FROM R

WHERE_ _ _ _ _ _ _ _ _

_ _ _ _ _ _ _ _ _A > 0 AND A < 100 .

4 Insert a given record in the file, respecting the files organization.5 Delete a record (identified by its rid), fix up the files organization if needed.

80

3.1.1 Cost model

I Performing these 5 database operations clearly involves block I/O, the majorcost factor.

I However, we have to additionally pay for CPU time used to search inside apage, compare a record field to a selection constant, etc.

I To analyze cost more accurately, we introduce the following parameters:

Parameter Description

b # of pages in the file

r # of records on a page

D time needed to read/write a disk page

C CPU time needed to process a record (e.g., compare a field value)

H CPU time taken to apply a hash function to a record

I Remarks: D 15ms C H 0.1s This is a coarse model to estimate the actual execution time

(we do not model network access, cache effects, burst I/O, . . .).81

! Aside: HashingA hashed file uses a hash function h to map a given record onto a specific

page of the file.

Example: h uses the lower 3 bits of the first field (of type INTEGER) of

the record to compute the corresponding page number:

h (42, true, "foo") 2 (42 = 1010102)h (14, true, "bar") 6 (14 = 11102)h (26, false, "baz") 2 (26 = 110102)

I The hash function determines the page number only; record placementinside a page is not prescribed by the hashed file.

I If a page p is filled to capacity, a chain of overflow pages is maintained(hanging off page p) to store additional records with h (. . . ) = p.

I To avoid immediate overflowing when a new record is inserted into ahashed file, pages are typically filled to 80% only when a heap file is

initially (re)organized into a hashed file.

(We will come back to hashing later.)

82

3.1.2 Scan

1 Heap fileScanning the records of a file involves reading all b pages as well as processing

each of the r records on each page:

Scanheap = b (D + r C)2 Sorted file

The sort order does not help much here. However, the scan retrieves the records

in sorted order (which can be big plus):

Scansort = b (D + r C)3 Hashed file

Again, the hash function does not help. We simply scan from the beginning

(skipping over the spare free space typically found in hashed files):

Scanhash = (100/80) =1.25

b (D + r C)

. Scanning a hashed fileIn which order does a scan of a hashed file retrieve its records?

83

3.1.3 Search with equality test (A = const)

1 Heap fileThe equality test is (a) on a primary key, (b) not on a primary key :

(a) Searchheap = 1/2 b (D + r C)(b) Searchheap = b (D + r C)

2 Sorted file (sorted on A)We assume the equality test to be on the field determining the sort order. The

sort order enables us to use binary search:

Searchsort = log2 b D + log2 r C(If more than one record qualifies, all other matches are stored right after the

first hit.)

3 Hashed file (hashed on A)Hashed files support equality searching best. The hash function directly leads us

to the page containing the hit (overflow chains ignored here):

(a) Searchhash = H +D + 1/2 r C(b) Searchhash = H +D + r C

(All qualifying records are on the same page or, if present, in its overflow chain.)84

3.1.4 Search with range selection (A lower AND A upper)1 Heap file

Qualifying records can appear anywhere in the file:

Rangeheap = b (D + r C)

2 Sorted file (sorted on A)Use equality search (with A = lower), then sequentially scan the file until a

record with A > upper is found:

Rangesort = log2 b D + log2 r C +n/r D + n C

(n denotes the number of hits in the range)

3 Hashed file (sorted on A)Hashing offers no help here as hash functions are designed to scatter records all

over the hashed file (e.g., h (7, . . . ) = 7, h (8, . . . ) = 0) :

Rangehash = 1.25 b (D + r C)

85

3.1.5 Insert

1 Heap fileWe can add the record to some arbitrary page (e.g., the last page). This involves

reading and writing the page:

Insertheap = 2 D + C

2 Sorted fileOn average, the new record will belong in the middle of the file. After insertion,

we have to shift all subsequent records (in the latter half of the file):

Insertsort = log2 b D + log2 r C search

+ 1/2 b (2 D + r C) shift latter half

3 Hashed fileWe pretend to search for the record, then read and write the page determined

by the hash function (we assume the spare 20% space on the page is sufficient

to hold the new record):

Inserthash = H +D search

+ C +D

86

3.1.6 Delete (record specified by its rid)

1 Heap fileIf we do not try to compact the file (because the file uses free space management)

after we have found and removed the record, the cost is:

Deleteheap = Dsearchby rid

+ C +D

2 Sorted fileAgain, we access the records page and then (on average) shift the latter half

the file to compact the file:

Deletesort = D + 1/2 b (2 D + r C) shift latter half

3 Hashed fileAccessing the page using the rid is even faster than the hash function, so the

hashed file behaves like the heap file:

Deletehash = D + C +D

87

I There is no single file organization that serves all 5 operations equally fast.Dilemma: more advanced file organizations make a real difference in speed!

range selections for increasing file sizes (D = 15ms, C = 0.1s, r = 100, n = 10):

0.01

0.1

1

10

100

1000

10000

10 100 1000 10000 100000

time

[s]

b [pages]

sorted fileheap/hashed file

deletions for increasing file sizes (as above, n = 1):

0.01

0.1

1

10

100

1000

10000

10 100 1000 10000 100000

time

[s]

b [pages]

sorted fileheap/hashed file

I There exist index structures which offer all the advantages of a sorted file andsupport insertions/deletions efficiently1: B+ trees.

1At the cost of a modest space overhead.88

3.2 Overview of indexes

I If the basic organization of a file does not support a particular operation, we canadditionally maintain an auxiliary structure, an index, which adds the needed

support.

I We will use indexes like guides. Each guide is specialized to accelerate searcheson a specific attribute A (or a combination of attributes) of the records in its

associated file:

1 Query the index for the location of a record with A = k (k is the searchkey),

2 The index responds with an associated index entry k(k contains enough information to access the actual record in the file),

3 Read the actual record by using the guiding information in k;the record will have an A-field with value k .2

k1

// index2// k

3// . . . , A = k, . . .

2This is true for so-called exact match indexes. In the more general case with similarity

indexes, the records are not guaranteed to contain the value k, they are only candidates for

having this value.89

I We can design the index entries, i.e., the k, in various ways:

Variant Index entry ka

k, . . . , A = k, . . .

bk, rid

c

k, [rid1, rid2, . . . ]

I Remarks:

With variant a, there is no need to store the data records in addition theindexthe index itself is a special file organization.

If we build multiple indexes for a file, at most one of these should use variant

a to avoid redundant storage of records. Variants b and c use rid(s) to point into the actual data file. Variant c leads to less index entries if multiple records match a search key k ,but index entries are of variable length.

90

Example:3

I The data file contains name, age, sal records, the file itself (index entry varianta) is hashed on field age (hash function h1).

I The index file contains sal, rid index entries (variant b), pointing into the datafile.

I This file organization + index efficiently supports equality searches on the ageand sal keys.

2 121

data filehashed on age

index file

h(age)=2

h(age)=0

h(age)=1

h(sal)=3

h(sal)=0

salage

3

entries

Jones, 40, 6003Tracy, 44, 5004

Basu, 33, 4003

Cass, 50, 5004

Smith, 44, 3000

Ashby, 25, 3000

6003

6003

2007

4003

3000

3000

5004

5004

Daniels, 22, 6003

h1 h2

Bristow, 29, 2007

3 1 . . . 3 refer to the index lookup scheme at the beginning of this section.91

3.3 Properties of indexes

3.3.1 Clustered vs. unclustered indexes

I Suppose, we have to support range selections on records such that lower A upper for field A.

I If we maintain an index on the A-field, we can1 query the index once for a record with A = lower , and then2 sequentially scan the data file from there until we encounter a record with field

A > upper .

I This will work provided that the data file is sorted on the field A:

(k*)

+B tree

index file

data filedatarecords

index entries

92

I If the data file associated with an index is sorted on the index search key, theindex is said to be clustered.

I In general, the cost for a range selection grows tremendously if the index on Ais unclustered. In this case, proximity of index entries does not imply proximity

in the data file. As before, we can query the index for a record with A = lower . To continue the

scan, however, we have to revisit the index entries which point us to data pages

scattered all over the data file:

(k*)

B tree+

index entries

index file

datarecords

data file

I Remarks: If the index entries (k) are of variant a, the index is obviously clustered by defini-tion.

A data file can have at most one clustered index (but any number of unclustered

indexes).93

3.3.2 Dense vs. sparse indexes

I A clustered index comes with more advantages than the improved speed forrange selections presented above. We can additionally design the index to be

space efficient:

To keep the size of the index file small, we maintain one index entry k perdata file page (not one index entry per data record). Key k is the smallest

search key on that page.

Indexes of this kind are called sparse (otherwise indexes are dense).

I To search a record with field A = k in a sparse A-index, we

1 locate the largest index entry k such that k k , then2 access the page pointed to by k , and3 scan this page (and the following pages, if needed) to find records with. . . , A = k, . . . .Since the data file is clustered (i.e., sorted) on field A, we are guaranteed to

find matching records in the proximity.

94

Example:

I Again, the data file contains name, age, sal records. We maintain a clusteredsparse index on field name and an unclustered dense index on field age. Both

use index entry variant b to point into the data file:Ashby, 25, 3000

Smith, 44, 3000

Ashby

CassSmith

22

25

30

4044

44

50

Sparse Indexon

Name Data File

Dense Indexon

Age

33

Bristow, 30, 2007Basu, 33, 4003

Cass, 50, 5004

Tracy, 44, 5004

Daniels, 22, 6003

Jones, 40, 6003

I Remarks: Sparse indexes need 23 orders of magnitude less space than dense indexes. We cannot build a sparse index that is unclustered (i.e., there is at most one

sparse index per file).

. SQL queries and index exploitationHow do you propose to evaluate query

`SELECT MAX(age) FROM employees

?

How about`SELECT MAX(name) FROM employees

?

95

3.3.3 Primary vs. secondary indexes

. TerminologyIn the literature, you may often find a distinction between primary

(mostly used for indexes on the primary key) and secondary (mostly

used for indexes on other attributes) indexes. The terminology, however,

is not very uniform, so some text books may use those terms for different

properties.

You might as well find primary denoting variant 1 of indexes accordingto Section 3.2, while secondary may be used to characterize the other

two variants 2 and 3.

96

3.3.4 Multi-attribute indexes

Each of the index techniques sketched so far and discussed in the sequel can be

applied to a combination of attribute values in a straightforward way:

I concatenate indexed attributes to form an index key,e.g. lastname,firstname 7 searchkey

I define index on searchkey

I index will support lookup based on both attribute values,e.g. . . . WHERE lastname=Doe AND firstname=John . . .

I possibly will also support lookup based on prefix of values,e.g. . . . WHERE lastname=Doe . . .

There are more sophisticated index structures that can provide support for symmetric

lookups for both attributes alone (or, to be more general, for all subsets of indexes

attributes). These are often called multi-dimensional indexes. A large number of

such indexes have been proposed, especially for geometric applications.

97

3.4 Indexes and SQL

The SQL-92 standard does not include any statement for the specification (creation,

dropping) of index structures. In fact, SQL does not even require SQL systems to

provide indexes at all!

Nonetheless, practically all SQL implementations support one or more kinds of in-

dexes. A typical SQL statement to create an index would look like this:

I Index specification in SQL1 CREATE INDEX IndAgeRating ON Students

2 WITH STRUCTURE = BTREE,

3 KEY = (age,gpa)

N.B. SQL-99 does not include indexes either.

98

Bibliography

Elmasri, R. and Navathe, S. (2000). Fundamentals of Database Systems. Addison-Wesley,

Reading, MA., 3 edition. Titel der deutschen Ausgabe von 2002: Grundlagen von

Datenbanken.

Harder, T. (1999). Datenbanksysteme: Konzepte und Techniken der Implementierung.

Springer.

Heuer, A. and Saake, G. (1999). Datenbanken: Implementierungstechniken. Intl Thomp-

son Publishing, Bonn.

Mitschang, B. (1995). Anfrageverarbeitung in Datenbanksystemen - Entwurfs- und Imple-

mentierungsaspekte. Vieweg.

Ramakrishnan, R. and Gehrke, J. (2003). Database Management Systems. McGraw-Hill,

New York, 3 edition.

99