1

PART IIISTORAGE AND INDEXING

CH. 8: OVERVIEW OF STORAGE AND INDEXING (introduction)CH. 9: STORING DATA: DISKS AND FILES (hardware)

CH. 10: TREE-STRUCTURED INDEXINGCH. 11: HASH-BASED INDEXING(two important types of indexing)

2

APPROACH

• NO FORMAL UNIFIED THEORY.

• PRESENTATION COVERS KEY IDEAS IN AD HOC FASHION.

• BASIC METHODOLOGY:

SOFTWARE ENGINEERING

3

SOFTWARE ENGINEERING

Engineering as Optimization or Rather as Nonlinear Programming

The goal of engineering is to create a design for an object or a system so as to optimize

a specified Criterion Function subject to certain Constraints.

4

FORMAL STATEMENT OF THE GOAL OF ENGINEERING•

• Max (CF( x1, x2, ... , xn, p1, p2, ... , pm)) (x1, x2, … , xn)

• S1( x1, x2, ... , xn, p1, p2, ... , pm) k1

• S2( x1, x2, ... , xn, p1, p2, ... , pm) k2 (constraint

• equations)

• ......................................................... • Sq( x1, x2, ... , xn, p1, p2, ... , pm) kq

where

pi : parameters xj : variables

If all the equations are linear: LINEAR PROGRAMMING PROBLEM

If any equation is nonlinear: NONLINEAR PROGRAMMING PROBLEM

5

ILLUSTRATION: 1-variable

SQ1 (X>k1)

SQ2 (X<k2)

CF

X

Figure 1. CF vs X

CFmax

XCFmax

6

PROBLEMS• CF generally unknown (subjective decision)• Parameters partly known• Variables partly known (subjective decision)• No algorithm for the nonlinear programming problem

CONCLUSION: The formal approach is to be used solely

as a guideline.

Actual Software Engineering:

Seek algorithms and techniques minimizing

- memory space requirements

- computing time

7

CONSTRAINTS• Database size requirements + Cost of High Speed

Memory Use of secondary storage for Database.

• Data processing to be done in HSM constant stream of I/O operations

• High cost (in time) of I/O minimize the number of I/Os

• Unavoidable defects in magnetic disk layer

guaranteed error detection and correction

8

TERMINOLOGY

• Attribute a value• Tuple 1: . . . . . . . . . . . . • Attribute k value• Instance of Relation A . . . . . . . . . . . . . . . . . . . . . . . . . • • Conceptual Schema: Tuple n . . . . . . . . . . . . . . . . . . . .

• . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Instance of Relation Z . . . . . . . . . . . . . . . . .. . . . . . . . . . . . .

• Field a value• Record # 1: . . . . . . . . . . . . • Field k value• File for Relation A . . . . . . . . . . . . . . . . . . . . . . . . . • • Physical Schema: Record # n . . . . . . . . . . . . . . . .

• . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • File for Relation Z . . . . . . . . . . . . . . . . .. . . . . .

9

BLOCKS AND PAGES• Confusing descriptions:

“A disk block is a contiguous sequence of bytes (residing on a disk) and is the unit in which data is written to a disk and read from a disk” (text p. 306).

“The unit of information read from or written to disk is a page” (text p.274).

• Disc concept seems more fundamental: block sizes are set by the OS at initialization and cannot be changed dynamically.

• Pages describe the subdivision of the buffer space set aside in the HSM to accommodate disc blocks during I/O transfers. Their size is set by the DBMS. Page size is normally set equal to block size

• The words “page” and “block” are often used interchangeably; No harm as long as their size is the same.

10

FILE ORGANIZATION AND INDEXING

Two (fairly independent) software tools for improving performance:

• File organization: a method of arranging the records in a file when the file is stored on disks. Five such methods are described in the text.

• Indexing; a technique (used in conjunction with certain file organizations) which can improve record searches by adding index files to the database (quite similar to book indexes). Several indexing methods are mentioned in the text, the main emphasis being on Hashing and B trees-based methods.

11

MAJOR CONCEPTS AND NOTATION

• Search Key: attribute value(s) of sought record(s), denoted k.

• Data Record(s): actual records as stored on disk.

• Record Id: disk address of a record permitting DBMS to locate it,

denoted rid.

• Data Entry: record stored in an index file permitting to locate the

sought record having search key value k; denoted k*.

12

SOME PROPERTIES OF INDEXES

• Clustering

An index is clustered when the ordering of data records is the same as or close to the ordering of the data entries. An index that is not clustered is said to be unclustered,

Clustered indexes are quite beneficial as they generally minimize search times.

• Primary and Secondary indexes

A primary index is an index on a set of fields that includes the primary key. Other indexes are called secondary indexes

13

CLUSTERD vs UNCLUSTERED INDEXES

14

ALTERNATIVE METHODS FOR STORING DATA ENTRIES

• Alternative (1): A data entry k* is an actual data record with search key value k (see left side of figure 8.2).

• Alternative (2): A data entry is a <k, rid> pair, where rid is the record id of a data record with search key value k (see right side of figure 8.2).

• Alternative (3): A data entry is a <k, rid-list> pair, where rid-list is a list of record ids of data records with search key value k (this alternative is not shown on figure 8.2. It corresponds precisely to the entries in a book index).

15

FIGURE 8.2Alternative 1

Alternative 2

16

HASH-BASED INDEXINGThe Hash Function: h(k) = bucket address

- Bucket: Primary page + optional chain linked overflow pages’.

- A ‘good’ hashing function: even distribution of keys into buckets.

- A poor hashing function: too many records per bucket producing long linked chains of overflow pages requiring multiple I/Os.

- The search key can be any sequence of one or more fields.

- There are several hash based indexing techniques (see chapter 11), with various ways to solve OF problem.

17

TREE-BASED INDEXING

18

TREE-BASED INDEXING- Data entries located in leaf levels, arranged in sorted order by

search key values.- Non-leaf pages contain node pointers separated by search key

values.- Hierarchical search data structure directs searches to correct page

of data entries.- Each node is a data page; retrieving a node = 1 disk I/O.- Number of I/O needed for search = path length (root-to-leaf) + no. of

pages with qualifying data entries.- B+ tree: balanced B tree: same length for every path (root-to-leaf).

height = path length.fan-out = average no. of children in non-leaf nodes.

Detail study in chapter 1.0+

19

COMPARISON OF FILEORGANIZATIONSFiles to be compared:

• Heap file

• Sorted files

• Clustered B+ tree files

• Heap file with unclustered B+ tree index

• Heap file with unclustered hash index

Operations considered:

• Scan

• Search with Equality Selection

• Search with Range Selection

• Insert a Record

• Delete a Record

20

COST MODEL - ASSUMPTIONS

B = # of (packed) data pages

R = # of records /page

D = avg time for I/O (≈15 ms)

C = avg time to process a record (≈100 ns)

H = avg time to apply hash function (≈ 100 ns)

21

COST MODEL - RESULTSFILE

TYPE

Scan Equality Search Range Search Insert Delete

Heap B(D + RC) BD≅ 0.5B(D+RC)

≅ 0.5 BD

B(D+RC) BD≅ 2D + C

≅ 2D

Search+C+D Search+D≅

Sorted B(D+RC)

≅ BD

Dlog2B+Clog2R

≅ Dlog2B

Dlog2B+#

matching pages

Search +

BD

Search + BD

Clustered 1.5 B(D+RC)

≅ 1.5 BD

DlogF1.5B+Clog2R

≅ DlogF1.5B

DlogF1.5B +#

matching pages

Search +

D

Search +

D

Unclustered tree index

≅

BD (R + 0.15) D(1+logF0.15B)

D(1+logF0.15B+#

matching records)

D(3+

logF0.15B)

Search +

2D

Unclustered hash index

BD(R + 0.125) 2D BD 4D Search +

2D