File Processing : Storage Media

2015, Spring

Pusan National University

Ki-Joune Li

STEMPNU

Major Functions of Computer

Computation Storage Communication Presentation

STEMPNU

Storage of Data

Major Challenges How to store and manage a large amount of data

Example : more than 100 peta bytes for EOS Project How to represent sophisticated data

STEMPNU

Modeling and Representation of Real World

Example Building DB about Korean History

Very complicated and Depending on viewpoint Database Course : 2010 Fall semester

Real World Computer World

STEMPNU

Managing Large Volume of Data

Large Volume of Data Cost for Storage Media

Not very important and negligible Processing Time

Comparison between main memory and disk access time RAM : several nanoseconds (10-9 sec) Disk : several milliseconds (10-3 sec)

Time is the most valuable resource Example

Retrieving a piece of data from 100 peta bytes DB

STEMPNU

Managing Large Volume of Data

Management of Data Secure Management

From hacking From any kinds of disasters

Consistency of Data Example

Failure during a flight reservation transaction Concurrent transaction

STEMPNU

Goals of File Systems

To provide with 1. efficient Data Structures for storing large and complex data

2. Access Methods for rapid search

3. Query Processing Methods

4. Robust Management of Transactions

STEMPNU

Memory Hierarchy

Large Data Volume Not be stored in main memory But in secondary memory

Memory Hierarchy

Cache Memory 256 K bytes

Main Memory 1G bytes

Secondary Memory 100 G bytes

Tertiary Memory 100 Tera bytes

Faster

Cheaper

STEMPNU

Flash Memory

Non-Volatile Data survives power failure, but Data can be written at a location only once, but location can be erased

and written to again Can support only a limited number of write/erase cycles. Erasing of memory has to be done to an entire bank of memory

Speed Reads are roughly as fast as main memory But writes are slow (few microseconds), erase is slower

Cost per unit of storage roughly similar to main memory Widely used in embedded devices such as digital cameras

STEMPNU

Optical Storage

Non-volatile : data is read optically from a spinning disk using a laser CD-ROM (800 MB), DVD (4.7 to 17 GB), CD-R, DVD-R CD-RW, DVD-RW, and DVD-RAM

Speed Reads and writes are slower than with magnetic disk

Juke-box systems Large numbers of removable disks, Few drives, and Mechanism for automatic loading/unloading of disks For storing large volumes of data

STEMPNU

Tape

Non-volatile Primarily Used for backup

Speed Sequential access : much slower than disk

Cost Very high capacity (40 to 300 GB tapes available) Tape can be removed from drive Drives are expensive

Tape jukeboxes hundreds of terabytes to even a petabyte

STEMPNU

Data Access with Secondary Memory

MainMemory

Access Request

Get Data

If in main memory

Disk

If not in main memory Access to Disk

Load on main memory

Get Data Hit Ratio rh = nh / na

How to increase hit ratio ?

STEMPNU

Why Hit Ratio is so important ?

Example

for(int i=0;i<1000;i++)

Nbytes=read(fd,buf,100);1000 disk accesses ?

1000 * 10-2 sec = 10 sec 1000 * 10-8 sec = 10-5 sec

when rh = 0 when rh = 1

STEMPNU

Physical Structure of Disk

512 bytes

200~400 sectors

2 * nDF

STEMPNU

Disk Access Time

Disk Access Time t = tS + tR + tT , where

tS : Seek Time Time to reposition the head over the correct track Average seek time is 1/2 the worst case seek time 4 to 10 milliseconds on typical disks

tR : Rotational Latency Time to reposition the head over the correct sector Average rotational latency : ½ r (to find index point) + ½ r = r In case of 15000 rpm : r =1*60sec/15000 = 4 msec

tT : Transfer Time Time to transfer data from disk to main memory via channel Proportional to the number of sectors to read Real transfer time is negligible

STEMPNU

Block-Oriented Disk Access

Examplefor(int i=0;i<1000;i++)

Nbytes=read(fd,buf,10);

1000 times

10 bytes

Buffer in main memory

1024 bytes

10 times

100 times

1 block (e.g. 1024 bytes)Number of Disk Accesses

STEMPNU

Disk Block

Unit of Disk Access Block Size

Normally multiple of sectors 1K, 4K, 16K or 64K bytes depending on configuration

Why not large block ? Limited by the size of available main memory Too large : unnecessary accesses of sectors

e.g. only 100 bytes, when block size is given as 64K 1 block : 128 sectors (about ½ track, ½ rotation, 2 msec) Too wasteful