Mobile Handset Storage and File System

2

Outline

Storage and File System Basics Android File System iOS File System

3

Storage Hierarchy

Almost all computers use a storage hierarchy Put fast but expensive and small storages close

to the CPU Put slower but larger and cheaper storages far

away from the CPU

4

Primary and Secondary Storages

Primary storage (or main memory or internal memory) is the storage directly accessible to the CPU Lose information when not powered Such as cache, registers and main memory

Secondary storage (or external memory or auxiliary storage) is not directly accessible by the CPU The computer usually uses input/output channels to access

secondary storage Does not lose the data when the device is powered down Such as hard disks, CD/DVD, flash memory (e.g. sdcard)

5

Flash Memory

Flash memory is a computer memory chip that maintains stored information without requiring a power source. It belongs to the secondary storage in the computer storage hierarchy

It can be electronically erased and overwritten Mobile devices use flash memory to store data There are two major types of flash memory

NAND flash memory and NOR flash memory They are named after the NAND and NOR logic gates

6

Flash Memory History

Flash memory was invented by Dr. Fujio Masuoka (Toshiba)First presented in IEEE International Electron

Devices Meeting (IEDM) 1984Intel introduced the first commercial NOR type

flash in 1988Toshiba announced NAND flash at IEDM 1987The first NAND-based removable media format

was SmartMedia in 1995

7

NOR Flash Memory

Random access to any memory location Use blocks as the storage units (the typical

block sizes are 64KB, 128KB or 256KB) Erasure must happen on block level, a block at

a time. Write happens on byte level Long write and erasure time Developed as a replacement for ROM (read

often, rarely updated)

8

NAND Flash Memory (1)

Use page (a group of memory words) as the basic unit to store data. The typical page sizes are 512, 2048 or 4096 bytes

Associated with each page are 12 to 16 bytes for checksum

Pages are combined into blocks Read and write happen on a page level.

Erasure can only happen on a block level

9

NAND Flash Memory (2)

Write and erasure time is reduced Suitable for replacing disks Typical block sizes:

16KB: 32 pages of (512 + 16 spare bytes)128KB: 64 pages of (2048 + 64 spare bytes)256KB: 64 pages of (4096 + 128 spare bytes)512KB: 128 pages of (4096 + 128 spare bytes)Spare bytes can be used for checksum

10

NOR VS NANDNOR NAND

Performance Very slow eraseSlow writeFast read

Fast eraseFast writeFast read

Reliability Standard reliability Low reliabilityNeeds bad block management

Erase Times 10,000 – 100,000 100,000 – 1,000,000

Life Span Less than 10% the life span of NAND

Over 10 times more than NOR

Access Random Sequential

Hardware Implementation

Easy Complicated

Spare Bytes No Yes (16 bytes)

11

File System

File system is a computer program which controls how data is stored and retrieved

Primary roles:Provides an abstraction for secondary storageProvides a logical organization of filesEnables sharing data between processes, users and

machines Protects data from unwanted access

12

Why Need File System

Disks are messy physical devices Errors, bad blocks, missed seeks, etc.

The job of OS is to hide the mess from higher level software It needs to handle low-level device control (start a disk

read, etc.) It needs to provide higher-level abstractions (files,

databases, etc.) The file system handles the mess for OS

13

File Concept

A file is a logically contiguous address space which stores a collection of data. It has following attributes: File name File identifier (a unique number for the file) File type File location (pointer to file location on disk) File size File protection (controls who can read, write or execute),

etc.

14

File Operations

Most common operations:CreateWriteReadReposition within a fileDelete Truncate

15

File Protection (1)

File system must implement some kind of protection to control who can access a file and how they can access it

Types of users Owner: the user who created the file Group: the users who is in the same group with the owner Others: any other users in the system Super user: administrator of the system

Types of access are read (r), write (w) and execute (x)

16

File Protection (2)

Protection in Unix file system

17

Directories

A directory, also known as folder, is a structure which allows the user to group files into separate collections

The root directory is the first or top-most directory in tree structured directories. It is the starting point where all branches originate fromE.g., the / directory in Unix systems

18

Tree-Structured Directories

19

Block (1)

A block is a sequence of bytes or bits and have a maximum length, a block size. It is the basic unit used by most file systems to store data

File systems define a block size (e.g., 4KB)Disk space is allocated in granularity of blocks

A “Master Block” stores the location of root directoryAlways at a well-known disk locationOften replicated across disk for reliability

20

Block (2)

A map stores which blocks are free, which are allocatedUsually a bitmap, one bit per block on the diskAlso stored on disk, cached in memory for

performance Remaining disk blocks are used to store files

and directories

21

Outline

Storage and File System Basics Android File System iOS File System

22

Overview

Android uses flash memory as its storage media, so it can use flash file systems such as exFAT, YAFFS2, JFFS2, etc.

Android is based on Linux kernel, so it can use a Linux file system, such as ext2, ext3, ext4, etc.

It may also use a proprietary file system developed by a manufacturer, depending on who made the device

The most commonly used file system on Android Yet Another Flash File System2 (YAFFS2)

23

YAFFS

YAFFS is a flash file system developed for NAND flashYAFFS1: designed for early NAND generations of

flash memory (512-byte page)YAFFS2: support new NAND with 2KB pages and

strictly sequential page writing order It uses chunk to manage data. Chunk is YAFFS

terminology for a page.

24

YAFFS1 Chunks

File data stored in fixed size “chunks”, i.e., NAND pages (512 bytes)

Two types of chunk:Data chunk: holding regular data file contentsFile header: a file’s metadata such as file name,

parent directory, etc.

25

YAFFS1 Tags Each chunk has tags with it. The tags comprise the

following fields (8 bytes in total):Field Bits Meaning

File ID 18 Identifies which file the chunk belongs to

Chunk ID 20Identifies where in the file this chunk belongs to. 0 means this chunk contains a file header, 1 means the first chunk and 2 is the next chunk and so on

Serial Number 2 Differentiates chunks with the same file ID and chunk ID

Byte Count 10 Number of bytes of data if this is a data chunk

Checksum 12 Checksum for tags

Reserved 2 Unused

64 Total

26

YAFFS1 Serial Number

When data is overwritten, the relevant chunks are replaced by writing new pages to the flash containing the new data. Then the old page is marked as “discarded”

If power loss/crash/other problem happens before the old page is marked as regarded, it is possible to have two pages with the same tags Solve the problem: Increase 2-bit serial number by 1 every

time a chunk is overwritten to distinguish the new data and old data

27

YAFFS1 Garbage Collection

A block with all discarded pages is an obvious candidate for garbage collection

Otherwise, valid pages are copied from a block and then mark the whole block discarded and ready for garbage collection

28

YAFFS1 Page Layout

Bytes Range Fields Details

0 - 511 Data Data, either file data or file header depending on tags

512 - 515 Tags Tags

516 Data Status If more than 4 bits are zero, this page is discarded.

517 Block Status Shows whether the block is damaged

518 - 519 Tags Tags

520 - 522 Checksum Checksum for second 256 bytes part of data

523 - 524 Tags Tags

525 - 527 Checksum Checksum for first 256 bytes part of data

29

YAFFS2 VS YAFFS1 (1) YAFFS2 is very similar in concept to YAFFS1 and

they share much of the same source code Add support for new NAND with 2KB pages Mark very newly written block with a sequence

number The sequence of the chunks can be inferred from the block

sequence number and chunk offset within the block When it detects two chunks with same file ID and chunk

ID, it can choose the new chunk by taking the greater sequence number

30

YAFFS2 VS YAFFS1 (2) Introduce concept of shrink headers for efficiency

When a file is resized to a smaller size, YAFFS1 will mark all of the affected chunks as discarded. But YAFFS2 writes a “shrink header”, which indicates that a certain number of pages before this header are invalid

Improve performance relative to YAFFS1 Write 1.5-5x Delete: 4x Garbage collection: 2x

31

Outline

Storage and File System Basics Android File System iOS File System

32

Overview

In 1985 Apple developed a new file system called hierarchical file system (HFS) for use in Mac OS

Hierarchical file system plus (HFS+) was introduced in 1998 for use in Mac OS 8.1

HFSX was introduced in Mac OS 10.3 in 2005. Now it becomes the file system for iOS

33

HFS Blocks

At the physical level, the disk is divided into blocks of 512 bytes

There are two types of blocks:Logical blocks: they are numbered from the first to

the last on the disk. And they are static and the same size as the physical blocks, 512 bytes

Allocation blocks: they are groups of logical blocks used by the HFS to track data in a more efficient way

34

HFS Structure (1) Logical blocks 0 and 1

the boot blocks which contain system startup information Logical blocks 2

contains the master directory block (MDB) which defines a wide variety of data such as date and time stamps for when the partition was created, the location of the bitmap, etc.

Logical block 3 the starting block of the bitmap which keeps track of which

allocation blocks are in use and which are free. Each allocation block is represented by a bit in the map: if the bit is set, the block is in use. Otherwise it is free to use.

35

HFS Structure (2) The extent overflow file

Keeps track of which allocation blocks are allocated to which files

Catalog fileDescribes the folder and file hierarchy on the disk.

It contains metadata about all the files and folders on the disk including information about modify, access and create times

36

HFS+ VS HFS HFS+ has three more parts in terms of the

structureAttributes file: contains attribute information of all

files and foldersStartup file: designed to assist in booting non-Mac

OS systems that don’t have HFS or HFS+ supportReserved block: reserved for use by Apple

37

HFSX VS HFS+ All Apple mobile devices use HFSX as the file

system. There is one major difference between HFSX and HFS+. HFSX is case sensitive.For example, Case_sensitive.doc and

Case_Sensitive.doc are treated as two different files. They can both exist on HFSX but not in HFS+

38

References (1) http://en.wikipedia.org/wiki/File_system https://cseweb.ucsd.edu/classes/fa06/cse120/lectures/120-fa06-l13.pdf http://codex.cs.yale.edu/avi/os-book/OS8/os8c/slide-dir/PDF-dir/ch10.pdf http://www.dubeiko.com/development/FileSystems/YAFFS/

HowYaffsWorks.pdf http://en.wikipedia.org/wiki/YAFFS http://www.forensicswiki.org/wiki/HFS%2B http://en.wikipedia.org/wiki/Mac_OS_X_Tiger http://en.wikipedia.org/wiki/HFS_Plus