Operating Systems Part – M14 File Systems IIinformatik.unibas.ch/uploads/media/M14.pdf ·...

Operating Systems Part – M14 File Systems II

Florina Ciorba 13 November 2015

Quiz for M13 of 10.11.15

•  What is reentrance? -  The property of a procedure to return the “same” result, independent of the

numbers of (simultaneous/not) calls •  What is thread safety? -  A piece of code of a multi-threaded program that only manipulates shared

data structures in a manner that guarantees safe execution by multiple threads at the same time.

•  How is a coroutine different from a subroutine? -  “Subroutines are special cases of … coroutines” Donald Knuth [that do not yield] -  Coroutines hold state between invocations

•  What is the task of a file system? -  Store large amounts of information -  Information survives process termination and is share-able

•  What levels does one decide upon in a file system? -  Logical file system?

•  Is the filename part of a file or an element of the directory list? -  It is part of the directory list ("vi .”)

Operating Systems Concepts 2 Computer Architecture and Operating Systems, Florina Ciorba, 13.11.15

Today’s Overview

•  Questions from Nov 10, 2015

•  File links

•  Allocation of file storage

•  Sparse files •  File system

•  Virtual file systems


Is Scala a reentrant language?

http://www.scala-lang.org/what-is-scala.html Scala is an object oriented and a functional language Scala uses •  Data-parallel operations on collections •  Actors for concurrency and distribution •  Futures for asynchronous (without blocking) programming

Reentrancy is not a property of a programming language, but of the code written by a user Reentrant code rules •  May not hold any static (or global) non-constant data. •  May not modify its own code. •  May not call non-reentrant computer programs or routines.


Concurrency at Programming Level: COBEGIN (Cont.)

Arithmetic expression: (a+b) * (c+d) – (e/f) BEGIN COBEGIN COBEGIN t1 = a + b; | t2 = c + d; COEND; t4 = t1 * t2; | t3 = e / f; COEND; t5 = t4 – t3;END


a + b

e / f

*

c + d

−

Can not have a pipe here

as t4 depends on t1 and t2

File Systems

•  Historically, operating systems and file systems have been viewed as distinct entities

•  From the perspective of the modern user, this distinction is often blurred [on purpose]


High Level Abstractions

•  Processes are an abstraction for processors

•  Virtual memory is an abstraction for memory

•  File systems are an abstraction for disk (disk blocks)

•  Partitions (or Volumes) can be viewed as the abstraction of virtual disks.


A Typical File System Organization

•  A directory can be viewed as a “symbol table” that translates file names into their directory entries


File Operations

•  Create •  Write •  Read •  Reposition within file – file seek •  Delete •  Truncate

•  Open(Fi) – search the directory structure on disk for entry Fi, and move the content of entry to memory.

•  Close (Fi) – move the content of entry Fi in memory to directory structure on disk


Directory Operations

•  Search for a file – need to find a particular entry or be able to find file names based on a pattern match. find

•  Create a file - and add its entry to the directory. touch

•  Delete a file – and remove it from the directory. rm

•  List a directory – list both the files in the directory and the directory contents for each file. ls

•  Rename a file – renaming may imply changing the position of the file entry in the directory structure. mv

•  Traverse the file system – the directory needs a logical structure such that every directory and every file within each directory can be accessing efficiently.


Directory Design Goal

To organize the logical structure to obtain: •  Efficiency – locating a file quickly •  Naming – convenient to users

•  Two users can have same name for different files •  The same file can have several different names

•  Grouping – logical grouping of files by properties, (e.g., all Java programs, all games, …)


File Systems as Acyclic Graphs

•  Pure (1|2-level) trees prohibit sharing of files or directories •  Acyclic graphs allow directories to have shared subdirectories and files

•  File/directory used by multiple users •  Either copy (consistency

problems!), or •  Always use with absolute path

name

•  Therefore introduce “links” (pointers)

•  Unix: yes •  MS-DOS: no (to avoid problems –

discussed next)

Operating Systems Concepts 12

root spelldict

count words listlist all w count

list rade w7

Computer Architecture and Operating Systems, Florina Ciorba, 13.11.15

Links in File Systems

Various types of “links” a)  Link consists of the name of the referenced file (using absolute or relative path) b)  Link points directly to the file (blocks) •  Symbolic links (a)

•  “Join” in MS/Windows ≥ Win 95 •  “Soft links” Unix

•  Direct links (b) •  “Hard links” Unix

Create links in Unix: ln –s (symbolic) or ln (direct)


General Graph Directory

•  When links are added to an existing tree-structured directory a general graph structure can be created


Link-related problems

•  Emergence of cycles with links •  How does such a system execute a traversal (e.g., backup, file statistics) in

the presence of multiple paths? •  Only links to files not directories; Garbage collection (expensive); Cycle

detection algorithm (expensive)

•  Consistency of directory entries •  Multiple absolute path names for a file

What happens to a symbolic link if the file is deleted? •  Link persists (dangling pointers to non-existent file) •  Leads nowhere (dangling pointers to middle of other files) Windows, Unix: Users must look for such links themselves

What happens to a file if a symbolic link is deleted?

•  Maintain a file reference list containing one entry for each reference to the file •  Disadvantages: variable and large list

•  Partitioning of the (no longer acyclic) graphs •  When deleting directory or link entries


Reference Counters for Hard (Nonsymbolic) Links

•  Hard links •  Multiple directory entries point to the same file •  Each entry is deleted individually •  When can/should the original file also be deleted?

•  Adding a file to a directory

•  Increments reference counter •  If a file is deleted from the directory

•  Decrements reference counter, delete if counter is 0 •  Raises the cycles problem

•  Reference counter may never be 0 (self-referencing) Unix: Hard links are not allowed for directories to preserve acyclic-graph structure


Example: Linux stat Error

In Linux, the file attributes (except file name) are not stored in the directory but in the file itself File: 'main.tex' Size: 44210 Blocks: 104 IO Block: 4096 regular file Device: 23h/35d Inode: 28576193 Links: 1Access: (0644/-rw-r--r--) Uid: (1000/tschudin) Gid: (1000/tschudin)Access: 2014-11-25 09:09:42.712992941 +0100Modify: 2014-11-25 09:10:27.492991725 +0100Change: 2014-11-25 09:10:27.492991725 +0100 Birth: - Inode: Index node, a data structure used to represent a file system object Stores attributes and disk block location(s) of the file system object data Inode number: An index in this data structure array


mode

owners (2)

timestamps (3)

size block

count

single indirect

double indirect

triple indirect

direct blocks. . .

data. . .data

data

data

data

data

. . .

. . .

. . .

. . .

data

data

data

data

Preview: Unix “inode” for File Data Location


12 pointers of index block

3 pointers of index block


attributes

Hard and Symbolic Links Demo


Hard links

ls –livi file1 (type this is a HL file)ls –li file1cat file1 (file’s data block)ln file1 file1_hlls –li file*stat –L –x file1_hlcat file1_hlvi file1 (add modification)cat file1cat file1_hl (modified content appears in HL)rm file1cat file1cat file1_hl

stat -f "%N: %HT%SY" *

Q: What happens when the initial file is moved?

Symbolic (soft) links ls –livi file2 (type this is a SL file)ls –li file2cat file2 (file’s data block)ln –s file2 file2_slls –li file*stat –L –x file2_slcat file2_slvi file2 (add modification)cat file2cat file2_sl (modified content appears in SL) rm file2cat file2 (no such file/dir)vi file2_slcat file2

stat -f "%N: %HT%SY" *

Q: What happens when the initial file is moved?


Remarks Regarding Storage Management

•  Links (file system) are pointers (programming) Directories (file system) as objects (programming)

•  Pointers in C •  malloc(), free() •  Common mistakes: no or memory allocated multiple times

•  In Java •  No explicit pointer, only hidden as object reference •  Java has to internally keep track of these links •  Also handled via reference counter •  Partitions can arise → garbage collection


Allocation of File Storage

Files are written sequentially: How does one know if there is enough free space on the hard disk? Historic approach – static allocation •  Reservation of a maximum file size before writing

•  Location of the (related) disk blocks is easy to define •  This can fail without a sufficiently large gap (space on disk) •  Files on the hard disk potentially “pushed around”

•  Can also make sense today for read-only file systems E.g., embedded computing, squash file system

Dynamic allocation: “on-demand”, i.e., one allocates a disk block as long as it is possible

Then one may end up with disk blocks scattered over the entire hard disk (fragmentation problem)


How to Map a File onto Hard Disk Data Blocks?

File: data blocks (typically 2-8 KB) 0… M-1 Hard disk: 0… N-1 blocks, typically M<<N Three strategies •  Contiguous allocation → File = amount of adjacent data blocks

•  Linked allocation → Data blocks form a linked list

•  Index allocation → Index of used blocks


Contiguous Allocation

•  Required fields are •  Start block S •  Size

•  “Random access” is simple •  File block n → Hard disk block S+n

•  Upon file creation: size must then be defined → files cannot grow (e.g., append)

•  Hard disk fragmentation by deleting files → waste of space •  Even though enough free blocks are available, new files cannot be created

if they are larger than the largest free block gap


Contiguous Allocation


Linked Allocation


•  Directory contains #of first and last disk block

•  (Question: Why is this useful?)

•  Each hard disk block contains pointer to the next block

•  File = linked list of hard disk blocks

•  Links for free blocks

directory

0 1 2 3

4 5 6 7

8 9 10 11

12 14 15

16 17 19

20 21 22 23

24 25 26 27

28 29 30 31

start9

end25

filejeep10

16 25

1

–1

13

18


Characteristics of Linked Allocation

•  Very good use of free blocks •  No gaps, no losses → no waste of space •  Fast •  Files of any size

•  However: poor “random access” access for pointers in a list •  First read n-1 blocks to arrive at block n

•  Variants •  All pointers are stored in a table •  Used in MS-DOS and OS/2: FAT – File allocation table

Robustness? FAT is critical, has often led to data loss


File-Allocation Table (FAT)


Indexed Allocation


Still pointers (index blocks), but now stored in an index table (rather than list)

•  File consists of •  Index block

•  Index block points to •  the file’s data blocks

•  Access to index block i •  Table entry i (fast)

•  Index block too large Inefficient for small files

•  Index block too small Too little space for large files index table


Indexed Allocation (Continued)


•  File directory contains address of index table: 19

•  There look up the addresses of the actual data blocks 9, 16, 1, 25, etc.

directory

0 1 2 3

4 5 7

8 9 10 11

12 13 14

16 17 18 19

20 21 22 23

24 25 26 27

28 29 30 31

index block19

filejeep

19

9 16 1

10 25 –1 –1 –1

15

6


Multilevel Indices

To keep index blocks small, to support large files •  Multi-level indices

Index blocks point to other index blocks

•  File = tree of hard disk blocks

Calculation example: •  Let HD block = 2 KB, up to 232 blocks •  Choose index block table size = HD block

•  2KB / 4 Bytes per entry → 512 index block entries •  1 level: File size max 512 entries pointing to * 2 KB data blocks (1 MB) •  2 levels: File size max 512 * 512 * 2 KB (0.5 GB)


Unix: Hybrid Index Tables (inode)

mode

owners (2)

timestamps (3)

size block

count

single indirect

double indirect

triple indirect

direct blocks. . .

data. . .data

data

data

data

data

. . .

. . .

. . .

. . .

data

data

data

data


Management of the Free Hard Disk Blocks

•  Bit map •  One bit per block (0 – free, 1 – occupied) •  Easy to find contiguous space

(why is this important?)

•  Linked list (free list) •  Pointers are saved in the free blocks (no waste of space) •  Not too hard to find individual free blocks •  Cumbersome to find contiguous area

Disk crash: can destroy memory structure (inodes, free blocks)


dskchk, scan, fsck – File System Check

•  Situation was precarious until recently File system corruption (e.g., the infamous Blue Screen, loss of a superblock) •  Approaches: redundant FAT, redundant superblocks

•  Unix: Introduces a “clean” bit: it is set when no data needs to be written in the main memory cache (sync) All updates to the file system metadata completed successfully

•  File system check needed to retrieve “lost” blocks Unix: lost+found directory, DOS: 00000001.CHK

Reduced through (log-based transaction oriented) journaling file system (NTFS, ext3 and later) •  All metadata change intentions are written sequentially to a log file (circular) •  Transaction: A set of operations for performing a specific task


Sparse file = File “with holes” (0 Bytes)

•  Simple map with an indexed allocation •  Supported by most Unix file systems

(but not ISO9600 CD format)

•  Examples •  Core dumps •  Not yet fully filled databases

•  How to recognize such files? Demo for Unix •  Compare the result of ‘ls’ and ‘du -sh’ or •  Find the same information with ‘stat’

•  See special options of cp or tar


Sparse Files (Program to Experiment)

#include <stdio.h>int main() /* Sparse file creation */{ FILE *fp; fp = fopen("sparse.dat", "w"); fseek(fp,131072-4,SEEK_CUR); /* 128 KB - 4 bytes */ fprintf(fp,"text"); fclose(fp); return 0;} Only the last block has content (the rest is 0). See sparse option of cpauto | always | never, and du (does not work in encrypted file systems)


File Descriptor Table

Kernel must keep an overview of all open files (cache, buffer management) •  System wide “open-file table”

•  Maintains all currently open files, see also lsof •  Caches the mapping of files to inodes

•  Per process “open-file table” •  Process-specific data such as •  Access mode (r/w) •  Current read/write position in the file (seek)

“File handle”, “file descriptor”: integer value as an index in this table


Overview: From File Descriptor to Inode

user space

read (4, ...)

system space disk space

data blocks

inode list

. . .

in-core inode

list

tables of open files

(per process)

file-structure table

sync


File descriptor Inode

File System: Mount

Previously seen: 1 hard disk partition with 1 file system •  Typically

•  Several partial file systems •  Multiple hard disk partitions •  Network file system

•  mount •  Engages the file system in the local address space •  Superblock: entry in the mount table pointing to the file system on a device

•  Represents a connected set of files that form a self-contained file system •  Provides access to inodes

Windows: Inherited from CP/M explicit reference to device (C:, D:) Unix: 1 tree


File System: Mount (Continued)


(a) Existing file system (b) Unmounted partition → file system unaccessible



Mount point under /usersAccess to file → /users/jane/doc/file.name


/

/priv /data

CD

/

/usr

/usr/bin

/mnt

/mnt/cd

Harddisk


An existing file system is mounted to another: Access to file → /mnt/cd/priv/file.name

File System: Virtual File System

Operating system must support multiple file system types •  Virtual file system: layer between logical file system and the underlying

(specific) levels (memory block and device levels) Contents generated on demand (e.g., procfs)

•  Functions •  Separates file system generic operations from their implementation by

defining a clear VFS interface •  Mechanism for uniquely representing a file throughout the network (vnode)

•  NTFS, FATxx

ext2, ext3 ISO9660 SMB, http (!)


File System: Virtual File System (Continued)


11.2 469

network

Figure 11.4 Schematic view of a virtual file system.

file-system type or the remote-file-system protocol is the third layer of the architecture.

Let's briefly examine the VFS architecture in Linux. The four main object types defined by the Linux VFS are:

The inode object, which represents an individual file

The file object, which represents an open file

The superblock object, which represents an entire file system

The dentry object, which represents an individual directory entry

For each of these four object types, the VFS defines a set of operations that must be implemented. Every object of one of these types contains a pointer to a f1.mction table. The function table lists the addresses of the actual functions that implement the defined operations for that particular object. For example, an abbreviated API for some of the operations for the file object include:

int open ( . . . ) -Open a file.

ssize_t read(. . . ) -Read from a file.

ssize_t write (. . . ) -Write to a file.

int mmap( ... ) -Memory-map a file.

An implementation of the file object for a specific file type is required to imple-ment each function specified in the definition of the file object. (The complete definition ofthe file object is specified in the struct f ile_operat ions, which is located in the file /usr/include/linux/fs .h.)

File System: Loop File System

Special “device”: loop | vnd (vnode disk, BSD) | lofi (loop file interface, Solaris) •  Loop device: data blocks are not delivered from a device, but from a software

module •  A file is accessible as a block device (e.g., CD/DVD ISO images)

•  Loop device software can manage its own memory blocks •  Read a file (!) •  Compression •  Encryption

•  Usefulness •  Managing and editing file system images offline •  Installing an OS onto a file system without repartitioning a disk

Example: File system in a file # mount –o loop test.ext3 /mnt/tmp


File System: Stackable FS

Modularity (layering) •  Set of independent (software) components joined by well-defined interfaces

Example: the pipe (shell)

•  File system modularity lags behind other modular OS interfaces (device drivers)

•  Usefulness •  Reduced FS development cost •  Building blocks for future FS services

•  Symmetric interfaces (upward / downward) •  Layer order can be interchanged


2.2. STACKING 9

release vnode fields vnode size operation countSunOS 2.0 (1985) 11 fields 32 bytes 24 operationsSunOS 4.0 (1988) 14 40 29SunOS 4.1 (1990) 14 40 30SVR4 without fill (1989) 11 40 37SVR4 with fill (1989) 19 72 69Rosenthal’s prototype (1990) 6 20 39

Table 2.1: A slightly expanded version of Rosenthal’s evaluation of vnode interface evolution in SunOS (derivedfrom [Ros90]). Fill indicates space left in SVR4 for future expansion; Rosenthal’s prototype is discussed in Sec-tion 11.6.1.

ule. Finally, ease-of-use requirements for stacking implythat layers are configured at run-time. We discuss theserequirements more in the next section, but for the inter-face they imply that the caller and calleemust bematchedat run-time; at least this level of dynamic binding is re-quired.Since file-system operations are often in the tight loop

of computation, efficiency is of primary concern.

2.2 StackingFile systems frequently implement very similar abstrac-tions. Nearly all file systems ultimately are groundedin disk access and file and directory allocation, for ex-ample. This observation motivates file-system stacking.If a complex filing service can be decomposed into sev-eral layers, then potentially each layer can be developedindependently. Furthermore, in the future, layers can beindividually upgraded as need or desire arises. Finally, aset of filing layers serve as building blocks for the con-struction of future services. Together, these examplesshow how stacking can reduce the cost of file-system de-velopment.An example of layered filing is seen in Figure 2.1. The

operating system vendor provided a standard file storagelayer (the Unix file-system, or UFS). On the left stack auser has configured a compression layer over this basicfile service.A key characteristic of a stackable layer is that it pos-

sess a symmetric interface; it should export an inter-face to its clients which is syntactically the same as thatwhich it depends upon from layers it stacks over. Lay-ers bounded by a symmetric interface can be insertedbetween any existing stack layers (subject to semanticconstraints, of course). For example, in the right-handstack of Figure 2.1, the user has “pulled apart” the com-pression layer and UFS and inserted an encryption layerfor more secure data storage.

OS

user

UFS

OS

UFS

compression

compression

encryption

user

Figure 2.1: Two file-system stacks providing encryptionand compression.

File System: RAID

RAID = Redundant Arrays of Independent/Inexpensive Disks •  Store multiple disk blocks, to enable restoring of data in case of a disk crash •  Reliability via redundancy

•  Striping Distribute data, e.g., 1 disk for each bit position, 8 disks for an 8 bytes stripe A group of disks is used as one storage unit

•  RAID level 0: no redundancy RAID level 1: disk mirroring (shadowing) RAID level 2: parity based error correction … level 6: can handle up to two hard disk outages

•  “Hot spare” defective disk will automatically be replaced


File System: RAID Levels


0: Block striped. No mirror. No parity. Min 2 disks. 1: Blocks mirrored. No stripe. No parity. Min 2 disks. 2: Bits striped (data disks). Stores Hamming ECC (ECC disks). 2 groups of disks (10+4, 4+3) 3: Bytes striped. Dedicated parity disk. Min 3 disks (2+1) 4: Blocks striped. Dedicated parity disk. Min 3 disks (2+1) 5: Blocks striped. Distributed parity. Min 3 disks. Good for DB – reads. 6: Blocks striped. Two distributed parity. 2 parity blocks : 1 data block


File System: RAID (0+1) and (1+0)


“Mirror of stripes” 2 disk groups Min 4 disks Data •  Striped within the group •  Mirrored across the group

“Stripe of mirrors” 1 group only 2 disks Min 4 disks Data •  Striped across the group •  Mirrored within the group Best option for critical apps (DB).


B A C D

B A C D

A C A C A C A C

B D B D B D B D

File System: /proc

File system as an organizational concept of Unix •  Linux process file system /proc •  Contents generated on demand (based on user requests)

Here: Use namespaces, even if no actual files •  All kernel data accessible without privileges

/proc/cpuinfo machine “profile”/proc/devices list/proc/..processnr.. directory for each process

•  Entries can be read with cat or described with echo

•  Windows: see registry


File System: Named Pipes

•  Pipes do not belong to file systems •  Pipes = internal (character based) communication channel

•  System call { int fd[2]; pipe(fd); //Produces pipe inode:

// fd[0] read, fd[1] write ...Generates an “anonymous (ordinary) pipe” Same by ls | wc

•  Named pipe (FIFO) •  No parent child relationship required •  Bidirectional communication % mkfifo /tmp/thepipename


File System: File Locks

Operating System can protect file access: file “locks” •  Data as “shared resource”

Simultaneous access by several programs

•  Protect/control access A “lock” signal can block the process accessing a file locked (by another process)

•  Numerous “locks”, e.g. SUN Solaris 2 Can “lock” even file regions

•  See man 3 lockf (POSIX file lock)


File System: Overview

There would be much more to say: •  Read-only (CD: ISO9660), write-once file system •  Distributed file systems: Andrew, Network File System, Storage Area Network •  Google FS, Amazon S3, Hadoop for “Big Data” •  Indexing (Google Desktop), Databases in OS (Longhorn) •  Integration with Application (email attachments no longer stored in multiple

places) Deduplication

•  Encrypted volumes, integrated backup Versioning, peer-to-peer, freenet

•  And others…


What have we learned today?

•  File links

•  Allocation of file storage

•  Sparse files •  File system

•  Virtual file systems

•  Memory management – next lecture (M15 24.11.2015)


File System: Virtual File System (Continued)

logical file system file systems logical devices physical devices

root

swap


Operating Systems Part – M14 File Systems IIinformatik.unibas.ch/uploads/media/M14.pdf ·...

Documents

Transcript of Operating Systems Part – M14 File Systems IIinformatik.unibas.ch/uploads/media/M14.pdf ·...