CSIT 301 (Blum)1 NTFS CSIT 301 (Blum)2 FAT Review FAT (16) uses up to 16 bits to address data on the hard drive (or partition thereof) 2 16 = 65,536 If you address 65,536 sectors, each having 512 bytes then you would have 65,536 512 = 33,554,432 bytes = 32,768 kilobytes = 32 megabytes (MB) CSIT 301 (Blum)3 Cluster Clusters are groups of sectors addressed in the FAT system. Within FAT(16) Sectors/ClusterCluster Size (KB)Partition Capacity (MB) (= 1 GB) (= 2 GB) CSIT 301 (Blum)4 The bigger the cluster, the more the slack The cluster size is the minimal space that can be used to store a file. With 32 sectors per cluster, a cluster was 16KB, much larger than many of the files that need to be stored on a typical partition. The unused portion of all of these clusters is called slack. While large clusters allowed for larger partitions, they resulted in unacceptable amounts of slack. CSIT 301 (Blum)5 More addresses needed To have large capacity partitions without sacrificing much of that capacity to slack, a larger address space is needed. FAT 32 can devote up to 28 bits to addressing (the other four bits are reserved for other purposes). Allows one to address 2 28 = 268,435,456 things CSIT 301 (Blum) is a lot Even if one addressed sectors then theoretically one could have a capacity of 268,435,456 512 bytes 137,438,953,472 bytes 134,217,728 kilobytes 131,072 megabytes 128 gigabytes And thats if youre addressing sectors, its even larger if youre addressing clusters. CSIT 301 (Blum)7 A very fat FAT The price one pays for having small clusters (which save on slack) is to have a large FAT table. The FAT table does not take up much room as far as disk space is concerned but it is something one probably wants in memory (disk cache). But a large FAT table will take up too much space in memory. So partition size, cluster size and FAT table size is a balancing act. CSIT 301 (Blum)8 FAT32 Table The table shows the FAT32 Table size for various choices of the partition size and cluster size. The size should be compared to the amount of memory since the FAT table is often in memory (disk cache). CSIT 301 (Blum)9 NTFS New Technology File System (NTFS) was built to provide features like: Reliability: introduced ideas like transactions (grouping certain updates together to maintain integrity) Security and Access Control: built-in features to manage who can access files and what type of access they have CSIT 301 (Blum)10 NTFS Features (Cont.) Large-capacity partitions: allows large partitions and even RAID (Redundant Array of Inexpensive Disks, treating multiple disk as one large disk) Slack reduction: allocates space differently from FAT Allows for long file names (not limited to 8- character names with 3 character extensions) Networking: built with networking in mind CSIT 301 (Blum)11 A more structured file system In NTFS files are more than just pools of data, they have structure The difference between FAT and NTFS is analogous to the difference between a flat file and a database. Just as in databases where one has data and metadata (the data about the data), NTFS has metadata files (files that contain data about other files). CSIT 301 (Blum)12 Partition/Volume Boot Sector/Record One of the first things made when an NTFS partition is created is the volume boot sector, which contains: BIOS parameter block: identifies the partition, how big it is, etc. Volume boot code: code that starts to load the operating system CSIT 301 (Blum)13 All else is files After the volume boot sector, just about everything else is a file. There are metadata files: files about files Created automatically when the partition is formatted Placed at the beginning (Actual or real) Data files CSIT 301 (Blum)14 MFT Think of the Master File Table (MFT) as a database containing records about all of the files (both data and metadata, including itself). Each files record holds the values of its attributes. The actual data in a data file is simply one of its attributes. CSIT 301 (Blum)15 The first several records The first several records in the MFT are about other important metadata files, including MFT itself MFT Mirror (1 st 16 records) Log file (keeps account of transactions) Attribution Definition Table (names file properties and says what they are) Root Directory Folder Bad cluster file Etc. CSIT 301 (Blum)16 MFT Zone There will be a record in the MFT for every file on the partition. Thus the MFT needs room to grow. Some space in the partition, called the MFT Zone, is reserved for this purpose. If one needs part of the MFT zone for storage, it will eventually be used. On the other hand, the MFT can grow to be larger than the MFT zone. It is then fragmented which could affect performance. CSIT 301 (Blum)17 Resident vs. Non-Resident Attributes The MFTs record size is fixed (between 1KB and 4KB), but the attributes may be of any size (especially since a data files data is an attribute). Attributes that are contained in the MFT are called resident. A small file may be entirely resident. Attributes that are linked to but not actually contained in the MFT are called non-resident. CSIT 301 (Blum)18 Extents Small files are contained within the MFT For larger files, the MFT contains a collection of pointers to the data runs of extents which actually hold the data. If the collection of pointers grows too large, then it is placed in a separate file and the MFT points to this file, which in turns points to the data runs. CSIT 301 (Blum)19 Some File Attributes File name: (can be up to 255 characters, allows a file to have aliases) Standard Information: read-only, hidden, archived, time stamps, etc. Security Descriptor: Access Control Lists (ACLs) who owns the file, who has what privilege, etc. Data: the actual data CSIT 301 (Blum)20 Security NTFS was designed with the idea of multiple users and security in mind. The features necessary to implement a security policy are built directly into the file system. In FAT32 a file may be hidden or read-only, but in NTFS a file can be hidden from user1, read-only to user2 and fully accessible to user3. CSIT 301 (Blum)21 Security Concepts Ownership: some user owns a file/folder and he or she grants permissions to other users. Permissions: what a user can do with a file/folder (read, read-write, delete, etc.) Users are placed in groups (possibly more than one) and permissions are assigned to groups Permissions can be inherited, e.g. new files gets permissions of folder it was created in Auditing: tracking information about users access to and modification of files CSIT 301 (Blum)22 ACLs An important security attribute of a file is its Access Control List (ACL). The ACL specifies which users can access the file and in what way they can access the file There are two types of ACL: System ACL: used for auditing purposes Discretionary ACL: explicit assigning of permissions to users or groups CSIT 301 (Blum)23 Permissions CSIT 301 (Blum)24 Versions NTFS version 1.1 (a.k.a. version 4.0) is used with Windows NT 4.0. Usually Windows 2000 uses a revised version known as NTFS version 5.0 CSIT 301 (Blum)25 Improvements (NTFS 5 over 4) Reparse Points: One can associate an action or actions with a file. So that if the file is accessed, the action is performed. Analogous to a trigger in a database Reparse points is very flexible, one example is redirection sending one to another file or directory, it may be on another drive or even have been archived. CSIT 301 (Blum)26 Improvements (NTFS 5 over 4) Improved Security and Permissions: one change is from static to dynamic permission inheritance. Static: a child inherits the parents permissions when it is created but is unaffected by subsequent changes in the parents permissions Dynamic: a change to the parents permission will affect the childs permissions Change Journals: improved auditing (journaling) of file/folder access activity. Encryption: Automatic encryption/decryption of files (when accessed by users with the appropriate permissions). CSIT 301 (Blum)27 Improvements (NTFS 5 over 4) Disk Quotas: Users or groups of users can be limited in the amount of disk space they can use. Sparse File Support: A sparse file is one that may be big but hold very little data (relative to its size). NTFS has utilities to help store sparse files more efficiently. Disk Defragmenter: Strictly speaking part of the operating system, it affects the file system. CSIT 301 (Blum)28 Transactions Dont forget NTFS is pretty much a database. Almost any activity involving the drive in anyway is going to affect a number of files. NTFS introduces the notion of a transaction the grouping together of various operations to form an atomic unit. In other words these operations should be viewed as all or nothing in order to maintain the file systems integrity. Recall the ACID test from databases? CSIT 301 (Blum)29 Logging and Committing There is a special metafile for logging all activity. When all of the components of a transaction are complete, this completion is indicated in the log file and the transaction is said to be committed. If something goes wrong (e.g. power failure) before a transaction is completed, the file system can undo the partially enacted transaction to return the file system to a consistent state. Doing so is said to be rolling back the transaction. It is also called transaction recovery. CSIT 301 (Blum)30 Effect on Performance Logging each activity which is great for security and integrity of the file system but does have some negative effects on performance. Each file access now requires another file access (writing to the log file). One way to save on performance but risk somewhat integrity is to cache the activity log changes rather than write to disk every time. The cached log results are written to disk periodically but not continuously. CSIT 301 (Blum)31 Recovery Recovery then involves three passes over the log file: Analysis pass: determine the part of the disk affected Redo pass: perform any transaction that was completed since the last checkpoint Undo pass: roll back any incomplete transactions CSIT 301 (Blum)32 Change Journal NTFS can record changes to files, these are kept in the Change Journal. Each change is assigned an ID, an Update Sequence Number (USN). It will record that a file was written to but not what was written. Otherwise it would be gargantuan. CSIT 301 (Blum)33 Fault Tolerance NTFS has a fault tolerance disk driver known as FTDISK. Thats where one can find the transaction recovery features. Also where one finds support for RAID (redundant array of inexpensive (or is that independent) disks). And where youll find dynamic bad cluster remapping. Basically the drive reads immediately after writing to ensure that the cluster written to was OK. If it was not, it writes it somewhere else and marks the cluster as bad. CSIT 301 (Blum)34 Compression NTFS has build-in utilities for file compression File compression takes advantage of patterns in data to reduce the amount of space required to store it. E.g. instead of ASCII code for text (each character 8 bits) one might use a variable length code with short codes for common letters like e and longer codes for uncommon letters like q or j. On average the files are much smaller. In NTFS one can compress any part of the partition. CSIT 301 (Blum)35 POSIX support NTFS offers POSIX support. POSIX stands for Portable Operating System Interface for UNIX It allows software developers to make sure that their code can be ported to a POSIX- compliant operating system, which includes most versions of UNIX. CSIT 301 (Blum)36 Supports Encryption NTFS supports Encrypting File System (EFS). EFS is really part of the operating system (Windows 2000). But the operating system works with the file system to make this feature easy to use. CSIT 301 (Blum)37 Disk Quota support As a genuinely multi-user file system, NTFS support disk quotas A quota can be set for a particular user or on a particular partition or the combination. Allows for limits and warnings. The user is warned when he or she exceeds the warning amount. The user is blocked (from writing?) when he or she exceeds the limit amount. Monitor and log events that cause a user to go over the "limit" or "warning" levels. CSIT 301 (Blum)38 DMA CSIT 301 (Blum)39 Transfer Mode The transfer mode describes the way in which the data moves from the hard disk through the interface (IDE/ATA) and to the memory. For example, it tells how fast data is transferred or what device is in charge of the transfer. There are two basic categories PIO (Programmed I/O) Mode The processor micro-manages data transfer DMA (Direct Memory Access) Mode The processor delegates data transfer CSIT 301 (Blum)40 Programmed I/O (PIO) Modes In the PIO category, the processor controls the data transfer. There are various PIO modes which differ mainly by speed. Through the early to mid-90s PIO was the standard way to transfer data to the hard disk. The original ATA standards document defined the first three modes. With ATA-2, two faster modes were introduced. CSIT 301 (Blum)41 Standard PIO Modes 3.3 MB/s = 3.3 10 6 bytes / second = (2 bytes / 600 s) Two bytes are transferred every 600 nanoseconds. CSIT 301 (Blum)42 External rates The PIO rates on the previous slide are external rates meaning that they reflect the rate that data in the hard disks buffer/cache can be transferred. Recall that access times to locate and read from a random sector are of the order of milliseconds. Reading a sector (512 bytes) in 20 ms would correspond to a rate of 25 KB/s. If one were not buffering and transferring consecutive data, the PIO mode rates would be sufficient. But we do transfer buffered data and the PIO transfer rates are considered prohibitively slow by todays standards. CSIT 301 (Blum)43 PIO is too, too slow PIO is slow in two ways: One does not achieve the same data transfer rates as with Ultra DMA, which is the standard transfer mode used for IDE/ATA today. Because the processor controls the details of the transfer in PIO, the processor is distracted from performing other tasks. Despite its slowness, PIO is still around because: PIO is simple (built into the BIOS so it does not require drivers). Backward compatibility Can be used as a backup when something goes wrong with DMA. CSIT 301 (Blum)44 DMA The alternative to PIO is DMA, Direct Memory Access. In DMA, a device transfers information to or from the memory directly rather than in a processor- controlled fashion. DMA has been around awhile but it was not always well supported early on. But speed requirements have made it preferred over PIO. CSIT 301 (Blum)45 Various Modes As with PIO, DMA has various modes differing mainly by speed. DMA modes split into two categories: Single-word modes which send one word (two bytes) at a time Multiple-word modes which send several words in rapid succession (rather like the idea of bursting that accounts for improved memory speeds). CSIT 301 (Blum)46 CSIT 301 (Blum)47 Assume its multiword The single-word DMA modes are too slow, today it is understood that DMA is multiword DMA and the term is rarely mentioned and usually implied. In fact, the single-word DMA modes were dropped from the standards with ATA-2. Ultra DMA is multiword. CSIT 301 (Blum)48 First Party The party of the first part shall be known in this contract as the party of the first part CSIT 301 (Blum)49 First Party vs. Third Party In Third-Party DMA there is a third device, the DMA controller mediating the transfer between the hard disk and memory. Third party DMA is slow and old fashioned. In First-Party DMA, a.k.a. bus mastering, the middle man is eliminated. The hard drive controls the transfer of data between itself and memory. The device (hard drive in this case) takes control of (masters) the bus along which the information is sent. CSIT 301 (Blum)50 Ultra DMA DMA only became the norm with the introduction of Ultra DMA. It just wasnt well supported before. DMA gained the advantage over PIO when Ultra DMA/33 doubled the interface transfer rate. Support for it also improved. Today, Ultra DMA is an industry standard. CSIT 301 (Blum)51 Ultra DMA uses DDR and CRC One feature that made Ultra DMA ultra was that it transferred data on both the positive and negative edges of the clock. Same idea as in DDR (Double Data Rate) memory As Ultra DMA pushed the limit on transfer rate, it made the occurrence of errors somewhat more likely. Thus it introduced a CRC (Cyclic Redundancy Check) as part of the standard. Recall CRC is error detection. If an error occurs in transmission the data is retransmitted. CSIT 301 (Blum)52 Ultra DMA transfer rates DMA finally beats PIO Mode 4s 16.7 MB/s with the added bonus of freeing up the processor. The modes are usually named after their maximum transfer rate and the interface they use: Ultra ATA/100 instead of Ultra DMA/Mode 5. CSIT 301 (Blum)53 New cable needed The faster speeds did require a change in the cable used to connect the drive. The 80-conductor ATA/EDE cable. Color Code Blue: connects to the host (motherboard or controller). Gray: connects to the slave drive (if there is one) Black: connects to the master drive CSIT 301 (Blum)54 80 wires and 40 pins?? Signals are varying currents. Currents produce magnetic fields, varying currents produce varying magnetic fields. Varying magnetic fields produce currents. Oops, the current in wire 1 begins to change the current in wire 2. There is interference (a.k.a. cross talk) The extra wires shield the signal-carrying wires from each other. CSIT 301 (Blum)55 Block Mode Certain BIOSs allow for a Block Mode setting. Block Mode allows 16 or 32 sectors (512 bytes each) to be handled using a single interrupt (even if the processor is not running the show it needs to know something has happened). CSIT 301 (Blum)56 IRQs Devices cannot interrupt the processor whenever they want. Instead they must request the processors attention. Interrupt Request Numbers are an addressing scheme so that the processor can determine which device wanted its attention when when it gets around to checking for interrupts. CSIT 301 (Blum)57 Resources Many devices might want to interrupt the processor, so devices are assigned IRQs. Usually this is done automatically. Occasionally two devices vie for the same slot and a conflict arises. Devices other than the hard drive transfer information to or from memory and they use DMA as well. Devices are assigned ranges of memory they can use for DMA so as to avoid conflicts. IRQs and DMA ranges are called system resources. CSIT 301 (Blum)58 CSIT 301 (Blum)59 CSIT 301 (Blum)60 CSIT 301 (Blum)61 Viewing Resources for a Device CSIT 301 (Blum)62 Viewing Resources for a Device CSIT 301 (Blum)63 Viewing Resources for a Device Double click CSIT 301 (Blum)64 Viewing Resources for a Device CSIT 301 (Blum)65 A Different Device CSIT 301 (Blum)66 Yet Another Device CSIT 301 (Blum)67 ATAPI The ATA/IDE Interface was originally designed for hard drives only. AT Attachment Packet Interface (ATAPI) is a protocol that allows devices that are not hard drives to use this interface. Instructions are sent to it in packets CSIT 301 (Blum)68 ATA Standard Summary Table CSIT 301 (Blum)69 Serial ATA Serial ATA (SATA or S-ATA) replace the parallel connection with a serial connection for hard drive and ATAPI devices. Similar to what USB-2 has done for external devices. Transfer rates starting at 150 MB/s. Serial cables can be longer and of course thinner, making connecting devices easier. They reduce issues of heating and interference. CSIT 301 (Blum)70 ReferencesPC Hardware in a Nutshell (Thompson and Thompson)