leveldb Log format
2
The log file contents are a sequence of 32KB blocks. The only exception is that the tail of the file may contain a partial block. Each block consists of a sequence of records: block := record* trailer? record := checksum: uint32 // crc32c of type and data[] length: uint16 type: uint8 // One of FULL, FIRST, MIDDLE, LAST data: uint8[length] record size = 4 + 2 + 1 + data[length] A record never starts within the last six bytes of a block (since it won't fit). Any leftover bytes here form the trailer, which must consist entirely of zero bytes and must be skipped by readers. Aside: if exactly seven bytes are left in the current block, and a new non-zero length record is added, the writer must emit a FIRST record (which contains zero bytes of user data) to fill up the trailing seven bytes of the block and then emit all of the user data in subsequent blocks. Èç¹û³öÏÖÏÂÃæµÄÇé¿ö£º seven bytes block i [+++++++++++++++++++++++++++--] block i + 1 [-----------------------------] - ±íʾ¿ÉÓÿռä + ±íʾ²»¿ÉÓÿռä leveldbµÄ½«»á°Ñ¿ªÊ¼µÄ7bytesдÈëµ½block iÖУ¬Ê£ÓàµÄdata²¿·ÖдÈëµ½block i+1ÖÐ More types may be added in the future. Some Readers may skip record types they do not understand, others may report that some data was skipped. FULL == 1 °üº¬Íê³ÉµÄrecord FIRST == 2 MIDDLE == 3 LAST == 4 The FULL record contains the contents of an entire user record. FIRST, MIDDLE, LAST are types used for user records that have been split into multiple fragments (typically because of block boundaries). FIRST is the type of the first fragment of a user record, LAST is the type of the last fragment of a user record, and MID is the type of all interior(ÄÚ²¿µÄ) fragments of a user record. Example: consider a sequence of user records: A: length 1000 B: length 97270 C: length 8000 A will be stored as a FULL record in the first block. ³õʼ¼ÙÉèblockΪ¿Õ£¬ÈçÏ£º block1 [++++++++++++++++++++++++] 32768=1024*32 ½«A²åÈëµ½block1ÖУ¬ÏÔÈ»AµÄÀàÐ;ÍÊÇFullÀàÐÍ,Íê³ÉÖ®ºó£¬ÈçÏÂͼËùʾ: record A
-
Upload
institute-of-computing-technology-chinese-academy-of-sciences -
Category
Education
-
view
787 -
download
2
description
leveldb Log format
Transcript of leveldb Log format
The log file contents are a sequence of 32KB blocks. The onlyexception is that the tail of the file may contain a partial block.
Each block consists of a sequence of records: block := record* trailer? record :=
checksum: uint32 // crc32c of type and data[]length: uint16type: uint8 // One of FULL, FIRST, MIDDLE, LASTdata: uint8[length]
record size = 4 + 2 + 1 + data[length]
A record never starts within the last six bytes of a block (since itwon't fit). Any leftover bytes here form the trailer, which mustconsist entirely of zero bytes and must be skipped by readers.
Aside: if exactly seven bytes are left in the current block, and a newnon-zero length record is added, the writer must emit a FIRST record(which contains zero bytes of user data) to fill up the trailing sevenbytes of the block and then emit all of the user data in subsequentblocks.Èç¹û³öÏÖÏÂÃæµÄÇé¿ö£º
seven bytesblock i [+++++++++++++++++++++++++++--]block i + 1 [-----------------------------]- ±íʾ¿ÉÓÿռä+ ±íʾ²»¿ÉÓÿռäleveldbµÄ½«»á°Ñ¿ªÊ¼µÄ7bytesдÈëµ½block iÖУ¬Ê£ÓàµÄdata²¿·ÖдÈëµ½block i+1ÖÐ
More types may be added in the future. Some Readers may skip recordtypes they do not understand, others may report that some data wasskipped.
FULL == 1 °üº¬Íê³ÉµÄrecordFIRST == 2MIDDLE == 3LAST == 4
The FULL record contains the contents of an entire user record.
FIRST, MIDDLE, LAST are types used for user records that have beensplit into multiple fragments (typically because of block boundaries).FIRST is the type of the first fragment of a user record, LAST is thetype of the last fragment of a user record, and MID is the type of allinterior(ÄÚ²¿µÄ) fragments of a user record.
Example: consider a sequence of user records: A: length 1000 B: length 97270 C: length 8000A will be stored as a FULL record in the first block.
³õʼ¼ÙÉèblockΪ¿Õ£¬ÈçÏ£ºblock1 [++++++++++++++++++++++++] 32768=1024*32½«A²åÈëµ½block1ÖУ¬ÏÔÈ»AµÄÀàÐ;ÍÊÇFullÀàÐÍ,Íê³ÉÖ®ºó£¬ÈçÏÂͼËùʾ:
record A
block1 [----+++++++++++++++++++] Ê£ÓàÈÝÁ¿32768-1000=31768 Ö®ºó²åÈërecord BµÄÇé¾°ÈçÏ£º
record A(FULL) record B(FIRST)block1 [----|------------------] blockÂú
record B (MIDDLE)block2 [-----------------------] blockÂú
record B(LAST)block3 [-------------------+++] Ê£ÓàÈÝÁ¿ 32768-(97270-31768-32768)=32768-32734=34byteÕâÀï¼ÆËã³öµÄ½á¹ûÊÇ34byte£¬µ«ÊÇÏÂÃæµÄÎĵµÖнâÊÍΪ6byte£¬È»ºórecord CÊý¾Ý½«Ð´ÈëеÄblockÖС£×ÜÖ®µÄ˼·¾ÍÊÇÈç¹ûÒ»¸öblockÊ£ÏÂÁËÉÙÓÚ6byteµÄ»°£¬Ê£ÏµIJ¿·Ö½«±»×÷Ϊtailer
ÕâÀïÐèҪעÒâµÄÊǼ´±ãÊÇblock2½ö½ö±»Ò»¸örecordÕ¼¾Ý£¬µ«ÊÇͬÑùÊÇ°´ÕÕrecordµÄ¸ñʽдÈëµÄ¡£
B will be split into three fragments: first fragment occupies the restof the first block, second fragment occupies the entirety of thesecond block, and the third fragment occupies a prefix of the thirdblock. This will leave six bytes free in the third block, which willbe left empty as the trailer.
C will be stored as a FULL record in the fourth block.
===================
Some benefits over the recordio format:
(1) We do not need any heuristics for resyncing - just go to nextblock boundary and scan. If there is a corruption, skip to the nextblock. As a side-benefit, we do not get confused when part of thecontents of one log file are embedded as a record inside another logfile.
(2) Splitting at approximate boundaries (e.g., for mapreduce) issimple: find the next block boundary and skip records until wehit a FULL or FIRST record.
(3) We do not need extra buffering for large records.
Some downsides compared to recordio format:
(1) No packing of tiny records. This could be fixed by adding a newrecord type, so it is a shortcoming of the current implementation,not necessarily the format.
(2) No compression. Again, this could be fixed by adding new record types.
