DataNode communicate with NameNode
-
Upload
jiang-yu -
Category
Data & Analytics
-
view
515 -
download
6
Transcript of DataNode communicate with NameNode
![Page 1: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/1.jpg)
NameNode与DataNode通信
![Page 2: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/2.jpg)
Protocol
• register: handshake,register when startup
• heartbeat: liveness,capacity,used,remaining. short time 3s default
• blockreport: diff. long time 6h
Protocol
![Page 3: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/3.jpg)
Inside NameNode
• FSNameSystem : INode,Metadata
• BlockManager:Block,DataNode,Heartbeat
• 两者关系不是割裂,紧密联系
![Page 4: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/4.jpg)
Inside NameNode
• DataNode在NameNode中对应的对象
• DataNode ——> DataNodeDescriptor
• Disk ——> DataNodeStorageInfo
• Block在DataNodeStorageInfo中以List存储,头结点为blockList
![Page 5: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/5.jpg)
INode
• Dir —> INodeDirectory只有meta,没有实际存储
• File —> INodeFile Blocks Replica
![Page 6: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/6.jpg)
INodeFile• INodeFile存储BlockInfo
• BlockInfo存储triplet : save memory, block report
![Page 7: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/7.jpg)
Heartbeat
• the function of heartbeat
![Page 8: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/8.jpg)
BlockReport
• The function of BlockReport
![Page 9: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/9.jpg)
BlockReport• toInvalidate:不在BlockMap中,该Block已经被删除
• toUC:在BlockMap中,状态为UC
• toCorrupt:在BlockMap中,状态为Corruput
• toAdd:在BlockMap中,triplet没有该节点的该磁盘
• toRemove:没有汇报,丢失的Block
![Page 10: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/10.jpg)
BlockReport流程
![Page 11: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/11.jpg)
BlockReport
![Page 12: DataNode communicate with NameNode](https://reader033.fdocuments.net/reader033/viewer/2022052413/55a5c0ed1a28abe8148b4634/html5/thumbnails/12.jpg)
BlockReport